kaltura/server — pull request #1868 by vixyvideo (2014-11-11 10:08:48+00:00)
Title: Move convert job threading to config

After upgrading I noticed that convert jobs were taking up to ~800% CPU load per convert Job instance.

Upon inspection I found out:
- The new ffmpeg has great multithreading support but without -threads it uses as much threads as it can.
- The new default flavors have -threads in the flavor params.
- When multiple -threads parameters are defined in the command, the last one is used.

After talking with @jessp01 we agreed this solution would be best:
- Remove -threads 4 from default flavors
- Add a optional threads option in the batch.ini
- When no threads option is present, it defaults to 4 so installed systems aren't affected.
- Overriding on a per flavor basis, is possible via the extra flavor params.

This brings convert jobs back to at least manageable levels. The only thing I can't seem to figure out, is:
a. flavors with no conversion_engines_extra_params use exactly 1 thread (h264b, AAC)
b. flavors with extra conversion_engines_extra_params span more than one thread. How can we affect this behavior further?

Example failing params:
-flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -trellis 1 -refs 1 -me_range 16 -keyint_min 20 -sc_threshold 40 -i_qfactor 0.71 -bt 800k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -level 30 -async 2 -vsync 2 | -flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -trellis 1 -refs 1 -me_range 16 -keyint_min 20 -sc_threshold 40 -i_qfactor 0.71 -bt 800k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -level 30 -async 2 -vsync 2