kaltura/server — pull request #1869 by jpluijmers (2014-11-11 10:13:13+00:00)
Title: Removed default threading and added threads param to the batch worker

After upgrading I noticed that convert jobs were taking up to ~800% CPU load per convert Job instance.

Upon inspection I found out:

The new ffmpeg has great multithreading support but without -threads it uses as much threads as it can.
The new default flavors have -threads in the flavor params.
When multiple -threads parameters are defined in the command, the last one is used.
After talking with @jessp01 we agreed this solution would be best:

Remove -threads 4 from default flavors
Add a optional threads option in the batch.ini
When no threads option is present, it defaults to 4 so installed systems aren't affected.
Overriding on a per flavor basis, is possible via the extra flavor params.
This brings convert jobs back to at least manageable levels. The only thing I can't seem to figure out, is:
a. flavors with no conversion_engines_extra_params use exactly 1 thread (h264b, AAC)
b. flavors with extra conversion_engines_extra_params span more than one thread. How can we affect this behavior further?

Example failing params:
-flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -trellis 1 -refs 1 -me_range 16 -keyint_min 20 -sc_threshold 40 -i_qfactor 0.71 -bt 800k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -level 30 -async 2 -vsync 2 | -flags +loop+mv4 -cmp 256 -partitions +parti4x4+partp8x8+partb8x8 -trellis 1 -refs 1 -me_range 16 -keyint_min 20 -sc_threshold 40 -i_qfactor 0.71 -bt 800k -maxrate 1200k -bufsize 1200k -rc_eq 'blurCplx^(1-qComp)' -level 30 -async 2 -vsync 2