drothlis/stb-tester — pull request #122 by wmanley (2014-03-21 20:54:25+00:00)
Title: Video transformation pipeline

A couple of commits broken out of #100 - Smart TV support

This commit relaxes the requirements of the user provided source pipeline. Previously it had to provide raw video frames whereas now stbt does decoding itself. This is convenient for users but the real advantage is that it allows us to buffer more intelligently.

raw video takes up lots of memory but it's very easy to manipulate. You can drop any frame (`GstBuffer`) and it will have no effect on the meaning of any previous or subsequent frames. Compressed video is the opposite. If you drop buffers in an h264 stream you will see artefacts until the next I frame comes along.

stbt has some queues set to leaky as our processing can be slower than the rate at which frames arrive. It is usually more important that the video frames being analysed are recent rather than consecutive so we drop frames. With this commit we are a little more selective and ensure that we only drop decoded frames rather than compressed ones to avoid artefacts, even under load. This makes the assumption that your machine can comfortable decode video in real time.

This is required to implement smart-tv-support where we wish to operate on decoded frames but want to avoid dropping any frames that we've done expensive transformations on - if we have to drop frames under load we prefer to do it as early as possible in the pipeline to waste the minimum amount of CPU time. With this we will be able to insert an additional transformation pipeline between the leaky queue and stbt's algorithms.

This feature also lays the groundwork for a performance measurement mode where you don't want to drop any frames as you want to analyse them all, for instance to measure animation smoothness. This should now be possible without having to buffer (potentially GBs of) raw frames in memory.

This is still not perfect. Ideally in the future we will be able to drop frames even before decoding them if we can identify that dropping them will not produce artifacts. This could be accomplished with a parser (e.g. h264parse) and flushing the non-leaky queue every time a non-delta frame is inserted. But this commit works well enough for now.