Torchvision.io video reader lack of frames

JuanFMontesinos: <p>Hi,<br>
I’ve realized that torchvision as well as other libraries suck as skvideo and opencv retrieve less amount of frames than ffmpeg.</p>
<p>Video example: <a href="https://drive.google.com/file/d/1DIRsDf1SrLOTGbVejoL-PEIlxDPP0LMC/view?usp=sharing" rel="nofollow noopener">https://drive.google.com/file/d/1DIRsDf1SrLOTGbVejoL-PEIlxDPP0LMC/view?usp=sharing</a></p>
<p>Context:<br>
I’m rencoding a dataset of youtube videos to 25.0 FPS via ffmpeg</p>
<p>Recording (.mkv) contains audio stream and video stream.<br>
Both streams are same duration (according to metadata info from ffprobe)<br>
Audio stream’s duration match the ones stated by metadata</p>
<p>Extracting frames via unix command line with ffmpeg provides a proper amount of frames (3688 in case of the given video example)</p>
<pre><code class="lang-auto">ffmpeg -i /media/jfm/Slave/SkDataset/videos/cello/1u3yHICR_BU.mkv  %05d.bmp
</code></pre>
<p>Extracting frames via imageio matches the one from unix ffmpeg.</p>
<pre><code class="lang-auto">w = mimread(PATH, memtest=False)
</code></pre>
<p>Using other librarias like <a href="http://torchvision.io" rel="nofollow noopener">torchvision.io</a> video reader, skvideo or even opencv videocapture gather less frames. The amount of frames are less than the expected ones. I’ve tried to debug the video reader from torchvision in order to see if it’s skipping frames with negative stamps but seems not to be the case. Altough by-default seeking point is 0. Anyway reproducing the video seems not to generate black frames indicating (I think) that the video stream contains only positive timestamps (which also makes sense since the whole video has been rencoded)</p>
<pre><code class="lang-auto">import skvideo.io
import skvideo.datasets
videodata = skvideo.io.vread(PATH)
</code></pre>
<p>Any idea? An example where this issue happens is given above.<br>
btw I think <a class="mention" href="/u/fmassa">@fmassa</a> is developing the video reader.</p>

ptrblck: <p>Unfortunately, I can’t comment on the mismatch of the extracted frames, but don’t think it’s an “OpenCV vs. FFmpeg” issue, since OpenCV can use <a href="https://docs.opencv.org/3.4.9/d0/da7/videoio_overview.html">FFmpeg</a>.<br>
That being said, it seems as if <code>torchvision.io</code> uses <code>pyav</code> as the default reader (<a href="https://github.com/pytorch/vision/blob/57c54075ca2a15acd83537d0776bff490ae8efc6/torchvision/__init__.py#L60">line of code</a>), which wraps FFmpeg as seen <a href="https://github.com/PyAV-Org/PyAV">here</a>.</p>

JuanFMontesinos: <p>Hi,<br>
I know (I’ve digged into the torchvision code but not in pyav’s).<br>
It seems all the libraries use ffmpeg as backend. That’s why my insight is it is related to stamps or the way libraries stop decoding. I realized that the frame counter of the library which properly extract the frames count the same frames than the ones which are wrong. So I think other libraries internally stop decoding once they reach that “last” frame.</p>
<p>Anyway thanks for having a look <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Thanks for the clarification, that’s interesting.<br>
In other words: if you are directly using <code>ffmpeg</code> in the terminal, you are getting the expected number of frames, but using a wrapper (via <code>torchvision.io</code>) you get a reduced number of frames?</p>

JuanFMontesinos: <p>Yep that’s the idea. I opened an issue <a href="https://github.com/pytorch/vision/issues/2490" rel="nofollow noopener">github</a> but I have no really time to dig into this. Thanks anyway <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>