Repository: pytorch/vision

Issue:
VideoClips fails silently when you don't have at least `clip_length_in_frames` in a video
## 🐛 Bug

`VideoClips` fails silently when you don't have at least `clip_length_in_frames` in a video.

## To Reproduce

Steps to reproduce the behavior:

1. Create an `HMDB51` instance (with its corresponding data; which uses `VideoClips`):

    ```python
    dataset = HMDB51(root="...", annotation_path="...", frames_per_clip=32, frame_rate=10)
    ```

2. Iterate the dataset with the dataloader, and there will be missing videos. In particular, you can inspect the attribute `dataset.video_clips.clips` and see some of them are empty.

This is because `VideoClips` tries to get clips of certain length, with a determined frame rate, and stride, but sometimes there aren't enough. It fails in this line because it returns an empty tensor:

https://github.com/pytorch/vision/blob/1aef87d01eec2c0989458387fa04baebcc86ea7b/torchvision/datasets/video_utils.py#L206

## Expected behavior

It's expected that it at least warns you about this problem, as opposed to failing silently. It can also try to pad somehow, maybe with zeros. In that line, it could save the clip as it is and then when getting it, check the length and pad.

## Environment

<details>

<summary>Environment</summary>

PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 440.82
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-lightning==0.8.5
[pip3] torch==1.5.1
[pip3] torchvision==0.6.0a0+35d732a
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.1.243             h6bb024c_0  
[conda] mkl                       2020.1                      217  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.1.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.18.5           py38ha1c710e_0  
[conda] numpy-base                1.18.5           py38hde5b4d6_0  
[conda] pytorch                   1.5.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] pytorch-lightning         0.8.5                      py_0    conda-forge
[conda] torchvision               0.6.1                py38_cu101    pytorch
</details>


Discussion:
Might be related to / resolved by #2460.
I think it's unrelated. #2460 happens when you ask for an out-of-bounds frame. The one I'm saying is about building what clips can you get from a video, which in turn returns an empty clip set because there aren't enough.
I believe that's a regression comming from TV 0.6 (with new async video reader). 
Can you try this with tv0.5? 
@bjuncek I think it's gonna be the same because the code in the `v0.5.0` git tag has the same part of the code that I pointed out above is still there. The `unfold` isn't gonna select any clip and thus not process the corresponding video.
Ah I see what you mean - there is an issue in 0.6 which has a rounding error so that some clips are skipped, but I see your point - yeah, I think raising a warning there is a good idea. Would you mind sending a PR? 
We can add a warning, but what about doing something about it? One thing could be to pad with zeros, another one to pad with a determined value or let the user pass a padding callback. Thoughts?
Another one could be to repeat the last frame.

Resolving patch:
diff --git a/torchvision/datasets/video_utils.py b/torchvision/datasets/video_utils.py
--- a/torchvision/datasets/video_utils.py
+++ b/torchvision/datasets/video_utils.py
@@ -1,5 +1,6 @@
 import bisect
 import math
+import warnings
 from fractions import Fraction
 from typing import List
 
@@ -204,6 +205,9 @@ def compute_clips_for_video(video_pts, num_frames, step, fps, frame_rate):
         )
         video_pts = video_pts[idxs]
         clips = unfold(video_pts, num_frames, step)
+        if not clips.numel():
+            warnings.warn("There aren't enough frames in the current video to get a clip for the given clip length and "
+                          "frames between clips. The video (and potentially others) will be skipped.")
         if isinstance(idxs, slice):
             idxs = [idxs] * len(clips)
         else:


Test patch:
diff --git a/test/test_datasets_video_utils.py b/test/test_datasets_video_utils.py
--- a/test/test_datasets_video_utils.py
+++ b/test/test_datasets_video_utils.py
@@ -119,6 +119,16 @@ def test_compute_clips_for_video(self):
         self.assertTrue(clips.equal(idxs))
         self.assertTrue(idxs.flatten().equal(resampled_idxs))
 
+        # case 3: frames aren't enough for a clip
+        num_frames = 32
+        orig_fps = 30
+        new_fps = 13
+        with self.assertWarns(UserWarning):
+            clips, idxs = VideoClips.compute_clips_for_video(video_pts, num_frames, num_frames,
+                                                             orig_fps, new_fps)
+        self.assertEqual(len(clips), 0)
+        self.assertEqual(len(idxs), 0)
+
 
 if __name__ == '__main__':
     unittest.main()