Loading videos from folders as a dataset object

Atcold: <h1>Aim</h1>
<p>I’m trying to gather some suggestions about how to implement a video loader implementing the <code>class torch.utils.data.Dataset</code>, so that it can be fed to <code>torch.utils.data.DataLoader</code> constructor.</p>
<h1>Set up</h1>
<p>Directory of multiple folders containing multiple MP4s with minimum size scaled to <code>256</code>.</p>
<h1>Proposal</h1>
<p>I could think of my multiple videos as a big long video of length <code>tot_nb_frames</code> which will be reshaped into something like <code>tot_nb_batches x batch_size x height x width x 3</code>, where <code>tot_nb_batches = tot_nb_frames // batch_size</code>.</p>
<p>Now I have that</p>
<pre><code class="lang-python">next(DataLoaderIter)
</code></pre>
<p>will call</p>
<pre><code class="lang-python">DataLoaderIter._next_indeces()
</code></pre>
<p>which <code>return</code> a <code>list</code> (why not a <code>tuple</code>???) of ordered numbers</p>
<pre><code class="lang-python">list(range(t * batch_size, (t + 1) * batch_size))
</code></pre>
<p>called <code>indeces</code>, with <code>t</code> in the interval <code>[0, tot_nb_batches]</code>.<br>
Now my</p>
<pre><code class="lang-python">dataset[i] for i in indeces
</code></pre>
<p>should <code>return</code> the correct next frame for each row of the batch, so <code>dataset</code> should have an appropriate internal mapping, which is based on <code>batch_size</code>, attribute of <code>DataLoader</code> and not of <code>Dataset</code>.</p>
<h1>Question / advices</h1>
<p>Can anyone provide feedback on this strategy? Does it sound reasonable, or am I missing something?</p>
<p>Given that the mapping is based on <code>batch_size</code> I am now thinking whether this should be performed by <code>DataLoaderIter</code> instead.<br>
Nevertheless, given a specific initial mapping, video readers should be initialised with different seeks. So, <code>DataLoaderIter</code> should call an initialisation method of <code>Dataset</code>, but I think that this is not currently supported.<br>
Oh, well, I could have a lazy approach. Initialise the reader the first time a specific frame is requested. And, yeah, the indexing should be done from the <code>DataLoaderIter</code> side, since the <code>Dataset</code> should not care about batching at all.</p>
<h1>More quircks</h1>
<p>Say I perform the mapping with the <code>Sampler</code>, I will have variable batch sizes, <code>batch_size</code> and <code>batch_size - 1</code>.</p>
<pre><code class="lang-auto">   0     5     10    15    20    25
0  &gt;xxxx|xxxxx|xxxxx|xxxxx|xxxxx|xxx
1  xxxxx|xxxxx|xxx&gt;x|xxxxx|xxxxx|xxx
2  xxxxx|xx&gt;xx|xxxxx|xxxxx|xxxxx|xxx
3  xxxxx|xxxxx|xxxxx|xxxxx|xxxx&gt;|xxx
4  xxxxx|xxxxx|xxxxx|x&gt;xxx|xxxx
</code></pre>
<p>After asking batch <code>23</code> the batch size should decrease by one. This is a mess, since <code>_next_indices()</code> is still going to ask for <code>batch_size</code> amount of data, screwing up everything.</p>
<h1>Hacky solution 1</h1>
<p>I could have the <code>DataLoader</code>'s <code>batch_size = 1</code>, and have the <code>Dataset</code> object returning batches (columns) itself, and <code>sqeeze()</code> the singleton later on. But it looks nasty…</p>
<h1>Hacky solution 2</h1>
<p>Given that the missing data (bottom right corner) is <code>&lt; batch_size</code>, and this gets as big as <code>128</code>, usually, I could simply return a duplicate of the last frame, at worse, <code>127</code> time, which is roughly <code>2</code> seconds of video, compared to the hours of data. So… I think it’s just fine. Otherwise, I could use the beginning of video <code>0</code>. I think I’ll opt for this way.<br>
The whole should look like this.</p>
<pre><code class="lang-auto">   0     5     10    15    20    25
0  0oooo|ooooo|ooooo|ooooo|ooooo|ooo
1  ooooo|ooooo|ooo&gt;x|xxxxx|xxxxx|xxx
2  xxxxx|xx&gt;xx|xxxxx|xxxxx|xxxxx|xxx
3  xxxxx|xxxxx|xxxxx|xxxxx|xxxx&gt;|xxx
4  xxxxx|xxxxx|xxxxx|x&gt;xxx|xxxx0|ooo
</code></pre>
<p>where <code>&gt;</code> represents the head of a generic video and <code>x</code> its subsequent frames, <code>0</code> represents the head of video zero, and <code>o</code> its frames.</p>

raivokoot: <p>This here is actually an implementation of the dataset class, which is made for videos. The implemented <strong>getitem</strong> function loads several frames of a video and returns them.</p>
<aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="15" height="15">
      <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/29647087?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="59" height="59">

<h3><a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" target="_blank" rel="noopener nofollow ugc">RaivoKoot/Video-Dataset-Loading-Pytorch</a></h3>


  <p><span class="label1">Generic PyTorch Dataset Implementation for Loading, Preprocessing and Augmenting Video Datasets</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>