Creating a random chunk of image sequences from a video

username: nivesh_gadipudi
cooked: <p>Some of us might have worked on KITTI vision benchmarks.<br>
Dataset Description:<br>
22 Sequences, Of which 11 are provided with ground truth labeling</p>
<p>number of frames in 11 sequences = [4541, 1101, 4661, 801, 271, 2761, 1101, 1101, 4071, 1591, 1201]</p>
<p>Let my dataloader be:</p>
<pre><code class="lang-auto">class KITTIData(Dataset):
    def __init__(self, basedir, sequences):
        self.basedir    = basedir
        self.sequences  = sequences
        self.seq_lens   = []
        self.cumulative = []
      
        for i in self.sequences:
            curdirlen = os.listdir(os.path.join(self.basedir, "sequences", i, "image_2"))
            self.seq_lens.append(len(curdirlen)-1)
        
        temp = 0
        for i in self.seq_lens:
            temp = temp + i
            self.cumulative.append(temp)
        
    def __len__(self):
        return np.sum(self.seq_lens)

    def __getitem__(self, index):
        seq_key   = liefunctions.BinarySearch(self.cumulative, index)
        seq_index = self.sequences[seq_key]

        if seq_key == 0:
            offset = index
        else:
            offset = index - self.cumulative[seq_key-1]
        
        t1   = offset
        t2   = offset + 1
        img1 = Image.open(os.path.join(self.basedir, "sequences", seq_index, "image_2", "%06d" % t1 + ".png")).convert('RGB')
        #img1 = ImageOps.grayscale(img1)
        img2 = Image.open(os.path.join(self.basedir, "sequences", seq_index, "image_2", "%06d" % t2 + ".png")).convert('RGB')
        #img2 = ImageOps.grayscale(img2)

        img1 = self.transform(img1)
        img2 = self.transform(img2)

        img = torch.cat((img1, img2), 0)  

        return img
</code></pre>
<ol>
<li>How can i modify this to generate randomly cropped sequences for example, (10 to 20), (45 to 60) …(1053 to 1080) frames from each sequence ?</li>
</ol>
<p>Any similar example for this where small chunks of randomly cropped sequences are added in the dataloader would help me a lot.</p>
<p>Thank you for your effort in advance!!</p>

username: nivesh_gadipudi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/tom">@tom</a> can i get some suggestions on this</p>

username: aktgpt
cooked: <p>If you have the image sequence as array-like shape <code>vid = [w, h, c,  n_frames]</code> you can use <code>np.split</code> and specify the indices to extract random cropped sequences.<br>
<code>random_sequence = np.split(vid, indices_or_sections=[10,20,45,60,...], axis=3)</code><br>
For this, you have to load the whole sequence into memory, which might be a bit memory-heavy.</p>
<p>Otherwise, you can just define <code>t1</code> and <code>t2</code> in <code>__getitem__</code> as:</p>
<pre><code class="lang-auto">t1 = random.sample(range(0, len(sequence-max_n_frames_wanted), 1)
t2 = random.sample(range(0, len(max_n_frames_wanted), 1)
</code></pre>
<p>and then load the frames sequentially in a <code>for</code> loop from <code>t1</code> to <code>t2</code> and use <code>torch.cat</code> in the end as you do.</p>
<p>Hope this helps.</p>

username: nivesh_gadipudi
cooked: <p>I did not get it.</p>
<p>The variables i am using <code>t1, t2</code> are frames at t, t+1 timestamps.<br>
From the above code:</p>
<pre><code class="lang-auto">t1 = random.sample(range(0, len(sequence-max_n_frames_wanted), 1)
t2 = random.sample(range(0, len(max_n_frames_wanted), 1)
</code></pre>
<p>i assume number of samples to be drawn for instance be k = 10 (argument is missing in random.sample()) or if at all we consider k = max_n_frames_wanted, sequence = 400.</p>
<p>t1 = [0, 1, 2, 3, 4,…389]<br>
t2 = [0, 1, 2, 3, 4,…9]</p>
<p>I hope i am getting it correctly.</p>
<p>But, my main concern is when we using random sampling the length in <code>def __len__(self)</code> varies and the index should point out to the sequence to retrieve image from disk.</p>

username: aktgpt
cooked: <p>First, you can set the <code>__len__(self)</code> to be the number of sequences you have, so in this case, 11. So instead of</p>
<aside class="quote no-group" data-username="nivesh_gadipudi" data-post="1" data-topic="116739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivesh_gadipudi/48/38188_2.png" class="avatar"> nivesh_gadipudi:</div>
<blockquote>
<p><code>np.sum(self.seq_lens)</code></p>
</blockquote>
</aside>
<p>you can specify it as <code>len(self.seq_lens)</code>. Sure, an epoch will be smaller, but you can train for more epochs.</p>
<p>Then, I assumed that you wanted to have a variable number of frames (but in serialized order) on each <code>__getitem__(self, idx)</code> call, so that’s what I specified as <code>t1</code> and <code>t2</code>. At each call, you would get a sequence index and then randomly pick  <code>t1</code> and <code>t2</code> in that sequence. Then take all the frames between <code>t1</code> and <code>t1+t2</code>.</p>
<p>If you want to fix the number of frames then you can simply take frames between <code>t1</code> and <code>t1+k</code>. Then, <code>t1</code> can be specified as:<br>
<code>t1 = random.sample(range(0, self.seq_lens[idx]-k), 1)</code></p>
<p>If you don’t want to take sequential <code>(t1, t1+1,... t1+k)</code> frames just replace <code>1</code> to <code>k</code> in <code>t1</code> and remove <code>t2</code>.</p>
<p>Hope this clears it up.</p>