How to operate variable number of inputs with variable length

username: yolle103
cooked: <p>hi, I have this data where for each sample there is a image list [img1, img2, img3, …]. For each image, the shape is different, and for each sample the number of images also varies.</p>
<p>I could come up with some conv net where different sized image could be reduce to a fix length representation (by adaptive pooling). But the only solution for number of images i can think of is just sample them to be the same.</p>
<p>Could anyone give me some idea that uses some neural network instead of sampling to deal with this issue?</p>

username: mattdeitke
cooked: <p>A few approaches that are often used would be to crop the image, by clipping off extraneous parts or resizing an image by squeezing or extending the sides.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5a5407eab50607ac7824ab843b61237448216e3c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a5407eab50607ac7824ab843b61237448216e3c" title="cr"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5a5407eab50607ac7824ab843b61237448216e3c_2_345x204.jpeg" alt="cr" data-base62-sha1="cT50tR7zbp7socTOFJra8tQlYm0" width="345" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5a5407eab50607ac7824ab843b61237448216e3c_2_345x204.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5a5407eab50607ac7824ab843b61237448216e3c_2_517x306.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5a5407eab50607ac7824ab843b61237448216e3c_2_690x408.jpeg 2x" data-dominant-color="D8DAD6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cr</span><span class="informations">1020×606 238 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It would often be best to choose a size that would work best for the output. Other, more interesting approaches, would be to take glances of the image using an RNN, which would allow you to make the input any size [<a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="noopener nofollow ugc">1</a>, <a href="https://arxiv.org/abs/1502.04623" rel="noopener nofollow ugc">2</a>, <a href="https://arxiv.org/abs/1412.7755" rel="noopener nofollow ugc">3</a>]. A more naive approach would be to add 0 padding to all of the extra space in the image; however, this would take up a lot of memory.</p>

username: yolle103
cooked: <p>Thank you for your reply! it is very useful to see that so many method can be done on variable size images. However, I still have the issue that within each sample I get variable number of images. So I can crop/resize or do some fancy stuff to make each image same size, I still can not make the number of images per sample the same.</p>

username: mattdeitke
cooked: <aside class="quote no-group" data-username="yolle103" data-post="3" data-topic="48957">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yolle103/48/13484_2.png" class="avatar"> yolle103:</div>
<blockquote>
<p>make the number of images per sample the same.</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">. Not entirely sure I understand what you mean. I assume your NN/CNN inputs a single image, like what is shown in the following graphic. Of course, this operation can be parallelized, but this is the general structure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/962db7700d95a9ba1a079838cd221bf60ca8bd86.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/962db7700d95a9ba1a079838cd221bf60ca8bd86" title="conv"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/962db7700d95a9ba1a079838cd221bf60ca8bd86_2_345x184.jpeg" alt="conv" data-base62-sha1="lqxzcKau9abg5Dw18tBH69aYwyG" width="345" height="184" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/962db7700d95a9ba1a079838cd221bf60ca8bd86_2_345x184.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/9/962db7700d95a9ba1a079838cd221bf60ca8bd86_2_517x276.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/962db7700d95a9ba1a079838cd221bf60ca8bd86_2_690x368.jpeg 2x" data-dominant-color="E5E5E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">conv</span><span class="informations">1600×856 354 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But I’m confused what you mean by making the number of samples the same. Are you using samples to refer to the number of images per mini-batch, the number of images per training set, or something along those lines?</p>

username: yolle103
cooked: <p>hi, what i meant is that my data is like sample -&gt; target, where each sample is a series of images. So in this case we are not predicting one target for one image, but one target for a series of images.</p>

username: shubhvachher
cooked: <p>haha. I think <a class="mention" href="/u/yolle103">@yolle103</a> is referring to something like training on “videos”. Each video is of a different duration and so different number of frames or “images” and he predicts one output for one video input. <a class="mention" href="/u/yolle103">@yolle103</a>, you could look at some approaches in lip reading research or similar “video” to target problems.</p>
<p>As an approach that I think is okay to start off with: You resize your “images” to fit a certain fixed shape and then sort your “videos” from shortest to longest. After this, pad the ends of the “video” training examples to the largest in the batch. Due to the sorting, this then would not be too much of an extra compute for your GPU. Your maximum video length per batch, though, will be different. For this, after passing it through a initial CNN feature extractor, you must use a sequence modelling model, like an RNN: GRU, LSTM. Now you have a nice, fixed size, output for each “video” to further train using!</p>
<p>[Of course, there are drawbacks to this “sorting” trick, for example, remember to set <strong>shuffling of your data to <code>False</code></strong> everywhere to avoid massive compute overheads. A way to overcome this as a drawback is to call it “curriculum learning” and pass it off as helping your model to converge faster <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">]</p>
<p><strong>As an example:</strong> Videos have been sorted from smallest to largest. Each frame has been reshaped to size 3x80x80 (3channels). batch size is 4 and video 1, video 2, video 3, video 4 are in one batch and are of sizes. 24x3x80x80, 26x3x80x80, 28x3x80x80, 29x3x80x80. Add zero frames of size 3x80x80 to video 1, 2 and 3 and so you get a batch to train of final size 4x29x3x80x80. Then use CNNs and finally RNNs because this number “29” will vary across your batches.</p>

username: yolle103
cooked: <p>yeah, great! This is exactly what I need, thanks!</p>