Custom batching in DataLoader based on a mask

username: pthelp
cooked: <p>Hello, I’m new to PyTorch and I apologize if this is a stupid question, but I am really stuck with this problem. I have a <code>Dataset</code> created from Numpy objects <code>X</code> and <code>y</code>, and I want to create a <code>DataLoader</code> to pass batches of data to my model. I have another Numpy array <code>users</code>, with the same length as <code>X</code> and <code>y</code>, which tells me which data instance comes from which user (think an array like <code>[0, 0, 0, 1, 1, 2, 2, 2,...]</code>).<br>
I want to batch the dataset such that each batch contains all instances from one user. The number of instances per user can vary, so the batch size is not fixed. I cannot figure out how to do this, but I feel like there should be an easy way that I’m just missing (is this what the <code>batch_sampler</code> argument is for?). Can anyone help with some pointers or a toy example?</p>
<p>Edited to clarify: There are multiple (X,y) pairs for each user in the dataset, and I want to assign all (X,y) pairs belonging to user 1 to batch 1, user 2 to batch 2, and so on. Hope that helps!</p>

username: nicofish
cooked: <p>DataLoader takes as parameters batch_size = foo. As such it returns batches of batch_size. The exception to this being the final batch. this happens if the length of your data_set is not divisible by the batch_size.</p>
<p>perhaps you may want to structure your data differently</p>
<p>consider this shape for your batches:<br>
[batch_size, seq_length, input_size]</p>
<p>batch_size = number of people per batch this is fixed<br>
seq_length = number of instances from that user<br>
input_size = length of those instances</p>
<p>Depending on what model you are using you may need to pad your seq_length and input_size so that they are all of the same length.</p>
<p>example:<br>
you have dataset of people names<br>
bob<br>
alice<br>
edward</p>
<p>where each row corresponds to one person.</p>
<p>In this case:<br>
input_size: is the length of your one hot encode for the name and<br>
seq_length: is the length of the name<br>
batch_size: is the number of people you will process at one time. This is fixed.</p>
<p>You did not provide very many details so this is the most I can do to help.</p>
<p>edit: example with names I gave did not account for encoding which would be necessary.</p>

username: pthelp
cooked: <p>Thanks <a class="mention" href="/u/nicofish">@nicofish</a>, my question was probably unclear, sorry about that. What I want to do is to have variable batch sizes in the DataLoader. I want to assign all X,y pairs belonging to user 1 to batch 1, user 2 to batch 2, and so on. The seq_length and input_size are fixed for each X, so that is not an issue.</p>

username: nicofish
cooked: <p>What is your end goal? When modeling, if you are using PyTorch to handle gradient descent, the model parameters will only be updated once per batch.</p>
<p>When you batch the data each batch has time steps equivalent to the batch size. Hence the model would be handling each person individually, and then would be updating all of the gradients once per batch at the end.</p>
<p>What you are doing is the same as using batch size = 1. In other words your gradients would be updated after every person. This is more computationally costly, and you lose some of the benefits from batching.</p>
<p>see last image here:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2019/09/icon-16x16.png" class="site-icon" width="15" height="15">
      <a href="https://machinelearningmastery.com/how-to-control-the-speed-and-stability-of-training-neural-networks-with-gradient-descent-batch-size/" target="_blank" rel="noopener nofollow ugc" title="06:00PM - 20 January 2019">Machine Learning Mastery – 20 Jan 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:96/72;"><img src="https://machinelearningmastery.com/wp-content/uploads/2018/11/Line-Plots-of-Classification-Accuracy-on-Train-and-Test-Datasets-With-Different-Batch-Sizes.png" class="thumbnail" width="96" height="72"></div>

<h3><a href="https://machinelearningmastery.com/how-to-control-the-speed-and-stability-of-training-neural-networks-with-gradient-descent-batch-size/" target="_blank" rel="noopener nofollow ugc">How to Control the Stability of Training Neural Networks With the Batch Size</a></h3>

<p>Neural networks are trained using gradient descent where the estimate of the error used to update the weights is calculated based on a subset of the training dataset. The number of examples from the training dataset used in the estimate of the error...</p>

  <p><span class="label1">Est. reading time: 19 minutes</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>observe how the accuracy in training is more erratic with lower batch size.</p>
<p>For more help please provide sample data, and explain what you are trying to accomplish.</p>

username: pthelp
cooked: <p>I am trying to implement multiple instance learning, as described in <a href="https://arxiv.org/pdf/1610.02501v1.pdf" rel="noopener nofollow ugc">this paper</a>. Each batch (user in my case) still has multiple instances, though the number of instances may be different across users (which means the batch size is not fixed). The gradients would be updated after every person, but after multiple instances, so the learning should be stable.</p>
<p>For some minimal sample data, assume the dataset is somewhat like this:</p>
<pre><code class="lang-auto">X = [[2,3], [1,1], [4,3], [6,2]]
y = [0, 1, 0, 1]
subjects = [0, 0, 1, 2]
</code></pre>
<p>then I want <code>X</code> to be batched based on <code>subjects</code>:</p>
<pre><code class="lang-auto">batch_0 = [[2,3], [1,1]]
batch_1 = [[4,3]]
batch_2 = [[6,2]]
</code></pre>
<p>In my actual dataset, there are about 15 instances per subject, so the batch size will not be too small.<br>
Thank you!</p>

username: nicofish
cooked: <p>In that case don’t use DataLoader. Batch the data yourself using a for loop and then feed each batch into the model one batch at a time with another loop.</p>
<p>do something like:</p>
<p>sudo code</p>
<pre><code class="lang-auto">for i in number of epochs:
    for j in number of subjects:
        index = which(X == j)
   
        #your model training function
        data = x[index]
        target = y[index]
        output = model(x)
        #compute your loss
        #do your back propagation
        #get your training and or validation loss
        #save model at epoch j
</code></pre>

username: pthelp
cooked: <p>I think that’s what I’ll fall back to, thanks. I was hoping there would be a more efficient way to do this using <code>DataLoader</code> - assigning individual samples to custom batches based on some criteria feels like it should not be such a unique use case.</p>

username: nicofish
cooked: <p>Its not that much less efficient. Consider that with DataLoader you still have to use a loop as it creates and iterable.<br>
so it would still look like:</p>
<pre><code class="lang-auto">for i in number epochs:
    for j, dat in enumerate(DataLoader):
        data = dat[0]
        target = data[1]
        #all the other training stuff
</code></pre>
<p>I also think the reason DataLoader isn’t necessary in this case is that you are simply doing batch size 1. That is why it doesn’t work.</p>
<p>Batch is a third dimension that you are adding to your data to group things together, in your case multiple people.</p>
<p>In the particular example you provided your subject 0 has data with sequence_length = 2 and input size = 2 (as you have x, y coordinates). I’m assuming the y isn’t your target data.</p>
<p>For example, if you are feeding data to an RNN or LSTM with batch_first = True. The data should have the following format:<br>
[batch_size, seq_length, input_size]</p>
<p>The shape of your input data for subject 0 would be  [1, 2, 2] and the shape for subject 1 would be [1, 1, 2]. As you can see batch size = 1.</p>
<p>I think you confused your sequence length (number of instances per person) with batch size. The same thing happened to me when I began with PyTorch.</p>
<p>Just remember each your people is a 2D matrix of shape (instances, a point). batch size is the number of 2D matrices you give to your model before performing gradient descent.</p>