Train and validate simultaneously on two datasets

aleurs: <p>I have two datasets each of 142.000 images, one for training and one for validation. What I have implemented is that it goes through the train data set (batchsize 11) and trains.<br>
But what I now want to do is the following:</p>
<ul>
<li>train on the first images 11 images of the train set</li>
<li>validate the model on the first 11 images from the validation set</li>
<li>then train on the second 11 images of the train set</li>
<li>then validate the model on the second 11 images from the validation set</li>
<li>etc.</li>
</ul>
<pre><code class="lang-auto">training_data_loader = DataLoader(dataset=train_set, num_workers=4, batch_size=11)
val_data_loader = DataLoader(dataset=val_set, num_workers=4, batch_size=11)
</code></pre>
<p>Here is my trainings code. Here I enumerate over the training_data_loader. Is there an efficient way to get the corresponding batch from the val_data_loader?<br>
I tried a for loop that iterates through the val_data_loader and gets me the correct batch, but that’s too time consuming. I have to directly access the correct batch from the val_data_loader. I tried it with the function <strong>getitem</strong>, but I didn’t found a solution that worked.</p>
<pre><code class="lang-auto">for iteration, batch in enumerate(training_data_loader, 0):
		input_train, target_train = Variable(batch[0]), Variable(batch[1])

		if cuda:
			input_train = input_train.cuda()
			target_train = target_train.cuda()
            
		optimizer.zero_grad()
        
		output_model_train = model(input_train)

		loss = criterion(output_model_train, target_train)
        
		loss.backward()
		optimizer.step()
</code></pre>

justusschock: <p>First of all: Do you really need this? Usually The validation is done after each epoch (epoch = whole iteration over dataset) on the whole validation dataset.</p>
<p>if you really want to do it the way you did it you could define a <a href="https://pytorch.org/docs/master/data.html#torch.utils.data.Dataset" rel="noopener nofollow ugc">Dataset</a> which handles train and validation data. If you do this you could use one dataloader and would get batches of always the same indices in train and validation set.</p>
<blockquote>
<p><strong>NOTE:</strong> This could lead to problems if the datasets have different sizes or you could validate the same images multiple times if you use shuffle=True (as usually done during training)</p>
</blockquote>

ptrblck: <p>If your <code>Dataset</code>s have the same sizes, which seems to be the case, you could just iterate both <code>DataLoaders</code> in the for loop.<br>
Note that the indices for each loader might be different if you use shuffling.</p>
<pre><code class="lang-auto">train_dataset = TensorDataset(torch.randn(100, 3, 24, 24))
val_dataset = TensorDataset(torch.randn(100, 3, 24, 24))

train_loader = DataLoader(train_dataset, batch_size=11, shuffle=False)
val_loader = DataLoader(val_dataset, batch_size=11, shuffle=False)

for batch_idx, (train_data, val_data) in enumerate(zip(train_loader, val_loader)):
    print('Batch idx {}\nTrain data shape {}\nVal data shape {}'.format(
        batch_idx, train_data[0].shape, val_data[0].shape))
</code></pre>

aleurs: <p>Yes, in the paper I’m implementing they calculate the loss after each iteration in the data set. If I would do it each epoch it would take too long.<br>
The train and validation datasets have the same size. Thanks for the tipp with shuffle.</p>

aleurs: <p>Thanks. That is what I was looking for :-).</p>

prashp: <p>Sorry to revive this old thread, but what is the solution if the Datasets don’t have the same size?</p>
<p>Currently I’m using this approach:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
   val_iter = itertools.cycle(validation_dataset_loader)
   for i_batch, data in enumerate(training_dataset_loader, 0):
      model.train()
      #training happens here
      
      model.eval()
      val_sample = next(val_iter)
      #evaluate performance on validation data
</code></pre>
<p>However I’m not sure if using the iterator here for the validation dataset is the best option, compared to the ‘enumerator’ approach, especially because the validation dataset is shuffled even when I explicitly set ‘shuffle=False’ when creating the validation DataLoader.</p>
<p>Thanks!</p>

ptrblck: <p>This approach seems to be valid for your use case.<br>
The shuffling should not be changed, if you wrap the <code>DataLoader</code> into <code>cycle</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(100).view(100, 1)
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
loader = DataLoader(
    dataset,
    batch_size=5,
    shuffle=False
)

my_iter = itertools.cycle(loader)

for _ in range(100):
    x = next(my_iter)
    print(x)
</code></pre>