Shuffling input

username: pedram1
cooked: <p>Hi All</p>
<p>I am working on hyperspectral image as input of my NN.<br>
this is the shape of my data set (65000,1,92) - 65000 is number of samples (signal) , 1 is channel and 92 is length of my signal.<br>
When I trained my model it turned out that error of test data is less than error of training.<br>
I assume that it has to do with how i shuffle my data. because signals are distributed randomly in my pixels. (every single pixel is a signal)<br>
below you can see my dataloader code, has anyone have any idea if it is OkAY?</p>
<p>data_set = torch.utils.data.TensorDataset(new_input_tensor,new_target_tensor)<br>
n_train = int(len(data_set)*0.80)<br>
n_test = int(len(data_set)) - n_train<br>
train_data_set, test_data_set = torch.utils.data.random_split(data_set, [n_train, n_test])<br>
train_loader  = torch.utils.data.DataLoader(train_data_set, batch_size = 2621, shuffle=True, drop_last=True)<br>
test_loader  = torch.utils.data.DataLoader(test_data_set, batch_size = 2621, shuffle=True, drop_last=True)</p>
<p>Thank you,</p>

username: ptrblck
cooked: <p>How large is the gap between the training and test error?<br>
If your model uses e.g. dropout layers, the loss might be (slightly) higher during training, as dropout reduces the model capacity of the model.</p>

username: pedram1
cooked: <p>Hi Patrick</p>
<p>Thank you for your reply.<br>
Train Epoch : 400/400 |Loss : 0.0128<br>
Test Epoch : 400/400 |Loss : 0.0032</p>
<p>Do you think the way I splitted  up the dataset between test and training set is right?<br>
Actually, Drop out is off.</p>

username: ptrblck
cooked: <p>The usage of <code>torch.utils.data.random_split</code> should be correct.<br>
How are you calculating the training loss? Are you printing the average of the complete training epoch or is this the value of the last batch?<br>
In the former case, are you observing also the training loss for each batch?</p>

username: pedram1
cooked: <p>I used these:<br>
##<span class="hashtag">#training</span> loop, scheduler learning was used to find the best learning rate. in evert 250 epoch learning rate changes by the factor of 0.1<br>
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=320, gamma=0.7, last_epoch=-1)</p>
<p>epoch_num = 0<br>
train_error = []<br>
test_error = []<br>
best_error = 100<br>
for epoch in range(num_epochs):<br>
print(scheduler.get_last_lr())<br>
loss_total = 0<br>
test_loss_total = 0<br>
model.train()<br>
for batch_idx, sample in enumerate(train_loader):<br>
inp = sample<br>
inp = inp.cuda()<br>
output = model(inp).cuda()<br>
loss = criterion(output, inp)<br>
loss_total += loss<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
scheduler.step()<br>
loss_total = loss_total / (batch_idx+1)<br>
train_error.append(loss_total.item())</p>
<p>model.eval()<br>
with torch.no_grad():<br>
for batch_idx, sample in enumerate(test_loader):<br>
inp = sample<br>
inp = inp.cuda()<br>
output = model(inp).cuda()<br>
loss = criterion(output, inp)<br>
test_loss_total += loss<br>
test_loss_total = test_loss_total / (batch_idx+1)<br>
if loss &lt; best_error:<br>
best_error = loss<br>
best_epoch = epoch<br>
print(‘Best loss at epoch’, best_epoch)<br>
model_save_name = ‘2020-05-25 Li_sample_no_image_processing (5)’<br>
path = F"/content/drive/My Drive/{model_save_name}"<br>
torch.save(model.state_dict(), path)<br>
test_error.append(test_loss_total.item())<br>
if epoch%10 == 9:<br>
epoch_num+=epoch_num<br>
print ((’\r Train Epoch : {}/{} \tLoss : {:.4f}’.format (epoch+1,num_epochs,loss_total)))<br>
print ((’\r Test Epoch : {}/{} \tLoss : {:.4f}’.format (epoch+1,num_epochs,test_loss_total)))</p>
<p>I do print the avg value over mini batch size</p>
<p>Thank you Patrick</p>

username: ptrblck
cooked: <p>You could try to add another loop after the training epoch to calculate the training loss after the model was trained without further training. This would calculate the current training loss for this epoch without using the running average, which might create a bias.</p>
<p>Let me know, if this reduces the gap or not.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: pedram1
cooked: <p>Thank you Patrick.<br>
Just may I ask you to clear this to me more? You say calculating the final error? I mean output of NN? But then how this can explain the problem that I was addresed, smaller error in test then training.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I think the gap might come from the training loss calculation which is using the running average of the batch losses in the current epoch (while the model is being trained), while the validation loss is calculated after the current epoch is finished.<br>
To exclude this possibility, you could also calculate the training loss after the epoch was finished (similar as the validation loss is calculated) and check, if the gap narrows.</p>

username: pedram1
cooked: <p>Thank you patrick. I’ll try that.</p>