Finished cifar10 and have some questions about mechanics and data loading

shaun: Hi, As always, as part of my first post I thank the developers for this amazing library that helps a lot of us in our deep learning escapades. I’ve finished running through the first tutorial involving the CIFAR10 dataset and have some questions. In this code block, Could some explain in detail on what is going on here. These might be more of python related questions than a pytorch question but I think its crucial to understand what is happening here. I understand whats happening in an abstract level but not on the code level. In particular, In the prediction code block, This code (net(images)) is similar to the training stage, so I’m not sure how we are “testing” because we don’t have testing mode. For example, in Keras for training we use model.fit and testing we use model.evaluate, and I’m not seeing a similar distinction here. I apologize for a whole lot of questions, most of them born out of ignorance and I’m sure I’ll have more as I start using pytorch for my problems. If I need to split them up into separate posts, please let me know and I’ll edit the post accordingly. Thanks and I appreciate everyone’s help!

jekbradbury: No, these are great questions. They’re all surrounded by double underscores (so they’re called “dunder” methods): 
__init__ is one of them, which defines the constructor; 
__str__ is another one – what you implement there defines what Python will do if you call str(obj). The __call__ dunder method defines what Python will do if you call
an instance of the class as if it were a function. Others questions:

shaun: @jekbradbury Thank you so much for your replies, I appreciate it. It cleared up a lot of stuff. I still have some questions maybe other people can pop in to the conversation. The following code is how we load the CIFAR10 dataset. For test, we just set train=False My initial intuition was just to set train_loader = train_loader[:small_number] but I got an error: Then I thought I could mess with the train_set directly but I got another error: Both these objects have a len function: So I’m not sure how to get a validation set out of this. So where is this flag being set and how do we know its not training again? I can think of two reasons on this works, but I’m not sure which one: Training block: Test block: During testing (aka prediction) we don’t compute the loss, run the backward, and run the optimization.step() which would mean we are just getting the class labels. So by omitting those steps we do the prediction? This makes sense to me after thinking about it, but it would be helpful if I could get confirmation that this is in fact what is happening. Thanks.

vabh: Train/test mode is something like this: <SCODE>net.train()
#train loop/function
for (images, labels) in train_loader:
    # train code


net.eval()
#test loop or function
for (images, labels) in test_loader
#    test code eg. outputs = net(images)
<ECODE>

shaun: 

vabh: The network in the example you linked does not have these layers which is why I suspect they did not call the eval function. Calling the train and eval functions won’t affect the model output. For models which have dropout/batchnorm layers, its quite imperative that you call the eval function before training.

tom: If the dataset is reasonably simple, you can split the dataset like so: Best regards Thomas

shaun: @tom

tom: the idea of using a validation set is that whatever you plan do to the test dataset would work for the validation set as well (really, you would use the test set’s DataLoader for the val_dataset). As such, my suggestion would be to feed it to a dataloader that works similar to the test one (for the MNIST example, in fact, you could make test_loader a parameter to the test function and feed the val_loader. That would also do the model.eval() call mentioned earlier). Personally, I would prefer to keep the workflow with Dataset->DataLoader->Validation as that is scalable and looks like an efficient use of my time, but you should certainly just do whatever works for you. Hope this helps, even if it’s just my very limited take on something that ultimately boils down to style preferences and I cannot vouch for my expertise in that. Best regards Thomas

deepcode: this works well, i believe something like this should be part of pytorch, at least as an example

superhans: Might be a bit late to the party, but to create a train/val/test split (of 45k/5k/10k) in CIFAR10, this should work : <SCODE>n_train = 45000
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)

# create a validation set
valset = deepcopy(trainset)
trainset.train_data = trainset.train_data[:n_train]
trainset.train_labels = trainset.train_labels[:n_train]

valset.train_data = valset.train_data[n_train:]
valset.train_labels = valset.train_labels[n_train:]

# create a test set
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)
valloader = torch.utils.data.DataLoader(valset, batch_size=128, shuffle=True, num_workers=2)
testloader = torch.utils.data.DataLoader(testset, batch_size=100, shuffle=False, num_workers=2)<ECODE>