Loading same data but getting different results

Omar_AlSuwaidi: <p>So I’m trying to manually split my training data into batches such that I can easily <strong>access</strong> them via <strong>indexing</strong>, and not relying on <code>DataLoader</code> to split them up for me, since that way I won’t be able to access the <strong>individual</strong> batches by indexing. So I tried the following:</p>
<pre><code class="lang-auto">train_data = datasets.ANY(root='data', transform=T_train, download=True)
BS = 200
num_batches = len(train_data) // BS
sequence = list(range(len(train_data)))
np.random.shuffle(sequence)  # To shuffle the training data
subsets = [Subset(train_data, sequence[i * BS: (i + 1) * BS]) for i in range(num_batches)]
train_loader = [DataLoader(sub, batch_size=BS) for sub in subsets]  # Create multiple batches, each with BS number of samples
</code></pre>
<p>Which works during training just fine.</p>
<p>However, when I attempted another way to manually split the training data I got different end results, even with <strong>all</strong> the same parameters and the following settings:</p>
<pre><code class="lang-auto">device = torch.device('cuda')
torch.manual_seed(0)
np.random.seed(0)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.cuda.empty_cache()
</code></pre>
<p>I only split the training data the following way this time:</p>
<pre><code class="lang-auto">train_data = list(datasets.ANY(root='data', transform=T_train, download=True))  # Cast into a list
BS = 200
num_batches = len(train_data) // BS
np.random.shuffle(train_data)  # To shuffle the training data
train_loader = [DataLoader(train_data[i*BS: (i+1)*BS], batch_size=BS) for i in range(num_batches)]
</code></pre>
<p>But this gives me different results than the first approach, even though (<em>I believe</em> ) that both approaches are identical in manually splitting the training data into batches. I even tried <strong>not shuffling</strong> at all and loading the data just as it is, but I still got different results (85.2% v.s 81.98% accuracy). I even manually checked that the loaded images from the batches match; and are the same using both methods.</p>
<p>Not only that, when I load the training data the conventional way as follows:</p>
<pre><code class="lang-auto">BS = 200
train_loader = DataLoader(train_data, batch_size=BS, shuffle=True)
</code></pre>
<p>I get even more drastic results!</p>
<p>Can somebody please explain to me why these differences arise, and how to fix it?</p>

nivek: <p>You have confirmed that “the loaded images from the batches match” between the first two methods. What happens afterward? Do you set your seed (e.g. <code>torch.manual_seed(0)</code>) before training?</p>

Omar_AlSuwaidi: <p>Hey, yeah the order of the images from each batch is the same in all batches using both approaches. And before training, I’ve set the following:</p>
<pre><code class="lang-auto">device = torch.device('cuda')
torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
np.random.seed(0)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.cuda.empty_cache()
</code></pre>
<p>Moreover, I would like to share an <strong>update</strong> that might shed some light:</p>
<p><code>T_train</code> transformation contains some random transformations (H_flip, crop) and when using it along with the first <code>train_loader</code>, the time taken during training was: 24.79s/it, while the second <code>train_loader</code> took: 10.88s/it (even though both have the exact same number of parameters updates/steps). So I decided to remove the random transformations from <code>T_train</code>; then the time taken using the first <code>train_loader</code> reduced to: 16.99s/it, while the second <code>train_loader</code> remained at: 10.87s/it. So somehow, the second <code>train_loader</code> still took the <strong>same</strong> time (with or without the random transformations). Thus, I decided to bring back the random transformations in <code>T_train</code> to <strong>visualize</strong> the image outputs from the second <code>train_loader</code> to verify if the random transformations were being applied, and indeed they were! So this is really confusing and I’m not quite why they’re giving different results.</p>

nivek: <p>I am unsure why your second <code>train_loader</code> still has the random transformations after you remove them. Perhaps you need clear all the variable and re-run everything?</p>
<p>Aside from that, the other pitfall may be nondeterministic algorithms:<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms</a></p>

H_MP: <p>I had same problem and in my case it was due to loading images</p>
<pre><code class="lang-auto">img = cv2.imread(path)
img = cv2.cvtColor(img, cv2_COLORBGR2GRAY)`
img = cv2.resize( img, (32,32) )
</code></pre>
<p>but when I change it to</p>
<pre><code class="lang-auto">transform_valid = transforms.Compose([
    transforms.ToPILImage(), 
    transforms.Resize((32, 32)),
    transforms.ToTensor()])

img = cv2.imread(path, 0)
img = custom_transfrom(img).unsqueeze(0)

</code></pre>
<p>my problem solved</p>