Question about batch size and loss function

username: Yolkandwhite
cooked: <p>I got my code running right but it takes too much time and loss value is too high</p>
<p>I found out that the dataloader isn’t getting the right batch size.<br>
It’s getting the whole data in the model.<br>
number of data is 3607 each (img and mask)<br>
I want the batch size to be 1<br>
How can I fix it??</p>
<pre><code class="lang-auto">class BasicDataset(Dataset):
    def __init__(self, imgs_dir, masks_dir):
        self.imgs_dir = imgs_dir
        self.masks_dir = masks_dir
        self.mriids = next(os.walk(self.imgs_dir))[2]
        self.maskids = next(os.walk(self.masks_dir))[2]
        
        def atoi(text):
            return int(text) if text.isdigit() else text

        def natural_keys(text):
            return [atoi(c) for c in re.split(r'(\d+)', text) ] 
        
        self.mriids = sorted(self.mriids, key = natural_keys)
        self.maskids = sorted(self.maskids, key = natural_keys)

    def __len__(self):
        return len(self.mriids)

    def __getitem__(self, idx):

        mriidx = self.mriids[idx] #img file name
        maskidx = self.maskids[idx] #mask file name
        
        mask_file = os.path.join(self.masks_dir, maskidx)
        img_file = os.path.join(self.imgs_dir, mriidx)
        
        img = Image.open(img_file).convert("RGB")
        mask = Image.open(mask_file).convert("L")
        
        mask = np.array(mask)
        img = np.array(img)
        
        mask = np.expand_dims(mask, axis=2)
        
        img = np.transpose(img, (2, 0, 1))
        mask = np.transpose(mask, (2, 0, 1))

#         obj_ids = np.unique(mask)
    
        obj_ids = np.unique(mask)
        obj_ids = obj_ids[1:]

        num_objs = len(obj_ids)
        labels = torch.ones((num_objs,), dtype=torch.int64)
        mask = torch.as_tensor(mask, dtype=torch.uint8)

        image_id = torch.tensor([idx])

        target = {}
        target["labels"] = labels
        target["masks"] = mask
        target["image_id"] = image_id
        
        return img, target
</code></pre>
<pre><code class="lang-auto">epochs = 100
batch_size = 1
lr = 0.00001
momentum = 0.99
</code></pre>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)
loss_func = nn.BCEWithLogitsLoss().to(device)
</code></pre>
<pre><code class="lang-auto">gen = BasicDataset('/home/intern/Desktop/YH/Brain_MRI/BrainMRI_train/MRI/MRI/', '/home/intern/Desktop/YH/Brain_MRI/BrainMRI_train/mask/mask/')
train_loader = DataLoader(gen, batch_size=batch_size)

total_batch = len(gen)
print(total_batch)

model.train()
print("start training")
for epoch in range(epochs):
    t0 = time.time()
    for mri, true_mask in train_loader:
        
        mri = mri.type(torch.FloatTensor)
        true_mask = true_mask["masks"]
        true_mask = true_mask.type(torch.FloatTensor)

        mri = mri.to(device)
        true_mask = true_mask.to(device)

        pred_mask = model(mri)
        loss = loss_func(pred_mask, true_mask)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    t1 = time.time()
    print('[Epoch:{}], loss = {}, time = {}'.format(epoch+1, loss, t1-t0))
print('training Finished!')
</code></pre>
<pre><code class="lang-auto">3607 
start training [Epoch:1], loss = nan, time = 141.68572974205017 [Epoch:2], loss = nan, time = 143.46247911453247 [Epoch:3], loss = nan, time = 143.64162826538086

--------------------------------------------------------------------------- 
KeyboardInterrupt                         Traceback (most recent call last)
&lt;ipython-input-72-0b9f410e970f&gt; in &lt;module&gt;()
     19 #         true_mask = true_mask.squeeze(1)
     20 
---&gt; 21         mri = mri.to(device)
     22         true_mask = true_mask.to(device)
     23 

KeyboardInterrupt: 
</code></pre>

username: Nisan_Aryal
cooked: <p>from what I see in the code…the “print(’[Epoch:{}], loss = {}, time = {}’.format(epoch+1, loss, t1-t0))”   …is outside the loop of train_loader so even though the batch size is 1 you are printing it only after the epoch is finish …may be because of this you are seeing that it is loading all the data as the printing is done after the epoch.</p>

username: Yolkandwhite
cooked: <p>I made this training code part, implementing my previous mnist code.<br>
It work good and fast(it takes about 12sec per 1 epoch)<br>
In that case even though the printing part was out side of loading loop and used cpu.<br>
MNIST has 60000 training image data so I don’t think this code(batch size is 100) below is getting the data 60000*100 times. or is it???</p>
<pre><code class="lang-auto"># dataset loader
data_loader = torch.utils.data.DataLoader(dataset=mnist_train,
                                          batch_size=batch_size,   #batch_size = 100
                                          shuffle=True,
                                          drop_last=True)
</code></pre>
<pre><code class="lang-auto">for epoch in range(training_epochs):
    avg_cost = 0
    total_batch = len(data_loader)

    for X, Y in data_loader:
        # reshape input image into [batch_size by 784]
        # label is not one-hot encoded
        X = X.view(-1, 28 * 28).to(device)
        Y = Y.to(device)

        optimizer.zero_grad()
        hypothesis = linear(X)
        cost = criterion(hypothesis, Y)
        cost.backward()
        optimizer.step()

        avg_cost += cost / total_batch

    print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.9f}'.format(avg_cost))

print('Learning finished')
</code></pre>

username: Nisan_Aryal
cooked: <p>total data is 60000 and batch size is 100 so to complete one epoch the no. of iteration will be 60000/100 = 600 …so if the batch size is 1 like the previous case then the no. of iteration will be 60000 … so it will take a lot of time …by the way i am confused what you want to ask? you want your batch size to be 1 from earlier post and i think the above code is working fine for batch 1 but as above code has 3607 data so the no. of iteration is 3607 for for MNIST data it takes around 12 sec (600 iteration) then for 3607 iteration it will take around 6 times more about 72 sec… i think i am confused what you want to ask please be a little specific so that i can help…</p>
<p>ps. the speed also depends on other factor such as size of network and other preprocessing steps…so the above calculation is just a rough estimation</p>

username: Yolkandwhite
cooked: <p>I think I’m misunderstanding the meaning of batch size…<br>
I googled about batch size and it says</p>
<p><em>Batch size  is a term used in machine learning and refers to the number of training examples utilized in one iteration.</em></p>
<p>If this is right than 100 training data should be loaded in one iteration.</p>
<p>What I thought the data in each iteration is like this.</p>
<p>(100/60000)<br>
(200/60000)<br>
(300/60000)<br>
…<br>
(60000/60000)</p>
<p>but I just found out that printing out the X, and Y in inner for loop, it was 600 data and which will lead to 100 iteration in one epoch.</p>
<p>result was same in the very top previous code about img and mask(batch size = 1).<br>
what I wanted to make was getting 1 data in inner loop instead of whole 3607 data.</p>
<p>If i’m misunderstanding the concept of batch size, then is the  batch size == number of iteration???</p>

username: Nisan_Aryal
cooked: <p>The meaning of batch size is loading [batch size] training data in one iteration. If your batch size is 100 then you should be getting 100 data at one iteration. batch size doesnt equal to no. of iteration unless there is a coincidence. well looking at the code i cant find the problem check the batch size once if the iteration is 100 then the batch size should be 600…make sure you arent confusing 100 with the epoch, the only variable i see that can produce 100 is epoch.</p>

username: Yolkandwhite
cooked: <p>As you can see in this code I printed out the number  of data coming in.</p>
<pre><code class="lang-auto"># parameters
training_epochs = 1
batch_size = 100
</code></pre>
<pre><code class="lang-auto"># dataset loader
data_loader = torch.utils.data.DataLoader(dataset=mnist_train,
                                          batch_size=batch_size,
                                          shuffle=True,
                                          drop_last=True)
</code></pre>
<pre><code class="lang-auto">for epoch in range(training_epochs):
    t0 = time.time()
    avg_cost = 0
    total_batch = len(data_loader)
    i = 1
    for X, Y in data_loader:
        print(i, end=" ")
        X = X.view(-1, 28 * 28).to(device)
        Y = Y.to(device)
        print(X.shape, Y.shape)

        optimizer.zero_grad()
        hypothesis = linear(X)
        cost = criterion(hypothesis, Y)
        cost.backward()
        optimizer.step()

        avg_cost += cost / total_batch
        i += 1
        t1 = time.time()

    print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.5f}'.format(avg_cost), 'time : ',t1-t0)
print('Learning finished')
</code></pre>
<h3><a name="out-1" class="anchor" href="#out-1"></a>out</h3>
<pre><code class="lang-auto">1 torch.Size([100, 784]) torch.Size([100])
2 torch.Size([100, 784]) torch.Size([100])
3 torch.Size([100, 784]) torch.Size([100])
4 torch.Size([100, 784]) torch.Size([100])
5 torch.Size([100, 784]) torch.Size([100])

...

598 torch.Size([100, 784]) torch.Size([100])
599 torch.Size([100, 784]) torch.Size([100])
600 torch.Size([100, 784]) torch.Size([100])
Epoch: 0001 cost = 0.33757 time :  28.640324354171753
Learning finished
</code></pre>
<p>seems it got 600 data in one iteration which batch size was 100.</p>
<aside class="quote no-group" data-username="Nisan_Aryal" data-post="6" data-topic="73838">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nisan_aryal/48/21895_2.png" class="avatar"> Nisan_Aryal:</div>
<blockquote>
<p>If your batch size is 100 then you should be getting 100 data at one iteration.</p>
</blockquote>
</aside>
<p>this is so confusing</p>

username: Nisan_Aryal
cooked: <p>the code is working fine you got confused.</p>
<p>first the file size is 28*28 pixel which equals 784. the tensor showing torch.Size([100, 784]) it tells that it is loading 100 images of size 784 in one iteration (iteration means one loop inside the data loader) the 600 that is printing is the no. of loop(iteration) it went through to complete the data in dataloader. so you see the dataloader is loading 100 images at one time. hope this clarifies your confusion</p>

username: Yolkandwhite
cooked: <p>Now I’m fully understood. I was confused in iteration in dataloader and outside loop of dataloader… Thank you very much for replying.</p>
<p>Can I ask you another question regarding to code on the very top?<br>
I changed epoch to 10 which will make 3607 data go through the model 10 times.<br>
I didn’t touch the rest of the hyperparameters.<br>
the average cost were all <strong>nan</strong>.</p>
<pre><code class="lang-auto">3607
start training
Epoch: 001 cost = nan time :  143.31351041793823
Epoch: 002 cost = nan time :  144.00150156021118
Epoch: 003 cost = nan time :  143.94768810272217
Epoch: 004 cost = nan time :  144.0049524307251
Epoch: 005 cost = nan time :  143.99128079414368
Epoch: 006 cost = nan time :  143.97400498390198
Epoch: 007 cost = nan time :  143.93819332122803
Epoch: 008 cost = nan time :  144.09143900871277
Epoch: 009 cost = nan time :  143.93545937538147
Epoch: 010 cost = nan time :  143.99345064163208
training Finished!
</code></pre>
<p>Do you have any idea why the average cost in each epoch is all nan?</p>

username: Nisan_Aryal
cooked: <p>from my experience nan occurs when the loss goes to infinity due to various reasons. print the loss(in your case cost) inside the data loader loop(where you printed 1 to 600) if the loss increases exponentially and reaches nan than decrease the learning rate and see again. normally if other things are right then decreasing learning rate will do. else there might be a problem with the model or with the data. hope it helps.</p>

username: Yolkandwhite
cooked: <p>I fixed learning rate to  0.0000001 and printed the loss values in the loop.</p>
<p>I found out that values of loss is somehow zigzaging positive and negative. It looks weird because I’ve leaned that binary crossentropy doesn’t gives negative value.<br>
and the negative loss values were much greater than positive loss value.</p>
<p>does this mean I am giving the wrong data??<br>
or is it just something to do with hyperparameters?</p>
<p>here are part of the outputs I got</p>
<pre><code class="lang-auto">
...

tensor(6.9173, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.1122, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(7.1797, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.8887, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.5463, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.6460, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.9631, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.4065, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.0100, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.5298, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.1861, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(4.7786, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.5828, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.4706, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.6942, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.4059, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.5020, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(6.0613, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.6552, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(5.9129, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-257.4940, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-251.9309, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-497.9838, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-362.6104, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-454.6986, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-897.4257, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-838.2051, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-843.5089, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-823.7191, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-781.8899, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-752.6201, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-602.7383, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-525.3030, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-152.3427, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)
tensor(-209.9590, device='cuda:0',
       grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)

...

</code></pre>

username: Nisan_Aryal
cooked: <p>the binary cross entropy is used in classification when there is two classes. see if the label  is between 0 and 1. it seems your input to calculate the loss is wrong.</p>

username: pourya_farzi
cooked: <aside class="quote no-group" data-username="Yolkandwhite" data-post="7" data-topic="73838">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yolkandwhite/48/17604_2.png" class="avatar"> Yolkandwhite:</div>
<blockquote>
<p><code>criterion</code></p>
</blockquote>
</aside>
<p>First of all, check your model, optimizer function, and type of loss function.<br>
Then specify what is your prediction shape and even your output shape. Perhaps , the loss function would be changed for the task you are working on. For example, with respect to classification, we use cross entropy loss while for regression we need to  apply MSE loss or L1 loss or …<br>
After that you are ready to go for this link to grasp how loss is computed.<br>
<a href="https://discuss.pytorch.org/t/interpreting-loss-value/17665/3">Interpreting loss value - PyTorch Forums</a></p>