Loss becomes zero after a few dozen pictures

7029279: <p>I am using Coco Dataset’s pictures and mask image that I crreated with the below script to do sematic segmentation.</p>
<p>Why is my cross entropy loss function returning zero after a few dozen pictures?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/028c3807fb5180099bdc656e4454c687aeafda01.png" alt="Screenshot from 2020-04-22 16-54-09" data-base62-sha1="mxn5QmNA4ekcixkF7S0lA6dAE9" width="420" height="360"></p>
<pre><code class="lang-auto">
def make_palatte (classes):
	plt_dict = {0: ["backgorund", (0,0,0)]}

	all_colors_map = []
	palette = []

	for i in range(255):
		for ii in range(255):
			for iii in range(255):
				adding = [i, ii, iii]
				if adding ==[0,0,0] or adding == [255, 255, 255]:
					pass # ["backgorund", (0,0,0)] 
				all_colors_map.append(adding)
			

	distance = len(all_colors_map)/(len(classes)+10)  # buffer
	distance = math.floor(distance)
	for one_class in classes:
		#print(one_class)
		id = one_class["id"] #starts with 1 
		name = one_class["name"] # word
		color = all_colors_map[int(id)*distance]
		palette.extend(color)
		plt_dict[id] = [name, tuple(color)]

		# 1 already taken by background

	palette.extend([255,255,255])
	plt_dict[len(plt_dict)+2] =["ambiguous", (255,255,255)] 
	
	return plt_dict, palette

def mask_maker (palette_dict, img_id, height, width, palette, segmentation, export_dir):
	
	im = Image.new("P", (height, width), color=(0,0,0)) # 0 0 0 &gt;&gt; background
	im.putpalette(palette)
	d =  ImageDraw.Draw(im)
	if len(segmentation) == 0:
		im.save(export_dir)
		return

	for segment in segmentation: #a["segmentation"] = xy coordinates
		#print(segment["segmentation"])
		xy_tup_list = []
		category_id = int(segment["category_id"])
		if len(segment["segmentation"]) == 0:
			im.save(export_dir)
			return 
		for idx, point in enumerate(segment["segmentation"][0]):
			if idx % 2 == 0: #STARTS WITH 0
				x = point
			if idx % 2 !=0:
				y = point
				xy_tup_list.append((x, y))
				x = None
				y = None		
		d.polygon(xy_tup_list, fill=category_id)
		#d.polygon(xy, fill=category_id)
	im.save(export_dir)
</code></pre>
<p>I randomly picked 1000train pictures and 600 val from COCO 2014 dataset excluding ones that have iscrowd=1. I am trying to do semantic segmentation on those 90 coco classes+ background.</p>
<p>I am using Pyramid Scenen Parsing Network which I pretty much copied from here except for the dataloader. <a href="https://github.com/YutaroOgawa/pytorch_advanced/blob/master/3_semantic_segmentation/3-7_PSPNet_training.ipynb" class="inline-onebox" rel="noopener nofollow ugc">pytorch_advanced/3_semantic_segmentation/3-7_PSPNet_training.ipynb at master · YutaroOgawa/pytorch_advanced · GitHub</a>.<br>
This original model did fine tuning with VOC pascal dataset. but i am trying to use COCO datset instead.</p>
<p>To create the mask, I basically drew polygon with PIL’s drawimage and assigned value by using “P” mode and original color pallet that has 90 colors and corresponding numbers.</p>
<p>below is my train function</p>
<pre><code class="lang-auto">def train_model(net, dataloaders_dict, criterion, scheduler, optimizer, num_epochs):


    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    print("available：", device)
    net.to(device)

    torch.backends.cudnn.benchmark = True


    num_train_imgs = len(dataloaders_dict["train"].dataset)
    num_val_imgs = len(dataloaders_dict["val"].dataset)
    batch_size = dataloaders_dict["train"].batch_size


    iteration = 1
    logs = []

    # multiple minibatch
    batch_multiplier = 3

    for epoch in range(num_epochs):

        t_epoch_start = time.time()
        t_iter_start = time.time()
        epoch_train_loss = 0.0  
        epoch_val_loss = 0.0  

        print('-------------')
        print('Epoch {}/{}'.format(epoch+1, num_epochs))
        print('-------------')

        for phase in ['train', 'val']:
            if phase == 'train':
                net.train()  
                scheduler.step()  
                optimizer.zero_grad()
                print('（train）')

            else:
                if((epoch+1) % 5 == 0):
                    net.eval()   
                    print('-------------')
                    print('（val）')
                else:
                    continue

            count = 0  # multiple minibatch
            for imges, anno_class_imges in dataloaders_dict[phase]:
                
                if imges.size()[0] == 1:
                    continue


                imges = imges.to(device)
                anno_class_imges = torch.squeeze(anno_class_imges)
                anno_class_imges = anno_class_imges.to(device)

                
                # multiple minibatch
                if (phase == 'train') and (count == 0):
                    optimizer.step()
                    optimizer.zero_grad()
                    count = batch_multiplier

          
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = net(imges)
                    loss = criterion(
                        outputs, anno_class_imges.long()) / batch_multiplier
                    print("loss:  " loss)
                    
                    # 
                    if phase == 'train':
                        loss.backward()  #
                        count -= 1  # multiple minibatch

                        if (iteration % 10 == 0):  
                            t_iter_finish = time.time()
                            duration = t_iter_finish - t_iter_start
                            print('iteration {} || Loss: {:.4f} || 10iter: {:.4f} sec.'.format(
                                iteration, loss.item()/batch_size*batch_multiplier, duration))
                            t_iter_start = time.time()

                        epoch_train_loss += loss.item() * batch_multiplier
                        iteration += 1


                    else:
                        epoch_val_loss += loss.item() * batch_multiplier

       
        t_epoch_finish = time.time()
        print('-------------')
        print('epoch {} || Epoch_TRAIN_Loss:{:.4f} ||Epoch_VAL_Loss:{:.4f}'.format(
            epoch+1, epoch_train_loss/num_train_imgs, epoch_val_loss/num_val_imgs))
        print('timer:  {:.4f} sec.'.format(t_epoch_finish - t_epoch_start))
        t_epoch_start = time.time()


        log_epoch = {'epoch': epoch+1, 'train_loss': epoch_train_loss /
                     num_train_imgs, 'val_loss': epoch_val_loss/num_val_imgs}
        logs.append(log_epoch)
        df = pd.DataFrame(logs)
        df.to_csv("log_output.csv")


    torch.save(net.state_dict(), 'weights/pspnet50_' +
               str(epoch+1) + '.pth')
num_epochs = 30
torch.cuda.empty_cache()
train_model(net, dataloaders_dict, criterion, scheduler, optimizer, num_epochs=num_epochs)

</code></pre>

ptrblck: <p>Could you explain your use case a bit, please?<br>
How did you create the masks? Which values do they have and which shape?<br>
What kind of model are you using and how many data samples are you using?</p>

7029279: <p>Thank you so much. I added some more information.</p>

7029279: <p>This is one of my mask images. What is kind of curious is that I have to do squeeze to get rid of one dimension before feeding it to GPU, which the original model did not have to do. Could this be related?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/758be0cd75643566a1503f6f56baee367421b92d.png" data-download-href="https://discuss.pytorch.org/uploads/default/758be0cd75643566a1503f6f56baee367421b92d" title="COCO_train2014_000000581097"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/758be0cd75643566a1503f6f56baee367421b92d.png" alt="COCO_train2014_000000581097" data-base62-sha1="gLRycdjSNSckNxG6mObGl8oyLbn" width="375" height="500" data-dominant-color="55DDB3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">COCO_train2014_000000581097</span><span class="informations">480×640 2.48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="7029279" data-post="4" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/7029279/48/23087_2.png" class="avatar"> 7029279:</div>
<blockquote>
<p>What is kind of curious is that I have to do squeeze to get rid of one dimension before feeding it to GPU, which the original model did not have to do.</p>
</blockquote>
</aside>
<p>That sounds weird.<br>
Which dimension are you squeezing and which shape does the target have before squeezing it?<br>
The expected shape for a multi-class segmentation for the target is <code>[batch_size, height, width]</code> and the target tensor should contain the class indices in the range <code>[0, nb_classes-1]</code>, if you are using <code>nn.CrossEntropyLoss</code>.</p>

7029279: <p>This is pre-squeeze size of my target data&gt;&gt; targets of size: : [4, 1, 300, 300] squeeze reduces this to [4,300,300]<br>
also I noticed that the size of the target image does not match the roiginal image.</p>
<p>Also a side question. Is there any way to make masks out of xy coordinates? do people code tehir own script everytime?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eafbfa3207e1dc1ac83d4bd8856fab1eb407497.jpeg" alt="COCO_train2014_000000581097" data-base62-sha1="be5Mv9kTsAOzleTijfwzQGhRye3" width="640" height="480"></p>

ptrblck: <p>What min and max values does the target contain?<br>
The shape looks OK after you squeeze it, if <code>target</code> is a <code>LongTensor</code> and contains the class indices in the range  <code>[0, nb_classes-1]</code>.</p>
<aside class="quote no-group" data-username="7029279" data-post="6" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/7029279/48/23087_2.png" class="avatar"> 7029279:</div>
<blockquote>
<p>Also a side question. Is there any way to make masks out of xy coordinates? do people code tehir own script everytime?</p>
</blockquote>
</aside>
<p>I would guess that you could find some good implementations in this discussion board or alternatively for numpy (which should be easily portable to PyTorch) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

7029279: <p>I dont know how to interpret these.<br>
print("max: ", torch.max(anno_class_imges), "min: ", torch.min(anno_class_imges))</p>
<p>max:  tensor(0.3294) min:  tensor(0.)</p>
<p>this is my datatransoform</p>
<pre><code class="lang-auto">
    def __init__(self, input_size):
        self.data_transform = transforms.Compose([
                    transforms.Resize((input_size, input_size)),
                    transforms.ToTensor()
                ])
        
</code></pre>

ptrblck: <p>Are you transforming the target image with the provided transformation or just the data tensor?<br>
In the former case, you will get a normalized target, which is wrong for a segmentation use case.<br>
You would have to map the colors of your target to class indices.<br>
E.g. “blue” -&gt; class0, “red” -&gt; class1 etc.<br>
<a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">Here</a> is an example on how to create a color mapping.</p>

7029279: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>class1</p>
</blockquote>
</aside>
<p>the guy I copied from yes does normalize the data, but I did not, so I just commented out those lines… I just made PIL image into tensors. I did map 90 colors  + background in the range of 0, 0, 0 to 255, 255, 255 and I did PIL’s putlpalette to each mask image I was creating.</p>

7029279: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">Here </a> is an example on how to create a color mapping.</p>
</blockquote>
</aside>
<p>Should my torch max and min values have integer values, instead of float? I thought I assigned numbers ranging from 0 to 91 in my color map. Also, you are using cmap in matplolib instead of PIL, is using PIL not reccomended?</p>

ptrblck: <aside class="quote no-group" data-username="7029279" data-post="11" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/7029279/48/23087_2.png" class="avatar"> 7029279:</div>
<blockquote>
<p>Should my torch max and min values have integer values, instead of float?</p>
</blockquote>
</aside>
<p>Yes, the target should be a <code>LongTensor</code>. The min values should be 0, while the max value should be <code>nb_classes-1</code>.</p>
<aside class="quote no-group" data-username="7029279" data-post="11" data-topic="77814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/7029279/48/23087_2.png" class="avatar"> 7029279:</div>
<blockquote>
<p>Also, you are using cmap in matplolib instead of PIL, is using PIL not reccomended?</p>
</blockquote>
</aside>
<p>No, use whatever works for you. I’m probably more familiar with the <code>matplotlib</code> color map than with <code>PIL</code>, but don’t have any recommendations.</p>

7029279: <p>thank you so much.You have been very kind. I’ll look into my code a little bit more.</p>

7029279: <p>I normalized the target and original image data and it somehow solved the problem which I don’t quiet understand.  closing the issue.</p>