Question about Multi-class Semantic Segmentation

username: sarmientoj24
cooked: <p>I basically have <strong>two masks</strong> but I do not know how to prepare it for a semantic segmentation model like DeepLab and U-Net.It has 5 classes (not including the background)</p>
<p>Color Mask<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f280916be2eb759ecc63b3fd9abafcf6cda10206.png" data-download-href="https://discuss.pytorch.org/uploads/default/f280916be2eb759ecc63b3fd9abafcf6cda10206" title="alligator1_color_mask"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f280916be2eb759ecc63b3fd9abafcf6cda10206.png" alt="alligator1_color_mask" data-base62-sha1="yBh0dXiiOKvGn5CUCgRuyGqXenQ" width="690" height="460" data-dominant-color="EAD9DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">alligator1_color_mask</span><span class="informations">900×600 2.04 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Mask<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0da6ef42c96c1a7162656a9861cc0862635c75c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0da6ef42c96c1a7162656a9861cc0862635c75c2" title="alligator1_mask"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0da6ef42c96c1a7162656a9861cc0862635c75c2.png" alt="alligator1_mask" data-base62-sha1="1WLRuT7hOkTKMUzUYl20gRAMods" width="690" height="460" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">alligator1_mask</span><span class="informations">900×600 1.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol>
<li>
<p>Is there a Pytorch function to transform the mask into something readily digestible by the model?</p>
</li>
<li>
<p>My model output is [batcth_size, n_channels, height, width]. What strategy should I use here? Should I create an n_chanbnel=n_classes set of masks with True False per pixel? False being the mask. <strong>Keep in mind that there are only few instances of multiple classes in the image. Usually, it only has one class at a time.</strong>. Should I use 1 channel with class numbers instead?</p>
</li>
<li>
<p>In line with strategy on <span class="hashtag-raw">#2</span>, what should be the loss function and output activation layer?</p>
</li>
</ol>

username: 7029279
cooked: <p>if you are using PIL in the getitem function in your Dataset class, you just have to open your mask image that has all the class masks with mode “P.” I am not sure what the advantage of using one-hot encoding-like mask image. I think its better to have one image with all labels drawn in it.<br>
Here is my code that makes mask images.</p>
<p>Loss function should be fine using BCE or categorical cross entropy.<br>
to answer your first question, your should transform your image to tensor and also perform normalization.</p>
<ul>
<li>Is there a Pytorch function to transform the mask into something readily digestible by the model?</li>
</ul>
<pre><code class="lang-auto">
def make_palatte (categories):
	plt_dict = {0: ["backgorund", (0,0,0)]} 

	all_colors_map = []
	palette = [0,0,0]

	for i in tqdm(range(255)):
		for ii in range(255):
			for iii in range(255):
				adding = [i, ii, +iii]
				all_colors_map.append(adding)
	random.shuffle(all_colors_map)

	distance = len(all_colors_map)/(len(categories)+10)  # 10&gt;&gt; buffer
	distance = math.floor(distance)
	
	for idx, one_categ in enumerate(categories):
		id = idx+1
		name = one_categ # word
		color = all_colors_map[(idx+1)*distance]
		palette.extend(color) 
		plt_dict[id] = [name, tuple(color)]  

	return plt_dict, palette


def mask_maker (palette_dict, img_id, height, width, palette, annotation, export_file_path):
	im = Image.new("P", (width, height), color=(0,0,0)) 

	im.putpalette(palette) 

	d =  ImageDraw.Draw(im)
	if len(annotation) == 0: 
		im.save(export_file_path)
		return
	
	#sort
	temp_non_label = []
	new_anno= []
	for anno in annotation:
		if anno["tags"] == "label":
			new_anno.append(anno)
		else:
			#temp_non_label.append(anno)
			pass

	new_anno.extend(temp_non_label)
	
	for region in new_anno: 
		category_id = region["tags"]
	
		
		for a in palette_dict: 
			if palette_dict[a][0] == category_id:
				category_index = a 
				break
			
		xy_tup_list = [] 
		for point in region["points"]:
			x = point["x"]
			y = point["y"]
			xy_tup_list.append((x, y))
		d.polygon(xy_tup_list, fill=category_index) 
		#print(category_id, "\n", export_dir)
#change to however you have your annotation.
	
	#plt.imshow(im)
	#plt.show()
	#print(im.getcolors())
	#print(im.getpalette())
	im.save(export_file_path) 
</code></pre>

username: sarmientoj24
cooked: <blockquote>
<p>your should transform your image to tensor and also perform normalization.<br>
Should i use normalization on tensor?</p>
</blockquote>
<p>Do you have a code/guide of Semantic segmentation?</p>
<p>Also, could you clarify the inputs of your function? I’m sorry I dont understand it.</p>

username: sarmientoj24
cooked: <aside class="quote no-group" data-username="7029279" data-post="2" data-topic="85298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/7029279/48/23087_2.png" class="avatar"> 7029279:</div>
<blockquote>
<pre><code class="lang-auto">def mask_maker (palette_dict, img_id, height, width, palette, annotation, export_file_path):
	im = Image.new("P", (width, height), color=(0,0,0)) 

	im.putpalette(palette) 

	d =  ImageDraw.Draw(im)
	if len(annotation) == 0: 
		im.save(export_file_path)
		return
	
	#sort
	temp_non_label = []
	new_anno= []
	for anno in annotation:
		if anno["tags"] == "label":
			new_anno.append(anno)
		else:
			#temp_non_label.append(anno)
			pass

	new_anno.extend(temp_non_label)
	
	for region in new_anno: 
		category_id = region["tags"]
	
		
		for a in palette_dict: 
			if palette_dict[a][0] == category_id:
				category_index = a 
				break
			
		xy_tup_list = [] 
		for point in region["points"]:
			x = point["x"]
			y = point["y"]
			xy_tup_list.append((x, y))
		d.polygon(xy_tup_list, fill=category_index) 
		#print(category_id, "\n", export_dir)
#change to however you have your annotation.
	
	#plt.imshow(im)
	#plt.show()
	#print(im.getcolors())
	#print(im.getpalette())
	im.save(export_file_path) 
</code></pre>
</blockquote>
</aside>
<p>what does mask maker do?</p>

username: 7029279
cooked: <p>mask maker makes color png mask imag like the one you have with multiple classes it takes in annotation infomation as input containg polygon coordinates and label names, like “car” and [“x”:10, “y”:20],[“x”:23, “y”:33],[“x”:1, “y”:5]. to make mode P image, you first have to make pallete, thats what the first function is for.</p>

username: sarmientoj24
cooked: <p>I basically dont have the annotations information and only the mask image</p>

username: 7029279
cooked: <p>Oh, I completely misunderstood you then. Then I guess you can just open the image with PIL with mode “P” and transform it into a tensor. your getitem function in Dataset class looks something like this.</p>
<pre><code class="lang-auto">        image_file_path = self.img_list[index]
        raw_img = Image.open(image_file_path)
        raw_img = raw_img.convert('RGB')  
        
        anno_file_path = self.mask_list[index]
        anns_img = Image.open(anno_file_path).convert('P')
        
        raw_img,anns_img  = self.transform(self.phase, raw_img, anns_img)
</code></pre>
<p>torchvision has totensor function.<br>
<a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor</a></p>

username: sarmientoj24
cooked: <p>since i have an n_channel mask of 5 (5 classes), what should the loss function be? also the activation? It seems that my only one channel in my model learns and all segmentations are on this channel</p>