Losing Mask Data

NoahKling: <p>Hi! Im new to PyTorch and Neural Networks as a whole so forgive me if I am wrong in some of my information/understandings (or how I am supposed to make this post). I am currently working with the BRaTs 2020 dataset, and I wanted some help regarding data transformations.</p>
<p>From what I can understand, my masks seem to be one hot encoded:<br>
Data type of mask: &lt;class ‘numpy.ndarray’&gt;<br>
Shape of mask: (240, 240, 3)<br>
Array dtype: uint8<br>
Array max val: 1<br>
Array min val: 0</p>
<p>The library I am using to build my models, segmentation_models.pytorch, expects that masks are not one hot encoded so that it can then do it itself (from what I’ve understood from git issues). So I tried to collapse it all into one channel making each pixel value assign to its class. I then try to sanity check and see if I can reverse it and output each of the masks as an image. Here, I manage to get outputs for only 2/3 masks. I’ve tried various solutions, whether via google or then bashing my head in using ChatGPT, nothing seems to work. Here is a code snippet regarding my data processing of the masks:</p>
<p>mask = mask.transpose((2, 0, 1))<br>
<span class="hashtag-raw">#Setting</span> label 4 to 3 to avoid issues<br>
mask[mask == 4] = 3</p>
<h1><a name="p-460596-convert-mask-from-one-hot-encoding-to-class-indices-1" class="anchor" href="#p-460596-convert-mask-from-one-hot-encoding-to-class-indices-1"></a>Convert mask from one-hot encoding to class indices</h1>
<p>mask = np.argmax(mask, axis=0)  # Now mask shape is (H, W)</p>
<h1><a name="p-460596-add-channel-dimension-to-mask-2" class="anchor" href="#p-460596-add-channel-dimension-to-mask-2"></a>Add channel dimension to mask</h1>
<p>mask = np.expand_dims(mask, axis=0)  # Now mask shape is (1, H, W)</p>
<h1><a name="p-460596-convert-mask-to-tensor-3" class="anchor" href="#p-460596-convert-mask-to-tensor-3"></a>Convert mask to tensor</h1>
<p>mask = torch.tensor(mask, dtype=torch.long)</p>
<p>Please let me know if im making a silly mistake or if you need more code/context, wouldnt let me post images but happy to show the pre and post visualizations I have. I’ve been testing whether the mask exists via this method:</p>
<p>for i in range(1000):<br>
sample_file_path = test_files[i]  # Access each file in the list<br>
with h5py.File(sample_file_path, ‘r’) as file:<br>
mask = file[‘mask’][()]<br>
# If mask is one-hot encoded, convert it to class indices<br>
mask = np.argmax(mask, axis=2)  # Assuming mask shape is (H, W, C)<br>
unique_labels = np.unique(mask)<br>
print(f"Unique labels in the mask for file {i}:", unique_labels)</p>
<p>which returns variations of [0,1,2] with 0 being background</p>

ptrblck: <p>Using <code>argmax</code> to create the indices should work and I’m also able to restore the original one-hot mask using this approach:</p>
<pre data-code-wrap="python"><code class="lang-python"># create one-hot encoded mask
mask = torch.zeros(3, 240, 240).long()
for i in range(mask.size(1)):
    for j in range(mask.size(2)):
        mask[torch.randint(0, 3, (1,)), i, j] = 1
print(mask.min(), mask.max())
# tensor(0) tensor(1)

indices = mask.argmax(dim=0)
print(indices.shape)
# torch.Size([240, 240])
print(indices.min(), indices.max())
# tensor(0) tensor(2)

out = torch.nn.functional.one_hot(indices, num_classes=3)
print(out.shape)
# torch.Size([240, 240, 3])
print(out.min(), out.max())
# tensor(0) tensor(1)

# compare
print((out.permute(2, 0, 1).contiguous() == mask).all())
# tensor(True)
</code></pre>

NoahKling: <p>Thats what I’ve read and what I’ve been trying however it seems to be refusing to work. I tried running your code and while it runs, it does not generate 3’s in the class indices for the 3rd mask. Is this because there is no background channel by default? All 3 channels here are a annotated mask. I also tried outputting mask before any transposing and after we restore the original one hot and the outputs look different, though I should mention that:</p>
<p><span class="hashtag-raw">#Sample</span> image to view<br>
sample_file_path = os.path.join(directory, h5_files[25070])<br>
data = {}<br>
with h5py.File(sample_file_path, ‘r’) as file:<br>
for key in file.keys():<br>
data[key] = file[key][()]</p>
<p><span class="hashtag-raw">#Transpose</span> the mask to have channels first</p>
<p>mask = data[‘mask’].transpose(2, 0, 1)<br>
print(mask.shape)<br>
#(3, 240, 240)<br>
<span class="hashtag-raw">#mask</span>[mask == 4] = 3 <span class="hashtag-raw">#Did</span> not affect results, Annotations comprise the GD-enhancing tumor (ET — label 4), the peritumoral edema<br>
#(ED — label 2), and the necrotic and non-enhancing tumor core (NCR/NET — label 1)<br>
mask = torch.tensor(mask, dtype=torch.long)<br>
print(mask.min(), mask.max())<br>
<span class="hashtag-raw">#tensor</span>(0) tensor(1)</p>
<p>indices = mask.argmax(dim=0)<br>
print(indices.shape)<br>
<span class="hashtag-raw">#torch</span>.Size([240, 240])<br>
print(indices.min(), indices.max())<br>
<span class="hashtag-raw">#tensor</span>(0) tensor(2)<br>
<span class="hashtag-raw">#Set</span> print options to display the full array<br>
torch.set_printoptions(profile=“full”)<br>
print(mask) <span class="hashtag-raw">#Did</span> command + f and search for [[, found 3 different channels, channel 3 contains pixels aka 1’s<br>
print(indices) <span class="hashtag-raw">#Contains</span> 0 (background) 1 2, but no 3’s for third mask<br>
torch.set_printoptions(profile=“default”)<br>
out = torch.nn.functional.one_hot(indices, num_classes=3)<br>
print(out.shape)<br>
<span class="hashtag-raw">#torch</span>.Size([240, 240, 3])<br>
print(out.min(), out.max())<br>
<span class="hashtag-raw">#tensor</span>(0) tensor(1)</p>
<p><span class="hashtag-raw">#compare</span><br>
print((out.permute(2, 0, 1).contiguous() == mask).all())<br>
<span class="hashtag-raw">#tensor</span>(False)</p>

NoahKling: <p>Follow up post, I’ve managed to solve it by doing something strange. My idea that maybe it was because there was no background class was correct (I think), however initially I tried to fix that by simply concatting a empty channel onto the mask. However I needed to make it the first channel, and now I seem to be getting my tensor max after argmax to be 3. I also ran the rest of the code and was able to vizualize all 3 masks from the indices (not converting it from one hot yet, not sure if ill need to after training my model). Anyways, any ideas on why this was the case? Why does adding a empty channel fix the issue?</p>

ptrblck: <p>Your codes are unfortunately not properly formatted so hard to read. If your tensor uses 3 channels, the corresponding indices will be <code>[0, 1, 2]</code> since Python is 0-index based. If you expect a class index of 3 you would need to work with 4 channels.</p>