Normalizing 16-bit Medical Images

StefanCepa995: <p>Hi there,</p>
<p>I just started using PyTorch and want to build a patch classifier for breast mammography. Thing is, my image patches are in range from [0, 65535] and I just found out that ToTensor() operation is treating my images as they are 8-bit. Here is the code I am currently using to load my dataset:</p>
<pre><code class="lang-auto">
data_transforms = {

    'train': transforms.Compose([

                                     transforms.RandomHorizontalFlip(),

                                     transforms.ToTensor(),

                                     ]),

    'val':  transforms.Compose([

                                     transforms.ToTensor(),

                                     ]),

    'test':  transforms.Compose([

                                     transforms.ToTensor(),

                                     ])

}
image_datasets = {x: datasets.ImageFolder(os.path.join(raw_images_root_dir, x), data_transforms[x])
                  for x in ['train', 'val', 'test']}
dataloaders = {x : torch.utils.data.DataLoader(image_datasets[x], batch_size=32, shuffle=True, num_workers=multiprocessing.cpu_count())
                for x in ['train', 'val', 'test']}
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>When I try to train my model I am gettig REALLY BAD RESULTS. Basically 50% accuracy throughout entire training. Here is my training setup:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()

learning_rate = 0.01

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.1)

model = train_model(model, criterion, optimizer, scheduler, dataloaders, dataset_sizes, device, num_epochs=11)
</code></pre>
<p>Here is my training info:</p>
<pre><code class="lang-auto">[INFO]: Epoch 0/10
[INFO]: Epoch [0/10], Step [0/134], Loss: 0.6922
[INFO]: Epoch [0/10], Step [20/134], Loss: 0.7699
[INFO]: Epoch [0/10], Step [40/134], Loss: 0.6767
[INFO]: Epoch [0/10], Step [60/134], Loss: 0.7273
[INFO]: Epoch [0/10], Step [80/134], Loss: 0.7482
[INFO]: Epoch [0/10], Step [100/134], Loss: 0.8035
[INFO]: Epoch [0/10], Step [120/134], Loss: 0.7248

[INFO]: train accuracy: 0.5035
[INFO]: train loss: 0.8703

[INFO]: Epoch [0/10], Step [0/34], Loss: 30.9444
[INFO]: Epoch [0/10], Step [20/34], Loss: 38.3757

[INFO]: val accuracy: 0.5000
[INFO]: val loss: 28.2381


[INFO]: Epoch 1/10
[INFO]: Epoch [1/10], Step [0/134], Loss: 0.6472
[INFO]: Epoch [1/10], Step [20/134], Loss: 0.6883
[INFO]: Epoch [1/10], Step [40/134], Loss: 0.6616
[INFO]: Epoch [1/10], Step [60/134], Loss: 0.6833
[INFO]: Epoch [1/10], Step [80/134], Loss: 0.6401
[INFO]: Epoch [1/10], Step [100/134], Loss: 0.6897
[INFO]: Epoch [1/10], Step [120/134], Loss: 0.6746

[INFO]: train accuracy: 0.5091
[INFO]: train loss: 0.7010

[INFO]: Epoch [1/10], Step [0/34], Loss: 0.9954
[INFO]: Epoch [1/10], Step [20/34], Loss: 0.7296

[INFO]: val accuracy: 0.5000
[INFO]: val loss: 0.7585


[INFO]: Epoch 2/10
[INFO]: Epoch [2/10], Step [0/134], Loss: 0.7247
[INFO]: Epoch [2/10], Step [20/134], Loss: 0.7023
[INFO]: Epoch [2/10], Step [40/134], Loss: 0.6870
[INFO]: Epoch [2/10], Step [60/134], Loss: 0.6869
[INFO]: Epoch [2/10], Step [80/134], Loss: 0.6935
[INFO]: Epoch [2/10], Step [100/134], Loss: 0.7037
[INFO]: Epoch [2/10], Step [120/134], Loss: 0.6893

[INFO]: train accuracy: 0.5119
[INFO]: train loss: 0.6927

[INFO]: Epoch [2/10], Step [0/34], Loss: 0.6981
[INFO]: Epoch [2/10], Step [20/34], Loss: 0.7100

[INFO]: val accuracy: 0.5000
[INFO]: val loss: 0.6977


[INFO]: Epoch 3/10
[INFO]: Epoch [3/10], Step [0/134], Loss: 0.7029
[INFO]: Epoch [3/10], Step [20/134], Loss: 0.6941
[INFO]: Epoch [3/10], Step [40/134], Loss: 0.7016
[INFO]: Epoch [3/10], Step [60/134], Loss: 0.6886
[INFO]: Epoch [3/10], Step [80/134], Loss: 0.7015
[INFO]: Epoch [3/10], Step [100/134], Loss: 0.6877
[INFO]: Epoch [3/10], Step [120/134], Loss: 0.6901

[INFO]: train accuracy: 0.5021
[INFO]: train loss: 0.6919

[INFO]: Epoch [3/10], Step [0/34], Loss: 0.6654
[INFO]: Epoch [3/10], Step [20/34], Loss: 0.7031

[INFO]: val accuracy: 0.5000
[INFO]: val loss: 0.6976


[INFO]: Epoch 4/10
[INFO]: Epoch [4/10], Step [0/134], Loss: 0.6859
</code></pre>
<p>I guess this is due to the fact that my images are NOT loaded correctly. When I inspect one of the images with following code:</p>
<pre><code class="lang-auto">dataiter = iter(dataloaders['train'])

images, labels = dataiter.next()
print(images[0].min())
print(images[0].max())
</code></pre>
<p>I get following output:</p>
<pre><code class="lang-auto">
tensor(1.)
tensor(1.)
</code></pre>
<p>How do I load my images correctly, taking into the fact that pixels are between [0, 65535] (16-bit).</p>
<p>Thank you!</p>

ivan-jgr: <p>in what format are your files?<br>
Usually medical images are in some specific format (e.g. DICOM) and you need to apply a window for transform it into gray scale images <a href="https://www.upstate.edu/radiology/education/rsna/intro/display.php" rel="noopener nofollow ugc">https://www.upstate.edu/radiology/education/rsna/intro/display.php</a></p>

StefanCepa995: <p>Yes, I am starting with DICOM images but when I read them with pydicom library they are being converted to N-dim NumPy arrays which have 16-bit pixels.</p>

ivan-jgr: <p>Yes, pydicom returns a <code>int16</code> array-type.</p>
<p>If you want to view it as a greyscale image in rgb format, then you need to know what window level you’re using or need, and adjust appropriately before saving or displaying.</p>
<p>First you have to extract parameters <code>slope</code> and <code>interception</code> from the DICOM file and convert to hounsfield units <code>hu = pixel_value * slope + interception</code> and then apply a correct window.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5_2_690x138.png" alt="image" data-base62-sha1="oVwOnfVWZtL75nfBH73TJoPqxyl" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5_2_690x138.png, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aeb4f90fc7a9c166cfab022e7642ac02c7f6e5f5.png 2x" data-dominant-color="3F76A8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">700×141 21.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This resources can be util for you:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.radiologycafe.com/radiology-basics/imaging-modalities/ct-overview/">
  <header class="source">
      <img src="https://www.radiologycafe.com/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://www.radiologycafe.com/radiology-basics/imaging-modalities/ct-overview/" target="_blank" rel="noopener nofollow ugc" title="04:24PM - 05 September 2017">Radiology Cafe – 5 Sep 17</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/315;"><img src="https://www.radiologycafe.com/wp-content/uploads/2021/05/radiology-basics-full-cover.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://www.radiologycafe.com/radiology-basics/imaging-modalities/ct-overview/" target="_blank" rel="noopener nofollow ugc">CT - Radiology Cafe</a></h3>

  <p>Basic principles of CT and how windowing works. The risks of CT scans in relation to radiation dose and contrast administration are explained.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and this code can be useful, specially see the step 2:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.programmersought.com/article/77814431060/">
  <header class="source">

      <a href="https://www.programmersought.com/article/77814431060/" target="_blank" rel="noopener nofollow ugc">programmersought.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.programmersought.com/article/77814431060/" target="_blank" rel="noopener nofollow ugc">Using pydicom to realize Dicom file reading and CT image window width and...</a></h3>

  <p>Using pydicom to realize Dicom file reading and CT image window width and window level adjustment, Programmer Sought, the best programmer technical posts sharing site.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


StefanCepa995: <p>Thank you so much for your help!</p>

sguerin: <p>Hi <a class="mention" href="/u/ivan-jgr">@ivan-jgr</a> , thanks for sharing. I read your proposed solution but I cant get all the information that I need from de Dicom file.</p>
<p>This is the <code>0x0028</code> group, from here I can take Rows  (<code>ds.Rows</code>) and Columns (<code>ds.Columns</code>)</p>
<pre><code class="lang-bash">(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0008) Number of Frames                    IS: "1"
(0028, 0010) Rows                                US: 512
(0028, 0011) Columns                             US: 512
(0028, 0100) Bits Allocated                      US: 16
(0028, 0101) Bits Stored                         US: 12
(0028, 0102) High Bit                            US: 11
(0028, 0103) Pixel Representation                US: 0
(0028, 0301) Burned In Annotation                CS: 'NO'
(0028, 2110) Lossy Image Compression             CS: '00'
(0028, 9001) Data Point Rows                     UL: 1
(0028, 9002) Data Point Columns                  UL: 0
(0028, 9003) Signal Domain Columns               CS: ''
(0028, 9108) Data Representation                 CS: ''
</code></pre>
<p>But calling <code>ds.WindowWidth</code> or <code>ds.WindowCenter</code> throws an error. I think It’s because that tag is in another group as shown below:</p>
<pre><code class="lang-bash">(0028, 9132)  Frame VOI LUT Sequence  1 item(s) ---- 
      (0028, 1050) Window Center                       DS: "1070.0"
      (0028, 1051) Window Width                        DS: "1860.0"
      (0028, 1055) Window Center &amp; Width Explanation   LO: 'User selection'
      (0028, 1056) VOI LUT Function                    CS: 'LINEAR'
</code></pre>
<p>When I open the image using a Dicom Viewer the tag is shown, but I don’t know how to call a (group, element) that is inside another (group, element) using Pydicom.</p>
<p>Thanks for the help!</p>

Pcamellon: <p>Hello Ivan!</p>
<aside class="quote no-group" data-username="ivan-jgr" data-post="4" data-topic="100833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivan-jgr/48/37403_2.png" class="avatar"> ivan-jgr:</div>
<blockquote>
<p>need to know what window level you’re using or need, and adjust appropriately before saving or displaying</p>
</blockquote>
</aside>
<p>Here are you suggesting to save the windowed images. Do we need to export as 8 bits?<br>
I want to train my model with 16 bits images. Is this possible?</p>

Pcamellon: <p>Hi! I’m planning do the same you did due to the same reason (getting bad results). Does Pytorch change my 16 bits images to 8 bits during transforms?<br>
I think when training DL models windowing is not necessary, we cant see so many shades of gray but machines do!</p>

ivan-jgr: <p>Hi <a class="mention" href="/u/pcamellon">@Pcamellon</a>,</p>
<p>Yes, you can train your model with 16 bits <em>tensors</em>, remember that in an image you can only  have [0, 255] value (i.e. 8 bits).</p>
<p>The usual approach is apply the window and save the image for feed the network with that images (you can Resize, and use the torch.transforms with that images) but if you want to use the original hounsfield values, you can write your own loader for the dicom files.</p>
<p>Also remember that the transformation ToTensor() normalize the images between [0, 1]. Torch will normalize [0, 65536] → [0, 1] or [0, 255] → [0, 1].</p>
<p>Hope this helps you.</p>

Pcamellon: <p>Thanks Ivan for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=12" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:" loading="lazy" width="20" height="20"> !</p>
<aside class="quote no-group" data-username="ivan-jgr" data-post="9" data-topic="100833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivan-jgr/48/37403_2.png" class="avatar"> ivan-jgr:</div>
<blockquote>
<p>remember that in an image you can only have [0, 255] value (i.e. 8 bits)</p>
</blockquote>
</aside>
<p>My images are 16 bits png, so the values go from [0, 65536]. Before ToTensor() transformation does PyTorch change these to 8 bits?</p>
<aside class="quote no-group" data-username="ivan-jgr" data-post="9" data-topic="100833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivan-jgr/48/37403_2.png" class="avatar"> ivan-jgr:</div>
<blockquote>
<p>Torch will normalize [0, 65536] → [0, 1] or [0, 255] → [0, 1]</p>
</blockquote>
</aside>
<p>So the transformation just divides each element from de image matrix representation by 65536 (16 bits in my case) and I get a representation of the same interval [0, 65536] but in float data type [0, 1]. Am I right?</p>

ivan-jgr: <aside class="quote no-group" data-username="Pcamellon" data-post="10" data-topic="100833">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pcamellon/48/43696_2.png" class="avatar"> Pcamellon:</div>
<blockquote>
<p>So the transformation just divides each element from de image matrix representation by 65536 (16 bits in my case) and I get a representation of the same interval [0, 65536] but in float data type [0, 1]. Am I right?</p>
</blockquote>
</aside>
<p>Exactly! ToTensor() doesn’t change your 16 bits to 8 bits, it will only scale between 0 and 1.</p>

nerdymuppet: <p><code>My images are 16 bits png, so the values go from [0, 65536]. Before ToTensor() transformation does PyTorch change these to 8 bits?</code></p>
<p>Exactly. The last Comment here is really the key. Write your own loader. <a href="https://medium.com/analytics-vidhya/creating-a-custom-dataset-and-dataloader-in-pytorch-76f210a1df5d" rel="noopener nofollow ugc">This is a great Resource to get started</a></p>

Pcamellon: <p>Thanks. I’ll check it out. I already finished the project. I decided go with 8 bits <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> Too much complications with 16 and it didn’t worth it.</p>

jiapei100: <p><strong>opencv</strong> is able to deal with <strong>uint16</strong>. However, <strong>transforms.Compose</strong> <strong>ONLY</strong> deals with <strong>PIL</strong>? Rather than <strong>np.ndarray</strong> ?</p>
<p>Take a look at here: <a href="https://pytorch.org/vision/stable/generated/torchvision.transforms.Compose.html" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/generated/torchvision.transforms.Compose.html</a> : <code>transforms.PILToTensor()</code></p>
<p>Hmmm…</p>
<p>And, it looks there is a <code>transforms.ToPILImage()</code>, which should be able to transform <code>np.ndarry</code> to <code>PIL</code>. But, unfortunately, it does <strong>NOT</strong> deal with <strong>uint16</strong>…</p>
<p>So many incompatibilities…</p>