Heatmap localization

Mahmoud_Salem: <p>i am trying to generate a heatmap for x-ray image but i don’t know how to get weights of pooling layer of the trained model<br>
i tried some images but the output image looks like a corrupted image.</p>
<p>this is the code…</p>
<pre><code class="lang-auto">class HeatmapGenerator ():
    
    #---- Initialize heatmap generator
    #---- pathModel - path to the trained densenet model
    #---- nnArchitecture - architecture name DENSE-NET121, DENSE-NET169, DENSE-NET201
    #---- nnClassCount - class count, 14 for chxray-14

    model = DenseNet()
    def __init__ (self, pathModel, nnArchitecture, nnClassCount, transCrop):
       
        #---- Initialize the network
        if nnArchitecture == 'DENSE-NET-121': model = densenet121(False).cuda()
        elif nnArchitecture == 'DENSE-NET-169': model = densenet169(False).cuda()
        elif nnArchitecture == 'DENSE-NET-201': model = densenet201(False).cuda()
          
        model = torch.nn.DataParallel(model).cuda()

        modelCheckpoint = torch.load(pathModel)
        model.load_state_dict(modelCheckpoint['best_model_wts'], strict=False)

        self.model = model.module.features
        self.model.eval()
        
        #---- Initialize the weights
        self.weights = list(self.model.parameters())[-2]

        #---- Initialize the image transform - resize + normalize
        normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        transformList = []
        transformList.append(transforms.Resize(transCrop))
        transformList.append(transforms.ToTensor())
        transformList.append(normalize)      
        
        self.transformSequence = transforms.Compose(transformList)
    
    #--------------------------------------------------------------------------------
     
    def generate (self, pathImageFile, pathOutputFile, transCrop):
        
        #---- Load image, transform, convert 
        imageData = Image.open(pathImageFile).convert('RGB')
        imageData = self.transformSequence(imageData)
        imageData = imageData.unsqueeze_(0)
        
        input = torch.autograd.Variable(imageData)
        
        self.model.cuda()
        output = self.model(input.cuda())
        
        #---- Generate heatmap
        heatmap = None
        for i in range (0, len(self.weights)):
            map = output[0,i,:,:]
            if i == 0: heatmap = self.weights[i] * map
            else: heatmap += self.weights[i] * map
        
        #---- Blend original and heatmap 
        npHeatmap = heatmap.cpu().data.numpy()

        imgOriginal = cv2.imread(pathImageFile, 1)
        imgOriginal = cv2.resize(imgOriginal, (transCrop, transCrop))
        
        cam = npHeatmap / np.max(npHeatmap)
        cam = cv2.resize(cam, (transCrop, transCrop))
        heatmap = cv2.applyColorMap(np.uint8(255*cam), cv2.COLORMAP_JET)
              
        img = heatmap * 0.5 + imgOriginal
            
        cv2.imwrite(pathOutputFile, img)
</code></pre>
<p>the bad images:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/695893c4837e6db5b96e9693fcd1d85a3db9242e.png" alt="00000250_011_heatmap" data-base62-sha1="f1VUqQwO7TCUIJF007dvyXVTxBY" width="224" height="224"></p>

ptrblck: <aside class="quote no-group" data-username="Mahmoud_Salem" data-post="1" data-topic="121965">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahmoud_salem/48/38571_2.png" class="avatar"> Mahmoud_Salem:</div>
<blockquote>
<p>but i don’t know how to get weights of pooling layer of the trained model</p>
</blockquote>
</aside>
<p>Pooling layers don’t have any parameters, so could you explain what kind or weights you would like to get and how they would be used?</p>

Mahmoud_Salem: <p>i am trying to generate heatmap with class activation map, so i need values from my trained model but i am not sure the values of which layer should i use…i searched on google and found that i need the values of last convolution layer…and i don’t know how to access this layer to get it’s value.<br>
i am still beginner in deep learning that’s why i am facing difficulties.</p>

carloalbertobarbano: <p>Are you trying to implement Grad-CAM?</p>

Mahmoud_Salem: <p>yes but i have a little experience with it…the problem is that i need some values from my trained model but i don’t know which of them and how to get them</p>

carloalbertobarbano: <p>You don’t need the weights (parameters) of a layer to compute the heatmap with gradcam. What you actually need is the activation maps (output of a layer). You can retrieve that with a forward hook <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.module.register_module_forward_hook — PyTorch 1.8.1 documentation</a></p>
<p>If you refer to the original grad-cam paper (<a href="https://arxiv.org/pdf/1610.02391.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1610.02391.pdf</a>) there are three steps to compute the activation map:</p>
<ol>
<li>Find the activations <code>A</code> and <code>y</code> of a conv (or other) layer and the output layer (done with a forward hook)</li>
<li>compute the importance weights <code>alpha = mean(dy/dA)</code> (Eq. 1)</li>
<li>compute the activation map <code>ReLU(sum(alpha*A))</code> (Eq.2)</li>
</ol>

Mahmoud_Salem: <p>first thing i have x-ray iamge</p>
<p>and want to get final conv-layer in denesenet model<br>
theb get the output of this layer so i can pass the images in it and get<br>
output of this layer then use it in my next process</p>
<p>i can do it using keras by this way but wan to try this way in pytorch and cannot write it<br>
by pytorch libaray</p>
<p>keras code:</p>
<pre><code class="lang-auto"># Get the 512 input weights to the softmax.
    class_weights = model.layers[-1].get_weights()[0]
    final_conv_layer = get_output_layer(model, "bn")
    get_output = kb.function([model.layers[0].input], [final_conv_layer.output, model.layers[-1].output])
    [conv_outputs, predictions] = get_output([np.array([img_transformed])])
    conv_outputs = conv_outputs[0, :, :, ]
</code></pre>

ptrblck: <p>To get the intermediate activation output of a layer you can use forward hooks as described by <a class="mention" href="/u/carloalbertobarbano">@carloalbertobarbano</a>. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">This post</a> gives an example how to use them.</p>