How to plot Saliency map from RNN model in NLP task?

Mengran_Wang: <p>Hi, I’m trying to reproduce results from this article “<a href="https://arxiv.org/pdf/1506.01066v2.pdf" rel="noopener nofollow ugc">Implementations of saliency models described in "Visualizing and Understanding Neural Models in NLP</a>”.<br>
By building an very simple RNN model (for binary classificaition):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e74c9fa8e58a8f543d2419420216571524545a9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1e74c9fa8e58a8f543d2419420216571524545a9" title="GRU_model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e74c9fa8e58a8f543d2419420216571524545a9_2_690x392.jpeg" alt="GRU_model" data-base62-sha1="4lqxGD1dm9eoMXDPEnkOAzpIPoZ" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e74c9fa8e58a8f543d2419420216571524545a9_2_690x392.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e74c9fa8e58a8f543d2419420216571524545a9_2_1035x588.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e74c9fa8e58a8f543d2419420216571524545a9_2_1380x784.jpeg 2x" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">GRU_model</span><span class="informations">2224×1264 311 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and training on IMDB dataset from torchtext datasets.IMDB, the model perform well on binary classification.<br>
Then I start to call saliency using the well-trained model by the following code (referd from <a href="https://medium.com/datadriveninvestor/visualizing-neural-networks-using-saliency-maps-in-pytorch-289d8e244ab4" rel="noopener nofollow ugc">https://medium.com/datadriveninvestor/visualizing-neural-networks-using-saliency-maps-in-pytorch-289d8e244ab4</a>,This Saliency is generated from CNN model )<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87a386ae64ea8730b56334b2f108cdbf7b2f12db.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/87a386ae64ea8730b56334b2f108cdbf7b2f12db" title="Saliency_error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87a386ae64ea8730b56334b2f108cdbf7b2f12db_2_690x194.jpeg" alt="Saliency_error" data-base62-sha1="jlUOjDHaBzN6dxa2Eh4kqTejvir" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87a386ae64ea8730b56334b2f108cdbf7b2f12db_2_690x194.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87a386ae64ea8730b56334b2f108cdbf7b2f12db_2_1035x291.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87a386ae64ea8730b56334b2f108cdbf7b2f12db_2_1380x388.jpeg 2x" data-dominant-color="1C1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Saliency_error</span><span class="informations">2750×776 218 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, when I run “model.eval()”, the error occur: RuntimeError: cudnn RNN backward can only be called in training mode.</p>
<p>I’m really confused why CNN can go through saliency easily but RNN goes very difficult? Is there any solution to generate saliency map for text classification (to show which word is most important for model prediction) by RNN model? Thanks a lot!</p>

shekhar: <p>Hi, can you paste your complete code here. Include the training phase of your code. Maybe you’re missing something there.</p>
<p>Also check if you’ve this same problem raised here <a href="https://discuss.pytorch.org/t/cudnn-rnn-backward-can-only-be-called-in-training-mode/37622" class="inline-onebox">Cudnn RNN backward can only be called in training mode</a></p>

Mengran_Wang: <p>Thanks shekhar! I read the problem you pasted here but it seems not same with my question. I want to figure out <strong>how to use backward()</strong> when doing <strong>saliency</strong> from <strong>RNN</strong> model.<br>
The complete code is <a href="https://github.com/amdreamer/Machine-Learning-Programs/blob/master/l3-textClassification_embedOutOfModel.py" rel="nofollow noopener">here</a>, and the Saliency part start form line 262.</p>

vdw: <p>Well, just to see if it work, stay in the training mode with <code>model.train()</code> so you can perform <code>backward()</code>. Just avoid doing the optimizer step to actually change the trainable parameters. I can see no harm here.</p>
<p>Apart from that, I’ve tried to visualize the importance of words when using an RNN using attention, where the attention weights indicate which word had the most affect on the prediction. It’s not unintuitive but <a href="https://arxiv.org/abs/1902.10186" rel="nofollow noopener">has been shown to have problems</a>.</p>

Mengran_Wang: <p>Thanks for your advices! It works! by perform backward() in model.train(), and go through forward path without updating weights of parameters, the saliency looks right.<br>
when I input “This film is horrible!” the saliency can tell the word “horrible” is most important for prediction the emotion of this review.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55d4cfcc8bfb72e818fc9a51700961e095560a02.png" alt="image" data-base62-sha1="cfiwYWNQMk16XvN2eF1keEK5yWS" width="511" height="234"><br>
The following is the updated code:</p>
<pre><code class="lang-auto"># tring Saliency now!
# for pre-processing the sentence (sentence to vectors)
def pre_process(sentence):
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    tensor = torch.LongTensor(indexed).to(device) # seq_len long tensor is important
    tensor = tensor.unsqueeze(1) # seq_len * batch_size(1)
    embedded = torch.tensor(embed(tensor),requires_grad=True) 
# here we need to get gradient for saliency computation, use requires_grad=True
    
    return embedded
# deal with input sentence
input_1 = u"This film is horrible!"
input_1 = u"This movie was sadly under-promoted but proved to be truly exceptional."
preprocess_1 = pre_process(input_1) # requires_grad = True
# we would run the model in evaluation mode

model.train() 
# if I set model.eval(), an error occur: RuntimeError: cudnn RNN backward can only be called in training mode

'''forward pass through the model to get the scores, note that RNNModel_GRU2 model doesn't perform sigmoid at the end
and we also don't need sigmoid, we need scores, so that's perfect for us.
'''
scores = model(preprocess_1)
'''
backward function on score_max performs the backward pass in the computation graph and calculates the gradient of 
score_max with respect to nodes in the computation graph
'''
scores.backward()
'''
Saliency would be the gradient with respect to the input now. 
But note that the input has 100 dim embdeddings. 
To derive a single class saliency value for each word (i, j),  
we take the maximum magnitude across all embedding dimensions.
'''
saliency, _ = torch.max(preprocess_1.grad.data.abs(),dim=2) # AttributeError: 'NoneType' object has no attribute 'data'
</code></pre>

Mengran_Wang: <p>There’s still a small problem when I avoided to do optimizer step after calling backward() in training mode. the problem is: <strong>for the same input sentence and the same model, the saliencies are different between the first time and second time I called</strong>. If the parameters are totally fixed, the results of saliency should be consistent. I guess that because my model contains a <strong>dropout layer</strong> and the randomness is generated in model.train() from drop out layer. <strong>Any ideas about how to deactivate drop out layer in model.train() mode?</strong><br>
Saliency for the first time calling backward():<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55d4cfcc8bfb72e818fc9a51700961e095560a02.png" alt="image" data-base62-sha1="cfiwYWNQMk16XvN2eF1keEK5yWS" width="511" height="234"><br>
Saliency for the second time calling backward():<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d4b89fed4b3a6817dd5fda3967f1ab352fe7bdb.png" alt="image" data-base62-sha1="fARSVxhXEUHeKlYs5SegRO7QS4X" width="497" height="217"><br>
The Saliency for each word is getting bigger.</p>
<p>Well actually I figured out how to do this, <strong>just set the drop out layer to eval by adding model.dropout.eval()</strong>, now the Saliency become consistent.</p>

vdw: <p>Yes, the Dropout layer was still a non-deterministic part of your network. This is one of the main reasons for <code>eval()</code>. I’ve never tried it, but I assume setting the dropout probability to <code>0.0</code> would have worked as well. Not sure, thought.</p>

Mengran_Wang: <p>Yeah, the problem is solved by setting drop out layer to eval mode but leaving the rest of the model to train model. Many thanks for your time~</p>