Masking in Transformer decoders with -inf rather than 0s

username: ams
cooked: <p>Hi,</p>
<p>I have a question about implementation of masking in Transformer decoders. I understand that the purpose of masking is so we dont peek at future tokens in the target sequence.<br>
I am trying to understand why the subsequent tokens that are masked are filled with -inf rather than 0s. <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L3427" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L3427</a><br>
I imagined that these positions would be filled with 0s so that when dot product is taken, then corresponding future positions would be 0, and so would the gradient. And then there would be no change in parameters in the corresponding future positions when we take a step along the gradient. With the -inf I think the gradients would be -inf/ill-defined? And the resulting solution would have nans in the subsequent positions.</p>
<p>Could you please help me understand the reasoning for filling with -inf?</p>
<p>Thanks!</p>

username: zhangguanheng66
cooked: <p>I guess it is related to <code>softmax</code> function. Usually, you have <code>softmax</code> in transformer/bert architecture and <code>-inf</code> will vanish in that case.</p>

username: vainaijr
cooked: <p>we could carry following experiment to see what is going on while computing gradient, let us assume we have</p>
<pre><code class="lang-auto">l = nn.Linear(2, 2, bias=False)
list(l.parameters())
</code></pre>
<p>[Parameter containing:<br>
tensor([[-0.0473,  0.0679],<br>
[-0.6429, -0.5586]], requires_grad=True)]</p>
<pre><code class="lang-auto">input = torch.tensor([[1.3032, 0.5701], [0.2671, 1.3900]])
</code></pre>
<pre><code class="lang-auto">o1 = l(input)
o1
</code></pre>
<p>tensor([[-0.0230, -1.1563],<br>
[ 0.0817, -0.9482]], grad_fn=<code>&lt;MmBackward&gt;</code>)</p>
<pre><code class="lang-auto">loss_fn = nn.CrossEntropyLoss(reduction='sum')
loss = loss_fn(o1, torch.tensor([0, 1]))
# assume that target is [0, 1]
</code></pre>
<p>softmax (nn.Softmax(dim=-1)) is,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30735f27e309974e0e35d3a3a9383c66f74298ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/30735f27e309974e0e35d3a3a9383c66f74298ee" title="Screenshot (248)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30735f27e309974e0e35d3a3a9383c66f74298ee_2_690x127.png" alt="Screenshot (248)" data-base62-sha1="6UC6hrWdduzfteRxJlGhBqsS5Gu" width="690" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30735f27e309974e0e35d3a3a9383c66f74298ee_2_690x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30735f27e309974e0e35d3a3a9383c66f74298ee_2_1035x190.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30735f27e309974e0e35d3a3a9383c66f74298ee_2_1380x254.png 2x" data-dominant-color="333333"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (248)</span><span class="informations">1808×334 10.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>output after passing input through linear and then softmax would be like,<br>
(w00, w01, w10, w11 are weights)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/212a1178626983e5bca4f7a8d84062c7c2b66bcc.png" data-download-href="https://discuss.pytorch.org/uploads/default/212a1178626983e5bca4f7a8d84062c7c2b66bcc" title="Screenshot (251)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/212a1178626983e5bca4f7a8d84062c7c2b66bcc_2_690x103.png" alt="Screenshot (251)" data-base62-sha1="4JnT0Req9aq3vLchxATguImELRq" width="690" height="103" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/212a1178626983e5bca4f7a8d84062c7c2b66bcc_2_690x103.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/212a1178626983e5bca4f7a8d84062c7c2b66bcc_2_1035x154.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/212a1178626983e5bca4f7a8d84062c7c2b66bcc_2_1380x206.png 2x" data-dominant-color="363636"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (251)</span><span class="informations">1780×266 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>after passing this through cross entropy loss with reduction=‘sum’, (we do not apply softmax externally, cross entropy has inbuilt softmax),<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf506c10687c285b7b1f00dfc189a167fca69936.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf506c10687c285b7b1f00dfc189a167fca69936" title="Screenshot (253)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf506c10687c285b7b1f00dfc189a167fca69936_2_690x90.png" alt="Screenshot (253)" data-base62-sha1="tzZ8G7ZlR3sKjyUDIs3BmKWprUy" width="690" height="90" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf506c10687c285b7b1f00dfc189a167fca69936_2_690x90.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf506c10687c285b7b1f00dfc189a167fca69936_2_1035x135.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf506c10687c285b7b1f00dfc189a167fca69936_2_1380x180.png 2x" data-dominant-color="343434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (253)</span><span class="informations">1738×229 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(since our target was torch.tensor([0, 1]), we considered 0th index in first row, and 1st index in second row)</p>
<p>after taking jacobian and substituting weights with their values,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74364c84c0e65ab752933c4b2c30533502211730.png" data-download-href="https://discuss.pytorch.org/uploads/default/74364c84c0e65ab752933c4b2c30533502211730" title="Screenshot (276)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74364c84c0e65ab752933c4b2c30533502211730_2_690x59.png" alt="Screenshot (276)" data-base62-sha1="gA3IJb1jECFfowu4Ny961tMaToQ" width="690" height="59" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74364c84c0e65ab752933c4b2c30533502211730_2_690x59.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74364c84c0e65ab752933c4b2c30533502211730_2_1035x88.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74364c84c0e65ab752933c4b2c30533502211730_2_1380x118.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (276)</span><span class="informations">1795×154 8.58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/daabc330eff701e2fd20668a4edfaa1c215d4492.png" data-download-href="https://discuss.pytorch.org/uploads/default/daabc330eff701e2fd20668a4edfaa1c215d4492" title="Screenshot (256)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/daabc330eff701e2fd20668a4edfaa1c215d4492_2_690x266.png" alt="Screenshot (256)" data-base62-sha1="vcs5s3gZYV1Cnzk2l50FdSGeNYm" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/daabc330eff701e2fd20668a4edfaa1c215d4492_2_690x266.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/daabc330eff701e2fd20668a4edfaa1c215d4492_2_1035x399.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/daabc330eff701e2fd20668a4edfaa1c215d4492_2_1380x532.png 2x" data-dominant-color="3C3C3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (256)</span><span class="informations">1659×640 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>if we do</p>
<pre><code class="lang-auto">loss.backward()
for param in l.parameters():
  print(param.grad)
</code></pre>
<p>then we get,</p>
<p>tensor([[-0.1206,  0.8855],<br>
[ 0.1206, -0.8855]])</p>
<p>which matches with our calculated gradient.</p>
<p>if our output after applying linear contained <code>float('-inf')</code>, that is it was something like,</p>
<pre><code class="lang-auto">o1 = torch.tensor([[float('-inf'), 2.], [1., 2.]], requires_grad=True)
</code></pre>
<p>then after applying cross entropy,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f05f856fa38a8f57653e20334ebe4d027a88d994.png" data-download-href="https://discuss.pytorch.org/uploads/default/f05f856fa38a8f57653e20334ebe4d027a88d994" title="Screenshot (270)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f05f856fa38a8f57653e20334ebe4d027a88d994_2_690x110.png" alt="Screenshot (270)" data-base62-sha1="yirfgqzAyvVY1QgexhYyg7u6F8M" width="690" height="110" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f05f856fa38a8f57653e20334ebe4d027a88d994_2_690x110.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f05f856fa38a8f57653e20334ebe4d027a88d994_2_1035x165.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f05f856fa38a8f57653e20334ebe4d027a88d994_2_1380x220.png 2x" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (270)</span><span class="informations">1778×284 12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>after computing jacobian,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d88e22df20005115f7b155477b2c5feb7ff3fe3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d88e22df20005115f7b155477b2c5feb7ff3fe3" title="Screenshot (272)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d88e22df20005115f7b155477b2c5feb7ff3fe3_2_690x242.png" alt="Screenshot (272)" data-base62-sha1="fCZjA8U1xjXK7YjXo5FalSqy7vR" width="690" height="242" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d88e22df20005115f7b155477b2c5feb7ff3fe3_2_690x242.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d88e22df20005115f7b155477b2c5feb7ff3fe3_2_1035x363.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d88e22df20005115f7b155477b2c5feb7ff3fe3_2_1380x484.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (272)</span><span class="informations">1517×534 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>substitute with values,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7608ce42358650bf8dd608fdd108cb47022ce312.png" data-download-href="https://discuss.pytorch.org/uploads/default/7608ce42358650bf8dd608fdd108cb47022ce312" title="Screenshot (274)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7608ce42358650bf8dd608fdd108cb47022ce312_2_690x132.png" alt="Screenshot (274)" data-base62-sha1="gQbcS8RE7DNF11kjRGTi4rWcEeK" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7608ce42358650bf8dd608fdd108cb47022ce312_2_690x132.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7608ce42358650bf8dd608fdd108cb47022ce312_2_1035x198.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/7608ce42358650bf8dd608fdd108cb47022ce312_2_1380x264.png 2x" data-dominant-color="3E3E3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (274)</span><span class="informations">1698×327 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>which matches with,</p>
<pre><code class="lang-auto">o1 = torch.tensor([[float('-inf'), 2.], [1., 2.]], requires_grad=True)
loss_fn = nn.CrossEntropyLoss(reduction='sum')
loss = loss_fn(o1, torch.tensor([0, 1]))
loss.backward()
o1.grad
</code></pre>
<p>tensor([[-1.0000,  1.0000],<br>
[ 0.2689, -0.2689]])</p>
<p>all the float(‘-inf’) terms vanish</p>

username: ams
cooked: <p>Ahh yes, I wasn’t thinking of the subsequent softmax operation. We want the elements of softmax output corresponding to future tokens to be 0, and not the dot product itself. Thanks for the prompt response!</p>

username: ams
cooked: <p>Thanks for the detailed response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> May I ask what software/package you used for the symbolic representation in your screenshots? Can you do that with pytorch?</p>

username: vainaijr
cooked: <p>I use sympy for symbolic mathematics.</p>