Crossattention problem

Gorgen: <p>Hello everyone:<br>
sorry to disturb us. When I used crossattention, I found all of scores are 1. To be honest, I doubt the crossattention layer I constructed was wrong. Here are coding and scores and models. Could you please give me some advice to address these problem？<br>
Thanks<br>
best wishes</p>
<p>class DotProductAttention(nn.Module):<br>
def <strong>init</strong>(self, key_size, num_hiddens, dropout, **kwargs):<br>
super(DotProductAttention, self).<strong>init</strong>(**kwargs)</p>
<pre><code>    self.key_size = key_size
    self.num_hiddens = num_hiddens
    self.dropout = dropout
    self.dropout = nn.Dropout(dropout)
    self.W_k = nn.Linear(key_size, num_hiddens, bias=False)

def forward(self, queries, keys, values):
    d = queries.shape[-1]
    queries = self.W_k(queries)

    # Set `transpose_b=True` to swap the last two dimensions of `keys`
    scores = torch.bmm(queries, keys.transpose(1, 2)) / math.sqrt(d)
    scores = nn.functional.softmax(scores, dim=-1)
    print('scores', scores.shape, scores)

    return torch.bmm(self.dropout(scores), values)
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:</p>
<pre><code>X = torch.normal(0, 1, (14, 1, 384))
Y = torch.normal(0, 1, (14, 1, 1024))
Y = torch.normal(0, 1, (14, 1, 1024))
print('X',X)
print('Y',Y)
print('Y',Y)

#hyparameters #t_a_a
key_size = 384
num_hiddens = 1024
dropout = 0.5
model = DotProductAttention(key_size, num_hiddens, dropout)
print(model)
output = model(X,Y,Y)
print('output shape: ', output.shape)
</code></pre>
<p>models summary and scores</p>
<p>DotProductAttention(<br>
(dropout): Dropout(p=0.5, inplace=False)<br>
(W_k): Linear(in_features=384, out_features=1024, bias=False)<br>
)<br>
scores torch.Size([14, 1, 1]) tensor([[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]],<br>
[[1.]]], grad_fn=)<br>
output shape:  torch.Size([14, 1, 1024])</p>

Gorgen: <p>By the way: X = torch.normal(0, 1, (14, 1, 384))， （14: seq_length, 1: batchsize, 384: feature dimension). how to fix it? Thanks, best wishes</p>

Matias_Vasquez: <p>The usual order is <code>Batch x Sequence Length x FeatureDimension</code>.</p>
<p>The way you are defining <code>X</code> and <code>Y</code>, it looks like you have 14 batches, each with only one element.</p>
<p>So the attention of one element with respect with only the same element will return 1,since there are no other elements to compare to (due to normalization and soft max, or else you would get the values squared, but always the same)</p>
<p>I have not tried it but I think this should work:</p>
<pre><code class="lang-python">#                       B, Seq, FeatDim
X = torch.normal(0, 1, (1,  14,      384))
Y = torch.normal(0, 1, (1,  14,    1024))
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

Gorgen: <p>Thanks. To be honest, My batchsize is 1, sequcelength is 14, feature dimension is 384.<br>
However, I try your way to exchange dimension. Miraculously it does print out different scores. So the fundamental question is that？<br>
if  batchsize is 1, sequence length is 14, feature dimension is 384. How to fix it?  Thanks, I am grateful for your help. Thanks, best wishes</p>
<h1>
<a name="b-seq-featdim-1" class="anchor" href="#b-seq-featdim-1"></a>B, Seq, FeatDim</h1>
<p>X = torch.normal(0, 1, (1,  14,      384))<br>
Y = torch.normal(0, 1, (1,  14,    1024))</p>

Matias_Vasquez: <p>Sorry I do not understand the question.</p>
<p>Batch = 1<br>
Sequence Length = 14<br>
Feature Dimension = 384</p>
<p>Then the correct order is like the one you posted here.</p>
<aside class="quote no-group" data-username="Gorgen" data-post="4" data-topic="148837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f14d63/48.png" class="avatar"> Gorgen:</div>
<blockquote>
<h1>B, Seq, FeatDim</h1>
<p>X = torch.normal(0, 1, (1, 14, 384))<br>
Y = torch.normal(0, 1, (1, 14, 1024))</p>
</blockquote>
</aside>
<p>What do you need to fix?</p>

Gorgen: <p>sorry. I am grateful for your detailed guidance for newborn children.</p>
<p>You said the input size has 14 batches. Thus, I guess you have missed  something. In fact, the input size (14,1, 384) is (seq_length, batchsize, feature_dimension).<br>
I just put one sample to prove the cross attention works?</p>
<p>By the way, the standard definition of cross attention needed to add 3 different weight layers (query, key, value) . However, in my model I just define query weight, miss key value weight layer. Is it ture?</p>
<p>Thanks<br>
best wishes<br>
jiachen</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7ce30b70b2349c42b09fcc806a072e8cd84f13e.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7ce30b70b2349c42b09fcc806a072e8cd84f13e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7ce30b70b2349c42b09fcc806a072e8cd84f13e_2_690x64.png" alt="image" data-base62-sha1="x4E3uzcfS9xee6y3IRioMo3nAKi" width="690" height="64" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7ce30b70b2349c42b09fcc806a072e8cd84f13e_2_690x64.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7ce30b70b2349c42b09fcc806a072e8cd84f13e_2_1035x96.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7ce30b70b2349c42b09fcc806a072e8cd84f13e.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1119×105 11.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Matias_Vasquez: <p>What I mean with this</p>
<aside class="quote no-group" data-username="Matias_Vasquez" data-post="3" data-topic="148837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/48/47847_2.png" class="avatar"> Matias_Vasquez:</div>
<blockquote>
<p>The way you are defining <code>X</code> and <code>Y</code>, it looks like you have 14 batches, each with only one element.</p>
</blockquote>
</aside>
<p>Is that when you do this like in your original post</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Gorgen" data-post="1" data-topic="148837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f14d63/48.png" class="avatar"> Gorgen:</div>
<blockquote>
<pre><code class="lang-auto">X = torch.normal(0, 1, (14, 1, 384))
Y = torch.normal(0, 1, (14, 1, 1024))
</code></pre>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/x.png?v=12" title=":x:" class="emoji only-emoji" alt=":x:" loading="lazy" width="20" height="20"></p>
<p>You do <strong>NOT</strong> have 1 batch, you are saying 14 batches.</p>
<p>The first number is the batches, so this is wrong.</p>
<p>If you change to this</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Gorgen" data-post="4" data-topic="148837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f14d63/48.png" class="avatar"> Gorgen:</div>
<blockquote>
<h1>B, Seq, FeatDim</h1>
<p>X = torch.normal(0, 1, (1, 14, 384))<br>
Y = torch.normal(0, 1, (1, 14, 1024))</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/heavy_check_mark.png?v=12" title=":heavy_check_mark:" class="emoji only-emoji" alt=":heavy_check_mark:" loading="lazy" width="20" height="20"></p>
<p>Then you have 1 batch. This is correct. 1 batch. 14 seq. 384 feature dimension.</p>

Matias_Vasquez: <aside class="quote no-group" data-username="Gorgen" data-post="6" data-topic="148837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f14d63/48.png" class="avatar"> Gorgen:</div>
<blockquote>
<p>In fact, the input size (14,1, 384) is (seq_length, batchsize, feature_dimension).</p>
</blockquote>
</aside>
<p>This is not correct, you cannot put the batch size in the middle.</p>
<p>It should be</p>
<ol>
<li>Batch</li>
<li>Sequence length</li>
<li>Feature dimension</li>
</ol>

Gorgen: <p>I am really grateful for your detailed guidance. Thanks, best wishes.</p>
<p>Thank you for the pytorch forum. It is such a nice platform.<br>
best wishes</p>