Transformer dimension probelem

username: Gorgen
cooked: <p>hello everyone: when i use the transformer to extract information.<br>
my src size is src = torch.size((368, 2, 256)), and tgt size is tgt = torch.size((768, 2, 256)). the last  dimension is not suitable to transfomer embeding dimension requirements.(512). thus, use nn.linear to project 256 to 512.</p>
<p>However, it report issue. RuntimeError: the feature number of src and tgt must be equal to d_model. How to fix it? Thanks, best wishes</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6_2_690x265.png" alt="image" data-base62-sha1="yf9qBaKlsMfLaZifGIUM8sNFiKy" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6_2_690x265.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6_2_1035x397.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/0/f00063e7c0fb2b16343d0a7d2bf69eb508fd54d6.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1152×443 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09.png" data-download-href="https://discuss.pytorch.org/uploads/default/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09_2_690x188.png" alt="image" data-base62-sha1="7pkGNG0mJDulBNZLocalNT0Q5rH" width="690" height="188" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09_2_690x188.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09_2_1035x282.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/3/33ec55096ca74bf5caa27c3dc1b1e5f79464fd09.png 2x" data-dominant-color="34353C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1161×317 41.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>In the screenshot it seems you re passing <code>nn.Embedding</code> modules to the forward method of <code>nn.Transformer</code>, which is wrong, so I don’t think you are running this code.<br>
In any case, based on the error message the <code>d_model</code> default value of <code>512</code> doesn’t fit the feature dimension and this should work:</p>
<pre><code class="lang-python">trans = nn.Transformer(d_model=256, nhead=16, num_encoder_layers=12)
src = torch.randn(368, 2, 256)
tgt = torch.randn(768, 2, 256)

out = trans(src, tgt)
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier.</p>

username: Gorgen
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="148139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>backticks</p>
</blockquote>
</aside>
<p>I am very grateful for your detailed guidance. Thanks. Best wishes</p>