ValueError: Expected input ... whenTraining bert

username: abadrawy
cooked: <p>I am running a batch size of 1, and the input and ouput<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e51549cc5a578fff67a15d65efa80026525eb1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e51549cc5a578fff67a15d65efa80026525eb1f" title="Future Group USA IRS 2020-06-21 at 16.23.18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e51549cc5a578fff67a15d65efa80026525eb1f_2_690x214.png" alt="Future Group USA IRS 2020-06-21 at 16.23.18" data-base62-sha1="8ThQSV7lc9IWbcrPGTgoPmqrGI7" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e51549cc5a578fff67a15d65efa80026525eb1f_2_690x214.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e51549cc5a578fff67a15d65efa80026525eb1f_2_1035x321.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e51549cc5a578fff67a15d65efa80026525eb1f_2_1380x428.png 2x" data-dominant-color="F5F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Future Group USA IRS 2020-06-21 at 16.23.18</span><span class="informations">2548×792 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  tensors look fine to me. so I am not sure what might be the problem<br>
ValueError: Expected input batch_size (1) to match target batch_size (512).</p>
<p>Here is the bert model used, incase it helps,<br>
BertForSequenceClassification(<br>
(bert): BertModel(<br>
(embeddings): BertEmbeddings(<br>
(word_embeddings): Embedding(30522, 768, padding_idx=0)<br>
(position_embeddings): Embedding(512, 768)<br>
(token_type_embeddings): Embedding(2, 768)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(encoder): BertEncoder(<br>
(layer): ModuleList(<br>
(0): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(1): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(2): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(3): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(4): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(5): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(6): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(7): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(8): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(9): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(10): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(11): BertLayer(<br>
(attention): BertAttention(<br>
(self): BertSelfAttention(<br>
(query): Linear(in_features=768, out_features=768, bias=True)<br>
(key): Linear(in_features=768, out_features=768, bias=True)<br>
(value): Linear(in_features=768, out_features=768, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(output): BertSelfOutput(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
(intermediate): BertIntermediate(<br>
(dense): Linear(in_features=768, out_features=3072, bias=True)<br>
)<br>
(output): BertOutput(<br>
(dense): Linear(in_features=3072, out_features=768, bias=True)<br>
(LayerNorm): BertLayerNorm()<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
)<br>
)<br>
)<br>
(pooler): BertPooler(<br>
(dense): Linear(in_features=768, out_features=768, bias=True)<br>
(activation): Tanh()<br>
)<br>
)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(classifier): Linear(in_features=768, out_features=512, bias=True)<br>
)</p>

username: chetan_patil
cooked: <p>The input batch-size should be <code>(512,1)</code> and not <code>(1,512)</code> .</p>

username: abadrawy
cooked: <p>I tried it but got another error, <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d02f73aec54c0fedc75b88e037727d394676308.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d02f73aec54c0fedc75b88e037727d394676308" title="Future Group USA IRS 2020-06-21 at 16.47.53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d02f73aec54c0fedc75b88e037727d394676308_2_690x240.png" alt="Future Group USA IRS 2020-06-21 at 16.47.53" data-base62-sha1="1R6yNdqAfindBbhgyZFWqKNzcBG" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d02f73aec54c0fedc75b88e037727d394676308_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d02f73aec54c0fedc75b88e037727d394676308_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d02f73aec54c0fedc75b88e037727d394676308_2_1380x480.png 2x" data-dominant-color="F6F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Future Group USA IRS 2020-06-21 at 16.47.53</span><span class="informations">2520×880 291 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: abadrawy
cooked: <p>nvm, I forgot to resize input mask as well. I think it’s working now. But would you mind explaining the intuition behind it?</p>

username: chetan_patil
cooked: <p>It’s actually simple.<br>
Pytorch expects inputs such that the first dimension is always the batchsize.</p>