Size mismatch between tensors - Using BERT model for binary classification

cvillanue: <p>Hello all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m currently working on a project using BERT (Bidirectional Encoder Representations from Transformers). The model is designed to output binary classification, where each instance can be classified into one of two possible classes. In the case of idiom recognition, the model is trained to classify each instance as either an idiom or not an idiom.</p>
<p>PIE Corpus Dataset: <a href="https://arxiv.org/abs/2006.09479" class="inline-onebox" rel="noopener nofollow ugc">[2006.09479] EPIE Dataset: A Corpus For Possible Idiomatic Expressions</a></p>
<p>This dataset contains possible idiomatic expressions instances from 717 idioms divided into two folders:</p>
<pre><code class="lang-auto">Formal Idioms - Idioms which undergo lexical changes.Static Idioms - Idioms which stay the same across instances.
</code></pre>
<p>Each folder contains 3 sentence aligned files with ‘*’ replaced with either ‘Static_Idioms’ or ‘Formal_Idioms’ *_Words.txt :- Original Sentences *_Candidates.txt :- Candidate Idiom whose instance is present in the corresponding sentence. *_Tags.txt :- Sequence labelling tags for each token of the sentence.</p>
<p>Here’s my problem. I’m getting this error when trying to train the BERT model:</p>
<pre><code class="lang-auto">    188     def __init__(self, *tensors: Tensor) -&gt; None:
--&gt; 189         assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors), "Size mismatch between tensors"
    190         self.tensors = tensors
    191 

AssertionError: Size mismatch between tensors
</code></pre>
<p>I understand that this error message indicates that there is a size mismatch between the input tensors. Specifically, the first dimension of all the input tensors should be the same, but they are not. I’ve checked the model node output: Linear(in_features=768, out_features=2, bias=True)</p>
<p>and the shapes:</p>
<pre><code class="lang-auto">print(input_ids.shape)
print(attention_masks.shape)
print(labels.shape)
which would output: 
</code></pre>
<p>torch.Size([1, 512])<br>
torch.Size([1, 512])<br>
torch.Size([3136, 1])</p>
<p>I’m using a DataLoader to create batches of input data however i’m now having doubts whether I’m correctly tokenizing the input sequences or If I need to resize the tensors (if that’s the case I don’t know how to go about it).</p>
<p>Here are the 2 sections of code I wanted to shed light on to see if anyone can catch something funky:</p>
<pre><code class="lang-auto">corpus_path = "Formal_Idioms_Corpus"
corpus_files = [os.path.join(corpus_path, f) for f in os.listdir(corpus_path) if f.endswith(".txt")]

corpus_texts = []
for file_path in corpus_files:
    with open(file_path, "r") as f:
        corpus_texts.append(f.read())

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

max_length = 512

# tokenize input sequences
tokenized_texts = []
for file_path in corpus_files:
    if file_path.endswith("_Words.txt"):
        with open(file_path, "r") as f:
            text = f.read()
        tokenized_text = tokenizer.tokenize(text)[:max_length]
        tokenized_texts.append(tokenized_text)

print(tokenized_texts)

# pad and truncate sequences
input_ids = []
attention_masks = []
for text in tokenized_texts:
    encoded_dict = tokenizer.encode_plus(text, add_special_tokens=True, max_length=max_length, pad_to_max_length=True, return_attention_mask=True)
    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])

input_ids = torch.tensor(input_ids, dtype=torch.long)
attention_masks = torch.tensor(attention_masks, dtype=torch.long)
</code></pre>
<p><strong>and this is the training section:</strong></p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, TensorDataset

# set the batch size and create a PyTorch DataLoader
batch_size = 16

data = TensorDataset(input_ids, attention_masks, labels)
dataloader = DataLoader(data, batch_size=batch_size, shuffle=True)

# set the optimizer and loss function
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-8)
loss_fn = nn.CrossEntropyLoss()

# train the model
model.train()
for epoch in range(num_epochs):
    for batch in dataloader:
        input_ids_batch = batch[0].to(device)
        attention_masks_batch = batch[1].to(device)
        labels_batch = batch[2].squeeze().to(device) # make sure to squeeze the labels tensor
        outputs = model(input_ids_batch, attention_masks_batch)
        logits = outputs[0]
        print('logits shape:', logits.shape)
        print('labels_batch shape:', labels_batch.shape)
        loss = loss_fn(logits, labels_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>Thanks for taking the time to read this — this my first time using the BERT model as well as using a corpus. I also wouldn’t mind sharing the google collab notebook on github if anyone wants to help out! <img src="https://discuss.pytorch.org/images/emoji/apple/star2.png?v=12" title=":star2:" class="emoji" alt=":star2:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I assume you are seeing the error when the forward pass is called in:</p>
<pre><code class="lang-python">outputs = model(input_ids_batch, attention_masks_batch)
</code></pre>
<p>If so, then note that the batch size seems to be <code>1</code> in your previous comment:</p>
<pre><code class="lang-python">print(input_ids.shape)
print(attention_masks.shape)
print(labels.shape)
#which would output: 

torch.Size([1, 512])
torch.Size([1, 512])
torch.Size([3136, 1])
</code></pre>
<p>(let’s ignore <code>labels</code> for now) while you are setting <code>batch_size=16</code> in your <code>DataLoader</code>, so something doesn’t seem right.<br>
Could you add print statements to the <code>DataLoader</code> loop and print the shapes of both input tensors in each iteration and check if all batch sizes are equal?</p>

cvillanue: <p>Hi ptrblck! Thank you for taking the time time comment on my post, I really appreciate it! This is driving me crazy lol</p>
<p>So here’s the snippet of code (ignoring labels) and printing statement to DataLoader.</p>
<pre><code class="lang-auto">batch_size = 16

data = TensorDataset(input_ids, attention_masks)
# Use the DataLoader class to create batches of the Dataset
dataloader = DataLoader(data, batch_size=batch_size, shuffle=True)

for batch in dataloader:
    inputs = batch[0]
    attention_masks = batch[1]

    print("Input shape: ", inputs.shape)
    print("Attention mask shape: ", attention_masks.shape)
    
</code></pre>
<p>This was the output:<br>
Input shape:  torch.Size([1, 512])<br>
Attention mask shape:  torch.Size([1, 512])</p>

ptrblck: <p>Assuming this code prints a single output it would mean that only a single batch with a single sample is available.<br>
What does <code>print(len(dataloader))</code> and <code>print(len(data))</code> return?</p>

cvillanue: <p>They both print out 1</p>

ptrblck: <p>This would mean that only a single sample is available in the entire dataset, which sounds wrong.<br>
Also, I don’t understand how:</p>
<pre><code class="lang-python">AssertionError: Size mismatch between tensors
</code></pre>
<p>could be raised as it seems at least both inputs have the same batch size.<br>
I would guess <code>tokenized_texts</code> is also a list containing a single sample?</p>

cvillanue: <p>Hi ptrblck,<br>
If that’s the case, would this suggest that only one sample is being processed at a time, rather than being batched together? It does sounds wonky…</p>
<p>Yes, I also just double checked on my end - There is only one sample in tokenized_texts.</p>

ptrblck: <aside class="quote no-group" data-username="cvillanue" data-post="7" data-topic="175843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cvillanue/48/60273_2.png" class="avatar"> cvillanue:</div>
<blockquote>
<p>If that’s the case, would this suggest that only one sample is being processed at a time, rather than being batched together? It does sounds wonky…</p>
</blockquote>
</aside>
<p>Yes, only one sample is processed but the main issue is that also your dataset only contains a single sample, which means there are no more samples to load and process.</p>
<aside class="quote no-group" data-username="cvillanue" data-post="7" data-topic="175843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cvillanue/48/60273_2.png" class="avatar"> cvillanue:</div>
<blockquote>
<p>There is only one sample in tokenized_texts.</p>
</blockquote>
</aside>
<p>OK, I also guess this is not expected?<br>
If so, then I would recommend checking why only a single sample is loaded and I would guess either <code>corpus_files</code> also contains a single file or the <code>if file_path.endswith("_Words.txt")</code> condition fails.</p>

cvillanue: <p>Ahhh! So, i’ve decided to play around with the code &amp; made some modifications where all the texts files in the Formal_Idioms_Corpus folder are read instead of the `_Words.txt ’ file.</p>
<pre><code class="lang-auto">corpus_path = "Formal_Idioms_Corpus/"

# create a list of file paths for all *_Words.txt files in the corpus
corpus_files = [os.path.join(corpus_path, f) for f in os.listdir(corpus_path) if f.endswith(".txt")]
</code></pre>
<p>this results in multiple samples in tokenized_texts.<br>
New input shape:<br>
Input shape:  torch.Size([4, 512])<br>
Attention mask shape:  torch.Size([4, 512])<br>
1<br>
4</p>

cvillanue: <p>Update: I realized that I’ve missed a crucial part during preprocessing &amp; that’s what caused the error. So I had to load the data from the three text files instead of one in the corpus and split the sentences into individual words/tokens (why did I not realize this before). For each sentence, i just needed to create a list of candidate idioms and their corresponding tags then convert list into a feature vector that can be fed into the model. I double checked sizes and everything matched!</p>
<p>torch.Size([81289, 64])<br>
torch.Size([81289, 64])<br>
torch.Size([81289])</p>
<p>Currently in the process of training the model (it’s taking a while though – so far i’m 36 minutes in running it on GPU).</p>

cvillanue: <p>Update2: after training and evaluating the model on the validation set… the training accuracy is reported as 0.000, which doesn’t seem right (maybe the model is over fitting the training data or i have to modify the architecture)…</p>
<pre><code class="lang-auto">Epoch 3: Train loss: 0.000 Train accuracy: 0.000 Validation accuracy: 1.000
Test accuracy: 1.000
</code></pre>