Getting same result every epoch when training Camembert with no error in the code

MABtt: <p>Hello everyone,<br>
For the last couple of days, I have been struggling with some code on how to train the french version of Bert (CamemBert) to classify some tweets (over 60.000) into 10 different classes. My biggest problem is that I don’t get any errors with the code below. It works fine and very well with other databases. With only 5 epochs, it gives higher results but with my database, it structures with a 0.5 F1 Score with even 10 epochs knowing that the 2 databases have the same structure which is even bizarre!</p>
<p>After mounting my drive and install all the needed packages, here is my code :</p>
<pre><code class="lang-auto">df = pd.read_csv('ALL.csv')
</code></pre>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>commentaire</th>
<th>classement</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>Nul à chier</td>
<td>Hate</td>
</tr>
<tr>
<td>1</td>
<td>VIDEO DE MERDE KFC EST TRES BIEN ET LE PDG VS …</td>
<td>Hate</td>
</tr>
<tr>
<td>2</td>
<td>il faut arreter de faire des videos clash roya…</td>
<td>Hate</td>
</tr>
<tr>
<td>3</td>
<td>9 ans ou pas le branleur je me fous en slip ,j…</td>
<td>Hate</td>
</tr>
<tr>
<td>4</td>
<td>Nul</td>
<td>Hate</td>
</tr>
</tbody>
</table>
</div><pre><code class="lang-auto">
#change all the label to int
possible_labels = df.classement.unique()

label_dict = {}
for index, possible_label in enumerate(possible_labels):
    label_dict[possible_label] = index

label_dict
</code></pre>
<p>Neutral              50000<br>
Hate                  3000<br>
Homophobia            3000<br>
Mockery               3000<br>
Racism                3000<br>
Troll                 3000<br>
Moral Harassment      3000<br>
Sexual Harassment     3000<br>
Threat                3000<br>
Insult                3000<br>
Name: classement, dtype: int64</p>
<pre><code class="lang-auto">#split data : training \ test

from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(df.index.values, 
                                                  df.label.values, 
                                                  test_size=0.15, 
                                                  stratify=df.label.values)


df['data_type'] = ['not_set']*df.shape[0]

df.loc[X_train, 'data_type'] = 'train'
df.loc[X_val, 'data_type'] = 'val'

df.groupby(['classement', 'label', 'data_type']).count()

</code></pre>
<p>{‘Hate’: 0,<br>
‘Homophobia’: 1,<br>
‘Insult’: 2,<br>
‘Mockery’: 3,<br>
‘Moral Harassment’: 5,<br>
‘Neutral’: 6,<br>
‘Racism’: 7,<br>
‘Sexual Harassment’: 8,<br>
‘Threat’: 9,<br>
‘Troll’: 10,<br>
nan: 4}</p>
<pre><code class="lang-auto">

#I think the problem cames from here
#I tied to convert the data into lists
msgTrain = df[df.data_type=='train'].commentaire.astype(str).values.tolist()
msgVal = df[df.data_type=='val'].commentaire.astype(str).values.tolist()

tokenizer = CamembertTokenizer.from_pretrained('camembert-base', do_lower_case=True)

encoded_data_train = tokenizer.batch_encode_plus(msgTrain,
                                                 add_special_tokens=True, 
                                                 return_attention_mask = True,
                                                 max_length=50,
                                                 padding=True, 
                                                 truncation=True, 
                                                 return_tensors = 'pt'
                                                 )


encoded_data_val = tokenizer.batch_encode_plus(msgVal, 
                                                 add_special_tokens=True,
                                                 return_attention_mask = True,
                                                 max_length=50,
                                                 padding=True, 
                                                 truncation=True, 
                                                 return_tensors = 'pt'
                                               )


input_ids_train = encoded_data_train['input_ids']
attention_masks_train = encoded_data_train['attention_mask']
labels_train = torch.tensor(df[df.data_type=='train'].label.values)

input_ids_val = encoded_data_val['input_ids']
attention_masks_val = encoded_data_val['attention_mask']
labels_val = torch.tensor(df[df.data_type=='val'].label.values)

dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)
dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)

model = CamembertForSequenceClassification.from_pretrained("camembert-base",
                                                      num_labels=len(label_dict),
                                                      #output_attentions=False,
                                                      #output_hidden_states=False
                                                      )

from torch.utils.data import DataLoader, RandomSampler, SequentialSampler

batch_size = 32


dataloader_train = DataLoader(dataset_train,
                              sampler=RandomSampler(dataset_train), 
                              batch_size=batch_size)

dataloader_validation = DataLoader(dataset_val, 
                                   sampler=SequentialSampler(dataset_val), 
                                   batch_size=batch_size)

from transformers import AdamW, get_linear_schedule_with_warmup

#optimizer
optimizer = AdamW(model.parameters(),
                  lr=1e-3,
                  eps=1e-6)


epochs = 10

scheduler = get_linear_schedule_with_warmup(optimizer, 
                                            num_warmup_steps=0,
                                            num_training_steps=len(dataloader_train)*epochs)

</code></pre>
<pre><code class="lang-auto">from sklearn.metrics import f1_score

def f1_score_func(preds, labels):#F1
    preds_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return f1_score(labels_flat, preds_flat, average='weighted')

def accuracy_per_class(preds, labels):#accurancy
    label_dict_inverse = {v: k for k, v in label_dict.items()}
    
    preds_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()#all class in one vec

    for label in np.unique(labels_flat):
        y_preds = preds_flat[labels_flat==label]
        y_true = labels_flat[labels_flat==label]
        print(f'Class: {label_dict_inverse[label]}')
        print(f'Accuracy: {len(y_preds[y_preds==label])}/{len(y_true)}\n')
</code></pre>
<pre><code class="lang-auto">import random

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
model = model.to(device) 


def evaluate(dataloader_val):

    model.eval()
    
    loss_val_total = 0
    predictions, true_vals = [], []
    

    for batch in dataloader_val:
        
        batch = tuple(b.to(device) for b in batch) 
        
        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }

        with torch.no_grad():       
            outputs = model(**inputs)
            
        loss = outputs[0]
        logits = outputs[1]
        loss_val_total += loss.item()

        logits = logits.detach().cpu().numpy()
        label_ids = inputs['labels'].cpu().numpy()
        predictions.append(logits)#recupr
        true_vals.append(label_ids)
    
    loss_val_avg = loss_val_total/len(dataloader_val) 
    
    predictions = np.concatenate(predictions, axis=0)
    true_vals = np.concatenate(true_vals, axis=0)
            
    return loss_val_avg, predictions, true_vals


for epoch in tqdm(range(1, epochs+1)):
    
    model.train() 
    
    loss_train_total = 0

    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)

    for batch in progress_bar:

        model.zero_grad() 
        
        batch = tuple(b.to(device) for b in batch)
        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }


        outputs = model(**inputs)
        
        loss = outputs[0]
        loss_train_total += loss.item() 
        
        loss.backward() 

        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        optimizer.step()
        scheduler.step()
        
        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})
         

        
    torch.save(model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model') #save epoch stat
        


    tqdm.write(f'\nEpoch {epoch}')#num epoch
    
    loss_train_avg = loss_train_total/len(dataloader_train)          
    tqdm.write(f'Training loss: {loss_train_avg}')
    
    val_loss, predictions, true_vals = evaluate(dataloader_validation)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'Validation loss: {val_loss}')
    tqdm.write(f'F1 Score (Weighted): {val_f1}')
</code></pre>
<p>Results are all the same in every epoch like this :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f37e318934f8308d8a1321e1888e910d015ac2e3.png" data-download-href="https://discuss.pytorch.org/uploads/default/f37e318934f8308d8a1321e1888e910d015ac2e3" title="Screenshot from 2021-09-10 19-03-12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37e318934f8308d8a1321e1888e910d015ac2e3_2_667x500.png" alt="Screenshot from 2021-09-10 19-03-12" data-base62-sha1="yK2opaMlL5w7pJeVOVRRYAL79Bh" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f37e318934f8308d8a1321e1888e910d015ac2e3_2_667x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f37e318934f8308d8a1321e1888e910d015ac2e3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f37e318934f8308d8a1321e1888e910d015ac2e3.png 2x" data-dominant-color="E7EEF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-09-10 19-03-12</span><span class="informations">990×742 99.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>That is all. Can anyone help me please ? maybe I did miss something important!</p>

gphilip: <p>Could you try printing out the set of classes that are predicted by your network? That is, out of the 10 (or 11, if you count “nan” as a valid class) classes in your input, see how many the net reports as having seen. It is possible that your network is predicting only one class, or a couple of classes.</p>
<p>Also: I seem to remember that the F1-score applies only to binary classification? But your input has more than two classes?</p>

MABtt: <p>It got even worse! I used my val set to see what are the predicted classes. I found that the net did not predicted any of them except for the neutral one (all of them! Which even make it harder to understand!) . The others i got just 0s!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3731052933d702d6ad25b486250b2aa6e6a5161.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b3731052933d702d6ad25b486250b2aa6e6a5161" title="Screenshot_20210910-212138_Chrome"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3731052933d702d6ad25b486250b2aa6e6a5161_2_286x500.jpeg" alt="Screenshot_20210910-212138_Chrome" data-base62-sha1="pBtZofqSTUQiPMMblBzW80qFFi9" width="286" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3731052933d702d6ad25b486250b2aa6e6a5161_2_286x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3731052933d702d6ad25b486250b2aa6e6a5161_2_429x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b3731052933d702d6ad25b486250b2aa6e6a5161_2_572x1000.jpeg 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20210910-212138_Chrome</span><span class="informations">720×1257 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As for the f1 score it is just calculated apon the precision and recall. Yes i have 10 class (11 if you count the nan!)</p>

gphilip: <aside class="quote no-group" data-username="MABtt" data-post="3" data-topic="131653">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/eb8c5e/48.png" class="avatar"> MABtt:</div>
<blockquote>
<p>I found that the net did not predicted any of them except for the neutral one</p>
</blockquote>
</aside>
<p>This is what I suspected, from the behaviour that you described in your post.</p>
<p>I am not familiar with BERT, but here are a couple of things you could try to fix this:</p>
<ul>
<li>See if you are missing some normalization step in input processing. Specifically, a normalization step that makes all the input numbers small. An example of such a step from computer vision would be: dividing all pixel values by 255 so that they are all at most 1.</li>
<li>More generally: many networks work best when we scale the input features so that they are all within a comparable range. Is there such a step involved in training BERT? If so, have you done this step?</li>
<li>Reduce the initial learning rate till your net starts finding more than one class. From your code above you have <code>lr</code> set to <code>1e-3</code>. Try reducing it to <code>1e-4</code>, then <code>1e-5</code>, and so on, till more classes start appearing in the predictions. Finding the right learning rate may require some trial and error.</li>
</ul>
<p>Edited to add: From the denominator it looks like all your validation instances have their (correct/ground truth) label set to “Neutral”. This does not look like a good thing for multiclass validation. Usually we want the validation set to have all classes in the same proportion (as far as possible) as the training set.</p>

MABtt: <blockquote>
<p>I am not familiar with BERT, but here are a couple of things you could try to fix this:</p>
<ul>
<li>See if you are missing some normalization steps in input processing. Specifically, a normalization step that makes all the input numbers small. An example of such a step from computer vision would be: dividing all pixel values by 255 so that they are all at most 1.</li>
</ul>
</blockquote>
<p>I really appreciate what you are doing with me. Thanks!!<br>
indeed I did not do any normalization nor preprocessing to my data I just take them as they are for tweeter with their emojis and all the characters that you can find in comments! but I don’t think that is the problem no? Camembert is trained with 148GB of text! well, I am not sure but I just supposed that!</p>
<blockquote>
<ul>
<li>More generally: many networks work best when we scale the input features so that they are all within a comparable range. Is there such a step involved in training BERT? If so, have you done this step?</li>
</ul>
</blockquote>
<p>Yes there is I already put it in the parameters of the model</p>
<pre><code class="lang-auto">encoded_data_train = tokenizer.batch_encode_plus(msgTrain,
                                                 add_special_tokens=True, 
                                                 return_attention_mask = True,
                                                 max_length=50, #Here it is it can be mac 512 and I already change it 
                                                   #many time but it gets the model training very slowly
                                                 padding=True, 
                                                 truncation=True, 
                                                 return_tensors = 'pt'
                                                 )
</code></pre>
<blockquote>
<ul>
<li>Reduce the initial learning rate till your net starts finding more than one class. From your code above you have <code>lr</code> set to <code>1e-3</code>. Try reducing it to <code>1e-4</code>, then <code>1e-5</code>, and so on, till more classes start appearing in the predictions. Finding the right learning rate may require some trial and error.</li>
</ul>
</blockquote>
<p>I also did this  <img src="https://discuss.pytorch.org/images/emoji/apple/roll_eyes.png?v=12" title=":roll_eyes:" class="emoji" alt=":roll_eyes:" loading="lazy" width="20" height="20"> but I got the same results!</p>
<blockquote>
<p>Edited to add: From the denominator it looks like all your validation instances have their (correct/ground truth) label set to “Neutral”. This does not look like a good thing for multiclass validation. Usually we want the validation set to have all classes in the same proportion (as far as possible) as the training set.</p>
</blockquote>
<p>I am not sure what do you mean. can you please explain more ?<br>
Thank you very much!!</p>

gphilip: <aside class="quote no-group" data-username="MABtt" data-post="5" data-topic="131653">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/eb8c5e/48.png" class="avatar"> MABtt:</div>
<blockquote>
<p>I don’t think that is the problem no? Camembert is trained with 148GB of text! well, I am not sure but I just supposed that!</p>
</blockquote>
</aside>
<p>As I said, I am not familiar with how BERT models work. The model <em>may</em> expect that you normalize the input in some manner. The model having been trained on large amounts of data does <em>not</em> mean that you don’t have to preprocess the input. For example, every pre-trained model from TorchVision <a href="https://pytorch.org/vision/stable/models.html#classification" rel="noopener nofollow ugc">expects that its input is normalized</a> in a very particular way:</p>
<pre><code>All pre-trained models expect input images normalized in the same way, 
i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where 
H and W are expected to be at least 224. The images have to be loaded
in to a range of [0, 1] and then normalized using 
`mean = [0.485, 0.456, 0.406]` and `std = [0.229, 0.224, 0.225]`.
</code></pre>
<p>Many of these vision models are trained on much more than 148GB of images. Camembert may also expect some such input normalization. You have to check its documentation to see if this is the case.</p>
<aside class="quote no-group quote-modified" data-username="MABtt" data-post="5" data-topic="131653">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/eb8c5e/48.png" class="avatar"> MABtt:</div>
<blockquote>
<blockquote>
<p>Edited to add: From the denominator it looks like all your validation instances have their (correct/ground truth) label set to “Neutral”. This does not look like a good thing for multiclass validation. Usually we want the validation set to have all classes in the same proportion (as far as possible) as the training set.</p>
</blockquote>
<p>I am not sure what do you mean. can you please explain more ?</p>
</blockquote>
</aside>
<p>This was  a mistake I made in interpreting the second <code>7501</code> in:</p>
<pre><code class="lang-auto">Class: Neutral
Accuracy: 7501/7501
</code></pre>
<p>After looking over the code which prints this, I think this should be fine (provided the logic there is OK; I didn’t look too closely).</p>
<p>Given that Camembert is a well-regarded pre-trained model, my current hypothesis is that the issue is with how you feed it your input. Most likely, you are missing to enforce some requirements on the input. So I would suggest that you go carefully through the documentation for Camembert, and compare your input code with other known successful examples of applying Camembert.</p>

MABtt: <p>Solved!! After all i just put the wrong type for the encoder here :</p>
<pre><code class="lang-auto">msgTrain = df[df.data_type=='train'].commentaire.astype(str).values.tolist()
msgVal = df[df.data_type=='val'].commentaire.astype(str).values.tolist()
</code></pre>
<p>It shouldn’t bee a list!!</p>
<p>I am not quite sure with wht is happening but … it worked!!</p>
<p>Thank you so much!<br>
Peace!!</p>