CUDA out of memory error for Bert Model 
Author: Backpackerice
Pull request: no
Labels: name: wontfix

Hi there,

I am building a BERT binary classification on SageMaker using Pytorch. Previously when I ran the model, I set the Batch size to 16 and the model were able to run successfully. However, yesterday after I stopped SageMaker and restarted the this morning, I can't run the model with Batch size as 16 any more. I am able to run the model with batch size 8. However, the model is not producing the same result (of course). I didn't change anything else in between. All other settings are the same. (Except I change the SageMaker volume from 30GB to 200GB.)

Does anyone know what may cause this problem? I really want to reproduce the result with batch size 16. 

Any answers will help and thank you in advance!

Comments:
I agree, I had a stable training pipeline for training on TPU and suddenly it broke because it ran out of memory when using the newer versions of Huggingface. I am using the Trainer class. For me the crash happens either during the first evaluation step or right after it.

Also because the Trainer is such a critical code that will be used in production systems in companies and various research labs, it is very important that the Trainer code is stable and is well tested for correctness, performance (iterations/sec) and memory use for training and evaluation. The tests should also cover the various devices it supports, i.e. CPU, GPU and TPU.

It would be great if these tests could run every time a change is made in the trainer code, so that we have confidence that the Trainer is stable. Over the last 3 months I have seen a lot of bugs popping into huggingface master and trying to debug Trainer bugs is very unproductive for Huggingface's users.

The commit id where I do not see an increase in device memory for Trainer 8fcbe486e1592321e868f872545c8fd9d359a515 . I have reverted back to this commit id and my training pipeline works again.

I think whats happening is something changed in the Trainer code and now it suddenly takes a bit more memory. Because most of the people select the training batch size = 1 less than when they start seeing memory failures, the setup becomes extra sensitive to any increase in memory used by the Trainer.
With the current master, I tried training with a lower batch size and it trained properly. Although, I lose convergence speed because I process less examples and the iterations/seconds remain almost the same as the larger batch size.
I would rather revert to the older commit than train with smaller batch sizes.

Hi @Backpackerice 
Would you mind sharing your code? It's hard to investigate a leak with just a general statement.

> Hi @Backpackerice
> Would you mind sharing your code? It's hard to investigate a leak with just a general statement.

Please find below my code:
To explain a little bit, this is trying to run a dual bert - two different two inputs (with attention or concat method). But when I ran into this Cuda issues, I was only using text input from review text (not agent text). 
` 

class ReviewClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = 2

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

        embedding_size = config.hidden_size

        self.classifier = nn.Linear(embedding_size, len(LABEL_NAME))
        self.init_weights()

    def forward(
            self,
            review_input_ids=None,
            review_attention_mask=None,
            review_token_type_ids=None,
            agent_input_ids=None,
            agent_attention_mask=None,
            agent_token_type_ids=None,
            labels=None,
    ):

        review_outputs = self.bert(
            review_input_ids,
            attention_mask=review_attention_mask,
            token_type_ids=review_token_type_ids,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
        )
        
        feature = review_outputs[1]  
        logits = self.classifier(feature)
        outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            pos_weight=torch.tensor(8.85) # N_negative/N_positive from entire training set
            loss_fct = nn.BCEWithLogitsLoss(pos_weight=pos_weight).cuda()
            loss = loss_fct(logits, labels)
            outputs = (loss,) + outputs
        return outputs  # (loss, logits, hidden_states, attentions)  `


> I think whats happening is something changed in the Trainer code and now it suddenly takes a bit more memory. Because most of the people select the training batch size = 1 less than when they start seeing memory failures, the setup becomes extra sensitive to any increase in memory used by the Trainer.
> With the current master, I tried training with a lower batch size and it trained properly. Although, I lose convergence speed because I process less examples and the iterations/seconds remain almost the same as the larger batch size.
> I would rather revert to the older commit than train with smaller batch sizes.

Currently I temporarily solve the issues but creating a new SageMaker. Seems like on the old SageMaker, some phantom python processes were hogging the GPU cards. Also it was acting wired even with the same setting, it will provides significantly different results. 

I encountered similar problems. What I did was to uninstall the latest version of transformers (v3.4.0) and install v3.1.0 instead. My code works fine with the old version.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.