Evaluation loop breaks after a certain number of samples
Author: megiandoni
Pull request: no

Hi everyone,

I'm using the Trainer (and SFT Trainer, I get the error with both) to finetune/evaluate on gsm8k. Training works well enough but evaluation stops after 100-200 out of 1300 samples and I get no error. The rest of the code simply doesn't run. 
I tried setting eval_accumulations_steps and per_eval_batch_size to 1, to rule out a memory issue, but it didn't help. 
This error occurs whenever I want to return predictions (to calculate metrics myself) or when I define a compute_metrics function. 

Here are my training args:

self.training_args = TrainingArguments(do_train=self.do_train,
                                        do_eval=self.do_eval,
                                        logging_strategy=IntervalStrategy.STEPS,
                                        logging_steps=50,
                                        logging_first_step=True,
                                        evaluation_strategy=IntervalStrategy.STEPS,
                                        eval_steps=100,
                                        learning_rate=2e-4,
                                        output_dir=self.tmp_dir,
                                        overwrite_output_dir=True,
                                        auto_find_batch_size=True,
                                        per_device_eval_batch_size=1,
                                        max_steps=self.max_iterations,
                                        warmup_ratio=0.05,
                                        weight_decay=self.weight_decay,
                                        optim=self.optimizer,
                                        report_to="wandb",
                                        debug="underflow_overflow",
                                        save_strategy='no',
                                        gradient_accumulation_steps=4, 
                                        dataloader_num_workers=4,
                                        fp16=True, 
                                        eval_accumulation_steps=1, 
                                        )

trainer = SFTTrainer(
            model=self.model,
            args=self.training_args,
            train_dataset=train,
            eval_dataset=val,
            formatting_func=formatting_prompts_func,
            tokenizer=self.tokenizer,
            max_seq_length=512,
            peft_config=lora_config,
            data_collator=collator,
            compute_metrics=self.compute_metrics
            )

And here's the formatting_prompts_func just in case it's relevant (doubt it tho)

def formatting_prompts_func(example):
    output_texts = []
    for i in range(len(example['question'])):
        text = f"### Question: {example['question'][i]}\n ### Answer: {example['answer'][i]}"
        output_texts.append(text)
    return output_texts

Comments:
Hi @megiandoni, thanks for raising an issue! 

So that we can best help you could you please: 
* Provide information about the running environment: run `transformers-cli env` in the terminal and copy-paste the output
* Provide a minimal code snippet to reproduce the error? At the moment, important information is missing from the example, in particular all instance attributes of `self` e.g. `self.compute_metrics`, `self.tokenizer` or `self.IntervalStrategy.STEPS`. 
* Some more information about the issue. What is the behaviour observed - do it just hang without executing? Or completely stop i.e. exits out of the evaluation loop? Could you share the logging output of the run? 

cc @muellerzr  @pacman100 

Hey @amyeroberts, thank you so much for your quick response! 
I'll do my best at answering your questions:

**Info about environment:**
- `transformers` version: 4.37.2
- Platform: Linux-6.1.0-13-amd64-x86_64-with-glibc2.36
- Python version: 3.11.6
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

**Info about my code:**
I've defined a "runner" class, and the tokenizer, trainer, metric functions and so on are it's attributes. If you want to reproduce my issue just ignore the "self". 
I'm using this model and tokenizer https://huggingface.co/cekal/mpt-7b-peft-compatible/tree/main

Here's the compute metrics function

def compute_metrics(self, eval_pred):
    print("Compute accuracy for GSM8K dataset.")
    preds, labels = eval_pred
    if isinstance(preds, tuple):
            preds = preds[0]
    preds = np.argmax(preds, axis=-1)
    preds= np.where(preds != -100, preds, self.tokenizer.pad_token_id)
    labels = np.where(labels != -100, labels, self.tokenizer.pad_token_id)
    decoded_results = self.tokenizer.batch_decode(preds, skip_special_tokens=True)
    decoded_labels = self.tokenizer.batch_decode(labels, skip_special_tokens=True)
    correct = [self.is_correct(pred, label) for pred, label in zip(decoded_results, decoded_labels)]
    accuracy = sum(correct) / len(correct)
    metrics = {'accuracy': accuracy}
    return metrics

Here is the code that creates the issue for me:

    `# Get tokenizer and model
    self.tokenizer = AutoTokenizer.from_pretrained(
        self.model_path,
        padding_side="right",
        use_fast=True
    )
    #if self.tokenizer.pad_token is None:
    self.tokenizer.pad_token = self.tokenizer.eos_token
    
    self.model = MptForCausalLM.from_pretrained(self.model_path, trust_remote_code=True, low_cpu_mem_usage=True, device_map="auto", torch_dtype=torch.float16)
    self.model.sequence_len = self.model.config.max_position_embeddings if hasattr(self.model.config, 'max_position_embeddings') else None
    self.model.resize_token_embeddings(len(self.tokenizer))

    # Get data
    dataset = load_dataset("gsm8k", "main", cache_dir=self.dataset_dir) 
    train, val = dataset["train"], dataset["test"]

    self.training_args = TrainingArguments(do_train=True,
                                    do_eval=True,
                                    logging_strategy=IntervalStrategy.STEPS,
                                    logging_steps=50,
                                    logging_first_step=True,
                                    evaluation_strategy=IntervalStrategy.STEPS,
                                    eval_steps=100,
                                    learning_rate=2e-4,
                                    output_dir=self.tmp_dir,
                                    overwrite_output_dir=True,
                                    auto_find_batch_size=True,
                                    per_device_eval_batch_size=1,
                                    max_steps=10,
                                    warmup_ratio=0.05,
                                    weight_decay=0.01,
                                    optim="paged_adamw_8bit",
                                    report_to="wandb",
                                    debug="underflow_overflow",
                                    save_strategy='no',
                                    gradient_accumulation_steps=4, 
                                    dataloader_num_workers=4,
                                    fp16=True, 
                                    eval_accumulation_steps=1,
                                    )

    # Initialize  Trainer
    lora_config = LoraConfig(r=8,
                    lora_alpha=16,
                    target_modules=["Wqkv", "out_proj", "up_proj", "down_proj"],
                    lora_dropout=0.1,
                    bias="all",
                    modules_to_save=["classifier"],
                    task_type="CAUSAL_LM")

    response_template = " ### Answer:"
    collator = DataCollatorForCompletionOnlyLM(tokenizer=self.tokenizer, response_template=response_template)
    trainer = SFTTrainer(
        model=self.model,
        args=self.training_args,
        train_dataset=train,
        eval_dataset=val,
        formatting_func=formatting_prompts_func,
        tokenizer=self.tokenizer,
        max_seq_length=512,
        peft_config=lora_config,
        data_collator=collator,
        compute_metrics=self.compute_metrics
        )
    
    # Train and evaluate
    if self.do_train:
        # Converts params with gradients to float32
        ModelUtils.convert_active_params_to_fp32(self.model) 
        trainer.train()
        # Convert params back to float 16
        for param in self.model.parameters():
            if param.dtype == torch.float32:
                param.data = param.data.to(torch.float16)
    if self.do_eval:
        trainer.evaluate(metric_key_prefix="eval")`

`IntervalStrategy.STEPS` comes from the IntervalStrategy class in transformers.trainer_utils. I've been using that instead of just writing "steps" cause that caused some other issues in the past.

**Issue**
The script execution is interrupted, no evaluation metrics are calculated or logged, and no error is thrown. 
<img width="671" alt="image" src="https://github.com/huggingface/transformers/assets/21020497/de0a9d3f-79f3-4ce4-9099-d97a76cb7adb">

I hope this helps, I'll gladly answer any additional questions!

same issue 
In my case
I found the problem is out of main memory (CPU memory)
you can use task manager to check it 
` eval_accumulation_steps` let torch move to main memory to avoid cuda OOM
but it still large

i according to this to fix the problem
[https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941/13](https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941/13)




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.