gradient calculation is not correct with gradient accumulation in  LM training 
Author: Jack47
Pull request: no
Labels: name: bug

### System Info

transformer 4.46.3

### Who can help?

@muellerzr 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

run any llm example with gradient accumulation opened

### Expected behavior
TLDR: should use num items in micro-batch instead of num items in batch when calculate cross entropy for micro-batch, currently implementation makes grad-norm lower than usual

with same settings, I compared the grad-norm between megatron and deepspeed:

![image](https://github.com/user-attachments/assets/b722f9d2-8301-4b9a-8cde-2b1fc2844c03)

get_batch_sample should return list instead of scalar, then we should use coordinate num_items_in_batch when call `loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)`

### How to fix
```
diff --git a/trainer.py b/trainer.py
index a19737c..ddecf05 100755
--- a/trainer.py
+++ b/trainer.py
@@ -2416,7 +2416,7 @@ class Trainer:
             epoch_iterator = iter(epoch_dataloader)
             # We chunkify the epoch iterator into gradient accumulation steps `n` batches
             remainder = num_examples % args.gradient_accumulation_steps
-            num_items_in_batch = None
+            num_items_in_batches = None
             if remainder == 0:
                 remainder = args.gradient_accumulation_steps
             update_step = -1
@@ -2424,7 +2424,8 @@ class Trainer:
             for _ in range(total_updates):
                 update_step += 1
                 num_batches = args.gradient_accumulation_steps if update_step != (total_updates - 1) else remainder
-                batch_samples, num_items_in_batch = self.get_batch_samples(epoch_iterator, num_batches)
+                batch_samples, num_items_in_batches = self.get_batch_samples(epoch_iterator, num_batches)
                 for i, inputs in enumerate(batch_samples):
                     step += 1
                     total_batched_samples += 1
@@ -5039,7 +5040,7 @@ class Trainer:
 
     def get_batch_samples(self, epoch_iterator, num_batches):
         batch_samples = []
-        num_items_in_batch = None
+        num_items_in_batches = None
         for _ in range(num_batches):
             try:
                 batch_samples += [next(epoch_iterator)]
@@ -5053,10 +5054,12 @@ class Trainer:
         if len(batch_samples) > 0 and "labels" in batch_samples[0]:
             # For now we don't support object detection
             try:
-                num_items_in_batch = sum([(batch["labels"].ne(-100)).sum() for batch in batch_samples])
+                num_items_in_batches = [(batch["labels"].ne(-100)).sum() for batch in batch_samples]
             except (TypeError, AttributeError):
                 pass
 
         if self.args.average_tokens_across_devices:
-            num_items_in_batch = self.accelerator.gather(num_items_in_batch).sum().item()
-        return batch_samples, num_items_in_batch
+            num_items_in_batches = self.accelerator.gather(num_items_in_batches).sum().item()
+        return batch_samples, num_items_in_batches
``` 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.