`LlamaConfig` is missing default `keys_to_ignore_at_inference`
Author: calico-1226
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I use `transformers.Trainer` to train a `LlamaForSequenceClassification` and provide a `compute_metrics` function, the program failed and returned the following error message:

```bash
Sizes of tensors must match except in dimension 0. Expected size 218 but got size 131 for tensor number 1 in the list.
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 75, in torch_pad_and_concatenate
    return torch.cat((tensor1, tensor2), dim=0)
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 116, in nested_concat
    return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer_pt_utils.py", line 114, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer.py", line 3235, in evaluation_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer.py", line 3029, in evaluate
    output = eval_loop(
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer.py", line 2300, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer.py", line 2019, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/user/Miniconda3/envs/safe-rlhf/lib/python3.10/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/home/user/Projects/classification/train.py", line 193, in train
    trainer.train()
  File "/home/user/Projects/classification/train.py", line 200, in <module>
    train()
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 218 but got size 131 for tensor number 1 in the list.
```

I have identified that the error occurred because the `LlamaConfig` is missing the default `keys_to_ignore_at_inference`, which resulted in the `logits` term incorrectly including the `past_key_values`. Then, when the `Trainer` attempts to `nested_concat` the `logits`, it encounters a dimension mismatch within the tuple. 

The related code in the  `evaluation_loop` to get `logits` term is as follows:

```python
if isinstance(outputs, dict):
        logits = tuple(v for k, v in outputs.items() if k not in ignore_keys + ["loss"])
    else:
        logits = outputs[1:]
```

Because in almost all cases, `logits` should not include `past_key_values`, it is reasonable to add default `keys_to_ignore_at_inference` in the `LlamaConfig`.


### Expected behavior

 Add default `keys_to_ignore_at_inference` in the `LlamaConfig`.

Comments: