fix a hidden bug of `GenerationConfig`, now the `generation_config.json` can be loaded successfully
Author: ParadoxZW
Pull request: yes

# What does this PR do?

I was developing an open-source LLM project, and I found that the `generation_config.json` file cannot be successfully loaded to controll model's generation process, though I've written some attributes in this file, such as `eos_token_id` (a newly initialized model object from `from_pretrained` api did not get correct `eos_token_id`).

I am aware of that there're many walkarounds to control the generation process instead of using `generation_config.json`. But I still want to use `generation_config.json` and no more other code, as it should be a standard way. So I dived into the source code of class `GenerationConfig` and spend hours to do debugging stuff.

The initialization process would be called several times during the initialization of a pretrained model. But I found the last time is somehow very strange. Using following code:

```Python
print('before')
logger.info(f"Configuration saved in {output_config_file}") # original L594 of `transformers/generation/configuration_utils.py`
print('after')
```

`before` was printed but `after` was not, as if the function is suddenly returned at here or broken. It gave me clue that there's some problem in the `__repr__` method of `GenerationConfig`. Continue to dive in, I finally located the bug:

```Python
try:
    return json.dumps(config_dict, indent=2, sort_keys=True) + "\n" # original L991 of `transformers/generation/configuration_utils.py`
except Exception:
    print(e)
    raise
```

It gave me the error info of `not supported between instances of 'str' and 'int'`. So it seems that there is some dirty code like `try:... except: pass` outside of the `GenerationConfig` class. Nevertheless, I can finally solve the problem by

```Python
return json.dumps(config_dict, indent=2, sort_keys=False) + "\n"
```

Although only one line of code need to be changed, I believe it's a very hidden bug that one may spend a entire afternoon to find it. Now we can successfully load `generation_config.json` and correctly configure model's generation behavior.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- generate: @gante
- Big Model Inference: @SunMarc

Comments:
Hi @ParadoxZW 👋 

The exception message you got, `not supported between instances of 'str' and 'int'`, seems to point at a failed attempt to sort `str` and `int`. In turn, this means `int` was used as a key of the json file, which is not intended for a `GenerationConfig`. May I ask for a short reproducer of the problem?

Sorting the keys is desirable to visually inspect the files, so I want to be sure what problem we are exactly fixing.

@gante Thanks for your reply.

I'm so sorry that the code cannot be public for now. But it is something like that

```Python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained(
    "../Test_model", 
    torch_dtype=torch.float16, 
    device_map="auto",
    trust_remote_code=True)
```

And I can provide the `config_dict` (a variable in `to_json_string(self, use_diff: bool = True)` function) that causes this error:

```
{
  'max_length': 4096, 
  'pad_token_id': 50256, 
  'bos_token_id': 50256, 
  'eos_token_id': 50295, 
  'transformers_version': '4.31.0', 
  'no_split_module_classes': ['ParallelBlock'], 
  'special_dtypes': {}, 
  'max_memory': {
    0: 1742864712, 
    1: 1742864712, 
    2: 1742864712, 
    3: 11071258624, 
    'cpu': 321020162048
  }
}
```

@ParadoxZW I see. The root issue is that the GPU indexes are stored as integers. Since `max_memory` is not a generation parameter, I will not accept this PR: it supports a custom use case at the cost of readability for all Hub users.

My suggestion would be to store `max_memory` as 
```
'max_memory': {
    '0': 1742864712, 
    '1': 1742864712, 
    '2': 1742864712, 
    '3': 11071258624, 
    'cpu': 321020162048
}
```
and to handle it on your side accordingly 🤗 

I've made another commit and maintained `sort_keys=True`, while solved this problem in a more nice way.

And it seems that `max_memory` is not set by me or any other custom configuration files. It somehow is automatically generated by the program.

Thanks for your reply @amyeroberts !

I've committed your suggestion and committed another change to deal with lists in the `config_dict`.

p.s.

> It seems to be flagging a more general issue about these values being stored in the config: why is this happening in the generation config and not the model's config?

When I was debugging, I found there are some inheritance relationships between the generation config and the model's config. It seems that model's config initialization process also calls the `to_json_string` method defined in `GenerationConfig` class (for some logging procedure I guess). Maybe I'm wrong. 😄 

😱 why there are failed tests?

@ParadoxZW Probably unrelated to this PR :) Seems the run just timed out. I've set it to re-run. 

OK! So will this PR be merged? 😄 @amyeroberts 

@ParadoxZW - yes, now that everything has passed :) 

For future reference of anyone visiting the PR - merged as fix was OK. Will wait for @gante to come back from vacation to confirm the reason for the extra params in the generation config. 

@amyeroberts some models, like Whisper, have a large set of generation options exclusive to the model. Mostly audio models.

The alternative to having custom options would be to create a `GenerationConfig` class for each model, like we do with the model config. However, since the exceptions are so infrequent, imposing a new additional file per model would be wasteful. It also enable our users to do all sort of weird things, at the expense of issues like this once in a while :D