Attribute Error: 'GenerationConfig' object has no attribute 'lang_to_id'
Author: mlstein-kim
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {'gradient_accumulation_steps': 1, 'offload_optimizer_device': 'nvme', 'offload_optimizer_nvme_path': '/nvme', 'offload_param_device': 'nvme', 'offload_param_nvme_path': '/nvme', 'zero3_init_flag': False, 'zero_stage': 2}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
class GenerateModel(tf.Module):
    def __init__(self, model):
        super(GenerateModel, self).__init__()
        self.model = model

    @tf.function(
        # shouldn't need static batch size, but throws exception without it (needs to be fixed)
        input_signature=[
            tf.TensorSpec((1, 80, 3000), tf.float32, name="input_features"),
        ],
    )
    def serving(self, input_features):
        outputs = self.model.generate(
            inputs=input_features,
            task="transcribe",
            language="
",
            max_new_tokens=450,  # change as needed
            return_dict_in_generate=True,
        )
        return {"sequences": outputs["sequences"]}
```
I have provided `task` and `language` param to the self.model.generate() function, 

Below is just a normal script that converts `whisper-tiny` model into tflite version.
```python
    model = TFWhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
    generate_model = GenerateModel(model=model)

    tf.saved_model.save(
        generate_model,
        args.from_huggingface_dir,
        signatures={"serving_default": generate_model.serving},
    )

    # Convert the model
    converter = tf.lite.TFLiteConverter.from_saved_model(args.from_huggingface_dir)
    converter.target_spec.supported_ops = [
        tf.lite.OpsSet.TFLITE_BUILTINS,  # enable TensorFlow Lite ops.
        tf.lite.OpsSet.SELECT_TF_OPS,  # enable TensorFlow ops.
    ]
    # Perform full integer 8-bit quantization
    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    tflite_model = converter.convert()

    # Save the model
    with open(args.to_tflite_path, "wb") as f:
        f.write(tflite_model)

    # Loading dataset
    ds = datasets.load_from_disk(args.dataset)

    feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny")
    tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny", predict_timestamps=True)
    processor = WhisperProcessor(feature_extractor, tokenizer)
    inputs = feature_extractor(
        ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="tf"
    )
    input_features = inputs.input_features
    labels = ds[0]["text"]

    # loaded model... now with generate!
    interpreter = tf.lite.Interpreter(args.to_tflite_path)

    tflite_generate = interpreter.get_signature_runner()
    generated_ids = tflite_generate(input_features=input_features)["sequences"]
    print("label: ", labels)
    print("prediction: ", generated_ids)
    transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

But the following error has occurred:

```sh
        outputs = self.model.generate(
    File "/home/chris/anaconda3/envs/pt212-cuda122/lib/python3.11/site-packages/transformers/models/whisper/modeling_tf_whisper.py", line 1486, in generate  *
        if generation_config.language in generation_config.lang_to_id.keys():

    AttributeError: 'GenerationConfig' object has no attribute 'lang_to_id'
```

### Expected behavior

I wanted to make sure the `task` and `language` parameters are passed to the function and according tokens are added prior to token generation so to generate korean transcripts successfully. Expected as in the order like below:

  "
": 50258 -> "
": 50264 -> "
": 50359 -> "
": 50363,


However 'lang_to_id' attribute doesn't seem to be included in the 'GenerationConfig' object and emits the error like above. Am I missing something here?

Comments:
Hey @kimbaang 👋 

This issue (https://github.com/huggingface/transformers/issues/25084) seems related -- can you check if the solutions presented there help with your problem? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.