Quantization model initialization routine tries to use quantization_config as if it is a dict
Author: Rexhaif
Pull request: no

The whole problem is here https://github.com/huggingface/transformers/blob/36f183ebab1261b388739d628aaa0b4150068df0/src/transformers/modeling_utils.py#L2389C34-L2389C34

One part of quantized model initialization tries to use `.get(...)` method of `quantization_config`. Presumably, this was made with the assumption that `quantization_config` would be a dict loaded from some json. However, this is not always the case.

For some reason it is not caught by tests but is was caught by me during training with modified [run_classification.py](https://github.com/huggingface/transformers/blob/36f183ebab1261b388739d628aaa0b4150068df0/examples/pytorch/text-classification/run_classification.py).

Here is customized model_init function:
```python
def model_init():
      if not model_args.use_lora and model_args.n_bits in {4, 8}:
          raise ValueError("4 and 8bit modes can only be used with LoRA")

      if model_args.n_bits not in {4, 8, 16, 32}:
          raise ValueError("Only 4, 8, 16, and 32bit modes are supported")

      model_kwargs = {}
      if model_args.n_bits == 4:
          model_kwargs['quantization_config'] = BitsAndBytesConfig(
              load_in_4bit=True,
              bnb_4bit_compute_dtype=torch.bfloat16,
              bnb_4bit_quant_type='nf4',
              bnb_4bit_use_double_quant=True
          )
          model_kwargs['torch_dtype'] = torch.bfloat16
      elif model_args.n_bits == 8:
          model_kwargs['quantization_config'] = BitsAndBytesConfig(
              load_in_8bit=True
          )

      model = AutoModelForSequenceClassification.from_pretrained(
          model_args.model_name_or_path,
          from_tf=bool(".ckpt" in model_args.model_name_or_path),
          config=config,
          cache_dir=model_args.cache_dir,
          revision=model_args.model_revision,
          token=model_args.token,
          trust_remote_code=model_args.trust_remote_code,
          ignore_mismatched_sizes=model_args.ignore_mismatched_sizes,
          **model_kwargs
      )

      if model_args.use_lora:
          if model_args.n_bits in {4, 8}:
              model = peft.prepare_model_for_kbit_training(model)

          lora_config = peft.LoraConfig(
              task_type=peft.TaskType.SEQ_CLS,
              r=128, lora_alpha=64,
              lora_dropout=0.05,
              inference_mode=False
          )
          model = peft.get_peft_model(model, lora_config)
          model.print_trainable_parameters()

      return model
```


And here is the stack trace:
```python
Traceback (most recent call last):
  File "/workspace/notebooks/wmt23/run_regression.py", line 786, in <module>
    main()
  File "/workspace/notebooks/wmt23/run_regression.py", line 724, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1509, in train
    self.model = self.call_model_init(trial)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1239, in call_model_init
    model = self.model_init()
  File "/workspace/notebooks/wmt23/run_regression.py", line 554, in model_init
    model = AutoModelForSequenceClassification.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 516, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2389, in from_pretrained
    quantization_method_from_config = config.quantization_config.get(
AttributeError: 'BitsAndBytesConfig' object has no attribute 'get'
```


Comments:
Thanks for the reproducer. Pinging @younesbelkada 

Hi @Rexhaif , thanks for reporting. I'm unable to reproduce this. Can you give me a minimal reproducible exmaple ? Make sure that you are on the main branch. It is strange that it is calling  `quantization_method_from_config = config.quantization_config.get(` as the config should not have the `quantization_config` field if we are not loading quantized weights. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.