Broken generation tests for quantized models
Author: poedator
Pull request: no

### System Info
Ubuntu-20, A100
Python 3.10.13, pytest-7.4.2, pluggy-1.3.0, torch==2.0.1, cuda=11.7
quite fresh transformers from repo

### Who can help
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada
- generate: @gante

### problem description
When working on PR on quantization, I noticed that some tests fail. Those tests also failed in `main` branch.
The tests are:
- 5 generation-related tests in BNB  with slight deviations from the target phrases. 
- 2 tests with `config_class` mismatch in BnB [FIXED  - see below]
- 11 generation-related tests in GPTQ  with slight deviations from the target phrases. 
```
=========================== BNB TESTS - GENERATION ============================
7b.0b57768f52b7775563f7cc78c4724e407b39593b.configuration_mpt.MPTConfig'> a...
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_generate_quality - AssertionError: "Hello my name is John Doe, and I'm a fan of the" != "Hello my name is John Doe, and I'm a big fan of"
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_generate_quality_config - AssertionError: "Hello my name is John Doe, and I'm a fan of the" != "Hello my name is John Doe, and I'm a big fan of"
7b.0b57768f52b7775563f7cc78c4724e407b39593b.configuration_mpt.MPTConfig'> a...
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_int8_from_pretrained - AssertionError: "Hello my name is John Doe, and I'm a fan of the" != "Hello my name is John Doe, and I'm a big fan of"
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_int8_serialization - AssertionError: "Hello my name is John Doe, and I'm a fan of the" != "Hello my name is John Doe, and I'm a big fan of"
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_int8_serialization_sharded - AssertionError: "Hello my name is John Doe, and I'm a fan of the" != "Hello my name is John Doe, and I'm a big fan of"

============ BNB MODEL CLASS MISMATCH =============
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8Test::test_get_keys_to_not_convert - ValueError: The model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has <class 'transformers_modules.mosaicml.mpt-
FAILED tests/quantization/bnb/test_mixed_int8.py::MixedInt8GPT2Test::test_get_keys_to_not_convert - ValueError: The model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has <class 'transformers_modules.mosaicml.mpt-

=========================== GPTQ TESTS ============================
FAILED tests/quantization/gptq/test_gptq.py::GPTQTest::test_change_loading_attributes - AssertionError: 'Hello my name is A.I. and I am a student of' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my name is ...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTest::test_generate_quality - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTest::test_serialization - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTest::test_serialization_big_model_inference - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMap::test_change_loading_attributes - AssertionError: 'Hello my name is A.I.I.I.I.' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my name is John, I am a stu...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMap::test_generate_quality - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMap::test_serialization - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMap::test_serialization_big_model_inference - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMapExllama::test_generate_quality - AssertionError: 'Hello my name is Alyson and I am a beautiful, sexy' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my n...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMapExllama::test_serialization - AssertionError: 'Hello my name is A.I. and I am a student of' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my name is ...
FAILED tests/quantization/gptq/test_gptq.py::GPTQTestDeviceMapExllama::test_serialization_big_model_inference - AssertionError: 'Hello my name is A.I.I.I.I.' not found in {'Hello my name is Alyson, I am a student in the', 'Hello my name is John, I am a professional photographer and I', 'Hello my name is Alyson and I am a very sweet,', 'Hello my name is John, I am a stu...
```
### Generation: possible causes and solutions
I tend to believe that generation problems are  caused by slight modifications in models and packets versions or some other random factors, not controlled for. [See how in GPTQ tests there are already several valid answers](https://github.com/huggingface/transformers/blob/6824461f2a35546a3d781fe60576e00f6db7bedf/tests/quantization/gptq/test_gptq.py#L87-L93). 
I propose to validate the new generation results. See my PR #26535

Comments:
### Model class mismatch: possible causes and solutions
[TODO] research on causes and propose solution
Error message with class names side-by-side:
```
(model has 
<class 'transformers_modules.mosaicml.mpt-7b.0b57768f52b7775563f7cc78c4724e407b39593b.configuration_mpt.MPTConfig'> 
and you passed 
<class 'transformers_modules.mosaicml.mpt-7b.72e5f594ce36f9cabfa2a9fd8f58b491eb467ee7.configuration_mpt.MPTConfig'>.
```
there is a mismatch in the snapshot hashes. And my cache has both:
```
*********$ ls ~/.cache/huggingface/hub/models--mosaicml--mpt-7b/snapshots/ -lah
total 16K
drwxrwxr-x 4 optimus dpt_ext_searchportal_dep45091 4.0K Oct  1 16:41 .
drwxrwxr-x 5 optimus dpt_ext_searchportal_dep45091 4.0K Oct  1 17:01 ..
drwxrwxr-x 2 optimus dpt_ext_searchportal_dep45091 4.0K Oct  1 17:01 0b57768f52b7775563f7cc78c4724e407b39593b
drwxrwxr-x 2 optimus dpt_ext_searchportal_dep45091 4.0K Oct  1 16:41 72e5f594ce36f9cabfa2a9fd8f58b491eb467ee7
```
After deleting cache, both snapshots re-appear and the error happens again.

Apparently, the model version was updated recently [link to commit 0b5776](https://huggingface.co/mosaicml/mpt-7b/commit/0b57768f52b7775563f7cc78c4724e407b39593b) and the changes only included an extra link in readme. So for this issue purposes it may be enough to update the model commit in the tests.  With the commit change, the tests pass OK. 

UPDATE: the fix has been made in https://github.com/huggingface/transformers/pull/26431 - see below
Yet it leaves open the question why `revision=` argument caused error in the first place. It may deserve a separate issue to be opened.

Hi @poedator 

Thanks a lot for the deep dive and the proposed fix;
with respect to the failing MPT test, the fix has been made in https://github.com/huggingface/transformers/pull/26431 

Regarding the other issues, I can see that our test report did not contain the failing tests you have mentioned, I can also see that you are using an A100. in my experience you can have some differences when using bnb on an A100 and T4 (ampere vs turing) - note the tests are run on a T4 GPU. Some other factors can play a role here but as long as new failing tests do not pop on our daily CI report we don't consider that a fix is needed.
 
Regarding your specific usecase, I believe you can continue developing with the changes values for generation tests to match generations on a A100, and before merging any of your PR that touches quantization related stuff I will run the slow tests myself on the same hardware that we use for testing

Thanks!

Hi, @younesbelkada 
Thanks for the feedback.  I will stop worrying about these tests while developing the quantization-related stuff. But if you see more developers seeing same results with Ampere, please consider updating the tests then.

On MPT test - I removed that commit from my PR.

OK perfect, thank you @poedator 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.