GGML (GGUF) Llama3 unit test fails
Author: jojje
Pull request: no

### System Info

* Commit: `d4564df1d4a6b355779f1a8ac250cb47cb4c38d8`
* Complete environment specification: [here](https://github.com/huggingface/transformers/blob/d4564df1d4a6b355779f1a8ac250cb47cb4c38d8/docker/transformers-all-latest-gpu/Dockerfile)
---

- `transformers` version: 4.41.2
- Platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

## Steps to reproduce:
1. Clone the project and enter the workspace `git clone https://github.com/huggingface/transformers && cd transformers`. (_main_ head was at d4564df1d as of this report)

2. Add the gguf package to the container (needed by your tests, but not included in your docker image for some reason), and (optionally) a non-root user so that volume mounted files can be cached on the host without requiring root access.
   ```
   cd docker/transformers-all-latest-gpu
   cat >> Dockerfile <<EOF
   RUN pip install gguf
   RUN useradd -m -s /bin/bash -u $(id -u) -g $(id -g) user
   USER user
   RUN mkdir ~/.cache
   EOF
   ```

3. Build the container image: `docker build -t transformers .`

4. Run the test using the tip of the checked out main branch.
   ```
   cd -  # return to the root of the workspace
   docker run --rm -ti \
   -v $PWD/.dockercache:/home/user/.cache \
   -v $PWD:/home/user/transformers \
   -w /home/user/transformers \
   -e RUN_SLOW=1 \
   transformers \
   pytest --verbose tests/quantization/ggml/test_ggml.py
   
Below is the test report summary.

```
===== test session starts ==================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/user/transformers/.hypothesis/examples'))
rootdir: /home/user/transformers
configfile: pyproject.toml
plugins: typeguard-4.3.0, hydra-core-1.3.2, dash-2.17.1, rich-0.1.1, xdist-3.6.1, timeout-2.3.1, hypothesis-6.103.2
collected 14 items

tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_llama3_q4_0 FAILED                              [  7%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_llama3_q4_0_tokenizer FAILED                    [ 14%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_mistral_q4_0 PASSED                             [ 21%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q2_k PASSED                                     [ 28%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q2_k_serialization PASSED                       [ 35%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q3_k PASSED                                     [ 42%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q4_0 PASSED                                     [ 50%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q4_k_m PASSED                                   [ 57%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q5_k PASSED                                     [ 64%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q6_k PASSED                                     [ 71%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q6_k_fp16 PASSED                                [ 78%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q8_0 PASSED                                     [ 85%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_qwen2_q4_0 FAILED                               [ 92%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_tokenization_xnli PASSED                        [100%]
```

The tests were run in your own [transformers-all-latest-gpu](https://github.com/huggingface/transformers/blob/main/docker/transformers-all-latest-gpu/Dockerfile) container, to ensure I was using the _exact_ same environment as your own CI tests.

[Here](https://github.com/user-attachments/files/15919478/pytest-output.txt) is the full verbose output from pytest regarding the test failures.

PS, this report relates to #31507 as well, as can be seen by the test report which has a specific test for that model which also fails.

### Expected behavior

All of the tests in `tests/quantization/ggml/test_ggml.py` passing. 

In particular, the two tests `test_llama3_q4_0` and `test_llama3_q4_0_tokenizer`, since that would indicate there is at least _some_ sort of way to run GGUF Llama-3 using the method in your documentation, and exemplified in the tests.

Comments:
cc @SunMarc 

Hi @jojje, thanks for the detailed report ! For the gguf tests, we are using transformers-quantization-latest-gpu docker file since we are trying to run quantization related tests. You will see that we indeed have `pip install gguf` in the Dockerfile. Could you try with that instead ? From the traceback, I see that qwen2 test (`tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_qwen2_q4_0`) is not passing and the reason is that `qwen2` is not in `GGUF_SUPPORTED_ARCHITECTURES` which is very strange. Could you try to debug from there ? I'm not able to reproduce this unfortunately. 



In current HEAD (72fb02c47) 
* The Qwen2 problem seems fixed.
* The Llama test still fails, but for a different reason; one that is trivially fixed. It seems the generated tokens have changed from the expected `"Hello, I am interested in [The Park]\nThe"` to `"Hello, I am new to this forum. I am"`.

So I consider the reported issue resolved.

The latest test results with more details in the attached log, as before.
```
72fb02c47 Fixed `log messages` that are resulting in TypeError due to too many arguments (#32017)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/user/transformers/.hypothesis/examples'))
rootdir: /home/user/transformers
configfile: pyproject.toml
plugins: dash-2.17.1, rich-0.1.1, hypothesis-6.108.2, xdist-3.6.1, timeout-2.3.1
collecting ... collected 14 items

tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_llama3_q4_0 FAILED [  7%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_llama3_q4_0_tokenizer PASSED [ 14%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_mistral_q4_0 PASSED [ 21%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q2_k PASSED [ 28%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q2_k_serialization PASSED [ 35%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q3_k PASSED [ 42%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q4_0 PASSED [ 50%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q4_k_m PASSED [ 57%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q5_k PASSED [ 64%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q6_k PASSED [ 71%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q6_k_fp16 PASSED [ 78%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_q8_0 PASSED [ 85%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_qwen2_q4_0 PASSED [ 92%]
tests/quantization/ggml/test_ggml.py::GgufIntegrationTests::test_tokenization_xnli PASSED [100%]
```

There is one other problem I found that would need fixing, and that is the fact that the latest version (0.9.1) of the `gguf` package on pypi is broken. This makes me wonder what _specific version_ you've baked into your `transformers-quantization-latest-gpu/Dockerfile` CI image. You're not pinning the gguf package, so whatever version was the latest when you built the CI container image will be what's baked in.

I worked around the gguf version problem by installing the gguf package directly from the source llamacpp source tree. For some reason _that_ results in a working gguf package, even for 0.9.1. So you may want to consider pinning all your python packages going forward to avoid external dependency drifts.

The exact package versions in my container are provided in the accompanying pytest log for completeness sake, to ensure the reason for our different test outcomes isn't due to different (unpinned) python dependencies.

[pytest.log](https://github.com/user-attachments/files/16273372/pytest.log)

PS. If you want me send a PR with the fixed test, let me know. Otherwise I trust you know how to fix it yourself.

Nice investigation for the gguf package issue ! This will help me a lot to debug that ! As for the PR, I will fix it after the gguf package issue is solved ! Thanks a lot again for this detailed report !