NotImplementedError: ggml_type 3 not implemented
Author: sandy1990418
Pull request: no
Labels: name: Feature request

name: GGUF

### Description 
When trying to use the model `Qwen/Qwen2-7B-Instruct-GGUF` with the gguf file `qwen2-7b-instruct-q4_0.gguf`, I encountered a NotImplementedError about ggml_type 3 not implemented. 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Install the required libraries:
`pip install transformers==4.42.3 gguf==0.6.0`

Load the model and tokenizer:
```
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Qwen/Qwen2-7B-Instruct-GGUF"
file_name = "qwen2-7b-instruct-q4_0.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file = file_name)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file = file_name)
```

### Expected behavior

The code should process ggml_type 3 without raising an error, or provide an alternative implementation.

### System Info

- transformers version: 4.42.3 
- gguf version: 0.6.0
- Platform: Ubuntu 20.04
- Python version: 3.9.16 
- PyTorch version (GPU?): 2.2.2(True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Comments:
@SunMarc 

Hey @sandy1990418, contrary to what the model name implies, this gguf model is of type `q4_1`, which is not supported yet in transformers (see `load_dequant_gguf_tensor` function) ! I'll leave it as a feature request in case someone wants to give it a try! I'll try to find some time to work on that too. If you want work on it, you would have to code the `dequantize_q4_1` function. 

Hi @SunMarc, I wrote the `dequantize_q4_1` function. How does this look? Please let me if I am missing something. If everything looks good, I can go ahead and create a PR.

```python
def dequantize_q4_1(data):
    # C implementation
    # https://github.com/ggerganov/ggml/blob/fca1caafea7de9fbd7efc733b9818f9cf2da3050/src/ggml-quants.c#L1106
    # C struct definition
    # https://github.com/ggerganov/ggml/blob/fca1caafea7de9fbd7efc733b9818f9cf2da3050/src/ggml-quants.h#L18
    block_size = GGML_BLOCK_SIZES["Q4_1"]
    num_blocks = len(data) // block_size

    data_f16 = np.frombuffer(data, dtype=np.float16).reshape(num_blocks, block_size // 2)
    data_u8 = np.frombuffer(data, dtype=np.uint8).reshape(num_blocks, block_size)

    # The scales are stored on the first 2 bytes
    scales = data_f16[:, 0].reshape(num_blocks, 1).astype(np.float32)    
    # scales = np.nan_to_num(scales)

    # The mins are stored on the second 2 bytes
    mins = data_f16[:, 1].reshape(num_blocks, 1).astype(np.float32)

    # the rest of the bytes corresponds to the quants - we discard the first four bytes
    quants = data_u8[:, 4:]

    ql = (quants[:, :] & 0xF).astype(np.int8)
    qr = (quants[:, :] >> 4).astype(np.int8)

    # Use hstack
    quants = np.hstack([ql, qr])

    return ((scales * quants) + mins).astype(np.float32)
```

And
```python
# The Blocksizes are reported in bytes
# Check out: https://github.com/ggerganov/llama.cpp/blob/8a56075b07a8b571bf95a912ffdce4c928c2b414/gguf-py/gguf/constants.py#L801
GGML_BLOCK_SIZES = {
    "Q8_0": 2 + 32,  # Q8_0 uses a blocksize of 32 (int8 tensors) + 2 bytes allocated for the scales
    "Q4_K": 144,
    # Q4_0 uses a blocksize of 32 but the 4-bit tensors are packed into 8-bit tensors + 2 bytes for the scales
    "Q4_0": 2 + 16,
    "Q4_1": 2 + 2 + 16,
    "Q6_K": 210,
    # See: https://github.com/99991/pygguf/commit/a417edbfc029a1bc270f984a694f9128c5afa8b9
    "Q2_K": 256 // 16 + 256 // 4 + 2 + 2,
    "Q3_K": 256 // 8 + 256 // 4 + 12 + 2,
    "Q5_K": 2 + 2 + 12 + 256 // 8 + 256 // 2,
}
```

Finally
```python
# Listed here: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
GGML_TYPES = {
    "F32": 0,
    "Q4_0": 2,
    "Q4_1": 3,
    "Q8_0": 8,
    "Q2_K": 10,
    "Q3_K": 11,
    "Q4_K": 12,
    "Q5_K": 13,
    "Q6_K": 14,
}
```

Hey @kskd1804, this looks pretty good ! It would be awesome if you could open a PR and add a few tests to check that the model is performing as expected / that we get a low perplexity ! 

Thanks @SunMarc! I've created a draft PR while I add the tests. 

I needed some help on writing the tests. This might be a dumb question but I'm actually confused on how the `EXPECTED_TEXT` variables were computed. 
https://github.com/huggingface/transformers/blob/8f0d26c55e5350be3aefdc94a106a24b147204bc/tests/quantization/ggml/test_ggml.py#L105

Since these models are probablistic, shouldn't they generate different outputs every time we run `generate`? If so, how can we assert equality of the generated output with a static value?