QuantizedCache fails with latest versions of optimum-quanto
Author: SimJeg
Pull request: no
Labels: name: bug

### System Info

transformers-4.46.3, optimum-quanto-0.2.6

### Who can help?

@ArthurZucker @SunMarc 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi ! I failed to use the `QuantizedCache` following the instructions on this [hf blog post](https://huggingface.co/blog/kv-cache-quantization#how-to-use-quantized-kv-cache-in-%F0%9F%A4%97-transformers).

Code:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16, device_map="cuda:0")
inputs = tokenizer("I like rock music because", return_tensors="pt").to(model.device)

out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="quantized", cache_config={"backend": "quanto", "nbits": 4})
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

out = model.generate(**inputs, do_sample=False, max_new_tokens=20)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
```

Error:

```bash
ValueError: shift must be specified for qtypes lower than 8-bit
```

### Expected behavior

It failed with optimum-quanto-0.2.6 and optimum-quanto-0.2.5 but worked with optimum-quanto-0.2.4.

Also running these in a notebook and adding %time before each `out = ...`, inference took 4.3sec with quantized cache compared to 0.6sec without. It is expected ? (running on A100 with CUDA 12.6)

Thanks ! 

Comments:
About the run times, first time I run my notebook cell I get some kernels errors:

```bash
/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/ops.py:66: UserWarning: An exception was raised while calling the optimized kernel for quanto::unpack: Error building extension 'quanto_cuda': [1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output gemm_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=quanto_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include -isystem /mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH -isystem /mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_72,code=sm_72 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_87,code=sm_87 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 -DENABLE_BF16 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --threads=8 -DQUANTO_CUDA_ARCH=800 -c /mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu -o gemm_cuda.cuda.o 
FAILED: gemm_cuda.cuda.o 
[/usr/local/cuda/bin/nvcc](/usr/local/cuda/bin/nvcc) --generate-dependencies-with-compile --dependency-output gemm_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=quanto_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC) -isystem [/usr/local/cuda/include](/usr/local/cuda/include) -isystem [/usr/include/python3.10](/usr/include/python3.10) -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_72,code=sm_72 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_87,code=sm_87 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 -DENABLE_BF16 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --threads=8 -DQUANTO_CUDA_ARCH=800 -c [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu) -o gemm_cuda.cuda.o 
[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu)(286): error: identifier "__hfma2" is undefined
        loaded[i] = __hfma2(loaded[i], scale2, zero2);
                    ^
          detected during instantiation of "void gemm_w4a16_T1<CTA_M,CTA_N,CTA_K,WARP_M,WARP_N,WARP_K,STAGES,G,SPLITK>(half *, half *, half *, half *, half *, int *, int, int, int) [with CTA_M=16, CTA_N=128, CTA_K=128, WARP_M=16, WARP_N=32, WARP_K=64, STAGES=4, G=128, SPLITK=2]" 

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu)(738): error: identifier "__hfma2" is undefined
        loaded[i] = __hfma2(loaded[i], scale2, zero2);
                    ^
          detected during instantiation of "void gemm_w4a16_T2<CTA_M,CTA_N,CTA_K,WARP_M,WARP_N,WARP_K,STAGES,G>(half *, half *, half *, half *, half *, int, int, int) [with CTA_M=64, CTA_N=128, CTA_K=64, WARP_M=64, WARP_N=32, WARP_K=64, STAGES=4, G=128]" 

2 errors detected in the compilation of "/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemm_cuda.cu".
[2/3] [/usr/local/cuda/bin/nvcc](/usr/local/cuda/bin/nvcc) --generate-dependencies-with-compile --dependency-output gemv_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=quanto_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC) -isystem [/usr/local/cuda/include](/usr/local/cuda/include) -isystem [/usr/include/python3.10](/usr/include/python3.10) -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_72,code=sm_72 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_87,code=sm_87 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 -DENABLE_BF16 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --threads=8 -DQUANTO_CUDA_ARCH=800 -c [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu) -o gemv_cuda.cuda.o 
FAILED: gemv_cuda.cuda.o 
[/usr/local/cuda/bin/nvcc](/usr/local/cuda/bin/nvcc) --generate-dependencies-with-compile --dependency-output gemv_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=quanto_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/TH) -isystem [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/torch/include/THC) -isystem [/usr/local/cuda/include](/usr/local/cuda/include) -isystem [/usr/include/python3.10](/usr/include/python3.10) -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_72,code=sm_72 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_87,code=sm_87 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 -DENABLE_BF16 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT162_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --threads=8 -DQUANTO_CUDA_ARCH=800 -c [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu) -o gemv_cuda.cuda.o 
[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(179): error: identifier "__hfma2" is undefined
                          = __hfma2(*reinterpret_cast<half2*>(dequantized_weight + y * NPerBlock + x * 2),
                            ^

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=1, BlockSize=256, GroupSize=128]" at line 264

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=2, BlockSize=256, GroupSize=128]" at line 269

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=3, BlockSize=256, GroupSize=128]" at line 274

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=4, BlockSize=256, GroupSize=128]" at line 279

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=5, BlockSize=256, GroupSize=128]" at line 284

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=6, BlockSize=256, GroupSize=128]" at line 289

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(151): error: identifier "__hfma2" is undefined
                      w = __hfma2(w, __half2half2(local_scale[idx]), __half2half2(local_scaled_zeros[idx]));
                          ^
          detected during instantiation of "void gemv_kernel<NPerBlock,Batch,BlockSize,GroupSize>(const half *, const uint32_t *, const half *, const half *, half *, int, int) [with NPerBlock=2, Batch=7, BlockSize=256, GroupSize=128]" at line 294

8 errors detected in the compilation of "/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu".
[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/awq/v2/gemv_cuda.cu)(83): warning #177-D: variable "kShuffleSize" was declared but never referenced
      static constexpr int kShuffleSize = 32;
                           ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

ninja: build stopped: subcommand failed.
 Falling back to default implementation.
  warnings.warn(message + " Falling back to default implementation.")
[/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/ops.py:66](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/ops.py:66): UserWarning: An exception was raised while calling the optimized kernel for quanto::unpack: [/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/build/quanto_cuda.so](/mnt/workspace/projects/kvpress/.venv/lib/python3.10/site-packages/optimum/quanto/library/extensions/cuda/build/quanto_cuda.so): cannot open shared object file: No such file or directory Falling back to default implementation.
  warnings.warn(message + " Falling back to default implementation.")
```

Hey @SimJeg, could you try to install transformers from source? I think it should be fixed on main. Thanks ! 

Installing transformers from source works for me, see https://github.com/NVIDIA/kvpress/blob/main/notebooks/per_layer_compression_demo.ipynb