Repository: huggingface/text-generation-inference

Issue:
Safe Tensor converting fails for LLaMa 13B and 30B
### System Info

```
2023-06-15T04:27:53.010592Z  INFO text_generation_launcher: Runtime environment:                                                                                                                        [30/661]
Target: x86_64-unknown-linux-gnu
Cargo version: 1.69.0
Commit sha: 5ce89059f8149eaf313c63e9ded4199670cd74bb
Docker label: sha-5ce8905
nvidia-smi:
Thu Jun 15 04:27:51 2023
   +-----------------------------------------------------------------------------+
   | NVIDIA-SMI 515.65.07    Driver Version: 515.65.07    CUDA Version: 11.8     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  NVIDIA A100-SXM...  On   | 00000000:10:00.0 Off |                  Off |
   | N/A   34C    P0    86W / 400W |  25302MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   1  NVIDIA A100-SXM...  On   | 00000000:16:00.0 Off |                  Off |
   | N/A   30C    P0    64W / 400W |      0MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   2  NVIDIA A100-SXM...  On   | 00000000:49:00.0 Off |                  Off |
   | N/A   31C    P0    73W / 400W |      0MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   3  NVIDIA A100-SXM...  On   | 00000000:4D:00.0 Off |                  Off |
   | N/A   31C    P0    71W / 400W |      0MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   4  NVIDIA A100-SXM...  On   | 00000000:C5:00.0 Off |                  Off |
   | N/A   34C    P0    91W / 400W |  32900MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   5  NVIDIA A100-SXM...  On   | 00000000:CA:00.0 Off |                  Off |
   | N/A   34C    P0    92W / 400W |  33044MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   6  NVIDIA A100-SXM...  On   | 00000000:E3:00.0 Off |                  Off |
   | N/A   33C    P0    96W / 400W |  33044MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
   |   7  NVIDIA A100-SXM...  On   | 00000000:E7:00.0 Off |                  Off |
   | N/A   35C    P0    89W / 400W |  32900MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+

   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
```

### Information

- [X] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

I used the official script to convert llama weights 7B~65B and get the following error when launching 13B and 30B only

I ran this command
`CUDA_VISIBLE_DEVICES=2,3 TRANSFORMERS_CACHE=~/ckpts/cache/ FLASH_ATTENTION=1 text-generation-launcher --model-id ~/ckpts/llama-hf/30b/ --num-shard 2 --port XXXXX`

```
2023-06-15T02:42:46.121030Z  INFO text_generation_launcher: Args { model_id: "~/ckpts/llama-hf/30b/", revision: None, sharded: None, num_shard: Some(2), quantize: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_input_length: 1000, max_total_tokens: 1512, max_batch_size: None, waiting_served_ratio: 1.2, max_batch_total_tokens: 32000, max_waiting_tokens: 20, port: 10208, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, env: false }
2023-06-15T02:42:46.121084Z  INFO text_generation_launcher: Sharding model on 2 processes
2023-06-15T02:42:46.121251Z  INFO text_generation_launcher: Starting download process.
2023-06-15T02:42:49.059320Z  WARN download: text_generation_launcher: No safetensors weights found for model ~/ckpts/llama-hf/30b/ at revision None. Converting PyTorch weights to safetensors.

2023-06-15T02:42:49.059564Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00001-of-00007.bin to ~/ckpts/llama-hf/30b/model-00001-of
-00007.safetensors.

2023-06-15T02:46:40.369256Z  INFO download: text_generation_launcher: Convert: [1/7] -- Took: 0:03:51.309267

2023-06-15T02:46:40.369344Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00002-of-00007.bin to ~/ckpts/llama-hf/30b/model-00002-of-00007.safetensors.

2023-06-15T02:50:24.190971Z  INFO download: text_generation_launcher: Convert: [2/7] -- Took: 0:03:43.820986

2023-06-15T02:50:24.191183Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00003-of-00007.bin to ~/ckpts/llama-hf/30b/model-00003-of-00007.safetensors.

2023-06-15T02:54:06.621353Z  INFO download: text_generation_launcher: Convert: [3/7] -- Took: 0:03:42.429557

2023-06-15T02:54:06.621511Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00004-of-00007.bin to ~/ckpts/llama-hf/30b/model-00004-of
-00007.safetensors.

11Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00004-of-00007.bin to ~/ckpts/llama-hf/30b/model-00004-of-00007.safetensors.

2023-06-15T02:57:45.265631Z  INFO download: text_generation_launcher: Convert: [4/7] -- Took: 0:03:38.643727

2023-06-15T02:57:45.265740Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00005-of-00007.bin to ~/ckpts/llama-hf/30b/model-00005-of-00007.safetensors.

2023-06-15T03:01:38.281861Z  INFO download: text_generation_launcher: Convert: [5/7] -- Took: 0:03:53.015641

2023-06-15T03:01:38.281982Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00006-of-00007.bin to ~/ckpts/llama-hf/30b/model-00006-of-00007.safetensors.

2023-06-15T03:03:43.722396Z  INFO download: text_generation_launcher: Convert: [6/7] -- Took: 0:02:05.440018

2023-06-15T03:03:43.722865Z  INFO download: text_generation_launcher: Convert ~/ckpts/llama-hf/30b/pytorch_model-00007-of-00007.bin to ~/ckpts/llama-hf/30b/model-00007-of-00007.safetensors.

2023-06-15T03:04:36.163221Z ERROR text_generation_launcher: Download encountered an error: Traceback (most recent call last):

  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())

  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 151, in download_weights
    utils.convert_files(local_pt_files, local_st_files)

  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 84, in convert_files
    convert_file(pt_file, sf_file)

  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 65, in convert_file
    check_file_size(pt_file, sf_file)

  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 20, in check_file_size
    raise RuntimeError(

RuntimeError: The file size different is more than 1%:
         - ~/ckpts/llama-hf/30b/pytorch_model-00007-of-00007.bin: 5900895281
         - ~/ckpts/llama-hf/30b/model-00007-of-00007.safetensors: 5687891896



Error: DownloadError
```



### Expected behavior

The launch works for LLaMa 7B and 65B well so its confusing why it wont work for 13B and 30B


Discussion:
Facing the same issue as of yesterday.
Hmm the `check_file_size` is pretty rough sanitation, the file might actually be OK but it's hard to tell without looking at the file.

You can try deactivating the check ? Remove the line 20 ?
Can you point to the actual model on the hub you're using too ? Because we don't have any issue with "official" checkpoints

In my case, we're using https://huggingface.co/ausboss/llama-30b-supercot
Initially we had the issue with https://huggingface.co/OpenAssistant/oasst-sft-6-llama-30b-xor however with a restart it was working miraculously..
Line 65 is the line that actually needs to be removed in order to skip the check for the safetensor file size. And here is a temporary patch for Docker for anyone that wants to disable it:

Dockerfile:
```Dockerfile
FROM ghcr.io/huggingface/text-generation-inference:0.8.2

RUN sed '65d' /opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py | tee /opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py
```
@Narsil 

I'm using the official LLaMa checkpoints that I converted using the official script.
The docker image I'm using is `https://ghcr.io/huggingface/text-generation-inference:latest`
and here's the environment I used for converting the llama ckpts:
- python 3.10
- torch 2.1
- transformers 4.30.1
- accelerate 0.20.3
- sentencepiece 0.1.99
- protobuf 3.20.0

The funny thing is the code works well with 7B and 65B models but fails for 13B and 30B
> Hmm the `check_file_size` is pretty rough sanitation, the file might actually be OK but it's hard to tell without looking at the file.
> 
> You can try deactivating the check ? Remove the line 20 ?

Throws a InvalidHeaderDeserialization. Loading the same files using LlamaForCausalLM works fine in a notebook.
> The funny thing is the code works well with 7B and 65B models but fails for 13B and 30B

I converted 30B like 3 times today for quantization purposes without a hitch.. (I'm not on docker though)
> Throws a InvalidHeaderDeserialization. Loading the same files using LlamaForCausalLM works fine in a notebook.

You're trying to load `pickle` files (which would work for `LLamaForCausalLM`) it seems
To reproduce the safetensors error with supercot:
```shell
git lfs install
git clone https://huggingface.co/ausboss/llama-30b-supercot

 

docker run -v ./llama-30b-supercot:/usr/src/llama-30b-supercot --gpus all --rm ghcr.io/huggingface/text-generation-inference:sha-5ce8905 --model-id "/usr/src/llama-30b-supercot" --quantize bitsandbytes --trust-remote-code
```
Note that the same behavior happens with the current latest tag as well as 0.8.2. It always fails for us on the first tensor, which is the smallest and is a pickle.

Resolving patch:
diff --git a/server/text_generation_server/models/__init__.py b/server/text_generation_server/models/__init__.py
--- a/server/text_generation_server/models/__init__.py
+++ b/server/text_generation_server/models/__init__.py
@@ -18,11 +18,43 @@
 from text_generation_server.models.t5 import T5Sharded
 from text_generation_server.models.gpt_neox import GPTNeoxSharded
 
+# The flag below controls whether to allow TF32 on matmul. This flag defaults to False
+# in PyTorch 1.12 and later.
+torch.backends.cuda.matmul.allow_tf32 = True
+
+# The flag below controls whether to allow TF32 on cuDNN. This flag defaults to True.
+torch.backends.cudnn.allow_tf32 = True
+
+# Disable gradients
+torch.set_grad_enabled(False)
+
+__all__ = [
+    "Model",
+    "BLOOMSharded",
+    "CausalLM",
+    "FlashCausalLM",
+    "GalacticaSharded",
+    "Seq2SeqLM",
+    "SantaCoder",
+    "OPTSharded",
+    "T5Sharded",
+    "get_model",
+]
+
+FLASH_ATT_ERROR_MESSAGE = (
+    "{} requires CUDA and Flash Attention kernels to be installed.\n"
+    "Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) "
+    "or install flash attention with `cd server && make install install-flash-attention`"
+)
+
 try:
-    if (
-        torch.cuda.is_available()
-        and not os.getenv("USE_FLASH_ATTENTION", "").lower() == "false"
-    ):
+    if not os.getenv("USE_FLASH_ATTENTION", "").lower() == "false":
+        if not torch.cuda.is_available():
+            FLASH_ATT_ERROR_MESSAGE = (
+                "{} requires CUDA. No compatible CUDA devices found."
+            )
+            raise ImportError("CUDA is not available")
+
         major, minor = torch.cuda.get_device_capability()
         is_sm75 = major == 7 and minor == 5
         is_sm8x = major == 8 and minor >= 0
@@ -30,6 +62,10 @@
 
         supported = is_sm75 or is_sm8x or is_sm90
         if not supported:
+            FLASH_ATT_ERROR_MESSAGE = (
+                "{} requires a CUDA device with capability 7.5, > 8.0 or 9.0. "
+                "No compatible CUDA device found."
+            )
             raise ImportError(
                 f"GPU with CUDA capability {major} {minor} is not supported"
             )
@@ -52,41 +88,12 @@
     )
     FLASH_ATTENTION = False
 
-__all__ = [
-    "Model",
-    "BLOOMSharded",
-    "CausalLM",
-    "FlashCausalLM",
-    "GalacticaSharded",
-    "Seq2SeqLM",
-    "SantaCoder",
-    "OPTSharded",
-    "T5Sharded",
-    "get_model",
-]
-
 if FLASH_ATTENTION:
     __all__.append(FlashNeoXSharded)
     __all__.append(FlashRWSharded)
     __all__.append(FlashSantacoderSharded)
     __all__.append(FlashLlama)
 
-FLASH_ATT_ERROR_MESSAGE = (
-    "{} requires Flash Attention CUDA kernels to be installed.\n"
-    "Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) "
-    "or install flash attention with `cd server && make install install-flash-attention`"
-)
-
-# The flag below controls whether to allow TF32 on matmul. This flag defaults to False
-# in PyTorch 1.12 and later.
-torch.backends.cuda.matmul.allow_tf32 = True
-
-# The flag below controls whether to allow TF32 on cuDNN. This flag defaults to True.
-torch.backends.cudnn.allow_tf32 = True
-
-# Disable gradients
-torch.set_grad_enabled(False)
-
 
 def get_model(
     model_id: str,
diff --git a/server/text_generation_server/utils/convert.py b/server/text_generation_server/utils/convert.py
--- a/server/text_generation_server/utils/convert.py
+++ b/server/text_generation_server/utils/convert.py
@@ -16,9 +16,9 @@ def check_file_size(source_file: Path, target_file: Path):
     source_file_size = source_file.stat().st_size
     target_file_size = target_file.stat().st_size
 
-    if (source_file_size - target_file_size) / source_file_size > 0.01:
+    if (source_file_size - target_file_size) / source_file_size > 0.05:
         raise RuntimeError(
-            f"""The file size different is more than 1%:
+            f"""The file size different is more than 5%:
          - {source_file}: {source_file_size}
          - {target_file}: {target_file_size}
          """
diff --git a/server/text_generation_server/utils/hub.py b/server/text_generation_server/utils/hub.py
--- a/server/text_generation_server/utils/hub.py
+++ b/server/text_generation_server/utils/hub.py
@@ -26,7 +26,10 @@ def weight_hub_files(
     filenames = [
         s.rfilename
         for s in info.siblings
-        if s.rfilename.endswith(extension) and len(s.rfilename.split("/")) == 1
+        if s.rfilename.endswith(extension)
+        and len(s.rfilename.split("/")) == 1
+        and "arguments" not in s.rfilename
+        and "args" not in s.rfilename
     ]
 
     if not filenames:


Test patch: