internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/text-generation-inference

Issue:
Add typical sampling
HF has TypicalWarper and `typical_p`. Would be nice to be able to pass that.


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/tokens.py b/server/text_generation_server/utils/tokens.py
--- a/server/text_generation_server/utils/tokens.py
+++ b/server/text_generation_server/utils/tokens.py
@@ -6,6 +6,7 @@
     TemperatureLogitsWarper,
     TopKLogitsWarper,
     TopPLogitsWarper,
+    TypicalLogitsWarper,
     RepetitionPenaltyLogitsProcessor,
     PreTrainedTokenizerBase,
 )
@@ -41,6 +42,7 @@ def __init__(
         repetition_penalty=1.0,
         top_k=None,
         top_p=None,
+        typical_p=None,
         do_sample=False,
         seed=0,
         device="cpu",
@@ -64,6 +66,9 @@ def __init__(
         if top_p is not None and top_p < 1.0:
             warpers.append(TopPLogitsWarper(top_p=top_p))
             sampling = True
+        if typical_p is not None and typical_p < 1.0:
+            warpers.append(TypicalLogitsWarper(mass=typical_p))
+            sampling = True
 
         self.warpers = warpers
         self.choice = Sampling(seed, device) if sampling else Greedy()
@@ -92,6 +97,7 @@ def from_pb(
             repetition_penalty=pb.repetition_penalty,
             top_k=pb.top_k,
             top_p=pb.top_p,
+            typical_p=pb.typical_p,
             do_sample=pb.do_sample,
             seed=pb.seed,
             device=device,


Test patch:
diff --git a/server/tests/conftest.py b/server/tests/conftest.py
--- a/server/tests/conftest.py
+++ b/server/tests/conftest.py
@@ -10,6 +10,7 @@ def default_pb_parameters():
         repetition_penalty=1.0,
         top_k=0,
         top_p=1.0,
+        typical_p=1.0,
         do_sample=False,
     )
Repository: huggingface/text-generation-inference

Issue:
Concurrent issue - batch size can not be greater than 1
1. when I use ab tool to test with c=2:
```
ab -n 10 -c 2 -p post_data.json -T application/json -H 'accept: application/json' http://localhost:8080/generate
```
a runtime error has been rasied when batch_size > 1?
```
data:{"error":"Request failed during generation: Server error: not enough values to unpack (expected 4, got 3)","error_type":"generation"}
```

![20230427101032](https://user-images.githubusercontent.com/738834/234742138-872541d1-e6f8-48f4-8fbf-3dce1ddcc7f3.png)




Discussion:
@njhill 
@paulcx what model are you using here?
> @paulcx what model are you using here?

bigscience/bloom-7b1
@paulcx @OlivierDehaene OK I see it, my bad, fix incoming.

Resolving patch:
diff --git a/server/text_generation_server/models/causal_lm.py b/server/text_generation_server/models/causal_lm.py
--- a/server/text_generation_server/models/causal_lm.py
+++ b/server/text_generation_server/models/causal_lm.py
@@ -335,7 +335,7 @@ def concatenate(cls, batches: List["CausalLMBatch"]) -> "CausalLMBatch":
                     [t.view(len(batch), -1, *t.shape[-2:]) for t in layer]
                     for layer in batch.past_key_values
                 ]
-            elif batch.past_key_values[0][0].shape == 3:
+            elif len(batch.past_key_values[0][0].shape) == 3:
                 for layer in batch.past_key_values:
                     for k, t in enumerate(layer):
                         layer[k] = t.view(len(batch), -1, *t.shape[-2:])


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Question about sharding / TP
@OlivierDehaene @Narsil is it expected that the output should be the same (or very close) when using the TP implementation for a given model vs non-sharded/single GPU?

Am seeing quite different output, this is for example with flan-ul2 or flan-t5-xxl with 2 GPUs, using float16 for both single and double GPU cases.

This is using a different fork of the code - I'm still investigating and will also try with the latest from the main branch of this repo as-is, but would be very helpful to know generally what you observe / what's expected.


Discussion:
> @OlivierDehaene @Narsil is it expected that the output should be the same (or very close) when using the TP implementation for a given model vs non-sharded/single GPU?

Yes everything should be very close.

Very close is hard to quantify though. Pytorch version can screw things up, dtype can mess up (some models are more susceptible to `bfloat16` to `float16`  change.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#LL316C15-L316C71
 https://github.com/huggingface/transformers/issues/20287
 
 Those might be relevant to your issue too. (Maybe the sharding accentuates some precision issue)
Thanks @Narsil!

Resolving patch:
diff --git a/server/text_generation_server/models/t5.py b/server/text_generation_server/models/t5.py
--- a/server/text_generation_server/models/t5.py
+++ b/server/text_generation_server/models/t5.py
@@ -40,7 +40,7 @@ def __init__(
         self.process_group, rank, world_size = initialize_torch_distributed()
         if torch.cuda.is_available():
             device = torch.device(f"cuda:{rank}")
-            dtype = torch.float16
+            dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32
         else:
             device = torch.device("cpu")
             dtype = torch.float32


Test patch:
Repository: huggingface/text-generation-inference

Issue:
NameError: name 'FastLinear' is not defined
### System Info

When running TGI on a custom OPT-based model  TGI crashes with:

> NameError: name 'FastLinear' is not defined

I'm not exactly a Python expert, but it appears the `FastLinear` class is simply not imported in https://github.com/huggingface/text-generation-inference/blob/7766fee9b15b6f600ee73db7f8bd72f7c62f335e/server/text_generation_server/models/custom_modeling/opt_modeling.py#L30-L35


```
Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 78, in serve
    server.serve(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 184, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 136, in serve_inner
    model = get_model(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/__init__.py", line 235, in get_model
    return OPTSharded(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/opt.py", line 60, in __init__
    model = OPTForCausalLM(config, weights)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/custom_modeling/opt_modeling.py", line 748, in __init__
    self.model = OPTModel(config, weights)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/custom_modeling/opt_modeling.py", line 690, in __init__
    self.decoder = OPTDecoder(config, weights)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/custom_modeling/opt_modeling.py", line 445, in __init__
    self.project_out = FastLinear.load(
NameError: name 'FastLinear' is not defined
```

### Information

- [X] Docker
- [X] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

```
$ docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.0.0 --model-id $model
```

### Expected behavior

The application should not crash.


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/models/custom_modeling/opt_modeling.py b/server/text_generation_server/models/custom_modeling/opt_modeling.py
--- a/server/text_generation_server/models/custom_modeling/opt_modeling.py
+++ b/server/text_generation_server/models/custom_modeling/opt_modeling.py
@@ -28,6 +28,7 @@
 from transformers.modeling_utils import PreTrainedModel
 from transformers import OPTConfig
 from text_generation_server.utils.layers import (
+    FastLinear,
     TensorParallelColumnLinear,
     TensorParallelEmbedding,
     TensorParallelRowLinear,


Test patch:
Repository: huggingface/text-generation-inference

Issue:
GPTQ used 8bits may have a bug
### System Info

docker image: ghcr.io/huggingface/text-generation-inference:0.9.4

### Information

- [X] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [X] My own modifications

### Reproduction

When I modify the default  bits (bits=4 -> bits=8) passed to the function quantize,

https://github.com/huggingface/text-generation-inference/blob/16fadcec5711ff232977b38c74a1c8829af6a63b/server/text_generation_server/cli.py#L219

then error message as shown below.

<img width="1093" alt="Snipaste_2023-08-07_12-10-11" src="https://github.com/huggingface/text-generation-inference/assets/32231230/466654af-a942-40d8-a68c-82d1dc392012">




### Expected behavior

I found a line of code in the quant_linear.py that might be wrong.

https://github.com/huggingface/text-generation-inference/blob/16fadcec5711ff232977b38c74a1c8829af6a63b/server/text_generation_server/utils/gptq/quant_linear.py#L266

the "4" in "self.infeatures = qweight.shape[0] * 32 // 4" mean 4bits. When I use 8bits gptq, the code may modify as "self.infeatures = qweight.shape[0] * 32 // 8"

Finally, I changed this line of code to "self.infeatures = qweight.shape[0] * 32 // self.bits" and obtain gptq used 8bits model successfully.


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/gptq/quant_linear.py b/server/text_generation_server/utils/gptq/quant_linear.py
--- a/server/text_generation_server/utils/gptq/quant_linear.py
+++ b/server/text_generation_server/utils/gptq/quant_linear.py
@@ -263,7 +263,7 @@ def __init__(self, qweight, qzeros, scales, g_idx, bias, bits, groupsize):
         self.groupsize = groupsize
 
         self.outfeatures = qweight.shape[1]
-        self.infeatures = qweight.shape[0] * 32 // 4
+        self.infeatures = qweight.shape[0] * 32 // bits
 
     @classmethod
     def new(cls, bits, groupsize, infeatures, outfeatures, bias):


Test patch:
Repository: huggingface/text-generation-inference

Issue:
small typo in galactica model loading
https://github.com/huggingface/text-generation-inference/blob/1fdc88ee908beb8ae0afe17810a17b9b4d8848e2/server/text_generation_server/models/__init__.py#L92

should be trust_remote_code


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/models/__init__.py b/server/text_generation_server/models/__init__.py
--- a/server/text_generation_server/models/__init__.py
+++ b/server/text_generation_server/models/__init__.py
@@ -89,7 +89,7 @@ def get_model(
             revision,
             quantize=quantize,
             dtype=dtype,
-            dtypetrust_remote_code=trust_remote_code,
+            trust_remote_code=trust_remote_code,
         )
 
     if model_id.startswith("bigcode/"):


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Unexpected results occur when using –rope-factor 2 –rope-scaling dynamic.
When I use -rope-factor 2 -rope-scaling dynamic to get LLaMA to achieve a 4k context length, the results are far worse on both long and short inputs than when I do not use these parameters (in the Chinese scene, some garbled text even appears). When I carefully read the code, I found the logic here to be a bit strange, it seems like it shouldn’t perform this action. I tried removing these two lines of code, and it seems to perform normally.

https://github.com/huggingface/text-generation-inference/blob/5df4c7c0d792738e0ed0fd3426770dea6d7233df/server/text_generation_server/utils/layers.py#L550

I’m not sure if my modification is correct, so I didn’t submit a PR.


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/layers.py b/server/text_generation_server/utils/layers.py
--- a/server/text_generation_server/utils/layers.py
+++ b/server/text_generation_server/utils/layers.py
@@ -502,8 +502,6 @@ def _update_cos_sin_cache(self, dtype, device, seqlen):
                     self.inv_freq = _create_inv_freq(self.dim, newbase, self.inv_freq.device)
                 self._seq_len_cached = seqlen
                 t = torch.arange(seqlen, device=device, dtype=self.inv_freq.dtype)
-                if self.scaling_factor is not None:
-                    t /= self.scaling_factor
                 # Don't do einsum, it converts fp32 to fp16
                 # freqs = torch.einsum("i,j->ij", t, self.inv_freq)
 


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Watermarking bug
### System Info

using singularity with the container `text-generation-inference:1.0.0` .

i get this error:

```
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 727, in warmup
    _, batch = self.generate_token(batch)
  File "/opt/conda/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/flash_causal_lm.py", line 834, in generate_token
    next_input_ids, next_token_logprobs = batch.next_token_chooser(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/tokens.py", line 224, in __call__
    scores = self.watermark_processor(input_ids, scores)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/logits_process.py", line 398, in __call__
    scores[i : i + 1] = processor(input_ids[i : i + 1], scores[i : i + 1])
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 88, in __call__
    greenlist_ids = self._get_greenlist_ids(
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/watermark.py", line 64, in _get_greenlist_ids
    greenlist_size = int(max_value * self.gamma)
ValueError: invalid literal for int() with base 10: '0.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50.50
```

i think `greenlist_size = int(max_value * self.gamma)` should be `greenlist_size = max_value * int(self.gamma)`

I also tried setting the env variables only `export WATERMARK_GAMMA=0.5 && export WATERMARK_DELTA=2.0` and the exact issue.


### Information

- [X] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

```bash
export USE_FLASH_ATTENTION=True && text-generation-launcher \
--model-id /scratch//models/huggyllama-llama-7b \
--port 61697 \
--max-best-of 5 \
--max-total-tokens 2048 \
--max-input-length 1024 \
--max-batch-prefill-tokens 1024 \
--max-concurrent-requests 20 \
--sharded false \
--num-shard 1  \
--dtype float16 \
 --disable-custom-kernels  \
--watermark-gamma 0.5 \
--watermark-delta 2
```

### Expected behavior

It should multiply the number not the string `0.5`


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/watermark.py b/server/text_generation_server/utils/watermark.py
--- a/server/text_generation_server/utils/watermark.py
+++ b/server/text_generation_server/utils/watermark.py
@@ -19,8 +19,8 @@
 from transformers import LogitsProcessor
 from typing import List, Union
 
-GAMMA = os.getenv("WATERMARK_GAMMA", 0.5)
-DELTA = os.getenv("WATERMARK_DELTA", 2.0)
+GAMMA = float(os.getenv("WATERMARK_GAMMA", 0.5))
+DELTA = float(os.getenv("WATERMARK_DELTA", 2.0))
 
 
 class WatermarkLogitsProcessor(LogitsProcessor):


Test patch:
Repository: huggingface/text-generation-inference

Issue:
missing 2 required positional arguments: 'top_n_tokens' and 'top_n_tokens_tensor' for the galactica model
### System Info

v1.0.3

### Information

- [ ] Docker
- [ ] The CLI directly

### Tasks

- [ ] An officially supported command
- [ ] My own modifications

### Reproduction

having a local galactica model in a folder called `facebook-galactica-30b-gptq` won't be detected since it will fail this check https://github.com/huggingface/text-generation-inference/blob/main/server/text_generation_server/models/__init__.py#L88

I suggest making it check `if "galactica" in  in model_id` instead.

### Expected behavior

expected to detect a galactica model


Discussion:
I changed the folder structure to `/models/facebook/galactica-30b-gptq/` only to discover another error.

```bash
File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 63, in Warmup
File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/galactica.py", line 135, in from_pb
    return cls(
TypeError: __init__() missing 2 required positional arguments: 'top_n_tokens' and 'top_n_tokens_tensor'
```

Resolving patch:
diff --git a/server/text_generation_server/models/galactica.py b/server/text_generation_server/models/galactica.py
--- a/server/text_generation_server/models/galactica.py
+++ b/server/text_generation_server/models/galactica.py
@@ -80,6 +80,7 @@ def from_pb(
         next_token_choosers = []
         stopping_criterias = []
         prefix_offsets = []
+        top_n_tokens = []
         read_offsets = []
         requests_idx_mapping = {}
 
@@ -96,6 +97,7 @@ def from_pb(
                 r.stopping_parameters, tokenizer
             )
             stopping_criterias.append(stopping_criteria)
+            top_n_tokens.append(r.top_n_tokens)
             max_truncation = max(max_truncation, r.truncate)
             max_decode_tokens += stopping_criteria.max_new_tokens
             padding_right_offset = max(
@@ -129,6 +131,9 @@ def from_pb(
         position_ids = tokenized_inputs["attention_mask"].long().cumsum(-1) - 1
         position_ids.masked_fill_(tokenized_inputs["attention_mask"] == 0, 1)
         all_input_ids = tokenized_inputs["input_ids"].T.split(1, dim=1)
+        top_n_tokens_tensor = torch.tensor(
+            top_n_tokens, device=device, dtype=torch.int64
+        )
 
         max_tokens = len(inputs) * max_input_length + max_decode_tokens
 
@@ -146,6 +151,8 @@ def from_pb(
             read_offsets=read_offsets,
             next_token_choosers=next_token_choosers,
             stopping_criterias=stopping_criterias,
+            top_n_tokens=top_n_tokens,
+            top_n_tokens_tensor=top_n_tokens_tensor,
             max_input_length=max_input_length.item(),
             padding_right_offset=padding_right_offset,
             max_tokens=max_tokens,


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Flash attention (v1) is broken
Defaulting window_size_left to "-1" breaks flash attention v1 since v1 only works if window_size_left is 0 (line 83).

https://github.com/huggingface/text-generation-inference/blob/5ba53d44a18983a4de32d122f4cb46f4a17d9ef6/server/text_generation_server/utils/flash_attn.py#L60C7-L60C7


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/flash_attn.py b/server/text_generation_server/utils/flash_attn.py
--- a/server/text_generation_server/utils/flash_attn.py
+++ b/server/text_generation_server/utils/flash_attn.py
@@ -80,7 +80,7 @@ def attention(
         )
 
     if HAS_FLASH_ATTN:
-        if window_size_left != 0:
+        if window_size_left != -1:
             raise NotImplementedError(
                 "window_size_left is only available with flash attn v2"
             )


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Support other means of passing image to IdeficsImageProcessor
### Feature request

IdeficsImageProcessor expects the image to be passed only in the form of a URL from which it can [fetch the image](https://github.com/huggingface/text-generation-inference/blob/7a6fad6aac67d9bf21fe75c034a6bcab5dbd88d2/server/text_generation_server/models/custom_modeling/idefics_image_processing.py#L182).

It would be beneficial to allow accepting images as base64 encoded strings.

### Motivation

We want to use IDEFICS on data not directly available under the HTTP endpoint. We either need to set up another server to unpack an image from our DVC and make it available under some URL or deploy a custom docker container with patched image download logic.

### Your contribution

We can submit a PR to accept images as base64 encoded strings. Roughly looking like this:

```python
if isinstance(image_url_or_urls, str):
    decoded_bytes = None
    try:
        decoded_bytes = BytesIO(base64.b64decode(image_url_or_urls))
    except:
        print("Not a base64 encoded image")
    if decoded_bytes is not None:
        return Image.open(decoded_bytes)
    else:
        response = requests.get(image_url_or_urls, stream=True, headers=headers)
        response.raise_for_status()
        return Image.open(BytesIO(response.content))
```



Discussion:

Resolving patch:
diff --git a/server/text_generation_server/models/custom_modeling/idefics_image_processing.py b/server/text_generation_server/models/custom_modeling/idefics_image_processing.py
--- a/server/text_generation_server/models/custom_modeling/idefics_image_processing.py
+++ b/server/text_generation_server/models/custom_modeling/idefics_image_processing.py
@@ -35,6 +35,7 @@
     valid_images,
 )
 from io import BytesIO
+import base64
 import requests
 from transformers import TensorType, is_torch_available
 
@@ -194,10 +195,17 @@ def fetch_images(self, image_url_or_urls: Union[str, List[str]]):
         if isinstance(image_url_or_urls, list):
             return [self.fetch_images(x) for x in image_url_or_urls]
         elif isinstance(image_url_or_urls, str):
-            response = requests.get(image_url_or_urls, stream=True, headers=headers, timeout=(1, 5))
-            response.raise_for_status()
+            image = image_url_or_urls
+
+            if image.startswith("http://") or image.startswith("https://"):
+                response = requests.get(image_url_or_urls, stream=True, headers=headers, timeout=(1, 5))
+                response.raise_for_status()
+                content = response.content
+            else:
+                content = base64.b64decode(image)
+
             try:
-                image = Image.open(BytesIO(response.content))
+                image = Image.open(BytesIO(content))
                 # image.verify()
             except Exception:
                 raise ValueError(f"Could not load image from url {image_url_or_urls}")    


Test patch: