internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/text-generation-inference

Issue:
GPTQ doesn't use exllama for all layers
I can no longer contribute here and am not looking at any of the additions since the license change, to avoid contaminating myself because we have our own fork.

But I thought you guys would like to know that there's what I think was an oversight in the GPTQ impl. In `get_multi_weights_col` in `weights.py`, the last element of the `weights` tuple is hardcoded as `False`. I'm pretty sure should instead be something like `bits == 4 and HAS_EXLLAMA`.

Without this, half of the layers still use the slow triton kernel.

Fixing this gives a huge speedup, for example about 2.5x for llama-2-70b-GPTQ.


Discussion:

Resolving patch:
diff --git a/server/text_generation_server/utils/weights.py b/server/text_generation_server/utils/weights.py
--- a/server/text_generation_server/utils/weights.py
+++ b/server/text_generation_server/utils/weights.py
@@ -212,7 +212,9 @@ def get_multi_weights_col(self, prefixes: List[str], quantize: str, dim: int):
                 g_idx = None
 
             bits, groupsize = self._get_gptq_params()
-            weight = (qweight, qzeros, scales, g_idx, bits, groupsize, False)
+            from text_generation_server.utils.layers import HAS_EXLLAMA
+            use_exllama = bits==4  and HAS_EXLLAMA and quantize == "gptq"
+            weight = (qweight, qzeros, scales, g_idx, bits, groupsize, use_exllama)
         else:
             w = [self.get_sharded(f"{p}.weight", dim=0) for p in prefixes]
             weight = torch.cat(w, dim=dim)


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Update Docker to torch 2.1?
### Feature request

H100s have trouble with gptq quants due to not having latest pytorch, can in the next TGI Docker we update torch to this, or have one special for this for use on h100s? 

### Motivation

Cant get tgi + gptq quant to work on h100s

### Your contribution

Sorry I dont have any contribution ^_^ 


Discussion:

Resolving patch:
diff --git a/integration-tests/conftest.py b/integration-tests/conftest.py
--- a/integration-tests/conftest.py
+++ b/integration-tests/conftest.py
@@ -318,6 +318,7 @@ def docker_launcher(
             ],
             volumes=volumes,
             ports={"80/tcp": port},
+            shm_size="1G"
         )
 
         yield ContainerLauncherHandle(client, container.name, port)


Test patch:
Repository: huggingface/text-generation-inference

Issue:
PEFT support does not work with local directories
### System Info

text-generation-inference version: main branch

The rest of the system info is probably not relevant.

### Information

- [X] Docker
- [X] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

Steps to reproduce:

1. Create a directory that contains PEFT weights, for example: `mypeft` which contains two files: `adapter_config.json` and `adapter_model.bin`
2. Try to download the base model using the following command: `text-generation-server download-weights mypeft`

### Expected behavior

It should be able to download the base model weights and merge the PEFT weights into the base model.

However, as mentioned in [this comment](https://github.com/huggingface/text-generation-inference/pull/762#issuecomment-1728042436):

> it seems the checks for PEFT model is only in the block checking if the model is not a local model

There is a workaround as documented in [this comment](https://github.com/huggingface/text-generation-inference/pull/762#issuecomment-1729689087)




Discussion:

Resolving patch:
diff --git a/server/text_generation_server/cli.py b/server/text_generation_server/cli.py
--- a/server/text_generation_server/cli.py
+++ b/server/text_generation_server/cli.py
@@ -150,6 +150,17 @@ def download_weights(
             if not extension == ".safetensors" or not auto_convert:
                 raise e
 
+    else:
+        # Try to load as a local PEFT model
+        try:
+            utils.download_and_unload_peft(
+                model_id, revision, trust_remote_code=trust_remote_code
+            )
+            utils.weight_files(model_id, revision, extension)
+            return
+        except (utils.LocalEntryNotFoundError, utils.EntryNotFoundError):
+            pass
+
     # Try to see if there are local pytorch weights
     try:
         # Get weights for a local model, a hub cached model and inside the WEIGHTS_CACHE_OVERRIDE


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Incorrectly multiplied timeout by 60 in the asynchronous client
### System Info

I'm testing TGI using Docker. Below is the exact command I'm utilizing:

```console
docker run --gpus '"device=1,2"' --shm-size 1g -p 8000:80 -v ~/tgi-test:/data ghcr.io/huggingface/text-generation-inference:1.4 --model-id mistralai/Mistral-7B-v0.1 --max-input-length 8000 --max-total-tokens 8001
```

### Information

- [ ] Docker
- [ ] The CLI directly

### Tasks

- [ ] An officially supported command
- [ ] My own modifications

### Reproduction

Given the generation request:

```python
async def test():
  start = time.time()
  try:
   response = await client.generate('1' * 6_000, max_new_tokens=1_800)
  except Exception as ex:
    pass
  print(time.time() - start)
```

And this async client definition:

```python
client = AsyncClient('http://localhost:8000', timeout=1)
```

It doesn't timeout after 1 second:

```python
>>> await test()
60.88534379005432
```

But if we create a client with a timeout of 2/60:

```python
client = AsyncClient('http://localhost:8000', timeout=(2/60))
```

It does timeout after 2 seconds:

```python
>>> await test()
2.0035104751586914
```

### Expected behavior

The function should have timed out after 1 second with this client definition:

```python
client = AsyncClient('http://localhost:8000', timeout=1)
```


Discussion:

Resolving patch:
diff --git a/clients/python/text_generation/client.py b/clients/python/text_generation/client.py
--- a/clients/python/text_generation/client.py
+++ b/clients/python/text_generation/client.py
@@ -424,7 +424,7 @@ def __init__(
         self.base_url = base_url
         self.headers = headers
         self.cookies = cookies
-        self.timeout = ClientTimeout(timeout * 60)
+        self.timeout = ClientTimeout(timeout)
 
     async def chat(
         self,


Test patch:
Repository: huggingface/text-generation-inference

Issue:
Unable to stop TGI after serving models
### System Info

I use the official docker image: ghcr.io/huggingface/text-generation-inference:2.0.1

### Information

- [X] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

I used the following command to serve the model. After TGI finished the model sharding/loading and started serving, I cannot use `Ctrl+C` to terminate the server. 

```bash
model=mistralai/Mixtral-8x7B-Instruct-v0.1
volume=/my_path_for_hf_cache
token="myhftokens"
    
docker run --gpus '"device=4,5"' \
    --shm-size 20g \
    -e HUGGING_FACE_HUB_TOKEN=$token \
    -p 8080:80 \
    -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0.1 \
    --model-id $model \
    --sharded true \
    --quantize eetq \
    --max-input-length 10240 \
    --max-batch-prefill-tokens 10240 \
    --max-total-tokens 32768 \
    --port 80
```

### Expected behavior

In previous version 1.3.0 and 1.4.0, I can use `Ctrl+C` to terminate the server while it is not the case for 2.0.1. My current solution is to use docker command to kill the container. Not sure if this is a good way?


Discussion:
Same here, it seems to come from #1716: https://github.com/huggingface/tgi-gaudi/pull/134#issuecomment-2095365083

Resolving patch:
diff --git a/server/text_generation_server/server.py b/server/text_generation_server/server.py
--- a/server/text_generation_server/server.py
+++ b/server/text_generation_server/server.py
@@ -35,9 +35,6 @@ def exit_gracefully(self, signum, frame):
         self.KEEP_PROCESSING = False
 
 
-signal_handler = SignalHandler()
-
-
 class TextGenerationService(generate_pb2_grpc.TextGenerationServiceServicer):
     def __init__(
         self,
@@ -251,7 +248,7 @@ async def serve_inner(
         await server.start()
 
         logger.info("Server started at {}".format(local_url))
-
+        signal_handler = SignalHandler()
         while signal_handler.KEEP_PROCESSING:
             await asyncio.sleep(0.5)
 


Test patch:
Repository: keras-team/keras

Issue:
Misiing import in list_pictures
`list_pictures` abborts with error `NameError: global name 're' is not defined`



Discussion:

Resolving patch:
diff --git a/keras/preprocessing/image.py b/keras/preprocessing/image.py
--- a/keras/preprocessing/image.py
+++ b/keras/preprocessing/image.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 
 import numpy as np
+import re
 from scipy import ndimage
 from scipy import linalg
 


Test patch:
Repository: keras-team/keras

Issue:
Python 3 compatibility problem with Image loading
Loading an Image using the `load_img` results in an error.

```
Traceback (most recent call last):
  File "keras/autoencoder.py", line 45, in <module>
    X_train, Y_train, X_test, Y_test, nb_classes = io.load_images(join(DATA_DIR, 'dataset0'))
  File "/home/jnphilipp/Documents/cnn/hieroglyphs/keras/utils/io.py", line 27, in load_images
    X_train.append(img_to_array(load_img(picture, True)))
  File "/home/jnphilipp/.local/lib/python3.4/site-packages/Keras-0.1.2-py3.4.egg/keras/preprocessing/image.py", line 107, in load_img
  File "/home/jnphilipp/.local/lib/python3.4/site-packages/PIL/Image.py", line 2330, in open
    % (filename if filename else fp))
OSError: cannot identify image file <_io.TextIOWrapper name='/home/jnphilipp/Documents/cnn/hieroglyphs/data/dataset0/train/P1_train0.png' mode='r' encoding='ISO-8859-1'>
```



Discussion:

Resolving patch:
diff --git a/keras/preprocessing/image.py b/keras/preprocessing/image.py
--- a/keras/preprocessing/image.py
+++ b/keras/preprocessing/image.py
@@ -104,7 +104,7 @@ def img_to_array(img):
 
 def load_img(path, grayscale=False):
     from PIL import Image
-    img = Image.open(open(path))
+    img = Image.open(path)
     if grayscale:
         img = img.convert('L')
     else: # Assure 3 channel even when loaded image is grayscale


Test patch:
Repository: keras-team/keras

Issue:
'float' object has no attribute 'get_value' problem.
When I was trying to print the configuration of a model by "model.get_config()" or trying to save my model as a 'json' file:

json_str = autoEncoder.to_json()
open('temp_model.json','w').write(json_str)
autoEncoder.save_weights('temp_model_weights.h5')

It raise the exception "float object has no attribute 'get_value' " in file 'optimizer.py', in class SGD(because I was using SGD as the optimizer), the definition of get_config() is:

 def get_config(self):
        return {"name": self.**class**.**name**,
                "lr": float(self.lr.get_value()),
                "momentum": float(self.momentum.get_value()),
                "decay": float(self.decay.get_value()),
                "nesterov": self.nesterov}

while the **init** of class SGD does not contain decay and nesterov

```
def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False, *args, **kwargs):
    super(SGD, self).__init__(**kwargs)
    self.__dict__.update(locals())
    self.iterations = shared_scalar(0)
    self.lr = shared_scalar(lr)
    self.momentum = shared_scalar(momentum)
```

Is it a bug? Can I fix the problem by adding 'self.decay = shared_scalar(decay)' or something like this?

Thank you very much!



Discussion:
well, it seems I can't fix it by myself....sad ￣﹏￣

`self.__dict__.update(locals())` sets up all the named parameters from `__init__` to the created object, so a correctly initialized object of class `SGD` should have both `decay` and `nesterov`. The problem here is that `decay` is a float and not a shared Theano variable. I'll fix this in a bit.


Resolving patch:
diff --git a/keras/optimizers.py b/keras/optimizers.py
--- a/keras/optimizers.py
+++ b/keras/optimizers.py
@@ -82,7 +82,7 @@ def get_config(self):
         return {"name": self.__class__.__name__,
                 "lr": float(self.lr.get_value()),
                 "momentum": float(self.momentum.get_value()),
-                "decay": float(self.decay.get_value()),
+                "decay": float(self.decay),
                 "nesterov": self.nesterov}
 
 


Test patch:
Repository: keras-team/keras

Issue:
A model doesn't know its input_shape after build
Reproduce the error as

``` python
model = Sequential()
model.add(Dense(1), input_shape=(784,))
model.build()

model.input_shape
```

Shouldn't the model know its `input_shape` after `build`? It knows `output_shape` for instance. Am I missing anything @matsuyamax ?



Discussion:

Resolving patch:
diff --git a/keras/layers/containers.py b/keras/layers/containers.py
--- a/keras/layers/containers.py
+++ b/keras/layers/containers.py
@@ -84,7 +84,11 @@ def get_input(self, train=False):
         if not hasattr(self.layers[0], 'input'):
             self.set_input()
         return self.layers[0].get_input(train)
-
+ 
+    @property
+    def input_shape(self):
+        return self.layers[0].input_shape
+    
     @property
     def input(self):
         return self.get_input()


Test patch:
Repository: keras-team/keras

Issue:
keras.utils.visualize_util
line 9: if type(model) == Sequential
Global name Sequential is not defined
line 25 elif type(model) == Graph:
Global name Graph is not defined



Discussion:
Looks like add:
    from keras.models import Sequential, Graph
would be fine.


Resolving patch:
diff --git a/keras/utils/visualize_util.py b/keras/utils/visualize_util.py
--- a/keras/utils/visualize_util.py
+++ b/keras/utils/visualize_util.py
@@ -1,7 +1,7 @@
 import pydot
 # old pydot will not work with python3, must use one
 # that works with python3 such as pydot2 or pydot
-
+from keras.models import Sequential, Graph
 
 def plot(model, to_file='model.png'):
 


Test patch:
Repository: keras-team/keras

Issue:
problem with K.common._FLOATX
I tried to run:

```
a = K.random_normal((100, 200))
```

and I got the error:

```
/home/eders/python/Theano/theano/sandbox/rng_mrg.pyc in get_substream_rstates(self, n_streams, dtype, inc_rstate)
   1167 
   1168         """
-> 1169         assert isinstance(dtype, str)
   1170         assert n_streams < 2**72
   1171         assert n_streams > 0

AssertionError:
```

I tried to print K.common._FLOATX to see what was going on and it is `u'float32'`. That little `u` upfront is making theano crash. I believe that when reading the type from json it was not converted to the right type of string. Anybody else had that problem? I'll check the code to see if I can fix it.



Discussion:

Resolving patch:
diff --git a/keras/backend/common.py b/keras/backend/common.py
--- a/keras/backend/common.py
+++ b/keras/backend/common.py
@@ -22,6 +22,8 @@ def set_floatx(floatx):
     global _FLOATX
     if floatx not in {'float32', 'float64'}:
         raise Exception('Unknown floatx type: ' + str(floatx))
+    if isinstance(floatx, unicode):
+        floatx = floatx.encode('ascii')
     _FLOATX = floatx
 
 


Test patch:
Repository: keras-team/keras

Issue:
add get_initial_states to Recurrent
GRU, LSTM and other conventional RNNs have well defined initial state sizes that can be calculated from output_dim, but NTMs, convolutional rnn (I'm working on that) don't. Instead of hard coding the initial states inside `get_output` I propose an extra method called get_initial_states.



Discussion:

Resolving patch:
diff --git a/keras/layers/recurrent.py b/keras/layers/recurrent.py
--- a/keras/layers/recurrent.py
+++ b/keras/layers/recurrent.py
@@ -41,6 +41,15 @@ def output_shape(self):
     def step(self, x, states):
         raise NotImplementedError
 
+    def get_initial_states(self, X):
+        # build an all-zero tensor of shape (samples, output_dim)
+        initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
+        initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
+        reducer = K.zeros((self.input_dim, self.output_dim))
+        initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
+        initial_states = [initial_state for _ in range(len(self.states))]
+        return initial_states
+
     def get_output(self, train=False):
         # input shape: (nb_samples, time (padded with zeros), input_dim)
         X = self.get_input(train)
@@ -64,12 +73,7 @@ def get_output(self, train=False):
         if self.stateful:
             initial_states = self.states
         else:
-            # build an all-zero tensor of shape (samples, output_dim)
-            initial_state = K.zeros_like(X)  # (samples, timesteps, input_dim)
-            initial_state = K.sum(initial_state, axis=1)  # (samples, input_dim)
-            reducer = K.zeros((self.input_dim, self.output_dim))
-            initial_state = K.dot(initial_state, reducer)  # (samples, output_dim)
-            initial_states = [initial_state for _ in range(len(self.states))]
+            initial_states = self.get_initial_states(X)
 
         last_output, outputs, states = K.rnn(self.step, X, initial_states,
                                              go_backwards=self.go_backwards,


Test patch:
Repository: keras-team/keras

Issue:
Bug in callbacks.py ModelCheckpoint class
There is a bug in the constructor of the ModelCheckpoint class (lines 247-259)

See code below (from master branch). If the mode parameter is not recognized, the warning message refers to a self.mode variable that is never initialized, and an exception is thrown:

`AttributeError: 'ModelCheckpoint' object has no attribute 'mode'`

--- Code from callbacks.py ---

```
 def __init__(self, filepath, monitor='val_loss', verbose=0,
                 save_best_only=False, mode='auto'):


        super(Callback, self).__init__()
        self.monitor = monitor
        self.verbose = verbose
        self.filepath = filepath
        self.save_best_only = save_best_only

        if mode not in ['auto', 'min', 'max']:
            warnings.warn('ModelCheckpoint mode %s is unknown, '
                          'fallback to auto mode.' % (self.mode),
                          RuntimeWarning)
```



Discussion:

Resolving patch:
diff --git a/keras/callbacks.py b/keras/callbacks.py
--- a/keras/callbacks.py
+++ b/keras/callbacks.py
@@ -255,7 +255,7 @@ def __init__(self, filepath, monitor='val_loss', verbose=0,
 
         if mode not in ['auto', 'min', 'max']:
             warnings.warn('ModelCheckpoint mode %s is unknown, '
-                          'fallback to auto mode.' % (self.mode),
+                          'fallback to auto mode.' % (mode),
                           RuntimeWarning)
             mode = 'auto'
 


Test patch:
Repository: keras-team/keras

Issue:
Textual information for labels?
I seem unable to use text for labels, whilst using to_categorical

```
Using Theano backend.
Traceback (most recent call last):
  File "playground.py", line 88, in <module>
    train_model_and_test(number_of_epochs, number_of_classes, train_data, train_label, augmented_data_generator)
  File "playground.py", line 62, in train_model_and_test
    train_label = np_utils.to_categorical(train_label, number_of_classes)
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/np_utils.py", line 12, in to_categorical
    y = np.asarray(y, dtype='int32')
  File "/usr/lib/python2.7/dist-packages/numpy/core/numeric.py", line 460, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: invalid literal for int() with base 10: 'yellow'
```



Discussion:
 The problem has been fixed, but I might as well ask - why on earth are you only allowing for int32 labels?

```

def to_categorical(y, nb_classes=None):
    '''Convert class vector (integers from 0 to nb_classes)
    to binary class matrix, for use with categorical_crossentropy.
    '''
    y = np.asarray(y, dtype='int32')
    if not nb_classes:
        nb_classes = np.max(y)+1
    Y = np.zeros((len(y), nb_classes))
    for i in range(len(y)):
        Y[i, y[i]] = 1.
    return Y
```

How to change (for reference)

```
    y = np.asarray(y, dtype='int32')
```

to 

```
    y = np.asarray(y, dtype='a16')
```

https://github.com/fchollet/keras/blob/master/keras/utils/np_utils.py#L8-L18

Seems a bit daft!

for my application, i have a varying number of classes for each sample. `y = np.asarray(y, dtype='int32')` causes `to_categorical` to fail with `ValueError: setting an array element with a sequence.` but if i remove the line completely, `to_categorical` behaves correctly.


Resolving patch:
diff --git a/keras/utils/np_utils.py b/keras/utils/np_utils.py
--- a/keras/utils/np_utils.py
+++ b/keras/utils/np_utils.py
@@ -9,7 +9,6 @@ def to_categorical(y, nb_classes=None):
     '''Convert class vector (integers from 0 to nb_classes)
     to binary class matrix, for use with categorical_crossentropy.
     '''
-    y = np.asarray(y, dtype='int32')
     if not nb_classes:
         nb_classes = np.max(y)+1
     Y = np.zeros((len(y), nb_classes))


Test patch:
Repository: keras-team/keras

Issue:
TensorBoard with Histograms crashes with Dropout Layers
The most recent version of Keras crashes after the first epoch when a TensorBoard callback is added that records histograms.

Failing Example Code (`examples/mnist_mlp.py` with added callback) and Crash Log:
https://gist.github.com/jdoerrie/713a2d53e14b01470aa664494186d7c7

Judging by the output the issue is related to `keras_learning_phase` not being fed. The code works normally when `histogram_freq` is set to 0 or the Dropout layers are removed.
- Operating System: Mac OS X 10.11.3
- Python: 3.5.1
- Keras: 1.0.0 (Master Checkout)
- Tensorflow: 0.7.1
- Running on CPU.



Discussion:
I'm on it, I will submit a pull request in a few minutes.


Resolving patch:
diff --git a/keras/callbacks.py b/keras/callbacks.py
--- a/keras/callbacks.py
+++ b/keras/callbacks.py
@@ -467,8 +467,14 @@ def on_epoch_end(self, epoch, logs={}):
             if epoch % self.histogram_freq == 0:
                 # TODO: implement batched calls to sess.run
                 # (current call will likely go OOM on GPU)
-                feed_dict = dict(zip(self.model.inputs,
-                                     self.model.validation_data))
+                if self.model.uses_learning_phase:
+                    cut_v_data = len(self.model.inputs)
+                    val_data = self.model.validation_data[:cut_v_data] + [0]
+                    tensors = self.model.inputs + [K.learning_phase()]
+                else:
+                    val_data = self.model.validation_data
+                    tensors = self.model.inputs
+                feed_dict = dict(zip(tensors, val_data))
                 result = self.sess.run([self.merged], feed_dict=feed_dict)
                 summary_str = result[0]
                 self.writer.add_summary(summary_str, epoch)


Test patch:
diff --git a/tests/keras/test_callbacks.py b/tests/keras/test_callbacks.py
--- a/tests/keras/test_callbacks.py
+++ b/tests/keras/test_callbacks.py
@@ -126,7 +126,7 @@ def test_LearningRateScheduler():
     assert (float(K.get_value(model.optimizer.lr)) - 0.2) < K.epsilon()
 
 
-@pytest.mark.skipif((K._BACKEND != 'tensorflow') or (sys.version_info[0] == 3),
+@pytest.mark.skipif((K._BACKEND != 'tensorflow'),
                     reason="Requires tensorflow backend")
 def test_TensorBoard():
     import shutil
@@ -252,8 +252,4 @@ def data_generator_graph(train):
     KTF.set_session(old_session)
 
 if __name__ == '__main__':
-    # pytest.main([__file__])
-    # test_ModelCheckpoint()
-    # test_EarlyStopping()
-    # test_LearningRateScheduler()
-    test_TensorBoard()
+    pytest.main([__file__])