Repository: huggingface/text-generation-inference

Issue:
GPTQ Continuous Batching not working
### System Info

I pulled the latest commits few hrs back and built the docker image locally. I tried to use the gptq models such as Bloke 33b with the new changes to TGI regarding gptq. I am able to inference with the model but it seems to only server 1 request at a time. I tried to issue 3 requests from 3 different devices and it waits till one is finished and then continues to the next one.

When GPTQ was initially released in TGI, this was working (sort of but still slow due to how things are for gptq). I tested it with few models on 0.8 version but now it seems to either be a bug or would like to know if its intended.

docker run --gpus all --shm-size 40g -p 1000:80 -v $volume:/data -e GPTQ_BITS=4 -e GPTQ_GROUPSIZE=128 hug:gptq --max-input-length 2048 --max-total-tokens 6000 --model-id TheBloke/WizardLM-33B-V1.0-Uncensored-GPTQ --revision gptq-4bit-128g-actorder_True --num-shard 4 --trust-remote-code --quantize gptq --max-batch-total-tokens 10000

Thank You!

### Information

- [ ] Docker
- [ ] The CLI directly

### Tasks

- [ ] An officially supported command
- [ ] My own modifications

### Reproduction

Running gptq model and inferencing with it on multiple devices at same time

### Expected behavior

Perform continuous batching with gptq models.


Discussion:
nvm it seems to work after i increased the --max-batch-total-tokens. I will do some more testing and close this asap if everything is good.

It seems like if theres a new request, it pauses the other request until its ready to stream the answer and then all streams can happen at the same time.


> It seems like if theres a new request, it pauses the other request until its ready to stream 

There are various parameters discoverable in `text-generation-launcher --help` that can manage how we sequence/stack/unstack operations.
What you said is correct, but there's a little more to it.
I tried it but I couldnt get anything to make it answer multiple queries at same time :( Any guidance on this? Like what parameters should i use other than what I have? I have about 90gb vram to work with and would like to use a 33b model. Hopefully once that exllama pr gets merged, it would boost the speeds and quantization will be more usable to serve multi user scenarios :) 

It's all your settings.

We are using GPTQ with *many* concurrent users in prod.
But it all boils down to your hardware + settings
@Narsil I got 4 A10G's (96GB VRAM) GPU's. Could you maybe give me like the docker command for smth you would be using to do concurrent requests? I think I should be able to use pretty much same numbers as you would be able to. And from there i can change them but would be nice to have a started point as you are using this in prod. 

This is what i was doing. Am I missing something?
docker run --gpus all --shm-size 40g -p 1000:80 -v $volume:/data -e GPTQ_BITS=4 -e GPTQ_GROUPSIZE=128 hug:gptq --max-input-length 2048 --max-total-tokens 6000 --model-id TheBloke/WizardLM-33B-V1.0-Uncensored-GPTQ --revision gptq-4bit-128g-actorder_True --num-shard 4 --trust-remote-code --quantize gptq --max-batch-total-tokens 17000

Also, is WizardCoder not supported yet for GPTQ? I get missing tensor error when using the Bloke's new quantized models that are suppose to be working with TGI. The llama ones work so thinking the WC is not yet implemented

Thanks!
Nvm, I figured it out and yes it indeed works. The quality seems to be dropped significantly though but gptq works :)

Still would like to know about WizardCoder support. 

Thanks!
> be dropped significantly though but gptq

Yes it's a model per model thing, possibly linked to seeding and examples too (but it really seems more model per model basis to my very surfacy investigation).

WizardCode is just GPTBigCode, no ? (So it's supported as santacoder models are)
@Narsil Hi! After your quantizing script upgrade, It is not possible to use it anymore. I just get same error for all models i try to quantize. 

weight lm_head.weight does not exist

I also saw that it says loaded model but nothing gets loaded but then realized its loading empty weights. But yea im no longer able to quantize. I tried messing with the code and I also see that its loading the whole thing on 1 gpu so is there way to use all the gpus instead? 

The error im seeing with GPTBigCodeForCausalLM models. I neither can load the gptq versions or quantize using the script myself. Guessing they are not supported for gptq yet as I am able to load them in float16 just fine

Also, the config is not loading trust_remote_code here:
config = AutoConfig.from_pretrained(
        model_id,
        trust_remote_code=True,
        device_map="auto",
    )

    with init_empty_weights():
        model = AutoModelForCausalLM.from_config(config, trust_remote_code=True, torch_dtype=torch.float16)
    model = model.eval()
    
    I had to manually add it for it to work or else it throws the error.

Overall, if you want to test this, try to use gptq version of starchat or wizardcoder. Neither can be loaded unfortunately. 

Resolving patch:
diff --git a/server/text_generation_server/utils/gptq/quantize.py b/server/text_generation_server/utils/gptq/quantize.py
--- a/server/text_generation_server/utils/gptq/quantize.py
+++ b/server/text_generation_server/utils/gptq/quantize.py
@@ -812,10 +812,13 @@ def inner(module, args):
                 tensor = weights.get_tensor(tensor_name)
                 setdeepattr(module, local_param, nn.Parameter(tensor))
             else:
+                tensor = current_tensor.to(device=torch.device("cuda:0"))
+                if current_tensor.requires_grad:
+                    tensor = nn.Parameter(tensor)
                 setdeepattr(
                     module,
                     local_param,
-                    nn.Parameter(current_tensor.to(device=torch.device("cuda:0"))),
+                    tensor
                 )
 
     return inner


Test patch: