Repository: huggingface/text-generation-inference

Issue:
TypeError: forward() got an unexpected keyword argument 'position_ids'
I started a server with 

```
text-generation-launcher --model-id facebook/galactica-30b --num-shard 1
```

However, when I now send a request like

```
curl localhost:3000/generate -H 'Content-Type: application/json' -d '{"inputs":"Hi my name is","parameters":{"max_new_tokens":60}}'
```

it consistenly returns `TypeError: forward() got an unexpected keyword argument 'position_ids'`, with the following traceback:

```
2023-02-20T14:41:54.081514Z ERROR shard-manager: text_generation_launcher: "Method Prefill encountered an error.
Traceback (most recent call last):
  File \"/home/user/miniconda3/envs/text_generation/bin/text-generation-server\", line 8, in <module>
    sys.exit(app())
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/typer/main.py\", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/click/core.py\", line 1130, in __call__
    return self.main(*args, **kwargs)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/typer/core.py\", line 778, in main
    return _main(
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/typer/core.py\", line 216, in _main
    rv = self.invoke(ctx)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/click/core.py\", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/click/core.py\", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/click/core.py\", line 760, in invoke
    return __callback(*args, **kwargs)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/typer/main.py\", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File \"/home/user/text-generation-inference-new/server/text_generation/cli.py\", line 55, in serve
    server.serve(model_id, revision, sharded, quantize, uds_path)
  File \"/home/user/text-generation-inference-new/server/text_generation/server.py\", line 130, in serve
    asyncio.run(serve_inner(model_id, revision, sharded, quantize))
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/asyncio/runners.py\", line 44, in run
    return loop.run_until_complete(main)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/asyncio/base_events.py\", line 634, in run_until_complete
    self.run_forever()
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/asyncio/base_events.py\", line 601, in run_forever
    self._run_once()
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/asyncio/base_events.py\", line 1905, in _run_once
    handle._run()
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/asyncio/events.py\", line 80, in _run
    self._context.run(self._callback, *self._args)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/grpc_interceptor/server.py\", line 153, in invoke_intercept_method
    return await self.intercept(
> File \"/home/user/text-generation-inference-new/server/text_generation/interceptor.py\", line 20, in intercept
    return await response
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py\", line 82, in _unary_interceptor
    raise error
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py\", line 73, in _unary_interceptor
    return await behavior(request_or_iterator, context)
  File \"/home/user/text-generation-inference-new/server/text_generation/server.py\", line 41, in Prefill
    generations, next_batch = self.model.generate_token(batch)
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/contextlib.py\", line 79, in inner
    return func(*args, **kwds)
  File \"/home/user/text-generation-inference-new/server/text_generation/models/causal_lm.py\", line 297, in generate_token
    logits, past = self.forward(
  File \"/home/user/text-generation-inference-new/server/text_generation/models/causal_lm.py\", line 284, in forward
    outputs = self.model.forward(
  File \"/home/user/miniconda3/envs/text_generation/lib/python3.9/site-packages/accelerate/hooks.py\", line 156, in new_forward
    output = old_forward(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'position_ids'
" rank=0
2023-02-20T14:41:54.081879Z ERROR batch{batch_size=1}:prefill:prefill{id=0 size=1}:prefill{id=0 size=1}: text_generation_client: router/client/src/lib.rs:29: Server error: forward() got an unexpected keyword argument 'position_ids'
2023-02-20T14:41:54.081947Z ERROR HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3000 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=curl/7.82.0 otel.kind=server trace_id=c742e54c2eddc1bfcc788ed10b3e0c52}:generate{req=Json(GenerateRequest { inputs: "Hi my name is", parameters: GenerateParameters { temperature: None, repetition_penalty: None, top_k: None, top_p: None, do_sample: false, max_new_tokens: 60, stop: [], details: false, seed: None } })}:generate{request=GenerateRequest { inputs: "Hi my name is", parameters: GenerateParameters { temperature: None, repetition_penalty: None, top_k: None, top_p: None, do_sample: false, max_new_tokens: 60, stop: [], details: false, seed: None } }}:generate_stream{request=GenerateRequest { inputs: "Hi my name is", parameters: GenerateParameters { temperature: None, repetition_penalty: None, top_k: None, top_p: None, do_sample: false, max_new_tokens: 60, stop: [], details: false, seed: None } }}:infer{batch_size=1}:send_error: text_generation_router::infer: router/src/infer.rs:338: Request failed during generation: Server error: forward() got an unexpected keyword argument 'position_ids'
```


Discussion:

Resolving patch:
diff --git a/server/text_generation/models/galactica.py b/server/text_generation/models/galactica.py
--- a/server/text_generation/models/galactica.py
+++ b/server/text_generation/models/galactica.py
@@ -2,7 +2,7 @@
 import torch
 import torch.distributed
 
-from typing import List, Optional, Type
+from typing import List, Optional, Type, Tuple
 
 from accelerate import init_empty_weights
 from safetensors import safe_open
@@ -145,6 +145,20 @@ def decode(self, generated_ids: List[int]) -> str:
             generated_ids, skip_special_tokens=False, cleanup_tokenization_spaces=False
         )
 
+    def forward(
+        self, input_ids, attention_mask, position_ids, past_key_values: Optional = None
+    ) -> Tuple[torch.Tensor, List[Tuple[torch.Tensor, torch.Tensor]]]:
+        """Overwrite forward to ignore position_ids"""
+
+        # Model Forward
+        outputs = self.model.forward(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            use_cache=True,
+        )
+        return outputs.logits, outputs.past_key_values
+
 
 class GalacticaSharded(Galactica):
     def __init__(
@@ -322,7 +336,6 @@ def forward(
         outputs = self.model.forward(
             input_ids=input_ids,
             attention_mask=attention_mask,
-            position_ids=position_ids,
             past_key_values=past_key_values,
             use_cache=True,
         )


Test patch: