aisingapore

/

sea-lion-7b-instruct

like 22

I tried to run the model as an HF inference endpoint. The first error I got was about the --trust-remote-code option missing, which I got trhough by setting envvar TRUST_REMOTE_CODE=true.

Afterwards, however, I run into what seems to be a configuration error. The tail end of the traceback in the logs is

- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once
- 2024-10-01T08:35:38.136+00:00     handle._run()
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/asyncio/events.py", line 84, in _run
- 2024-10-01T08:35:38.136+00:00     self._context.run(self._callback, *self._args)
- 2024-10-01T08:35:38.136+00:00 > File "/opt/conda/lib/python3.11/site-packages/text_generation_server/server.py", line 229, in serve_inner
- 2024-10-01T08:35:38.136+00:00     model = get_model_with_lora_adapters(
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/__init__.py", line 1219, in get_model_with_lora_adapters
- 2024-10-01T08:35:38.136+00:00     model = get_model(
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/__init__.py", line 632, in get_model
- 2024-10-01T08:35:38.136+00:00     return CausalLM(
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/causal_lm.py", line 569, in __init__
- 2024-10-01T08:35:38.136+00:00     model = model_class(prefix, config, weights)
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/custom_modeling/mpt_modeling.py", line 1099, in __init__
- 2024-10-01T08:35:38.136+00:00     self.transformer = MPTModel(prefix, config, weights)
- 2024-10-01T08:35:38.136+00:00   File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/custom_modeling/mpt_modeling.py", line 791, in __init__
- 2024-10-01T08:35:38.136+00:00     self.attn_impl = config.attn_config.attn_impl
- 2024-10-01T08:35:38.136+00:00 AttributeError: 'dict' object has no attribute 'attn_impl'

Any idea?