codellama

/

CodeLlama-7b-Instruct-hf

like 180

Did you found any way to make it run.?

While trying to host in sagemaker, I get the following error message:

"""
The tokenizer class you load from this checkpoint is 'CodeLlamaTokenizer'.
The class this function is called from is 'LlamaTokenizer'.
"""

I believe it probably has to do with this disclaimer in the "Model usage":
To use this model, please make sure to install transformers from main until the next version is released:
pip install git+https://github.com/huggingface/transformers.git@main accelerate

Since the scripts gets aHugging Face Deep Learning Container image from here:
get_huggingface_llm_image_uri("huggingface",version="0.9.3")

Hugging face team probably has to update this version to include new transformers class.
I could be totally wrong anyway.

@lucasbonservizzi is right. Code Llama just landed in Transformers main branch. There is not yet even a release so there is no sagemaker container available.

@philschmid Do you have any instructions on how would I be able to create my own container with TGI and the new transformers branch in order to try Code llama on SageMaker?

I got this error : You are using a model of type llama to instantiate a model of type . This is not supported for all configurations of models and can yield errors.

Is it related to the same issue, and is there any ETA on the release please?

@ArthurZ @philschmid any ETA for a new sagemaker container release to fix this bug please ?

Try changing the huggingface version to 0.8.2 in the deployment script

Try changing the huggingface version to 0.8.2 in the deployment script

I haven't tried your suggestion yet. However, it seems awkward that version 0.8.2 works and 0.9.3 doesn't.
As I previously commented, the error seems to be that CodeLlama models seems to be using a brand new class of tokenizer (CodeLlamaTokenizer), which has not been implemented on the newest TGI version (0.9.3) let alone an older version like 0.8.2. Anyway, I could be mistaken and downgrading might work.
In addition, if anyone has any tutorials on how to build your own TGI container it would be nice to learn. I haven't found out any tutorials on how to do it.

@lucasbonservizzi I tested the new TGI release (1.0.3) and it seems to work pretty well.
https://github.com/aws/deep-learning-containers/pull/3286

The code snippet you get from "deploy" is updated as well.