mms.service.PredictionException: Tokenizer class LlamaTokenizer does not exist or is not currently imported.
Author: ashwincv0112
Pull request: no

### System Info

I am trying to run a Prediction(Inference) by accessing a deployed AWS Sagemaker endpoint for CodeLlama-Python-7B model.
While I was able to finetune and  deploy the endpoint. I am getting the above error while running a prediction.
Could someone help on this. I changed the tokenised_class in the model repo to LlamaTokenizer while running this.


Also, tried with CodeLlamaTokenizer, getting the same error there as well.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

hub = {
  'HF_TASK':'text-generation'                           # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_output_path,  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   env=hub,
   transformers_version="4.26",             # Transformers version
    pytorch_version="1.13",                  # PyTorch version
    py_version="py39",                                 # Python version used
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="<GPU Instance>"
)


predictor.predict(sentiment_input)

### Expected behavior

Able to get the Model output for the provided Prompt (by accessing the Sagemaker endpoint)

Comments:
Hey! I think the transformers version you are using is wrong. Llama was part of the `4.32` release. Make sure to use this transformers version if you wanna use Llama. Codellama is `4.33.1`

Thank you for the response.... 
Also, could you also let me know the corresponding version for 
pytorch_version="1.13" and 
py_version="py39"

Mmmm torch should be alright, python as well

ok. thanks. I will try it and let you know...
Appreciate your help!

Hi Arthur,

I tried deploying the endpoint with the above mentioned transformer version (4.33.1 for CodeLlama)

huggingface_model = HuggingFaceModel(
   model_data=model_output_path,  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   env=hub,
   transformers_version="4.33.1",             # Transformers version
    pytorch_version="1.13",                  # PyTorch version
    py_version="py39",                                 # Python version used
)


I am getting the below error (I did upgrade the Sagemaker as mentioned):

`ValueError: Unsupported huggingface version: 4.33.1. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.6.1, 4.10.2, 4.11.0, 4.12.3, 4.17.0, 4.26.0, 4.28.1, 4.6, 4.10, 4.11, 4.12, 4.17, 4.26, 4.28.`


Ouch, that's outside my knowledge so I'll ping @philschmid 

@ashwincv0112 we recommend using TGI to deploy LLMs, see here for an example: https://www.philschmid.de/sagemaker-llama-llm

@philschmid thank you for the information.

We are actually trying to deploy a trained model saved in S3 to Sagemaker endpoint.

Just wanted to confirm, would the above approach work for that as well.


Yes see here for an example: https://www.philschmid.de/sagemaker-mistral#4-deploy-fine-tuned-mistral-7b-on-amazon-sagemaker

Thank you too much. I will try the same and let you know. Appreciate it!!!

> Yes see here for an example: https://www.philschmid.de/sagemaker-mistral#4-deploy-fine-tuned-mistral-7b-on-amazon-sagemaker

Thank you @philschmid , I am able to deploy the endpoint and inference on the same.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.