aws-neuron

/

zephyr-7b-seqlen-2048-bs-4-cores-2

like 2

Follow

AWS Inferentia and Trainium 64

I have followed the commands given in https://huggingface.co/blog/text-generation-inference-on-inferentia2 but I am still getting the message

WARNING:sagemaker:Your model is not compiled. Please compile your model before using Inferentia.

when attempting to run the deploy line:

llm = llm_model.deploy(
        initial_instance_count=1,
        instance_type=instance_type,
        container_startup_health_check_timeout=health_check_timeout,
    )

Hey @charlescearl ,

Leaving this here for anyone else that comes across it. The warning from sagemaker is something that you will (currently) see for every Inferentia model you try to load regardless.