Package: sagemaker-huggingface-inference-toolkit
SageMaker Hugging Face Inference ToolkitSageMaker Hugging Face Inference Toolkit is an open-source library for serving 🤗 Transformers and Diffusers models on Amazon SageMaker. This library provides default pre-processing, predict and postprocessing for certain 🤗 Transformers and Diffusers models and tasks. It utilizes theSageMaker Inference Toolkitfor starting up the model server, which is responsible for handling inference requests.For Training, seeRun training on Amazon SageMaker.For the Dockerfiles used for building SageMaker Hugging Face Containers, seeAWS Deep Learning Containers.For information on running Hugging Face jobs on Amazon SageMaker, please refer to the🤗 Transformers documentation.For notebook examples:SageMaker Notebook Examples.💻  Getting Started with 🤗 Inference Toolkitneeds to be adjusted -> currently pseudo codeInstall Amazon SageMaker Python SDKpipinstallsagemaker--upgradeCreate a Amazon SageMaker endpoint with a trained model.fromsagemaker.huggingfaceimportHuggingFaceModel# create Hugging Face Model Classhuggingface_model=HuggingFaceModel(transformers_version='4.6',pytorch_version='1.7',py_version='py36',model_data='s3://my-trained-model/artifacts/model.tar.gz',role=role,)# deploy model to SageMaker Inferencehuggingface_model.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")Create a Amazon SageMaker endpoint with a model from the🤗 Hub.note: This is an experimental feature, where the model will be loaded after the endpoint is created. Not all sagemaker features are supported, e.g. MMEfromsagemaker.huggingfaceimportHuggingFaceModel# Hub Model configuration. https://huggingface.co/modelshub={'HF_MODEL_ID':'distilbert-base-uncased-distilled-squad','HF_TASK':'question-answering'}# create Hugging Face Model Classhuggingface_model=HuggingFaceModel(transformers_version='4.6',pytorch_version='1.7',py_version='py36',env=hub,role=role,)# deploy model to SageMaker Inferencehuggingface_model.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")🛠️ Environment variablesThe SageMaker Hugging Face Inference Toolkit implements various additional environment variables to simplify your deployment experience. A full list of environment variables is given below.HF_TASKTheHF_TASKenvironment variable defines the task for the used 🤗 Transformers pipeline. A full list of tasks can be findhere.HF_TASK="question-answering"HF_MODEL_IDTheHF_MODEL_IDenvironment variable defines the model id, which will be automatically loaded fromhuggingface.co/modelswhen creating or SageMaker Endpoint. The 🤗 Hub provides +10 000 models all available through this environment variable.HF_MODEL_ID="distilbert-base-uncased-finetuned-sst-2-english"HF_MODEL_REVISIONTheHF_MODEL_REVISIONis an extension toHF_MODEL_IDand allows you to define/pin a revision of the model to make sure you always load the same model on your SageMaker Endpoint.HF_MODEL_REVISION="03b4d196c19d0a73c7e0322684e97db1ec397613"HF_API_TOKENTheHF_API_TOKENenvironment variable defines the your Hugging Face authorization token. TheHF_API_TOKENis used as a HTTP bearer authorization for remote files, like private models. You can find your token at yoursettings page.HF_API_TOKEN="api_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"HF_TRUST_REMOTE_CODETheHF_TRUST_REMOTE_CODEenvironment variable defines wether or not to allow for custom models defined on the Hub in their own modeling files. Allowed values are"True"and"False"HF_TRUST_REMOTE_CODE="True"HF_OPTIMUM_BATCH_SIZETheHF_OPTIMUM_BATCH_SIZEenvironment variable defines the batch size, which is used when compiling the model to Neuron. The default value is1. Not required when model is already converted.HF_OPTIMUM_BATCH_SIZE="1"HF_OPTIMUM_SEQUENCE_LENGTHTheHF_OPTIMUM_SEQUENCE_LENGTHenvironment variable defines the sequence length, which is used when compiling the model to Neuron. There is no default value. Not required when model is already converted.HF_OPTIMUM_SEQUENCE_LENGTH="128"🧑🏻‍💻 User defined code/modulesThe Hugging Face Inference Toolkit allows user to override the default methods of theHuggingFaceHandlerService. Therefore, they need to create a folder namedcode/with aninference.pyfile in it. You can find an example for it insagemaker/17_customer_inference_script.
For example:model.tar.gz/|-pytorch_model.bin|-....|-code/|-inference.py|-requirements.txtIn this example,pytorch_model.binis the model file saved from training,inference.pyis the custom inference module, andrequirements.txtis a requirements file to add additional dependencies.
The custom module can override the following methods:model_fn(model_dir): overrides the default method for loading the model, the return valuemodelwill be used in thepredict()for predicitions. It receives argument themodel_dir, the path to your unzippedmodel.tar.gz.transform_fn(model, data, content_type, accept_type): overrides the default transform function with a custom implementation. Customers using this would have to implementpreprocess,predictandpostprocesssteps in thetransform_fn.NOTE: This method can't be combined withinput_fn,predict_fnoroutput_fnmentioned below.input_fn(input_data, content_type): overrides the default method for preprocessing, the return valuedatawill be used in thepredict()method for predicitions. The input isinput_data, the raw body of your request andcontent_type, the content type form the request Header.predict_fn(processed_data, model): overrides the default method for predictions, the return valuepredictionswill be used in thepostprocess()method. The input isprocessed_data, the result of thepreprocess()method.output_fn(prediction, accept): overrides the default method for postprocessing, the return valueresultwill be the respond of your request(e.g.JSON). The inputs arepredictions, the result of thepredict()method andacceptthe return accept type from the HTTP Request, e.g.application/json🤝 ContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull
requests to us.📜  LicenseSageMaker Hugging Face Inference Toolkit is licensed under the Apache 2.0 License.🧑🏻‍💻 Development EnvironmentInstall all test and development packages withpip3install-e".[test,dev]"Run Model Locallymanually changeMMS_CONFIG_FILEwget -O sagemaker-mms.properties https://raw.githubusercontent.com/aws/deep-learning-containers/master/huggingface/build_artifacts/inference/config.propertiesRun Container, e.g.text-to-imageHF_MODEL_ID="stabilityai/stable-diffusion-xl-base-1.0" HF_TASK="text-to-image" python src/sagemaker_huggingface_inference_toolkit/serving.pyAdjusthandler_service.pyand comment outif content_type in content_types.UTF8_TYPES:thats needed for SageMaker but cannot be used locallySend requestcurl --request POST \
  --url http://localhost:8080/invocations \
  --header 'Accept: image/png' \
  --header 'Content-Type: application/json' \
  --data '"{\"inputs\": \"Camera\"}" \
  --output image.png