Deploying Huggingface Sagemaker Models with Elastic Inference

Question:
When I try to deploy a HuggingFace Sagemaker model with elastic inference (denoted by the accelerator_type parameter) I get an error.
Deploy Snippet:
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.t2.medium",
    accelerator_type='ml.eia2.medium'
)

Error Msg:
~/miniconda3/envs/ner/lib/python3.8/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name)
    305     """Checks if the arg is in the available options, and raises a ``ValueError`` if not."""
    306     if arg not in available_options:
--> 307         raise ValueError(
    308             "Unsupported {arg_name}: {arg}. You may need to upgrade your SDK version "
    309             "(pip install -U sagemaker) for newer {arg_name}s. Supported {arg_name}(s): "

ValueError: Unsupported image scope: eia. You may need to upgrade your SDK version (pip install -U sagemaker) for newer image scopes. Supported image scope(s): training, inference.

The model deploys successfully if I do not provide an accelerator (i.e., no Elastic Inference).
Do the HuggingFace Sagemaker models support EI? If yes, how might I deploy the model successfully with EI? And if not, is EI support on the roadmap?
Much thanks in advance!

Context:
Hey @schopra,
Sadly speaking we don’t have EI DLCs yet. We are working on it and it is on the roadmap with one of the highest priorities.
I would update this thread here when I got any news.


How to deploy a T5 model to AWS SageMaker for fast inference?

Question:
Hi,
I just watched the video of the Workshop: Going Production: Deploying, Scaling & Monitoring Hugging Face Transformer models 2 (11/02/2021) from Hugging Face.
With the informations about how to deploy (timeline start: 28:14), I created a notebook instance (type: ml.m5.xlarge) on AWS SageMaker where I did upload the notebook lab3_autoscaling.ipynb from huggingface-sagemaker-workshop-series >> workshop_2_going_production in github.
I ran it and got a inference time of about 70ms for the QA model (distilbert-base-uncased-distilled-squad). Great!
Then, I changed the model to be loaded from the HF model hub to t5-base with the following code:
hub = {
  'HF_MODEL_ID':'t5-base', # model_id from hf.co/models
  'HF_TASK':'translation' # NLP task you want to use for predictions
}

I did make the deploy through the following code:
# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)

And then, I did launch an inference… but the inference time goes up to more than 700ms!
As in the video (timeline start: 57:05), @philschmid said that there are still models that can not be deployed this way, I would like to check if T5 models (up to ByT5) are optimized or not for inference in AWS SageMaker (quantization through ONNX for example or not)?
If they are not yet optimized (as it looks like), when will they be?

awssagemaker_qavsT5o1924×1164 171 KB

Note: I noticed the same problem about T5 inference through the Inference API (see this thread: How to get Accelerated Inference API for T5 models? ).

Context: for large DL models such as transformers, inference on CPU is slower than on GPU. And T5 is much bigger than the distillbert used in the demo. 700ms is actually not that bad for a CPU transformer  try replacing m5.xlarge by g4dn.xlarge to reduce latency.


Question:
Hi there,
I have been trying to use the new serverless feature from Sagemaker Inference, following the different steps very well explained by @juliensimon in his video (using same Image for the container and same ServerlessConfig) to use an HuggingFace model (not fine-tuned on my side). However after having successfully created/deployed all resources (Model, EndpointConfig, Endpoint) and trying to invokeEndpoint, I encountered this error :
'Message': 'An exception occurred from internal dependency. Please contact customer support regarding request ...'

And when looking on cloudwatch, I also get this message :
python: can't open file "/usr/local/bin/deep_learning_container.py": [Errno 13] Permission denied

Error that I don’t get when using invokeEndpoint for a non-serverless Inference Endpoint.
Did someone already encounter this error ?
Thanks in advance !!

Context:
Hey @YannAgora,
Thanks for opening the thread. We encountered this error as well. See blog post 13

found limitation when testing: Currently, Transformer models > 512MB create errors

We already reported this error to the SageMaker team.
Which model are you trying to use? with which memory configuration?


Return all class labels from SageMaker invoke_endpoint

Question:
I’ve deployed the Hate-speech-CNERG/bert-base-uncased-hatexplain multi-class text classification model (along with others) to a SageMaker endpoint. Using the TextClassificationPipeline you’re able to pass in the return_all_scores=True parameter to see all class labels for the input.
However when using the SageMakerRuntime.invoke_endpoint function, I can only get one class per input. Anyone have any thoughts on an equivalent to the return_all_scores parameter?

Context:
Hi Will, when deploying a model to a Sagemaker endpoint you can provide a custom inference script in which you can control the behaviour of the model for inference requests. You can find the documentation on how to do this here: Deploy models to Amazon SageMaker
In your particular scenario I would think hat you can override the predict_fn and/or the output_fn methods to get to the desired result. You can find an example for an inference.py file here 2 (note that this particular file is for text summarization, so it won’t apply for your use case).
To see what code you need to get all scores returned you can refer back to the code from the TextClassification Pipeline, that you have already mentioned: transformers/text_classification.py at master · huggingface/transformers · GitHub 2. If you implement this code into your inference script you should be all set.
Hope that helps, let me know if any questions.
Cheers
Heiko


How can I create lambda function code after deploying bart transformer for summarization in sagemaker?

Question:
I deployed bart-large-cnn model in aws sagemaker for summarization task by following the code.

from sagemaker.huggingface import HuggingFaceModel
import sagemaker
role = sagemaker.get_execution_role()
Hub Model configuration. Models - Hugging Face 2

hub = {
‘HF_MODEL_ID’:‘facebook/bart-large-cnn’, # model_id from Models - Hugging Face
‘HF_TASK’:‘summarization’ # NLP task you want to use for predictions
}
create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
env=hub,
role=role, # iam role with permissions to create an Endpoint
transformers_version=“4.6”, # transformers version used
pytorch_version=“1.7”, # pytorch version used
py_version=“py36”, # python version of the DLC
)

deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1,
instance_type=“ml.m5.xlarge”
)
Then tested with text to extract summaries and it performed well.
What code i needed to add in lambda function, for creating a rest api to call the summariZation model endpoint using aws lambda and api gateway.

Context:
Hello @vihaary,
Create to hear it worked as expected. Invoking your endpoint from an AWS Lambda function is pretty easy you can either use the boto3 sagemaker-runtime or install the sagemaker sdk into your lambda function.
boto3 snippet:
import boto3
import json
client = boto3.client('sagemaker-runtime')

response = client.invoke_endpoint(
EndpointName=ENDPOINT_NAME,
ContentType="application/json",
Accept="application/json",
Body=json.dumps(payload),
)
print(response['Body'].read().decode())

sagemaker snippet
from sagemaker.huggingface import HuggingFacePredictor

predictor = HuggingFacePredictor(ENDPOINT_NAME)
response = predictor.predict(payload)
print(response)


Question:
Hi all. I’ve been using the HF SDK for SM, and it’s working well. I have two questions:

It looks like the SDK always deploys Endpoint resources in “real-time” mode. Is support for the new “serverless” mode forthcoming? Could help small devs like me save a lot of money.

There doesn’t seem to be a way to connect HF to an existing Endpoint resource. When my python app restarts, a new Endpoint resource must be created. Are there any plans to allow the SDK to gain control of an existing Endpoint? That would be great because it takes a lot of time to redeploy new Endpoint resources. (And the old ones have to be cleaned up manually too.)

Thank you!

Context:
Hi! regarding endpoint re-use: is your goal to connect to an endpoint that is already live?
You can use the Predictor class for this:
from sagemaker.huggingface.model import HuggingFacePredictor
predictor = HuggingFacePredictor(endpoint_name="<my existing endpoints>")


Question:
I need some help with deploying HG Pipelines on AWS Sagemaker
This is how I deploy a pre-trained model in a pipeline in my local environment and get a prediction.
local_model = BertForSequenceClassification.from_pretrained(“local_model_path”)
tokenizer = BertTokenizer.from_pretrained(“"bert-base-uncased", padding=True, truncation=True)
pipe = pipeline("text-classification", model= local_model, tokenizer=.tokenizer)
output = pipe(input)

I could not find a way to deploy this HG Pipeline on Sagemaker. I followed the following documentation Deploy models to Amazon SageMaker and tried to use HuggingFaceModel. However, I am not sure how I can provide the other pipeline parameters.
huggingface_model = HuggingFaceModel(
   model_data="s3://model.tar.gz",
   role=role,
   transformers_version="4.12",
   pytorch_version="1.9",
   py_version='py38'
)

Where and how can I provide the task (text-classification) and tokenizer information? If this is not possible, should I first tokenize my data and give that as an input to predictor? predictor.predict(data_tokenized)?
Any other suggestions? Thanks so much!!!

Context:
Hi Tom, I think you are on the right path. Provided your model is already on S3 and packaged as tar.gz file you can just use the deploy() method to deploy the model. To get predictions you can use predictor.predict, as you have already pointed out.
This notebook by @philschmid should be useful: notebooks/deploy_transformer_model_from_s3.ipynb at master · huggingface/notebooks · GitHub 1
Hope that helps, let me know if there are any questions.
Cheers
Heiko


Question:
I’m trying to overwrite predict_fn for a named-entity recognition task. Mostly because I provide very long sequences. In this case, I need to use the tokenizer to break the long sequence down, and then merge the predictions of all the sub-sequences.
I call the tokenizer as:
sentences = tokenizer(sentence, max_length=max_length, stride=stride, truncation=True,
                      return_overflowing_tokens=True)

Since I have a stride, I need to properly care for overlapping tokens and their predictions.
I can access the model, as it is received as a parameter of the predict_fn, but how do I access the tokenizer?
Thanks!

Context:
Hi Rogerio
model_fn actually takes the model directory as a parameter, not the model itself (see here: Deploy models to Amazon SageMaker). This means that you can load the tokenizer within model_fn like so:
tokenizer = AutoTokenizer.from_pretrained(model_dir)
Cheers
Heiko


Can’t deploy conversational HF model on AWS - Logs say model-path not a valid directory

Question:
I am using the given template code to make a SageMaker endpoint from my conversational HF model:
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'rachelcorey/DialoGPT-medium-kramer',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
        transformers_version='4.12',
        pytorch_version='1.9',
	py_version='py38',
	env={ 'HF_TASK':'conversational' },
	role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.t2.medium' # ec2 instance type
)

The code has been revised based on the answers from this thread: Deploying a conversational pipeline on AWS - #3 by philschmid which is doing exactly what I want to do. I’ve installed the proper versions of pytorch and transformers.
When I run the code, I get this error:
UnexpectedStatusException: Error hosting endpoint huggingface-pytorch-inference-2022-01-12-15-00-34-671: Failed. Reason:  The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..

When I go into the logs, I see this message about 100000 times:
ERROR - Given model-path /opt/ml/model is not a valid directory. Point to a valid model-path directory.

And also this message a couple times, but I assume it’s related to the previous error:
subprocess.CalledProcessError: Command '['model-archiver', '--model-name', 'model', '--handler', 'sagemaker_huggingface_inference_toolkit.handler_service', '--model-path', '/opt/ml/model', '--export-path', '/.sagemaker/mms/models', '--archive-format', 'no-archive', '--f']' returned non-zero exit status 1.

I tried to create this directory in opt/ml/ before running the code but it has no effect on the issue it seems.
According to some research, this is the directory that SageMaker puts trained models… However, my model is on HF, not trained by SageMaker… should I put the model files in that directory? Not sure what to do here. Do I ask AWS support for help with this instead of posting here? Any help is much appreciated, thank you in advance!

Context:
Hey @rachelcorey,
you are not passing in your hub configuration containing your model_id into the HuggingFaceModel meaning you are trying to create an endpoint without a model at all.
you just need to change
	env={ 'HF_TASK':'conversational' } => env=hub


Error 403 when downloading model for Sagemaker batch inference

Question:
I am creating a batch job with the code below. However it fails immediately with 403 forbidden client error. My cloudwatch has the following output (full traceback below)
This is an experimental beta features, which allows downloading model from the
Hugging Face Hub on start up. It loads the model defined in the env var `HF_MODEL_ID'

.
immediately followed by:
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2
after which the batch job fails. Deploying to an endpoint is working fine.
The full code for batch job:
from sagemaker.huggingface import HuggingFaceModel
# Hub Model configuration. https://huggingface.co/models
hub = {
  'HF_MODEL_ID':'sentence-transformers/all-mpnet-base-v2',
  'HF_TASK':'feature-extraction'
}
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.6',
    pytorch_version='1.7',
    py_version='py36',
    env=hub,
    role=role,
)

batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    output_path='s3://kj-temp/hf/out', # we are using the same s3 path to save the output with the input
    strategy='SingleRecord'
    )

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=test_input,
    content_type='application/json',
    split_type='Line',

wait = False)

and the full traceback:

Traceback (most recent call last):
  File "/usr/local/bin/dockerd-entrypoint.py", line 23, in <module>
    serving.main()
  File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/serving.py", line 34, in main
    _start_mms()
  File "/opt/conda/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/opt/conda/lib/python3.6/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/opt/conda/lib/python3.6/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/conda/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/serving.py", line 30, in _start_mms
    mms_model_server.start_model_server(handler_service=HANDLER_SERVICE)
  File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/mms_model_server.py", line 75, in start_model_server
    use_auth_token=HF_API_TOKEN,
  File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/transformers_utils.py", line 154, in _load_model_from_hub
    model_info = _api.model_info(repo_id=model_id, revision=revision, token=use_auth_token)
  File "/opt/conda/lib/python3.6/site-packages/huggingface_hub/hf_api.py", line 155, in model_info
    r.raise_for_status()
  File "/opt/conda/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

Context: Can you please retry, there was an issue with loading Sentence Transformers from the Hub.


ClientErro:400 when using batch transformer for inference

Question:
Hi everyone,
I try to do sentiment analysis on a bunch of data and follow the example notebook notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub   and below is my code:
from sagemaker.huggingface.model import HuggingFaceModel

hub = {
 'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',
 'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
env=hub,
role=role,
transformers_version='4.6',
pytorch_version="1.7",
py_version='py36',
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
 instance_count=1,
 instance_type='ml.p3.2xlarge',
 output_path=output_s3_path,
 strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
 data=input_s3_path,
 content_type="application/json",
 split_type="Line")

My input jsonl file is formatted as

image1485×231 19.8 KB

The model successfully infer when I feed with  first 10 rows  (totally this data is about 6k rows)
but it throw errors when I expand to 100 rows.
I’m super new to sagemaker and hugging face ,  can anyone tell me what I’m missing ? Thank you

Context:
sorry it’s still me, just attach more pics so the context might be better understood.
I heard BERT has limitation on text length so I truncated each line to 460 words

image1501×577 14.6 KB


Deploying a conversational pipeline on AWS

Question:
I am following the instructions for deploying a custom Microsoft/DialoGPT-medium on AWS Sagemaker. As a first step, I am using the instructions included on the model hub, but they do not work and appear to be out of date.
The code that follows leads to this error:
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "ConversationalPipeline expects a Conversation or list of Conversations as an input"
}

Code from the documentation:
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'microsoft/DialoGPT-medium',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	}
})

I can make those lists into Conversation objects, but even still I get an error. I am not even sure they will be json serializable anymore.
The larger issue is that I want to use a finetuned  Microsoft/DialoGPT-medium that I trained on my local machine. I am following a Hugging Face youtube tutorial for that one, but once again the code presented in the video does not work out of the box.
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz",  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   transformers_version="4.6",                           # Transformers version used
   pytorch_version="1.7",                                # PyTorch version used
   py_version='py36',                                    # Python version used
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)

This fails with the following error:
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "(\"You need to define one of the following [\u0027feature-extraction\u0027, \u0027text-classification\u0027, \u0027token-classification\u0027, \u0027question-answering\u0027, \u0027table-question-answering\u0027, \u0027fill-mask\u0027, \u0027summarization\u0027, \u0027translation\u0027, \u0027text2text-generation\u0027, \u0027text-generation\u0027, \u0027zero-shot-classification\u0027, \u0027conversational\u0027, \u0027image-classification\u0027] as env \u0027TASK\u0027.\", 403)"

This is my fourth time deploying a Hugging Face model to AWS Sagemaker, and the process is so incredibly complex and non-intuitive. I feel like the entire process is held up by toothpicks. Thanks for any light you can shed.

Context:
I think you forgot to add the env task when creating your HuggingFaceModel:
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz",  # path to your trained SageMaker model
   role=role,                                            # IAM role with permissions to create an endpoint
   transformers_version="4.6",                           # Transformers version used
   pytorch_version="1.7",                                # PyTorch version used
   py_version='py36',                                    # Python version used
   env={ 'HF_TASK':'conversational' },
)


ClientError:400 when using batch transformer on sagemaker for inference

Question:
Hello everyone,
I’m new to hugging face and try to do sentiment analysis on bunch of texts with batch transformer job , here is my code and I follow through example notebook : notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub 2
Here I rewrite the  input json file  as {“inputs”: “xxxxxxxxxx”}, and truncate  long text into 460 words since BERT model has limitation on text length,

image1490×495 12.3 KB

following is my code :

hub = {
    'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',
    'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role,
   transformers_version='4.6',
   pytorch_version="1.7",
   py_version='py36',
)
# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    output_path=output_s3_path,
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=input_s3_path,
    content_type="application/json",
    split_type="Line")

I test it with first 10 rows (data is 6k rows totally). It did successfully output result, however when I expand input to 100 rows , it prompts client error as this:
No older events at this moment.  Retry
2022-01-07T13:33:06.824-08:00 2022-01-07T21:33:03.067:[sagemaker logs]: MaxConcurrentTransforms=1, MaxPayloadInMB=6, BatchStrategy=SINGLE_RECORD
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.019:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: ClientError: 400
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.019:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl:
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.019:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: Message:
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.020:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: {
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.020:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: “code”: 400,
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.020:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: “type”: “InternalServerException”,
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.020:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: “message”: “CUDA error: device-side assert triggered”
2022-01-07T13:33:10.825-08:00 2022-01-07T21:33:10.021:[sagemaker logs]: soa-pax-processed/sentiment_hf/month_test.jsonl: }
Can anyone help me on what I’m mssing? Any help is highly appreciated!

Context:
Hey @miOmiO,
Happy to help you here! to narrow down your issue. I think the first step would be to check if the dataset was created in the sample (notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub 1) works or if it also errors out.
Additionally, could you bump the version of the HuggingFaceModel to the latest one? For transformers_version that’s 4.12.3 and for pytorch_version its 1.9.1 maybe this already solves your issue. You can find the list of available containers here: Reference 3
Also worth testing is to replace your model with a different model, e.g. distilbert-base-uncased-finetuned-sst-2-english


Training Spot Instance: meaning of parameters max_run and max_wait

Question:
Hi,
in the notebook sagemaker >> 05_spot_instances >> sagemaker-notebook.ipynb, we need to setup the parameters max_run and max_wait as said in the Hugging Face doc Spot instances.
However, there is no explanation about their meaning.
I searched more information the AWS SageMaker doc and found the parameters MaxRuntimeInSeconds and MaxWaitTimeInSeconds:

MaxRuntimeInSeconds: The maximum length of time, in seconds, that a training or compilation job can run.

MaxWaitTimeInSeconds: The maximum length of time, in seconds, that a managed Spot training job has to complete. It is the amount of time spent waiting for Spot capacity plus the amount of time the job can run. It must be equal to or greater than MaxRuntimeInSeconds . If the job does not complete during this time, Amazon SageMaker ends the job.

I guess that:

max_run = MaxRuntimeInSeconds

max_wait = MaxWaitTimeInSeconds

max_run
My question is: about max_run and in the case of the training of a Hugging Face model with Trainer(), what is a “training or compilation job”?

Is it the whole training job from the first step to the final one (ie, all the epochs)?
Is it the training job by checkpoint (by epoch or by the number of defined steps) and separately, the evaluation job at the end of each checkpoint?

max_wait
My question is: about max_wait and in the case of the training of a Hugging Face model with Trainer(), what is a “managed Spot training job”?

(same question 1 as for max_run) whole training job?
(same question 2 as for max_run) training job by checkpoint?

cc @philschmid

Context:
@pierreguillou To rephrase what @OlivierCR said:
max_run : answer 1: the whole training job
max_wait: none of your answers. This is the max time to get a spot instance allocated, plus the max_run


Slow inference using most recent docker image

Question:
We have been deploying a BERT model to a SageMaker endpoint using the g4dn.xlarge instance. The deploy is managed with terraform and we use the following image:
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.7-transformers4.6-gpu-py36-cu110-ubuntu18.04

We’re really happy with the model latency, which is about 0.05 seconds for the typical request.
Now we were experimenting with using the more recent image:
{account}.dkr.ecr.{region}.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-gpu-py38-cu111-ubuntu20.04

and found that model latency dropped by a factor of about 10, everything else being equal. I noticed in the sagemaker sdk docs that it says the supported versions of pytorch, transformers and python respectively are 1.7.1, 4.6.1 and 3.6 respectively.
So my question is more of a curiosity - why does the model latency drop on this most recent image? Is there some issue using the GPU here?
Thanks,
Owen

Context:
Hey @ojturner,
Thank you for opening the Thread. Performance drop should not appear! Can you share which model (at least the architecture and task) you use and which instance type?


Question:
Hi @philschmid,
I liked very much your post “Hugging Face Transformers BERT fine-tuning using Amazon SageMaker and Training Compiler 5” (and its promises of saving time and money when training NLP models  ) and I did try it with a T5 model.
However, I got the following error: The training task fails due to a missing XLA configuration
I found an article from AWS on “Training Job Fails Due to Missing XLA Configuration 4” but it did not fix the problem.
Do you have any suggestion? Thank you.

Context:
Hey @pierreguillou,
could you share how you created your Training Job (python estiomator + hyperparameters) and the training script you use?
Iin addition to this, I am not sure if T5 is well supported yet. As mentioned in my blog

The Amazon Training Compiler works best with Encoder Type models, like BERT , RoBERTa , ALBERT , DistilBERT .

The Training Compiler currently works best with encoder-type models. In the test I ran a couple of weeks back T5 performed worse than the default training.


Question:
Hi all,
I asked a question in the Discord a few days ago and was told to move it here instead.
In the wake of the Log4j vulnerability, should we expect any updated versions of the HuggingFace Deep Learning Container images used for the SageMaker integration 1, or have y’all confirmed that none of your versions are vulnerable in this way?
All the best,
Charles

Context:
Hey @charlesatftl,
I was in contact with AWS and the SageMaker Team and got the following response from Yogesh Sharma Engineering Manager for the DLCs 2

Hello,
We ran a canary scan to exercise caution and find out if HF DLCs are impacted by the log4j vulnerability. I can confirm that both Hugging Face TensorFlow and Hugging Face PyTorch DLCs are not impacted by the log4j issue.
Some of DLC’s upstream libraries use log4j v1.2 which is old but is not impacted by this CVE (it impacts v2.x). Teams at AWS have decided that we will upgrade the log4j version early January to v2.16 just to be on the latest safe version.

Let us know if you have further questions, thanks.


Sagemaker not being able to download EleutherAI/gpt-j-6B model from the Hugging Face Hub on start up

Question:
Trying to create a basic inference on Sagemaker, but can see that the model downloads 30 -40% and then the download restarts again and the loop keeps on happening . And after 20 -30 mins it just fails.
Tried with different instances as well, but still the same issue persists.
Any help would be really appreciated.
Here is he exact code,
hub = {
‘HF_MODEL_ID’:‘EleutherAI/gpt-j-6B’,
‘HF_TASK’:‘text-generation’
}

create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
transformers_version=‘4.6.1’,
pytorch_version=‘1.7.1’,
py_version=‘py36’,
env=hub,
role=role,
)

deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type=‘ml.m5.4xlarge’ # ec2 instance type
)
predictor.predict({
‘inputs’: "Can you please let us know more details about your "
})

Context: Pinging @philschmid here


Batch Tansform and accents in the json file

Question:
Hi.
I used the notebook lab2_batch_transform.ipynb of @philschmid to launch batch for inferences. My dataset is composed of texts in Portuguese (ie, with accents).
When I download the json file created and open it with Sublime Text or note, I see that all letters with accents were converted as following:
(...)
{"inputs": "forma\u00e7\u00e3o ...}
(...)

In this example, forma\u00e7\u00e3o is formação.
What do you think? I can use my json file or I need to solve this problem in order to get (real) letters with accents in my json file? Thanks.

Context:
Thanks @philschmid. I tested your code with encoding='utf-8' but it did not change the content of my json file with strange letters instead of letters with accents.
However, I just found the complementary code in this post 1 that solves my problem: ensure_ascii=False as an argument of json.dump().
Here the code I use now (taken from notebook lab2_batch_transform.ipynb and modified with the 2 cited arguments):
with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+", encoding='utf-8') as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        json.dump(row, outfile, ensure_ascii=False)
        outfile.write('\n')


Batch Tansform and API token for private model

Question:
Hi.
I used the notebook lab2_batch_transform.ipynb 2 of @philschmid to launch batch for inferences.
The model I used in in private mode in the HF model hub. I do not see in the following code from the notebook which argument to use in order to pass the API_TOKEN:
from sagemaker.huggingface.model import HuggingFaceModel

# Hub Model configuration. <https://huggingface.co/models>
hub = {
    'HF_MODEL_ID':model_checkpoint,
    'HF_TASK':'text2text-generation'
}

Who knows the answer? Thanks.

Context:
I just found the answer in github at Environment variables: HF_API_TOKEN
from sagemaker.huggingface.model import HuggingFaceModel

# Hub Model configuration. <https://huggingface.co/models>
hub = {
    'HF_MODEL_ID':model_checkpoint,
    'HF_TASK':'text2text-generation',
    'HF_API_TOKEN':API_TOKEN
}


How to export training logs from CloudWatch

Question:
Hi,
When I launch a model training in AWS SageMaker & HF Training DLC, there is a training job created in AWS SageMaker that has a link to CloudWatch > Logs > Logs groups > /aws/sagemaker/TrainingJobs > job name of my training > Log events where I can see the logs of my training.
Great.
Now, I would like to export all these logs to a txt file. Strange but I do not see a button for that.
Who knows how to do it? Thanks.

Context:
Hi! you can either export logs to S3 via the log export feature:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html 1
You can also query the logs via SDK, for example CloudWatchLogs — Boto3 Docs 1.20.23 documentation 2


Spot instances with Sagemaker batch transform?

Question:
Is it possible to use spot instances for batch transform? Don’t see in either of these places
https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/sagemaker.huggingface.html
ht tp s://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html
Trying to pass “use_spot_instances=True” to either the HuggingFaceModel, HuggingFaceModel.transformer or HuggingFaceModel.transformer.transform() gives an error. I am using this notebook as an example.

      github.com

    huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb

    {
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Huggingface Sagemaker-sdk - Run a batch transform inference job with 🤗 Transformers\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "1. [Introduction](#Introduction)  \n",
    "2. [Run Batch Transform after training a model](#Run-Batch-Transform-after-training-a-model)  \n",
    "3. [Run Batch Transform Inference Job with a fine-tuned model using `jsonl`](#Run-Batch-Transform-Inference-Job-with-a-fine-tuned-model-using-jsonl)   \n",
    "\n",
    "Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy two transformer model for inference. \n",
    "In the first example we deploy a trained Hugging Face Transformer model on to SageMaker for inference.\n",
    "In the second example we directly deploy one of the 10 000+ Hugging Face Transformers from the [Hub](https://huggingface.co/models) to Amazon SageMaker for Inference.<"
   ],

  This file has been truncated. show original

THanks

Context: not possible today. If you want to use Spot for inference, you can write inference code as a custom python script and run inference in the Training API / HuggingFace Estimator


How to access to /opt/ml/model before the end of the model training?

Question:
Hi,
In hyperparameters of my notebook in AWS SageMaker & HF Training DLC, I defined:
'output_dir': '/opt/ml/model'
In the CloudWatch logs, I can see for example:
Model weights saved in /opt/ml/model/checkpoint-6000/pytorch_model.bin
I would like to test this checkpoint-6000 in another notebook without waiting for the end of the model training.
How can I access to the content of /opt/ml/model/checkpoint-6000/ within the AWS site or inside another notebook? (I do not want to use aws cli in a terminal).
Thanks.

Context:
Hi @pierreguillou ; the content of opt/ml/model is accessible only at the end of training (and it will be compressed in a model.tar.gz, which will be enormous if you save dozen/hundreds of transformers checkpoints there). If you want to export models to s3 during training, without interruption, in the same file format as saved locally, you need to save them in /opt/ml/checkpoints, and specify the S3 sync location in your SDK call with the parameter checkpoint_s3_uri. Then use the AWS CLI or boto3 download_file to bring the from S3 back to your notebook.
See here the doc of the checkpoint feature, which is in my opinion one of the best features of SageMaker

Use Checkpoints in Amazon SageMaker - Amazon SageMaker 2
notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub


Question:
Hi,
in the notebook 06_sagemaker_metrics 1 / sagemaker-notebook.ipynb, there is the code to get training and eval metrics at the end of the training from the HuggingFaceEstimator.

How we can get them DURING the training?
Great, but I don’t understand how we can get them DURING the training to check how good (or not) the training is (for example, to detect overfitting and then, stop training before the last epoch).
My idea was to create a duplicate notebook (without running fit() in this duplicated one) for that purpose. The following text in the notebook seems to say that it is possible but how can we get specifiying the exact training job name in the TrainingJobAnalytics API call? Thanks.

Note that you can also copy this code and run it from a different place (as long as connected to the cloud and authorized to use the API), by specifiying the exact training job name in the TrainingJobAnalytics API call.)

Problem: “Warning: No metrics called eval_loss found”
I have a second question.
I used the metrics code (copy/paste) from  06_sagemaker_metrics 1 / sagemaker-notebook.ipynb in a NER finetuning notebook on AWS SageMaker.
The code of my NER notebook uses directly the script run_ner.py from github (through the argument git_config in my Hugging Face Estimator).
metric_definitions=[
    {'Name': 'loss', 'Regex': "'loss': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'learning_rate', 'Regex': "'learning_rate': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_loss', 'Regex': "'eval_loss': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_accuracy', 'Regex': "'eval_accuracy': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_f1', 'Regex': "'eval_f1': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_precision', 'Regex': "'eval_precision': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_recall', 'Regex': "'eval_recall': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_runtime', 'Regex': "'eval_runtime': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_samples_per_second', 'Regex': "'eval_samples_per_second': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'epoch', 'Regex': "'epoch': ([0-9]+(.|e\-)[0-9]+),?"}]

git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.12.3'}

huggingface_estimator = HuggingFace(
    entry_point          = 'run_ner.py',
    source_dir           = './examples/pytorch/token-classification',
    git_config           = git_config,
    (...),
    metric_definitions   = metric_definitions,
)

I have no problem of training but when I want to display the metrics, most of them were not found (see the following screen shot):

metrics2038×940 59.8 KB

I compared the code relative to the logs in the 2 scripts and they are different.
In the train.py:
  # Set up logging
   logger = logging.getLogger(__name__)

   logging.basicConfig(
        level=logging.getLevelName("INFO"),
        handlers=[logging.StreamHandler(sys.stdout)],
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )

In the run_ner.py:
logger = logging.getLogger(__name__)

# Setup logging
logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

This is the reason of the problem?

Context:
@pierreguillou - how to find the metrics - in addition to dashboarding them in Cloudwatch via the path provided by Philipp (job detail page in the console, then “algorithm metrics” link in the bottom), you can also pull them in real time with the SDK
from sagemaker.analytics import TrainingJobAnalytics

df = TrainingJobAnalytics( training_job_name="jobname").dataframe()

Capture d’écran 2021-12-09 à 17.07.251144×1134 93.4 KB


Finetuning sentence embedding model with SageMaker - how to compute loss?

Question:
I’m looking for a model that will return an embedding vector that can be used in downstream classification tasks. I have been able to deploy the pretrained model sentence-transformers/all-mpnet-base-v2 · Hugging Face 1
to an endpoint and get embeddings from it. However, when I try to finetune the model with huggingface_estimator I get  an error as it doesn’t appear this model returns a loss since it doesn’t have labels (the data I am supplying is simply more domain specific text examples. How can I pass a loss when fine tuning a sentence transformer?
So fist I used the HuggingFaceModel from the sagemaker toolkit
from sagemaker.huggingface import HuggingFaceModel
hub = {
  'HF_MODEL_ID':'sentence-transformers/all-mpnet-base-v2',
  'HF_TASK':'feature-extraction'
}
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    transformers_version='4.6',
    pytorch_version='1.7',
    py_version='py36',
    env=hub,
    role=role,
)

In this example from amazon Fine-tune and host Hugging Face BERT models on Amazon SageMaker | AWS Machine Learning Blog 2 they are first fine tuning the model using the huggingfacee_estimator, and then they create a new HuggingFaceModel object and pass the model_data from the huggingface_estimator fine tuning job. Like this:
from sagemaker.huggingface.model import HuggingFaceModel

huggingface_model = sagemaker.huggingface.HuggingFaceModel(
env={ 'HF_TASK':'sentiment-analysis' },
model_data=huggingface_estimator.model_data,
role=role, # iam role with permissions to create an Endpoint
transformers_version="4.6.1", # transformers version used
pytorch_version="1.7.1", # pytorch version used
py_version='py36', # python version
)

My problem is when I try to finetune on my data, since it is not a classifier, the model output is not return a loss and I get an invalid key error from compute_loss() in transformer/trainer.py transformers/trainer.py at 3977b58437b8ce1ea1da6e31747d888efec2419b · huggingface/transformers · GitHub

KeyError: 'loss'

Does it make sense to finetune an embedding model? Is there a way to pass a loss function and have it included in the model output.
I am just passing it additional unlabelled data. How would one do this in sagemaker given that it is a feature extraction model and not a classification/prediction model?
Thanks

Context:
Hello @kjackson,
How did you try to fine-tune it the code you shared is only on how to deploy it.
We have a detailed “Getting Started” example with video support to run your first training on Amazon SageMaker: Get started 4 This might help you get started.

Does it make sense to finetune an embedding model? Is there a way to pass a loss function and have it included in the model output.

Yes it makes sense to further fine-tune a language model to let it “learn” a new context/domain


Huggingface_hub integration: ModuleNotFoundError: No module named ‘huggingface_hub’

Question:
Hi Philipp,
I have been trying to use the new functionally of push to hub on my script and I could not even past the installation, I ran the: !pip install “sagemaker==2.69.0” “transformers==4.12.3” --upgrade command and for some reason sagemaker is not getting updated.
I am using a notebook instance.
Thanks,
Jorge

Context:
Hey @Jorgeutd,
The sagemaker notebooks are currently containing an incompatible/old version of boto3 with the latest dataset and transformers.
Thats how I solved it:
!pip install "sagemaker>=2.69.0" "transformers==4.12.3" --upgrade
# using older dataset due to incompatibility of sagemaker notebook & aws-cli with > s3fs and fsspec to >= 2021.10
!pip install  "datasets==1.13" --upgrade

BTW. we also have an example notebook on how to push models to the hub during sagemaker training.

      github.com

    huggingface/notebooks/blob/master/sagemaker/14_train_and_push_to_hub/sagemaker-notebook.ipynb 1

    {
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HuggingFace Hub meets Amazon SageMaker\n",
    "### Fine-tune a Multi-Class Classification with `Trainer` and `emotion` dataset and push it to the [Hugging Face Hub](https://huggingface.co/models)"
   ]
  },
  {
   "attachments": {
    "emotion-widget.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAABRYAAAM4CAYAAAC5rb6iAAAMamlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnltSSWiBCEgJvQnSq5QQWgABqYKNkAQSSowJQcVeFhVcKyKKFV0VUXR1BWRREXtZFHtfLKgo66IuiqLyJiSg677yvfN9c+fPmTP/KZm5dwYArV6eVJqHagOQLymQJUSEsMampbNIHYAMDAAGtIA7jy+XsuPjYwCUwf7v8u4GQJT9VScl1z/H/6voCoRyPgDIeIgzBXJ+PsTNAOAb+FJZAQBEpd5yaoFUiedCrCeDAUJcpsTZKrxLiTNVuGnAJimBA/FlAMg0Hk+WDYDmPahnFfKzIY/mJ4hdJAKxBACtERAH8kU8AcTK2Efk509W4gqI7aC9FGIYD/DJ/IYz+2/8mUP8PF72EFblNSDkULFcmseb/n+W5n9Lfp5i0IcNbDSRLDJBmT+s4a3cydFKTIO4S5IZG6esNcS9YoGq7gCgVJEiMllljxrz5RxYP8CE2EXAC42G2BjicElebIxan5klDudCDFcLOk1cwE2C2ADixUJ5WKLaZotscoLaF1qXJeOw1fqzPNmAX6WvB4rcZLaa/41IyFXzY5pFoqRUiKkQWxWKU2Ih1oTYWZ6bGK22GVUk4sQO2sgUCcr4rSBOEEoiQlT8WGGWLDxBbV+SLx/MF9siEnNj1fhAgSgpUlUf7CSfNxA/zAW7LJSwkwd5hPKxMYO5CIShYarcsedCSXKimqdXWhCSoJqLU6V58Wp73EKYF6HUW0DsIS9MVM/FUwrg4lTx41nSgvgkVZx4UQ4vKl4VD74CxAAOCAUsoIAtE0wGOUDc2lXfBX+pRsIBD8hANhACJ7VmcEbqwIgEPhNBEfgDIiGQD80LGRgVgkKo/zykVT2dQNbAaOHAjFzwFOJ8EA3y4G/FwCzJkLcU8ARqxP/wzoOND+PNg005/u/1g9qvGjbUxKg1ikGPLK1BS2IYMZQYSQwn2uNGeCDuj8fAZzBsbrgP7juYx1d7wlNCG+ER4TqhnXB7kni+7LsoR4N2yB+urkXmt7XAbSCnJx6CB0B2yIwzcSPghHtAP2w8CHr2hFqOOm5lVVjfcf8tg2/+DbUdxYWCUoZRgil238/UdND0HGJR1vrb+qhizRyqN2do5Hv/nG+qL4B99PeW2GLsIHYGO46dw5qwesDCjmEN2EXsiBIPra4nA6tr0FvCQDy5kEf8D388tU9lJeUuNS6dLp9UYwXCaQXKjceZLJ0uE2eLClhs+HUQsrgSvvMIlpuLmysAym+N6vX1ljnwDUGY57/qFpgDEDC9v7+/6asuGr5zDx6B2//OV51tB3xNnAfg7Fq+Qlao0uHKBwG+JbTgTjMEpsAS2MF83IAX8AfBIAxEgTiQBNLARFhlEVznMjAVzATzQDEoBSvAGrAebAbbwC6wFxwA9aAJHAenwQVwGVwHd+Hq6QAvQTd4B/oQBCEhdISBGCJmiDXiiLghPkggEobEIAlIGpKBZCMSRIHMRBYgpcgqZD2yFalGfkYOI8eRc0gbcht5iHQib5CPKIbSUD3UBLVBR6I+KBuNRpPQCWg2OgUtQheiy9AKtArdg9ahx9EL6HW0HX2J9mAA08CYmDnmhPlgHCwOS8eyMBk2GyvByrEqrBZrhP/zVawd68I+4EScgbNwJ7iCI/FknI9PwWfjS/H1+C68Dj+JX8Uf4t34FwKdYExwJPgRuISxhGzCVEIxoZywg3CIcArupQ7COyKRyCTaEr3hXkwj5hBnEJcSNxL3EZuJbcTHxB4SiWRIciQFkOJIPFIBqZi0jrSHdIx0hdRB6iVrkM3IbuRwcjpZQp5PLifvJh8lXyE/I/dRtCnWFD9KHEVAmU5ZTtlOaaRconRQ+qg6VFtqADWJmkOdR62g1lJPUe9R32poaFho+GqM0RBrzNWo0NivcVbjocYHmi7NgcahjacpaMtoO2nNtNu0t3Q63YYeTE+nF9CX0avpJ+gP6L2aDE1nTa6mQHOOZqVmneYVzVdaFC1rLbbWRK0irXKtg1qXtLq0Kdo22hxtnvZs7Urtw9o3tXt0GDquOnE6+TpLdXbrnNN5rkvStdEN0xXoLtTdpntC9zEDY1gyOAw+YwFjO+MUo0OPqGerx9XL0SvV26vXqtetr6vvoZ+iP02/Uv+IfjsTY9owucw85nLmAeYN5sdhJsPYw4TDlgyrHXZl2HuD4QbBBkKDEoN9BtcNPhqyDMMMcw1XGtYb3jfCjRyMxhhNNdpkdMqoa7jecP/h/OElww8Mv2OMGjsYJxjPMN5mfNG4x8TUJMJEarLO5IRJlynTNNg0x7TM9KhppxnDLNBMbFZmdszsBUufxWblsSpYJ1nd5sbmkeYK863mreZ9FrYWyRbzLfZZ3LekWvpYZlmWWbZYdluZWY22mmlVY3XHmmLtYy2yXmt9xvq9ja1Nqs0im3qb57YGtlzbItsa23t2dLsguyl2VXbX7In2Pva59hvtLzugDp4OIodKh0uOqKOXo9hxo2PbCMII3xGSEVUjbjrRnNhOhU41Tg+dmc4xzvOd651fjbQamT5y5cgzI7+4eLrkuWx3ueuq6xrlOt+10fWNm4Mb363S7Zo73T3cfY57g/trD0cPoccmj1ueDM/Rnos8Wzw/e3l7ybxqvTq9rbwzvDd43/TR84n3Wepz1pfgG+I7x7fJ94Ofl1+B3wG/P/2d/HP9d/s/H2U7Sjhq+6jHARYBvICtAe2BrMCMwC2B7UHmQbygqqBHwZbBguAdwc/Y9uwc9h72qxCXEFnIoZD3HD/OLE5zKBYaEVoS2hqmG5Yctj7sQbhFeHZ4TXh3hGfEjIjmSEJkdOTKyJtcEy6fW83tjvKOmhV1MpoWnRi9PvpRjEOMLKZxNDo6avTq0fdirWMlsfVxII4btzrufrxt/JT4X8cQx8SPqRzzNME1YWbCmURG4qTE3YnvkkKSlifdTbZLViS3pGiljE+pTnmfGpq6KrV97Mixs8ZeSDNKE6c1pJPSU9J3pPeMCxu3ZlzHeM/xxeNvTLCdMG3CuYlGE/MmHpmkNYk36WAGISM1Y3fGJ14cr4rXk8nN3JDZzefw1/JfCoIFZYJOYYBwlfBZVkDWqqzn2QHZq7M7RUGiclGXmCNeL36dE5mzOed9blzuztz+vNS8ffnk/Iz8wxJdSa7k5GTTydMmt0kdpcXS9il+U9ZM6ZZFy3bIEfkEeUOBHjzUX1TYKX5QPCwMLKws7J2aMvXgNJ1pkmkXpztMXzL9WVF40U8z8Bn8GS0zzWfOm/lwFnvW1tnI7MzZLXMs5yyc0zE3Yu6uedR5ufN+m+8yf9X8vxakLmhcaLJw7sLHP0T8UFOsWSwrvrnIf9Hmxfhi8eLWJe5L1i35UiIoOV/qUlpe+mkpf+n5H11/rPixf1nWstblXss3rSCukKy4sTJo5a5VOquKVj1ePXp1XRmrrKTsrzWT1pwr9yjfvJa6VrG2vSKmomGd1boV6z6tF62/XhlSuW+D8YYlG95vFGy8sil4U+1mk82lmz9uEW+5tTVia12VTVX5NuK2wm1Pt6dsP/OTz0/VO4x2lO74vFOys31Xwq6T1d7V1buNdy+vQWsUNZ17xu+5vDd0b0OtU+3Wfcx9pfvBfsX+Fz9n/HzjQPSBloM+B2t/sf5lwyHGoZI6pG56XXe9qL69Ia2h7XDU4ZZG/8ZDvzr/urPJvKnyiP6R5UepRxce7T9WdKynWdrcdTz7+OOWSS13T4w9ce3kmJOtp6JPnT0dfvrEGfaZY2cDzjad8zt3+LzP+foLXhfqLnpePPSb52+HWr1a6y55X2q47Hu5sW1U29ErQVeOXw29evoa99qF67HX224k37h1c/zN9luCW89v591+fafwTt/dufcI90rua98vf2D8oOp3+9/3tXu1H3kY+vDio8RHdx/zH798In/yqWPhU/rT8mdmz6qfuz1v6gzvvPxi3IuOl9KXfV3Ff+j8seGV3atf/gz+82L32O6O17LX/W+WvjV8u/Mvj79aeuJ7HrzLf9f3vqTXsHfXB58PZz6mfnzWN/UT6VPFZ/vPjV+iv9zrz+/vl/JkvIGjAAYbmpUFwJudANDTAGDAMwR1nOouOCCI6v46gMB/wqr74oB4AVALO+UxntMMwH7YbOZCbtgrj/BJwQB1dx9qapFnubupuGjwJkTo7e9/awIAqRGAz7L+/r6N/f2ft8NgbwPQPEV1B1UKEd4ZtoQq0e3VE+aC70R1P/0mx+97oIzAA3zf/wt4tJFlaPiOogAAAIplWElmTU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUAAAABAAAARgEoAAMAAAABAAIAAIdpAAQAAAABAAAATgAAAAAAAACQAAAAAQAAAJAAAAABAAOShgAHAAAAEgAAAHigAgAEAAAAAQAABRagAwAEAAAAAQAAAzgAAAAAQVNDSUkAAABTY3JlZW5zaG90qFMKUgAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAAddpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3J

  This file has been truncated. show original


Different Summary Outputs Locally vs API for the Same Text

Question:
Hi Team,
Whilst using the inference API to produce summaries of call using a private model I sometimes get different outputs to when I load the model and tokeniser locally even though I’m using the exact same parameters.
To generate the summary locally I run:
device = 'cuda' if cuda.is_available() else 'cpu'
inputs = tokenizer(txt, return_tensors='pt')
summary_ids = model.generate(inputs['input_ids'].to(device), no_repeat_ngram_size = 2, max_length=75, top_k = 50, top_p=0.95, early_stopping = True)
summary_3 = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]

to generate the summary using pipeline / inference api I run:
output = query({
    "inputs": txt,
    "parameters" : {"max_length": 75,
        "no_repeat_ngram_size": 2,
         "early_stopping": True,
        "top_k": 50,
        "top_p": 0.95},
})

output[0]['summary_text']

or
summarizer = pipeline("summarization",  model="kaizan/production-bart-large-cnn-samsum", use_auth_token=TOKEN)
output = summarizer(txt, max_length=75, no_repeat_ngram_size = 2, top_k = 50, top_p = 0.95, early_stopping = True)
output[0]['summary_text']

In the pipeline / inference API case I get exactly the same output but when I run it manually I get a different output which makes me think there is a variable / seed value that might be set in the pipeline somewhere which I’m not dealing with when running this set up manually. Does anyone know what the variable causing this difference could be?
Thanks,
Karim

Context:
Hey Karim,
thanks for opening the Thread. When you said “Inference API” are you talking about the hosted inference API (Overview — Api inference documentation) or did you deploy a Model to Amazon SageMaker?


Push_to_hub() from HuggingFaceEstimator in AWS Sagemaker

Question:
Hi,
I guess there is an easy answer but I did not find it.
Use case: I did train a model in a HF Training DLC in AWS SageMaker through a HuggingFaceEstimator.fit(). Now, I would like to upload this trained model to the model hub of Hugging Face.
What is the equivalent  code to the model.push_to_hub() for a HuggingFaceEstimator in AWS Sagemaker?

Context:
Hello,
If you want to push your model to the Hugging Face Hub you can do this directly inside the train.py as shown here: notebooks/sagemaker/14_train_and_push_to_hub at master · huggingface/notebooks · GitHub
or you need to download the model.tar.gz and push it to hub afterwards Documentation. There is currently no method inside the sagemaker-sdk to push models to the hub after training.


Question:
Hello @philschmid I hope you are doing well. Question for you, do you an example of the expected format of the data in order to been able to use this script ( run_ner.py) in sagemaer training?
Thanks,
Jorge

Context:
Hey,
you can find the data format of all examples/ always inside the script. For run_ner.py it is here: transformers/run_ner.py at b518aaf193938247f698a7c4522afe42b025225a · huggingface/transformers · GitHub 6
    if data_args.text_column_name is not None:
        text_column_name = data_args.text_column_name
    elif "tokens" in column_names:
        text_column_name = "tokens"
    else:
        text_column_name = column_names[0]

    if data_args.label_column_name is not None:
        label_column_name = data_args.label_column_name
    elif f"{data_args.task_name}_tags" in column_names:
        label_column_name = f"{data_args.task_name}_tags"
    else:
        label_column_name = column_names[1]

In detail, you can either define text_column_name & label_column_name as hyperparameter to the define the column/key of your text/token and label field is. If you are not defining something it will pick index 0 for text/token and 1 for the label.
You can provide your dataset in data file formats, which are compatible with the datasets library, e.g. csv, json more to this here: Loading a Dataset — datasets 1.11.0 documentation


Question:
Hi,
I saw that there is a difference about how to get arguments between script in

Prepare a  Transformers fine-tuning script 1 that uses argparse.ArgumentParser() and parser.add_argument()

and the ones in transformers/examples/pytorch/ that uses HfArgumentParser() and parser.parse_args_into_dataclasses()

but I need some explanation.

Script in “Prepare a  Transformers fine-tuning script”
SageMaker doesn’t support argparse actions: what does it means?
The `hyperparameters` defined in the [Hugging Face Estimator](https://huggingface.co/docs/sagemaker/train#create-an-huggingface-estimator)
are passed as named arguments and processed by `ArgumentParser()` .

import transformers
import datasets
import argparse
import os

if __name__ == "__main__":

    parser = argparse.ArgumentParser()

    # hyperparameters sent by the client are passed as command-line arguments to the script
    parser.add_argument("--epochs", type=int, default=3)
    parser.add_argument("--per_device_train_batch_size", type=int, default=32)
    parser.add_argument("--model_name_or_path", type=str)

Note that SageMaker doesn’t support argparse actions.
For example, if you want to use a boolean hyperparameter,
specify type as bool in your script and provide an explicit True or False value.

Script in transformers/examples/pytorch
For example, in the script run_ner.py, the formulation is different.
from transformers import (
    (...),
    HfArgumentParser,
    (...)
)

(...)

@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
    """

    model_name_or_path: str = field(
        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
    )
    config_name: Optional[str] = field(
        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
    )

  (...)

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

Someone could explain the differences and if in SageMaker, we must rephrase the arguments section of the scripts in transformers/examples/pytorch/ as formulated in Prepare a  Transformers fine-tuning script 1 or not? Thanks.

Context:
pierreguillou:

SageMaker doesn’t support argparse actions : what does it means?

This means you cannot use parser.add_argument("--args", action="store_true")

The HfArgumentParser is a custom on top implementation on the argsparser to make it easy create python scripts for Transformers. You can use the HfArgumentParser if you want and feel confident, for example with the HfArgumentParser you don’t need to define the TrainingArguments as parser.add_argument since they are added behind the scenes.
For the SageMaker examples we went with the default argsparser since it is easier and faster to get started for non Transformers experts and it might have been difficult to understand that you don’t need to define per_device_train_batch_size in the train.py but can use it as hyperparameter in the notebook.

Someone could explain the differences and if in SageMaker, we must rephrase the arguments section of the scripts in transformers/examples/pytorch/ as formulated in Prepare a  Transformers fine-tuning script  or not?

No you don’t need to rephrase them, since the HfArgumentParser is creating add_argument behind the scenes it works with SageMaker. So you can decide how you would like to structure your script


Question:
Hi all,
Is there a timeline for when Transformers 4.6.0 will be supported in the HuggingFace SDK on SageMaker?
I’ve recently been having issues with CUDA running out of memory while training a distilBert model:

RuntimeError: CUDA out of memory. Tried to allocate 6.87 GiB (GPU 0; 15.78 GiB total capacity; 7.35 GiB already allocated; 2.79 GiB free; 11.78 GiB reserved in total by PyTorch)

It seems like this has been acknowledged and fixed in a recent commit 3 - also similarly described here 3. Looks like this has also been added to Transformers 4.6.0 and I can confirm that using this latest version (without the HuggingFace SDK) fixes the OOM issues for me.
When configuring the HuggingFace estimator, it seems like the latest supported version of Transformers is version 4.5.0

ValueError: Unsupported huggingface version: 4.6.0. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.4.2, 4.5.0, 4.4, 4.5.

Does anyone have an idea of when we can expect version 4.6.0 to be supported?
Thanks!

Context:
hey @nreamaroon,
We already opened a PR for a DLC with transformers 4.6.0. I hope we can get as merged as soon as possible.

      github.com/aws/deep-learning-containers

      [huggingface_tensorflow, huggingface_pytorch] update for Transformers to 4.6.0  10

      aws:master ← philschmid:patch-1

        opened

        May 13, 2021

          philschmid

          +8
          -8

    … 4.6.0

*Issue #, if available:*

## PR Checklist
- [x] I've prepended PR …tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
- [x] (If applicable) I've documented below the DLC image/dockerfile this relates to
- [x] (If applicable) I've documented below the tests I've run on the DLC image
- [x] (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list.  See [https://www.apache.org/legal/resolved.html](https://www.apache.org/legal/resolved.html).
- [x] (If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.

## Pytest Marker Checklist
- [ ] (If applicable) I have added the marker `@pytest.mark.model("<model-type>")` to the new tests which I have added, to specify the Deep Learning model that is used in the test (use `"N/A"` if the test doesn't use a model)
- [ ] (If applicable) I have added the marker `@pytest.mark.integration("<feature-being-tested>")` to the new tests which I have added, to specify the feature that will be tested
- [ ] (If applicable) I have added the marker `@pytest.mark.multinode(<integer-num-nodes>)` to the new tests which I have added, to specify the number of nodes used on a multi-node test
- [ ] (If applicable) I have added the marker `@pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">)` to the new tests which I have added, if a test is specifically applicable to only one processor type

#### EIA/NEURON Checklist
* When creating a PR:
- [ ] I've modified `src/config/build_config.py` in my PR branch by setting `ENABLE_EI_MODE = True` or `ENABLE_NEURON_MODE = True`
* When PR is reviewed and ready to be merged:
- [ ] I've reverted the code change on the config file mentioned above
#### Benchmark Checklist
* When creating a PR:
- [ ] I've modified `src/config/test_config.py` in my PR branch by setting `ENABLE_BENCHMARK_DEV_MODE = True`
* When PR is reviewed and ready to be merged:
- [ ] I've reverted the code change on the config file mentioned above

## Reviewer Checklist
* For reviewer, before merging, please cross-check:
- [ ] I've verified the code change on the config file mentioned above has already been reverted

*Description:*

This PRs updates the `transformers` version inside the `huggingface` PyTorch and TensorFlow DLC.

*Tests run:*

https://github.com/huggingface/transformers/tree/master/tests/sagemaker

| ID                                  | Description                                                       | Platform                   | #GPUS | Collected & evaluated metrics            |
|-------------------------------------|-------------------------------------------------------------------|-----------------------------|-------|------------------------------------------|
| pytorch-transfromers-test-single    | test bert finetuning using BERT fromtransformerlib+PT             | SageMaker createTrainingJob | 1     | train_runtime, eval_accuracy & eval_loss |
| pytorch-transfromers-test-2-ddp     | test bert finetuning using BERT from transformer lib+ PT DPP      | SageMaker createTrainingJob | 16    | train_runtime, eval_accuracy & eval_loss |
| pytorch-transfromers-test-2-smd     | test bert finetuning using BERT from transformer lib+ PT SM DDP   | SageMaker createTrainingJob | 16    | train_runtime, eval_accuracy & eval_loss |
| pytorch-transfromers-test-1-smp     | test roberta finetuning using BERT from transformer lib+ PT SM MP | SageMaker createTrainingJob | 8     | train_runtime, eval_accuracy & eval_loss |
| tensorflow-transfromers-test-single | Test bert finetuning using BERT from transformer lib+TF           | SageMaker createTrainingJob | 1     | train_runtime, eval_accuracy & eval_loss |
| tensorflow-transfromers-test-2-smd  | test bert finetuning using BERT from transformer lib+ TF SM DDP   | SageMaker createTrainingJob | 16    | train_runtime, eval_accuracy & eval_loss |

*DLC image/dockerfile:*

The Hugging Face DLCs for PyTorch and TensorFlow

*Additional context:*
![Bildschirmfoto 2021-04-06 um 14 20 34](https://user-images.githubusercontent.com/32632186/114026575-afec6480-9876-11eb-892e-55144707367d.png)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Cuda memory error on unchanged workshop 1 notebooks

Question:
I am running notebooks 1 and 3 unchanged from huggingface-sagemaker-workshop-series/workshop_1_getting_started_with_amazon_sagemaker at main · philschmid/huggingface-sagemaker-workshop-series · GitHub 2
And I get the following error:

RuntimeError: CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 15.78 GiB total capacity; 14.80 GiB already allocated; 44.75 MiB free; 14.83 GiB reserved in total by PyTorch)

I am trying with different batch sizes and learning rates, but can someone help me understand why not everyone got the same error if we’re all using the same AWS resources?

Context:
Hello @kjackson,
I run the notebook now twice as it on “main” and never got any error.
2021-12-01 08:46:23 Uploading - Uploading generated training model
2021-12-01 08:48:23 Completed - Training job completed
ProfilerReport-1638347816: NoIssuesFound
Training seconds: 490
Billable seconds: 490


Question:
after 0.07 epochs the training job stops and gives the following error :-
[INFO|trainer.py:1885] 2021-11-19 18:43:48,502 >> Saving model checkpoint to /opt/ml/model/checkpoint-3500
[INFO|configuration_utils.py:351] 2021-11-19 18:43:48,503 >> Configuration saved in /opt/ml/model/checkpoint-3500/config.json
#015Downloading:   0%|          | 0.00/7.78k [00:00<?, ?B/s]#015Downloading: 28.8kB [00:00, 16.0MB/s]
#015Downloading:   0%|          | 0.00/4.47k [00:00<?, ?B/s]#015Downloading: 28.7kB [00:00, 17.6MB/s]

2021-11-19 18:44:10 Uploading - Uploading generated training model
2021-11-19 18:44:10 Failed - Training job failed
ProfilerReport-1637341687: Stopping
---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<ipython-input-13-6a9d8eb3a402> in <module>
     33
     34 # starting the train job
---> 35 huggingface_estimator.fit()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    690         self.jobs.append(self.latest_training_job)
    691         if wait:
--> 692             self.latest_training_job.wait(logs=logs)
    693
    694     def _compilation_job_name(self):

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
   1650         # If logs are requested, call logs_for_jobs.
   1651         if logs != "None":
-> 1652             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   1653         else:
   1654             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3776
   3777         if wait:
-> 3778             self._check_job_status(job_name, description, "TrainingJobStatus")
   3779             if dot:
   3780                 print()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3333                 ),
   3334                 allowed_statuses=["Completed", "Stopped"],
-> 3335                 actual_status=status,
   3336             )
   3337

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-11-19-17-08-07-355: Failed. Reason: ClientError: Artifact upload failed:Error 5: Received a failed archive status.

Thank you for your help.

Context: You could use checkpointing with sagemaker 3, which automatically syncs the checkpoints to S3 when they are created and only save the model without checkpoints at the end to /opt/ml/model


Some issues when training model on Sagemaker

Question:
Hello world,
I’m getting two issues when I fine-tuning on my model using this sagemaker notebook.

No GUI login prompt out when running notebook_login(), instead I’m getting this:

error2991×230 16.6 KB

As a workaround, I’m using hardcoded token

Hit ResourceLimitExceeded Error when running huggingface_estimater.fit(…):

error984×607 36.8 KB

For item 2, 1. I have opened an issue on AWS support to request for increasing the limit but I will expect a slow reply from them. Is there any other way to get around this while getting GPU boost from Sagemaker?
FYI, I’m using my own AWS account (Free Tier account but having some credits).
Thanks.

Context:
Hello @ivanlau,
thanks for opening the thread.
To 1. where are you running the sagemaker notebook?
To 2. I think you can go with the ml.g4dn.xlarge it also has 1 GPU and shouldn’t need a limit increase for that.


Question:
I fine-tuned BERT for text-classification on a custom dataset using HuggingFace and Tensorflow, and now I’m trying to deploy the model for inference through SageMaker. I followed this HuggingFace tutorial 5 but I get the following error. I spent a while looking through the SageMaker HuggingFace documentation 3 to no avail. The error says that model_uri is set to None, but model_uri is not a parameter that I can pass, and I just want it to pull my model from the HuggingFace Hub.
I also tried downloading the model from the Hub, zipping it, uploading it to S3, and passing model_data=“model.tar.gz”, but that didn’t work either.

image1885×856 134 KB

Any help would be greatly appreciated!

Context: Resolved: I just needed to add an image_uri!


Returning Multiple Answers for a QA Model on SageMaker

Question:
Hi,
I’ve currently fine-tuned this 1 Question-Answering model to fit a specific business use case we have (identifying the name of a company from a piece of text). When it comes to inference, I’ve found as @sgugger has very clearly explained in this notebook 1 that sometimes the best answer isn’t the one with the best start and end logits as sometimes the highest scoring combination can produce an answer that is too long or too short (just one character).
As such when I was predicting using this model locally I created a return_best_combinaton function that finds the most practical answer using the list of logit scores.
When I used this model using the SageMaker API I realised it just returns one single answer with a score assigned to it. I wanted to check how this answer is produced? (happy to just be directed to the source code if it’s available) and wether it’s possible to return n number of likely answers instead of just 1.
Thanks every so much,
Karim

Context:
The Inference Toolkit uses the transformers pipelines under the hood. So if your are deploying a model for Question-Answering it would use the pipeline("question-answering"). You can find the code for this here: transformers/question_answering.py at master · huggingface/transformers · GitHub 3
But if you want to use your own function return_best_combinaton you could create a custom inference.py with your own “prediction” step.

      huggingface.co

Deploy models to Amazon SageMaker 3


Error while finding module specification for ‘run_glue.py’

Question:
Code written:-
!pip install git+https://github.com/huggingface/transformers
!python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I hate you'))"
!git clone https://github.com/huggingface/transformers.git
%cd "transformers/examples/pytorch/text-classification/"

!python3 -m run_glue.py --model_name_or_path "microsoft/deberta-v3-large" --task_name "mnli" --do_train --do_eval --evaluation_strategy steps --max_seq_length 256 --warmup_steps 50 --per_device_train_batch_size 8 --learning_rate 6e-6 --num_train_epochs 2  --output_dir "ds_results" --overwrite_output_dir --logging_steps 1000 --logging_dir "ds_results"

These commands were primarily taken from (and slightly modified) - microsoft/deberta-v3-large · Hugging Face
Error:-
/home/ec2-user/anaconda3/envs/pytorch_p36/bin/python: Error while finding module specification for 'run_glue.py' (AttributeError: module 'run_glue' has no attribute '__path__')

I have tried this on my own system it didn’t give this error. Am I missing something?  @lewtun

Context: I think the problem is that you’re using the -m flag which tries to import run_glue.py as a module and then run it. Does it work if you remove the -m flag?


Question:
Hey!
Been experiencing this error and have tried diagnosing but not sure where the problem might be.
UnexpectedStatusException: Error hosting endpoint summarization-endpoint: Failed. Reason:  Please make sure all images included in the model for the production variant AllTraffic exist, and that the execution role used to create the model has permissions to access them..

I initially thought it had to do with IAM Permissions but I am no longer sure that is the case since I added all permissions that might be relevant and I don’t think it’s an issue of the resource not being assigned to the right policy. The model is being created but the endpoint is not being processed correctly. I also considered whether my model.tar.gz was corrupted but even when I tried uploading a model directly from the Hugging Face Hub, I am met with this error message. For some reason as well, no CloudWatch logs are being saved despite the CloudWatch Log Group being created for /aws/sagemaker/Endpoints/summarization-endpoint and having all relevant permissions.
The script is below:
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import BytesDeserializer
import sagemaker

model_name = 'summarization-model'
endpoint_name = 'summarization-endpoint'

role = sagemaker.get_execution_role()

# Hub Model configuration. https://huggingface.co/models
# hub = {
# 	'HF_MODEL_ID':'google/pegasus-large',
# 	'HF_TASK':'summarization'
# }

# # create Hugging Face Model Class
# huggingface_model = HuggingFaceModel(
# 	transformers_version='4.6.1',
# 	pytorch_version='1.7.1',
# 	py_version='py36',
# 	env=hub,
# 	role=role,
# )

# # create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://qfn-transcription/ujjawal_files/model.tar.gz",  # path to your trained sagemaker model
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6.1", # transformers version used
   pytorch_version="1.7.1", # pytorch version used
   py_version='py36',
   name=model_name
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.xlarge',#'ml.m5.xlarge',ml.inf1.xlarge
    endpoint_name=endpoint_name,
)

predictor.predict({
	'inputs': "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
})

Thanks!
IAM Permissions:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "iam:GetRole",
                "iam:PassRole",
                "sagemaker:GetRecord"
            ],
            "Resource": [
                "arn:aws:sagemaker:*:216283767174:feature-group/*",
                "arn:aws:s3:::qfn-transcription/*",
                "arn:aws:iam::216283767174:role/callTranscriptionsRole"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "sagemaker:CreateModel",
                "logs:GetLogRecord",
                "logs:DescribeSubscriptionFilters",
                "logs:StartQuery",
                "logs:DescribeMetricFilters",
                "ecr:BatchDeleteImage",
                "logs:ListLogDeliveries",
                "ecr:DeleteRepository",
                "logs:CreateLogStream",
                "logs:TagLogGroup",
                "logs:CancelExportTask",
                "logs:GetLogEvents",
                "logs:FilterLogEvents",
                "logs:DescribeDestinations",
                "sagemaker:CreateEndpoint",
                "logs:StopQuery",
                "cloudwatch:GetMetricStatistics",
                "logs:CreateLogGroup",
                "ecr:PutImage",
                "logs:PutMetricFilter",
                "logs:CreateLogDelivery",
                "servicecatalog:ListAcceptedPortfolioShares",
                "sagemaker:CreateEndpointConfig",
                "logs:PutResourcePolicy",
                "logs:DescribeExportTasks",
                "sagemaker:ListActions",
                "logs:GetQueryResults",
                "sagemaker:DescribeEndpointConfig",
                "logs:UpdateLogDelivery",
                "ecr:BatchGetImage",
                "logs:PutSubscriptionFilter",
                "ecr:InitiateLayerUpload",
                "logs:ListTagsLogGroup",
                "sagemaker:EnableSagemakerServicecatalogPortfolio",
                "logs:DescribeLogStreams",
                "ecr:UploadLayerPart",
                "logs:GetLogDelivery",
                "cloudwatch:ListMetrics",
                "servicecatalog:AcceptPortfolioShare",
                "logs:CreateExportTask",
                "ecr:CompleteLayerUpload",
                "logs:AssociateKmsKey",
                "sagemaker:DescribeEndpoint",
                "logs:DescribeQueryDefinitions",
                "logs:PutDestination",
                "logs:DescribeResourcePolicies",
                "ecr:DeleteRepositoryPolicy",
                "logs:DescribeQueries",
                "logs:DisassociateKmsKey",
                "sagemaker:DeleteApp",
                "logs:UntagLogGroup",
                "logs:DescribeLogGroups",
                "logs:PutDestinationPolicy",
                "logs:TestMetricFilter",
                "logs:PutQueryDefinition",
                "logs:DeleteDestination",
                "logs:PutLogEvents",
                "s3:ListAllMyBuckets",
                "ecr:SetRepositoryPolicy",
                "logs:PutRetentionPolicy",
                "logs:GetLogGroupFields"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "sagemaker:CreateApp"
            ],
            "Resource": [
                "arn:aws:sagemaker:*:216283767174:app/*/*/*/*",
                "arn:aws:s3:::qfn-transcription/*"
            ]
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": "sagemaker:DescribeApp",
            "Resource": "arn:aws:sagemaker:*:216283767174:app/*/*/*/*"
        },
        {
            "Sid": "VisualEditor4",
            "Effect": "Allow",
            "Action": [
                "sagemaker:DescribeTrainingJob",
                "sagemaker:CreateMonitoringSchedule",
                "sagemaker:PutRecord",
                "sagemaker:CreateTrainingJob",
                "sagemaker:CreateProcessingJob"
            ],
            "Resource": [
                "arn:aws:sagemaker:*:216283767174:feature-group/*",
                "arn:aws:sagemaker:*:216283767174:monitoring-schedule/*",
                "arn:aws:sagemaker:*:216283767174:processing-job/*",
                "arn:aws:sagemaker:*:216283767174:training-job/*"
            ]
        },
        {
            "Sid": "VisualEditor5",
            "Effect": "Allow",
            "Action": [
                "sagemaker:DescribeNotebookInstanceLifecycleConfig",
                "sagemaker:StopNotebookInstance",
                "sagemaker:DescribeNotebookInstance"
            ],
            "Resource": [
                "arn:aws:sagemaker:*:216283767174:feature-group/*",
                "arn:aws:sagemaker:*:216283767174:notebook-instance-lifecycle-config/*",
                "arn:aws:sagemaker:*:216283767174:notebook-instance/*"
            ]
        },
        {
            "Sid": "VisualEditor6",
            "Effect": "Allow",
            "Action": [
                "ecr:SetRepositoryPolicy",
                "ecr:CompleteLayerUpload",
                "ecr:BatchGetImage",
                "ecr:BatchDeleteImage",
                "ecr:UploadLayerPart",
                "ecr:DeleteRepositoryPolicy",
                "ecr:InitiateLayerUpload",
                "ecr:DeleteRepository",
                "ecr:PutImage"
            ],
            "Resource": "arn:aws:ecr:*:*:repository/*"
        }
    ]
}```

Context:
Hey @ujjirox,
The error is 100% related to your IAM Permissions. See:

amazon web services - 'all images for the production variant AllTraffic exist, the execution role used to create the model has permissions to access them' - Stack Overflow 18
https://github.com/aws/sagemaker-python-sdk/issues/1835 8
https://github.com/aws/sagemaker-python-sdk/discussions/2365 9

I saw that you edited you IAM permissions manually (VisualEditorX), e.g I miss ecr:GetDownloadUrlForLayer for downloading the ecr correctly.
Can you test the deployment with the AmazonSageMakerFullAccess see here: SageMaker Roles - Amazon SageMaker 9

The IAM managed policy, AmazonSageMakerFullAccess , used in the following procedure only grants the execution role permission to perform certain Amazon S3 actions on buckets or objects with SageMaker , Sagemaker , sagemaker , or aws-glue in the name. To learn how to add an additional policy to an execution role to grant it access to other Amazon S3 buckets and objects, see Add Additional Amazon S3 Permissions to an SageMaker Execution Role 5.

If this works you can take a look at more detailed permissions here: SageMaker Roles - Amazon SageMaker 9
And then create a new clean role.


Truncation of input data for Summarization pipeline

Question:
I’m using bert-large-cnn for a summarization task. I have been truncating the input text in order to avoid exceeding the maximum sequence length:
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
model = BartForConditionalGeneration.from_pretrained(
    "facebook/bart-large-cnn"
)

inputs = tokenizer(
    input_text, return_tensors="pt", max_length=1024, truncation=True
)

outputs = model.generate(
    inputs["input_ids"],
    max_length=300,
    min_length=100,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=True,
)

summary = " ".join(
    tokenizer.decode(
        g, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )
    for g in outputs
)

How can I replicate this in SageMaker? I don’t see a way to pass configuration values to the tokenizer when calling predict on an instance of HuggingFaceModel? Here is my code so far:
env = {
    "HF_MODEL_ID": "facebook/bart-large-cnn",
    "HF_TASK": "summarization",
}

huggingface_model = HuggingFaceModel(
    env=env,
    role=role,
    transformers_version="4.6",
    pytorch_version="1.7",
    py_version="py36",
)

data = {
    "inputs": input_text,
    "parameters": {
        "max_length": 300,
        "min_length": 100,
        "length_penalty": 2.0,
        "num_beams": 4,
    }
}

result = predictor.predict(data)

Thank you!

Context:
I could reproduce the issue and also found the root cause of it. The issue that
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

And there is currently no way to pass in the max_length to the inference toolkit.
There are now 2 options to solve this you could either for the model into your own repository and add a tokenizer_config.json similar to this one tokenizer_config.json · distilbert-base-uncased-finetuned-sst-2-english at main 2.
or you could provide a custom inference.py as entry_point when creating the HuggingFaceModel.
e.g.
huggingface_model = HuggingFaceModel(
    env=env,
    role=role,
    transformers_version="4.6",
    pytorch_version="1.7",
    py_version="py36",
    entry_point="inference.py",
)

The inference.py then need to contain predict_fn and a model_fn. Pseudo code below.
def model_fn(model_dir):
    """"model_dir is the location where the model is stored"""
    tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
    model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
    return model, tokenizer

def predict_fn(data, model):
    """model is the return of the model_fn and data is the json from the request as python dict"""
   model,tokenzier = model
    outputs = model.generate(
    inputs["input_ids"],
    max_length=300,
    min_length=100,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=True,
    )

   summary = " ".join(
    tokenizer.decode(
        g, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )
    for g in outputs
    )
    return {"summary":summary}

You can find more documentation here: Deploy models to Amazon SageMaker


How to deploy a huggingface model from S3 outside a Jupyter Notebook

Question:
I’ve successfully deployed my model from S3 in a jupyter notebook. Unfortunately my organization requires that all production AWS apps need to be 100% terraform. The guide here says “you can also instantiate Hugging Face endpoints with lower-level SDK such as boto3 and AWS CLI , Terraform and with CloudFormation templates.”
So I’m fairly sure that it’s possible to do this, but I can’t find any documentation anywhere regarding terraform deployment.

Context:
Hey @sdegrace,
We have an example for AWS CDK already: cdk-samples/sagemaker-endpoint-huggingface at master · philschmid/cdk-samples · GitHub 4, CDK is similar to Terraform.
But if your company wants to use terraform you can easily create it.
For a successful deployment of a SageMaker endpoint you need:

a SageMaker model: Terraform documentation 2

a Endpoint Configuration: Terraform documentation

a SageMaker Endpoint: Terraform documentation

below you can find “pseudo” code of how this is going to look like:
resource "aws_sagemaker_model" "huggingface" {
  name               = "bert"
  execution_role_arn = "arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001"

  primary_container {
    # CPU Image
    image="763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04"
# GPU Image  image = "763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:1.9.1-transformers4.12.3-gpu-py38-cu111-ubuntu20.04"
   model_data_url="s3://your-model"
  }
}

resource "aws_sagemaker_endpoint_configuration" "huggingface" {
  name = "bert"

  production_variants {
    variant_name           = "variant-1"
    model_name             = aws_sagemaker_model.huggingface.name
    initial_instance_count = 1
    instance_type          = "ml.t2.medium"
  }
}

resource "aws_sagemaker_endpoint" "huggingface" {
  name                 = "bert"
  endpoint_config_name = aws_sagemaker_endpoint_configuration.huggingface.name
}


Question:
Hi!
I am trying to setup a lambda and a batch transform.  These are the steps that was executed in Lambda:

HuggingFaceModel()
HuggingFaceModel.transformer()
HuggingFaceModel.transfomer.transform()

The issue is that step #3 waits for a response return from sagemaker’s batch transform job.  This takes several minutes which is billed for lambda.  For other batch transform jobs using boto3.sagemaker.create_transform_job, the status is returned immediately after a successful send.  Does anyone have any experience with the batch transform on lambda and having it returned immediately without waiting for a response?  Thanks!

Context:
Hello @vng510,
you can still use the sagemaker classes in your lambda functions.

 vng510:

HuggingFaceModel()
HuggingFaceModel.transformer()
HuggingFaceModel.transfomer.transform()

HuggingFaceModel.transformer.transform() accepts a parameter called wait which defines wether the call should wait until the job completes.
So when you add HuggingFaceModel.transformer.transform(wait=False) it will create the Batch Job but doesn’t wait for anything.
https://sagemaker.readthedocs.io/en/stable/api/inference/transformer.html#sagemaker.transformer.Transformer.transform 1


Training on Sagemaker with Trainer() Instance

Question:
Hello everyone,
I wanted to train a NLP classifer on our server but it takes around 9 hours for a training. So I wanted to switch the training process to Sagemaker. When I just copy my code with the Trainer() instance (trainer.train()) I get the following error:
ImportError: torch>=1.5.0 is required for a normal functioning of this module, but found torch==1.4.0.
And it looks like I cannot update my torch on Sagemaker to 1.5.0.
On my research I found out that many trainings are done with the HugginFace estimator. Do I always have to use this and change my “local server code”?

Context:
How did you start your training? Which service di you use?
SageMaker has different options there are Notebook instances which are just hoster Jupyter Services and then there is also the Training Platform, which uses the HuggingFace estimator as shown in all examples.


How to fix “ValueError: Need either a GLUE task or a training/validation file.”

Question:
I’m not sure what is causing this error during training. I’m using the GLUE task, and I think I have the correct transformers version, but I keep getting this error when I try to train the model: “ValueError: Need either a GLUE task or a training/validation file.”
Here’s the notebook: aws-sagemaker-deploy-sentiment-analysis/huggingface_pl_sentiment_analysis.ipynb at main · JayThibs/aws-sagemaker-deploy-sentiment-analysis · GitHub 1

Context:
You are not passing your datasets to your training job when starting it using the sagemaker SDK.
Currently, you have
huggingface_estimator.fit() # put this inside fit if using train.py: {'train': training_input_path, 'test': test_input_path}

=> Saying to sagemaker you are not passing data from s3.
You need to change this to
huggingface_estimator.fit({'train': training_input_path, 'test': test_input_path})


Question:
Ive been working closely with AWS to solve this issue. They told me to post here. Ive been trying to get multi instance working with AWS Sagemaker x Hugging Face estimators. My code works okay for single instance non distributed training and single instance distributed training. It does not work for multi instance distributed training. I am using the huggingface-pytorch-training:1.7-transformers4.6-gpu-py36-cu110-ubuntu18.04 image. The image is in our internal ECR because we run in a VPC.
Here is the code I am using. Its calling the same train.py from this repo (SageMaker-HuggingFace-Workshop/train.py at main · C24IO/SageMaker-HuggingFace-Workshop · GitHub 1). I get a FileNotFoundError error after training when the script is trying to load the model. I must be forgetting to set the correct path somewhere.
import sagemaker
import time
from sagemaker.huggingface import HuggingFace
import logging
import os
from sagemaker.s3 import S3Uploader

role = 'ROLE'
default_bucket = 'BUCKET_NAME'

local_train_dataset = "amazon_us_reviews_apparel_v1_00_train.json"
local_test_dataset = "amazon_us_reviews_apparel_v1_00_test.json"

# s3 uris for datasets
remote_train_dataset = f"s3://{default_bucket}/"
remote_test_dataset = f"s3://{default_bucket}/"

# upload datasets
S3Uploader.upload(local_train_dataset,remote_train_dataset)
S3Uploader.upload(local_test_dataset,remote_test_dataset)

print(f"train dataset uploaded to: {remote_train_dataset}/{local_train_dataset}")
print(f"test dataset uploaded to: {remote_test_dataset}/{local_test_dataset}")

# hyperparameters, which are passed into the training job
hyperparameters={'epochs': 1,                          # number of training epochs
                 'train_batch_size': 32,               # batch size for training
                 'eval_batch_size': 64,                # batch size for evaluation
                 'learning_rate': 3e-5,                # learning rate used during training
                 'model_id':'distilbert-base-uncased', # pre-trained model
                 'fp16': True,                         # Whether to use 16-bit (mixed) precision training
                 'train_file': local_train_dataset,    # training dataset
                 'test_file': local_test_dataset,      # test dataset
                 }

metric_definitions=[
    {'Name': 'eval_loss',               'Regex': "'eval_loss': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_accuracy',           'Regex': "'eval_accuracy': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_f1',                 'Regex': "'eval_f1': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_precision',          'Regex': "'eval_precision': ([0-9]+(.|e\-)[0-9]+),?"}]

# define Training Job Name
job_name = f'huggingface-workshop-{time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())}'

discovery_bucket_kms = 'KMS'
subnets = ['subnet-xxx']
security_group_ids = ['sg-xxx','sg-xxz','sg-xxy','sg-xx10']
distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}

logging.debug('Creating the Estimator')

# create the Estimator
huggingface_estimator = HuggingFace(
    entry_point          = 'train.py',
    source_dir           = 'scripts',
    instance_type        = 'ml.p3.16xlarge',
    instance_count       = 2,
    base_job_name        = job_name,
    role                 = role,
    transformers_version = '4.6',
    pytorch_version      = '1.7',
    py_version           = 'py36',
    hyperparameters      = hyperparameters,
    metric_definitions   = metric_definitions,
    sagemaker_session=sess,
    distribution = distribution,

    # SECURITY CONFIGS
    output_kms_key = discovery_bucket_kms,
    subnets = subnets,
    security_group_ids = security_group_ids,
    enable_network_isolation = True,
    encrypt_inter_container_traffic = True,

    image_uri = 'INTERNAL_ECR_URI'
    )

# define a data input dictonary with our uploaded s3 uris
training_data = {
    'train': remote_train_dataset,
    'test': remote_test_dataset
}
logging.debug('Running Fit')
huggingface_estimator.fit(training_data)

Context:
I figured out my issues. The error I was getting was from loading the best model after training using the flag load_best_model_at_end=True. The model is not saved on every node when doing multi instance training, so the FileNoteFound error occurs. This flag will cause issues in the trainer if youre using transformers <= 4.6.
This issue is solved when using version of the AWS Deep Learning containers with a version of Hugging Face greater than 4.6 such as 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-training:1.9.0-transformers4.11.0-gpu-py38-cu111-ubuntu20.04, or by setting load_best_model_at_end=False
Thanks for the help!


Hitting Deployed Endpoint *Outside* of Notebook

Question:
All the tutorials tend to end at:
predictor.predict({"input": "YOUR_TEXT_GOES_HERE"})

It’s great that the notebooks deliver you to inference, but I have no idea how to hit this endpoint outside of the context of a Jupyter Notebook. I basically have Amazon AWS Java sdk code that does this:
AmazonSageMakerRuntime runtime = AmazonSageMakerRuntimeClientBuilder.defaultClient();

String body = "{\"instances\": [{\"data\": { \"input\": \"Hello World\"}}]}";

ByteBuffer bodyBuffer = ByteBuffer.wrap(body.getBytes());

InvokeEndpointRequest request = new InvokeEndpointRequest()
        .withEndpointName("huggingface-pytorch-training-....")
        .withBody(bodyBuffer);

InvokeEndpointResult invokeEndpointResult = runtime.invokeEndpoint(request);

Unfortunately, I get an error:
{
 "code": 400,
  "type": "InternalServerException",
  "message": "Content type  is not supported by this framework.\n\n            Please implement input_fn to to deserialize the request data or an output_fn to\n            serialize the response. For more information, see the SageMaker Python SDK README."
}

Am I missing something?

Context:
Hey @rosenjcb,
Thank you for opening this thread. Yes you can use the endpoint with the aws sdk for this you can use the InvokeEndpoint method. Java doc
It looks like you are already doing this and there are only a few missing parts i guess.
The Endpoint expects a JSON as HTTP Body and as the error says you are missing the Content-Type: application/json for that.
I have to say i have no JAVA experience at all but i found this on StackOverflow:
InvokeEndpointRequest invokeEndpointRequest = new InvokeEndpointRequest();
invokeEndpointRequest.setContentType("application/x-image");
ByteBuffer buf = ByteBuffer.wrap(image);

invokeEndpointRequest.setBody(buf);
invokeEndpointRequest.setEndpointName(endpointName);
invokeEndpointRequest.setAccept("application/json");

AmazonSageMakerRuntime amazonSageMaker = AmazonSageMakerRuntimeClientBuilder.defaultClient();
InvokeEndpointResult invokeEndpointResult = amazonSageMaker.invokeEndpoint(invokeEndpointRequest)

maybe this helps you crafting your request.
You can also find a example on using the aws sdk with python below
        response = client.invoke_endpoint(
            EndpointName=ENDPOINT_NAME,
            ContentType="application/json",
            Accept="application/json",
            Body=JSON_STRING,
        )


Sagemaker downloads huggingface model image every time on running fit

Question: Sagemaker downloads huggingface model image every time on running huggingface_estimator.fit()

Context: In a local notebook, you could always save the model locally and refer to it using a file path (e.g. /Users/sanjanag/models/bert-base). But since this is a job running on ECS, I don’t think you get that choice. It runs from scratch every time.