Question:
import sagemaker
from sagemaker.huggingface import HuggingFace

# gets role for executing training job
role = sagemaker.get_execution_role()
hyperparameters = {
    'epochs': 1,
    'train_batch_size': 128,
	'model_name_or_path':'EleutherAI/gpt-j-6B',
	'output_dir':'/opt/ml/model'
	# add your remaining hyperparameters
	# more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/pytorch/language-modeling
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}

# creates Hugging Face estimator
huggingface_estimator = HuggingFace(
	entry_point='run_clm.py',
	source_dir='./examples/pytorch/language-modeling',
	instance_type='ml.p3.2xlarge',
	instance_count=1,
	role=role,
	git_config=git_config,
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	hyperparameters = hyperparameters
)

# starting the train job
huggingface_estimator.fit({'training': 's3://domain-gen-data/domain-gen-training.jsonl'})

above is the code and below is the error
2021-08-31 07:31:41 Starting - Starting the training job...
2021-08-31 07:32:07 Starting - Launching requested ML instancesProfilerReport-1630395096: InProgress
......
2021-08-31 07:33:08 Starting - Preparing the instances for training......
2021-08-31 07:34:08 Downloading - Downloading input data...
2021-08-31 07:34:28 Training - Downloading the training image..................
2021-08-31 07:37:33 Training - Training image download completed. Training in progress.bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2021-08-31 07:37:34,584 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2021-08-31 07:37:34,615 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2021-08-31 07:37:36,036 sagemaker_pytorch_container.training INFO     Invoking user training script.
2021-08-31 07:37:36,481 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:
/opt/conda/bin/python3.6 -m pip install -r requirements.txt
Requirement already satisfied: datasets>=1.1.3 in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.6.2)
Requirement already satisfied: sentencepiece!=0.1.92 in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (0.1.91)
Requirement already satisfied: protobuf in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.17.1)
Requirement already satisfied: multiprocess in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.70.11.1)
Requirement already satisfied: pyarrow>=1.0.0<4.0.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.0.0)
Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: xxhash in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2.0.2)
Requirement already satisfied: tqdm<4.50.0,>=4.27 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.49.0)
Requirement already satisfied: huggingface-hub<0.1.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.0.8)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (20.9)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.0.1)
Requirement already satisfied: fsspec in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2021.5.0)
Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2.25.1)
Requirement already satisfied: dataclasses in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.8)
Requirement already satisfied: pandas in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (1.1.5)
Requirement already satisfied: dill in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: filelock in /opt/conda/lib/python3.6/site-packages (from huggingface-hub<0.1.0->datasets>=1.1.3->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (2020.12.5)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (1.25.11)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: six>=1.9 in /opt/conda/lib/python3.6/site-packages (from protobuf->-r requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata->datasets>=1.1.3->-r requirements.txt (line 1)) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata->datasets>=1.1.3->-r requirements.txt (line 1)) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->datasets>=1.1.3->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from pandas->datasets>=1.1.3->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.6/site-packages (from pandas->datasets>=1.1.3->-r requirements.txt (line 1)) (2021.1)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

2021-08-31 07:37:39,041 sagemaker-training-toolkit INFO     Invoking user script

Training Env:

{
    "additional_framework_parameters": {},
    "channel_input_dirs": {
        "training": "/opt/ml/input/data/training"
    },
    "current_host": "algo-1",
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1"
    ],
    "hyperparameters": {
        "train_batch_size": 128,
        "output_dir": "/opt/ml/model",
        "epochs": 1,
        "model_name_or_path": "EleutherAI/gpt-j-6B"
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {
        "training": {
            "TrainingInputMode": "File",
            "S3DistributionType": "FullyReplicated",
            "RecordWrapperType": "None"
        }
    },
    "input_dir": "/opt/ml/input",
    "is_master": true,
    "job_name": "huggingface-pytorch-training-2021-08-31-07-31-36-059",
    "log_level": 20,
    "master_hostname": "algo-1",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-us-east-1-765248384165/huggingface-pytorch-training-2021-08-31-07-31-36-059/source/sourcedir.tar.gz",
    "module_name": "run_clm",
    "network_interface_name": "eth0",
    "num_cpus": 8,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-1",
        "hosts": [
            "algo-1"
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "run_clm.py"
}

Environment variables:

SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={"epochs":1,"model_name_or_path":"EleutherAI/gpt-j-6B","output_dir":"/opt/ml/model","train_batch_size":128}
SM_USER_ENTRY_POINT=run_clm.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["training"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=run_clm
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-east-1-765248384165/huggingface-pytorch-training-2021-08-31-07-31-36-059/source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{"epochs":1,"model_name_or_path":"EleutherAI/gpt-j-6B","output_dir":"/opt/ml/model","train_batch_size":128},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"huggingface-pytorch-training-2021-08-31-07-31-36-059","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-east-1-765248384165/huggingface-pytorch-training-2021-08-31-07-31-36-059/source/sourcedir.tar.gz","module_name":"run_clm","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"run_clm.py"}
SM_USER_ARGS=["--epochs","1","--model_name_or_path","EleutherAI/gpt-j-6B","--output_dir","/opt/ml/model","--train_batch_size","128"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TRAINING=/opt/ml/input/data/training
SM_HP_TRAIN_BATCH_SIZE=128
SM_HP_OUTPUT_DIR=/opt/ml/model
SM_HP_EPOCHS=1
SM_HP_MODEL_NAME_OR_PATH=EleutherAI/gpt-j-6B
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages

Invoking script with the following command:

/opt/conda/bin/python3.6 run_clm.py --epochs 1 --model_name_or_path EleutherAI/gpt-j-6B --output_dir /opt/ml/model --train_batch_size 128

Traceback (most recent call last):
  File "run_clm.py", line 468, in <module>
    main()
  File "run_clm.py", line 182, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.6/site-packages/transformers/hf_argparser.py", line 187, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 12, in __init__
  File "run_clm.py", line 161, in __post_init__
    raise ValueError("Need either a dataset name or a training/validation file.")
ValueError: Need either a dataset name or a training/validation file.

2021-08-31 07:37:44,472 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 run_clm.py --epochs 1 --model_name_or_path EleutherAI/gpt-j-6B --output_dir /opt/ml/model --train_batch_size 128"
Traceback (most recent call last):
  File "run_clm.py", line 468, in <module>
    main()
  File "run_clm.py", line 182, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.6/site-packages/transformers/hf_argparser.py", line 187, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 12, in __init__
  File "run_clm.py", line 161, in __post_init__
    raise ValueError("Need either a dataset name or a training/validation file.")
ValueError: Need either a dataset name or a training/validation file.

2021-08-31 07:37:49 Uploading - Uploading generated training model
2021-08-31 07:37:49 Failed - Training job failed
---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<ipython-input-5-33c7f1decb60> in <module>
     31
     32 # starting the train job
---> 33 huggingface_estimator.fit({'training': 's3://domain-gen-data/domain-gen-training.jsonl'})

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    680         self.jobs.append(self.latest_training_job)
    681         if wait:
--> 682             self.latest_training_job.wait(logs=logs)
    683
    684     def _compilation_job_name(self):

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
   1625         # If logs are requested, call logs_for_jobs.
   1626         if logs != "None":
-> 1627             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   1628         else:
   1629             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3731
   3732         if wait:
-> 3733             self._check_job_status(job_name, description, "TrainingJobStatus")
   3734             if dot:
   3735                 print()

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3291                 ),
   3292                 allowed_statuses=["Completed", "Stopped"],
-> 3293                 actual_status=status,
   3294             )
   3295

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-08-31-07-31-36-059: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 run_clm.py --epochs 1 --model_name_or_path EleutherAI/gpt-j-6B --output_dir /opt/ml/model --train_batch_size 128"
Traceback (most recent call last):
  File "run_clm.py", line 468, in <module>
    main()
  File "run_clm.py", line 182, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.6/site-packages/transformers/hf_argparser.py", line 187, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 12, in __init__
  File "run_clm.py", line 161, in __post_init__
    raise ValueError("Need either a dataset name or a training/validation file.")
ValueError: Need either a dataset name or a training/validation file.

Context:
Hello @danurahul,
Thanks for opening the thread. EleutherAI/gpt-j-6B is not yet trainable with Amazon SageMaker, since the PR is not yet merged into transformers for GPT-J and when it is merged, we need to update the DLC or you have to include the new version of transformers in the requirements.txt.
In addition to this is GPT-J-6B 22GB big and won’t fit on a singe ml.p3.2xlarge instance it would be possible to train it, when merged with distributed training, see Run training on Amazon SageMaker 9
Also even adjusting the two mentioned points above your script would still not work since you are missing a few hyperparameters. The most crucial on is train_file which should have your input file as value, in your case it would be:
/opt/ml/input/data/training/domain-gen-training.jsonl


Batch transform inference job - downloading model from the Hugging Face Hub on start up

Question:
I try to run

      github.com

    huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb 9

    {
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Huggingface Sagemaker-sdk - Run a batch transform inference job with 🤗 Transformers\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "1. [Introduction](#Introduction)  \n",
    "2. [Run Batch Transform after training a model](#Run-Batch-Transform-after-training-a-model)  \n",
    "3. [Run Batch Transform Inference Job with a fine-tuned model using `jsonl`](#Run-Batch-Transform-Inference-Job-with-a-fine-tuned-model-using-jsonl)   \n",
    "\n",
    "Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy two transformer model for inference. \n",
    "In the first example we deploy a trained Hugging Face Transformer model on to SageMaker for inference.\n",
    "In the second example we directly deploy one of the 10 000+ Hugging Face Transformers from the [Hub](https://huggingface.co/models) to Amazon SageMaker for Inference.<"
   ],

  This file has been truncated. show original

However I get this error:
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: [https://huggingface.co/api/models/cardiffnlp/twitter-roberta-base-sentiment]
It looks like an instance created by AWS SageMaker (client) is not allowed to download the model. Something specific I need to add in the role I use with AWS Sagemaker?
or there is an issue with a token in
model_info = _api.model_info(repo_id=model_id, revision=revision, token=use_auth_token)
Thanks,
Kate

Context:
I rerun the notebook: notebooks/sagemaker/12_batch_transform_inference at master · huggingface/notebooks · GitHub 16
and for me it worked. Can you please test again.


Infer with SageMaker for a Private Model

Question:
Hello,
I have a private model I wish to use for inference using Amazon SageMaker. I’ve found the documentation and code snippets are great for public models but for private models I’m just provided with the same code snippet to use which doesn’t have any reference to my authentication tokens and so when I try running it I get a 404 error on the generated instance in SageMaker. Does anyone know how to add to this code to make it work for private models?
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'private_model',
	'HF_TASK':'text2text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': "The answer to the universe is"
})

Thanks so much,
Karim

Context:
Hey @kmfoda,
If you want to use private models you also need to define HF_API_TOKEN in the hub dictionary. More here GitHub - aws/sagemaker-huggingface-inference-toolkit 2.


Training model file too large and fail to deploy

Question:
hi, community!
I have trained a Bert classification model with below config on sagemaker:
from sagemaker.huggingface import HuggingFace

# hyperparameters, which are passed into the training job
hyperparameters={'epochs': 5,
                 'train_batch_size': 4,
                 'model_name':'bert-base-uncased',
                 'num_labels':31,
                 }

huggingface_estimator = HuggingFace(entry_point='train.py',
                            source_dir='./scripts',
                            instance_type='ml.p3.16xlarge',
                            instance_count=1,
                            role=role,
                            # add volume size set up
                            volume_size=200,
                            transformers_version='4.6',
                            pytorch_version='1.7',
                            py_version='py36',
                            hyperparameters = hyperparameters)
# starting the train job with our uploaded datasets as input
huggingface_estimator.fit({'train': training_input_path, 'test': test_input_path})

this works perfectly. however, I have two problems:

the generated model file is huge: 64gb. why is that?
I can’t deploy endpoint successfully, I am guessing due to the model size too big, any idea on how to solve?

Context:
Hey @jackieliu930,
Yes, I guess your artifact is so big because all saved checkpoints during training are included. You can either change your checkpoint saving strategy in your train.py or the location where the checkpoints are saved.
Or you could load your model.tar.gz and remove all checkpoints from it and then upload it so s3 again. Documentation here: Deploy models to Amazon SageMaker 2
Another solution would be to upload your model to Models - Hugging Face and then deploy using HF_MODE_ID and HF_TASK.


Train end-to-end text classication on sagemaker

Question:
hi,
I’m following the guidence on training text-classfication using my own dataset,
refer to notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub 2
I have two questions:

should the dataset contain label column only support int? in other words, I need to preprocess my data, convert categories to 1,2,3…?
do I need to specify the class number? if so, where?

thanks!
jackie

Context:
Hey @jackieliu930,

Yes, the labels need to be int values
Yes you need to modify the .from_pretrained method here: notebooks/train.py at 3fdb8bd61ed2f2b499dcd55034b1ee58be5cfabb · huggingface/notebooks · GitHub 3

You could also use the run_glue.py from examples 2 using the git_config then you don’t need to provide your own training script.


Question:
Hi,
I am interested in deploying a HuggingFace Model on AWS SageMaker. Let’s say for example I deploy “google/pegasus-large” on AWS. You have very generously given the code to deploy this shown below. I was wondering if, as part of the predict function we have additional arguments. I would like to incorporate a custom length penalty as well as repetition penalty. Would you be able to share where in this code these arguments would be inserted?
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'google/pegasus-large',
	'HF_TASK':'summarization'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
})

This is what the inference looks like right now in my notebook:
batch = tokenizer([source], truncation=True, padding='longest', return_tensors="pt").to('cuda')
predicted = tokenizer.batch_decode(model.generate(**batch, length_penalty = 1.5, repetition_penalty=4.), skip_special_tokens=True, )[0]

Thanks!

Context:
Hey @ujjirox,
Nice to hear that you want to use Amazon SageMaker for deploying your Summarization model, and yes it is possible.
The Inference Toolkit supports the functionalities as the transformers pipelines. You can provide all of your additional prediction parameters in the parameters attribute. I attached an example below of how the request body looks like.
{
	"inputs": "Hugging Face, the winner of VentureBeat’s Innovation in Natural Language Process/Understanding Award for 2021, is looking to level the playing field. The team, launched by Clément Delangue and Julien Chaumond in 2016, was recognized for its work in democratizing NLP, the global market value for which is expected to hit $35.1 billion by 2026. This week, Google’s former head of Ethical AI Margaret Mitchell joined the team.",
	"parameters": {
		"repetition_penalty": 4.0,
		"length_penalty": 1.5
	}
}

You can find more information/parameter at Pipelines — transformers 4.10.1 documentation 11 or at our sagemaker documentation Deploy models to Amazon SageMaker 6
Here is an end-to-end example for the google/pegasus-large how to deploy and use your parameters.
Note: the code snippet is scrollable
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'google/pegasus-large',
	'HF_TASK':'summarization'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

long_text= """
Hugging Face, the winner of VentureBeat’s Innovation in Natural Language Process/Understanding Award for 2021,
is looking to level the playing field. The team, launched by Clément Delangue and Julien Chaumond in 2016,
was recognized for its work in democratizing NLP, the global market value for which is expected to
hit $35.1 billion by 2026. This week, Google’s former head of Ethical AI Margaret Mitchell joined the team.
"""

parameters = {'repetition_penalty':4.,'length_penalty': 1.5}

predictor.predict({"inputs":long_text,"parameters":parameters})


Finetune bart for text summary has nan loss

Question:
hi,
I‘m using Sagemaker to finetune on amazon-review(chinese) for text summary and trying to use mt5 as backbone. but got loss nan, and wondering why
training code:

import sagemaker
import boto3
from sagemaker.huggingface import HuggingFace

# gets role for executing training job
#iam_client = boto3.client('iam')

sess = sagemaker.Session()
role = sagemaker.get_execution_role()

print(f"IAM role arn used for running training: {role}")
print(f"S3 bucket used for storing artifacts: {sess.default_bucket()}")

hyperparameters = {
'model_name_or_path':'google/mt5-small',
'output_dir':'/opt/ml/model',
'dataset_name': 'amazon_reviews_multi',
'dataset_config_name': 'zh',
'output_dir': '/opt/ml/model',
'do_train': True,
'do_eval': True,
'do_predict': True,
'predict_with_generate': True,
'num_train_epochs': 5,
'learning_rate': 5e-5,
'seed': 7,
'fp16': True,

# add your remaining hyperparameters
# more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/pytorch/seq2seq
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}

# creates Hugging Face estimator
huggingface_estimator = HuggingFace(
entry_point='run_summarization.py',
source_dir='/home/ec2-user/SageMaker/transformers/examples/seq2seq',
instance_type='ml.p3.2xlarge',
instance_count=1,
role=role,
git_config=git_config,
transformers_version='4.6.1',
pytorch_version='1.7.1',
py_version='py36',
hyperparameters = hyperparameters
)

# starting the train job
huggingface_estimator.fit()

log:
IAM role arn used for running training: arn:aws:iam::847380964353:role/spot-bot-SpotSageMakerExecutionRole-917OYJPI7O18
S3 bucket used for storing artifacts: sagemaker-us-west-2-847380964353
2021-09-16 05:19:41 Starting - Starting the training job...
2021-09-16 05:20:05 Starting - Launching requested ML instancesProfilerReport-1631769575: InProgress
...
2021-09-16 05:20:39 Starting - Preparing the instances for training............
2021-09-16 05:22:29 Downloading - Downloading input data
2021-09-16 05:22:29 Training - Downloading the training image.................bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2021-09-16 05:25:28,430 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2021-09-16 05:25:28,454 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2021-09-16 05:25:31,492 sagemaker_pytorch_container.training INFO     Invoking user training script.
2021-09-16 05:25:31,980 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:
/opt/conda/bin/python3.6 -m pip install -r requirements.txt

2021-09-16 05:25:27 Training - Training image download completed. Training in progress.Requirement already satisfied: datasets>=1.1.3 in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.6.2)
Requirement already satisfied: sentencepiece!=0.1.92 in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (0.1.91)
Requirement already satisfied: protobuf in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.17.1)
Collecting sacrebleu>=1.4.12
  Downloading sacrebleu-2.0.0-py3-none-any.whl (90 kB)
Collecting rouge-score
  Downloading rouge_score-0.0.4-py2.py3-none-any.whl (22 kB)
Collecting nltk
  Downloading nltk-3.6.2-py3-none-any.whl (1.5 MB)
Requirement already satisfied: pandas in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (1.1.5)
Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2.25.1)
Requirement already satisfied: fsspec in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2021.5.0)
Requirement already satisfied: huggingface-hub<0.1.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.0.8)
Requirement already satisfied: xxhash in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (2.0.2)
Requirement already satisfied: tqdm<4.50.0,>=4.27 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.49.0)
Requirement already satisfied: dill in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: multiprocess in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.70.11.1)
Requirement already satisfied: dataclasses in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (0.8)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.0.1)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (20.9)
Requirement already satisfied: pyarrow>=1.0.0<4.0.0 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (4.0.0)
Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.6/site-packages (from datasets>=1.1.3->-r requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: regex in /opt/conda/lib/python3.6/site-packages (from sacrebleu>=1.4.12->-r requirements.txt (line 4)) (2021.4.4)
Collecting portalocker
  Downloading portalocker-2.3.2-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: colorama in /opt/conda/lib/python3.6/site-packages (from sacrebleu>=1.4.12->-r requirements.txt (line 4)) (0.4.3)
Collecting tabulate>=0.8.9
  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
Requirement already satisfied: filelock in /opt/conda/lib/python3.6/site-packages (from huggingface-hub<0.1.0->datasets>=1.1.3->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (2020.12.5)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.19.0->datasets>=1.1.3->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: six>=1.9 in /opt/conda/lib/python3.6/site-packages (from protobuf->-r requirements.txt (line 3)) (1.16.0)
Collecting absl-py
  Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
Requirement already satisfied: joblib in /opt/conda/lib/python3.6/site-packages (from nltk->-r requirements.txt (line 6)) (1.0.1)
Requirement already satisfied: click in /opt/conda/lib/python3.6/site-packages (from nltk->-r requirements.txt (line 6)) (7.1.2)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata->datasets>=1.1.3->-r requirements.txt (line 1)) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.6/site-packages (from importlib-metadata->datasets>=1.1.3->-r requirements.txt (line 1)) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->datasets>=1.1.3->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from pandas->datasets>=1.1.3->-r requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.6/site-packages (from pandas->datasets>=1.1.3->-r requirements.txt (line 1)) (2021.1)
Installing collected packages: tabulate, portalocker, nltk, absl-py, sacrebleu, rouge-score
  Attempting uninstall: tabulate
    Found existing installation: tabulate 0.8.7
    Uninstalling tabulate-0.8.7:
      Successfully uninstalled tabulate-0.8.7
Successfully installed absl-py-0.13.0 nltk-3.6.2 portalocker-2.3.2 rouge-score-0.0.4 sacrebleu-2.0.0 tabulate-0.8.9
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
aws-parallelcluster 2.10.4 requires tabulate<=0.8.7,>=0.8.2, but you have tabulate 0.8.9 which is incompatible.
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

2021-09-16 05:25:39,039 sagemaker-training-toolkit INFO     Invoking user script

Training Env:

{
    "additional_framework_parameters": {},
    "channel_input_dirs": {},
    "current_host": "algo-1",
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1"
    ],
    "hyperparameters": {
        "predict_with_generate": true,
        "seed": 7,
        "do_predict": true,
        "do_train": true,
        "dataset_name": "amazon_reviews_multi",
        "num_train_epochs": 5,
        "do_eval": true,
        "dataset_config_name": "zh",
        "output_dir": "/opt/ml/model",
        "learning_rate": 5e-05,
        "model_name_or_path": "google/mt5-small",
        "fp16": true
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {},
    "input_dir": "/opt/ml/input",
    "is_master": true,
    "job_name": "huggingface-pytorch-training-2021-09-16-05-19-35-810",
    "log_level": 20,
    "master_hostname": "algo-1",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-us-west-2-847380964353/huggingface-pytorch-training-2021-09-16-05-19-35-810/source/sourcedir.tar.gz",
    "module_name": "run_summarization",
    "network_interface_name": "eth0",
    "num_cpus": 8,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-1",
        "hosts": [
            "algo-1"
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "run_summarization.py"
}

Environment variables:

SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={"dataset_config_name":"zh","dataset_name":"amazon_reviews_multi","do_eval":true,"do_predict":true,"do_train":true,"fp16":true,"learning_rate":5e-05,"model_name_or_path":"google/mt5-small","num_train_epochs":5,"output_dir":"/opt/ml/model","predict_with_generate":true,"seed":7}
SM_USER_ENTRY_POINT=run_summarization.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=[]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=run_summarization
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-west-2-847380964353/huggingface-pytorch-training-2021-09-16-05-19-35-810/source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{"dataset_config_name":"zh","dataset_name":"amazon_reviews_multi","do_eval":true,"do_predict":true,"do_train":true,"fp16":true,"learning_rate":5e-05,"model_name_or_path":"google/mt5-small","num_train_epochs":5,"output_dir":"/opt/ml/model","predict_with_generate":true,"seed":7},"input_config_dir":"/opt/ml/input/config","input_data_config":{},"input_dir":"/opt/ml/input","is_master":true,"job_name":"huggingface-pytorch-training-2021-09-16-05-19-35-810","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-west-2-847380964353/huggingface-pytorch-training-2021-09-16-05-19-35-810/source/sourcedir.tar.gz","module_name":"run_summarization","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"run_summarization.py"}
SM_USER_ARGS=["--dataset_config_name","zh","--dataset_name","amazon_reviews_multi","--do_eval","True","--do_predict","True","--do_train","True","--fp16","True","--learning_rate","5e-05","--model_name_or_path","google/mt5-small","--num_train_epochs","5","--output_dir","/opt/ml/model","--predict_with_generate","True","--seed","7"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_HP_PREDICT_WITH_GENERATE=true
SM_HP_SEED=7
SM_HP_DO_PREDICT=true
SM_HP_DO_TRAIN=true
SM_HP_DATASET_NAME=amazon_reviews_multi
SM_HP_NUM_TRAIN_EPOCHS=5
SM_HP_DO_EVAL=true
SM_HP_DATASET_CONFIG_NAME=zh
SM_HP_OUTPUT_DIR=/opt/ml/model
SM_HP_LEARNING_RATE=5e-05
SM_HP_MODEL_NAME_OR_PATH=google/mt5-small
SM_HP_FP16=true
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages

Invoking script with the following command:

/opt/conda/bin/python3.6 run_summarization.py --dataset_config_name zh --dataset_name amazon_reviews_multi --do_eval True --do_predict True --do_train True --fp16 True --learning_rate 5e-05 --model_name_or_path google/mt5-small --num_train_epochs 5 --output_dir /opt/ml/model --predict_with_generate True --seed 7

09/16/2021 05:25:45 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: True
09/16/2021 05:25:45 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/opt/ml/model', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=True, evaluation_strategy=<IntervalStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Sep16_05-25-45_algo-1', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=500, save_total_limit=None, no_cuda=False, seed=7, fp16=True, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=[], dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/opt/ml/model', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name='length', report_to=[], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, use_legacy_prediction_loop=False, push_to_hub=False, resume_from_checkpoint=None, mp_parameters='', sortish_sampler=False, predict_with_generate=True)
Downloading and preparing dataset amazon_reviews_multi/zh (download: 109.09 MiB, generated: 52.01 MiB, post-processed: Unknown size, total: 161.10 MiB) to /root/.cache/huggingface/datasets/amazon_reviews_multi/zh/1.0.0/724e94f4b0c6c405ce7e476a6c5ef4f87db30799ad49f765094cf9770e0f7609...
Dataset amazon_reviews_multi downloaded and prepared to /root/.cache/huggingface/datasets/amazon_reviews_multi/zh/1.0.0/724e94f4b0c6c405ce7e476a6c5ef4f87db30799ad49f765094cf9770e0f7609. Subsequent calls will reuse this data.
https://huggingface.co/google/mt5-small/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpsal7nwx8
storing https://huggingface.co/google/mt5-small/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/97693496c1a0cae463bd18428187f9e9924d2dfbadaa46e4d468634a0fc95a41.dadce13f8f85f4825168354a04675d4b177749f8f11b167e87676777695d4fe4
creating metadata file for /root/.cache/huggingface/transformers/97693496c1a0cae463bd18428187f9e9924d2dfbadaa46e4d468634a0fc95a41.dadce13f8f85f4825168354a04675d4b177749f8f11b167e87676777695d4fe4
loading configuration file https://huggingface.co/google/mt5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/97693496c1a0cae463bd18428187f9e9924d2dfbadaa46e4d468634a0fc95a41.dadce13f8f85f4825168354a04675d4b177749f8f11b167e87676777695d4fe4
Model config MT5Config {
  "architectures": [
    "MT5ForConditionalGeneration"
  ],
  "d_ff": 1024,
  "d_kv": 64,
  "d_model": 512,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "mt5",
  "num_decoder_layers": 8,
  "num_heads": 6,
  "num_layers": 8,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "tokenizer_class": "T5Tokenizer",
  "transformers_version": "4.6.1",
  "use_cache": true,
  "vocab_size": 250112
}

loading configuration file https://huggingface.co/google/mt5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/97693496c1a0cae463bd18428187f9e9924d2dfbadaa46e4d468634a0fc95a41.dadce13f8f85f4825168354a04675d4b177749f8f11b167e87676777695d4fe4
Model config MT5Config {
  "architectures": [
    "MT5ForConditionalGeneration"
  ],
  "d_ff": 1024,
  "d_kv": 64,
  "d_model": 512,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "mt5",
  "num_decoder_layers": 8,
  "num_heads": 6,
  "num_layers": 8,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "tokenizer_class": "T5Tokenizer",
  "transformers_version": "4.6.1",
  "use_cache": true,
  "vocab_size": 250112
}

https://huggingface.co/google/mt5-small/resolve/main/spiece.model not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp1dhqa6qf
storing https://huggingface.co/google/mt5-small/resolve/main/spiece.model in cache at /root/.cache/huggingface/transformers/37d0f67f084f8c5fc5589e0bba5ff3c6307af833bb0b7f4eb33fbfd8d4038a9d.84ea7af2df68dc8db434d3160aab65cce8ac63ce5b6f7743f8c9a4a14b4f77e2
creating metadata file for /root/.cache/huggingface/transformers/37d0f67f084f8c5fc5589e0bba5ff3c6307af833bb0b7f4eb33fbfd8d4038a9d.84ea7af2df68dc8db434d3160aab65cce8ac63ce5b6f7743f8c9a4a14b4f77e2
https://huggingface.co/google/mt5-small/resolve/main/special_tokens_map.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpqgr1lhnz
storing https://huggingface.co/google/mt5-small/resolve/main/special_tokens_map.json in cache at /root/.cache/huggingface/transformers/685ac0ca8568ec593a48b61b0a3c272beee9bc194a3c7241d15dcadb5f875e53.f76030f3ec1b96a8199b2593390c610e76ca8028ef3d24680000619ffb646276
creating metadata file for /root/.cache/huggingface/transformers/685ac0ca8568ec593a48b61b0a3c272beee9bc194a3c7241d15dcadb5f875e53.f76030f3ec1b96a8199b2593390c610e76ca8028ef3d24680000619ffb646276
https://huggingface.co/google/mt5-small/resolve/main/tokenizer_config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpvibtracc
storing https://huggingface.co/google/mt5-small/resolve/main/tokenizer_config.json in cache at /root/.cache/huggingface/transformers/6a9e52d6dd21568e37b65fc180ada927968e8f7124f0acd6efcaf90cd2e0f4bb.4b81e5d952ad810ca1de2b3e362b9a26a5cc77b4b75daf20caf69fb838751c32
creating metadata file for /root/.cache/huggingface/transformers/6a9e52d6dd21568e37b65fc180ada927968e8f7124f0acd6efcaf90cd2e0f4bb.4b81e5d952ad810ca1de2b3e362b9a26a5cc77b4b75daf20caf69fb838751c32
loading file https://huggingface.co/google/mt5-small/resolve/main/spiece.model from cache at /root/.cache/huggingface/transformers/37d0f67f084f8c5fc5589e0bba5ff3c6307af833bb0b7f4eb33fbfd8d4038a9d.84ea7af2df68dc8db434d3160aab65cce8ac63ce5b6f7743f8c9a4a14b4f77e2
loading file https://huggingface.co/google/mt5-small/resolve/main/tokenizer.json from cache at None
loading file https://huggingface.co/google/mt5-small/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/google/mt5-small/resolve/main/special_tokens_map.json from cache at /root/.cache/huggingface/transformers/685ac0ca8568ec593a48b61b0a3c272beee9bc194a3c7241d15dcadb5f875e53.f76030f3ec1b96a8199b2593390c610e76ca8028ef3d24680000619ffb646276
loading file https://huggingface.co/google/mt5-small/resolve/main/tokenizer_config.json from cache at /root/.cache/huggingface/transformers/6a9e52d6dd21568e37b65fc180ada927968e8f7124f0acd6efcaf90cd2e0f4bb.4b81e5d952ad810ca1de2b3e362b9a26a5cc77b4b75daf20caf69fb838751c32
https://huggingface.co/google/mt5-small/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmplpu5vm63
storing https://huggingface.co/google/mt5-small/resolve/main/pytorch_model.bin in cache at /root/.cache/huggingface/transformers/8e7b2a80ddcb5611b27d8c89e1e8e33a947e105415051402a22b9c8d7d1caeb0.e22331f3a065b885b30ae3dd1ff11ccaf7fbc444485f6eb07ef5e0138bca8b70
creating metadata file for /root/.cache/huggingface/transformers/8e7b2a80ddcb5611b27d8c89e1e8e33a947e105415051402a22b9c8d7d1caeb0.e22331f3a065b885b30ae3dd1ff11ccaf7fbc444485f6eb07ef5e0138bca8b70
loading weights file https://huggingface.co/google/mt5-small/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/8e7b2a80ddcb5611b27d8c89e1e8e33a947e105415051402a22b9c8d7d1caeb0.e22331f3a065b885b30ae3dd1ff11ccaf7fbc444485f6eb07ef5e0138bca8b70
All model checkpoint weights were used when initializing MT5ForConditionalGeneration.

All the weights of MT5ForConditionalGeneration were initialized from the model checkpoint at google/mt5-small.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MT5ForConditionalGeneration for predictions without further training.
Using amp fp16 backend
***** Running training *****
  Num examples = 200000
  Num Epochs = 5
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 125000
[2021-09-16 05:26:59.501 algo-1:31 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2021-09-16 05:26:59.621 algo-1:31 INFO profiler_config_parser.py:102] User has disabled profiler.
[2021-09-16 05:26:59.621 algo-1:31 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.
[2021-09-16 05:26:59.622 algo-1:31 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.
[2021-09-16 05:26:59.624 algo-1:31 INFO hook.py:255] Saving to /opt/ml/output/tensors
[2021-09-16 05:26:59.624 algo-1:31 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.
[2021-09-16 05:26:59.881 algo-1:31 INFO hook.py:591] name:shared.weight count_params:128057344
[2021-09-16 05:26:59.881 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.SelfAttention.q.weight count_params:196608
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.SelfAttention.k.weight count_params:196608
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.SelfAttention.v.weight count_params:196608
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.SelfAttention.o.weight count_params:196608
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight count_params:192
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.0.layer_norm.weight count_params:512
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.1.DenseReluDense.wi_0.weight count_params:524288
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.1.DenseReluDense.wi_1.weight count_params:524288
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.1.DenseReluDense.wo.weight count_params:524288
[2021-09-16 05:26:59.882 algo-1:31 INFO hook.py:591] name:encoder.block.0.layer.1.layer_norm.weight count_params:512
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.0.SelfAttention.q.weight count_params:196608
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.0.SelfAttention.k.weight count_params:196608
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.0.SelfAttention.v.weight count_params:196608
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.0.SelfAttention.o.weight count_params:196608
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.0.layer_norm.weight count_params:512
[2021-09-16 05:26:59.883 algo-1:31 INFO hook.py:591] name:encoder.block.1.layer.1.DenseReluDense.wi_0.weight count_params:524288
name:encoder.block.5.layer.0.SelfAttention.q.weight count_params:196608
[2021-09-16 05:26:59.887 algo-1:31 INFO hook.py:591] name:encoder.block.5.layer.0.SelfAttention.k.weight count_params:196608
[2021-09-16 05:26:59.887 algo-1:31 INFO hook.py:591] name:encoder.block.5.layer.0.SelfAttention.v.weight count_params:196608
[2021-09-16 05:26:59.887 algo-1:31 INFO hook.py:591] name:encoder.block.5.layer.0.SelfAttention.o.weight count_params:196608
[2021-09-16 05:26:59.887 algo-1:31 INFO hook.py:591] ....arams:512
[2021-09-16 05:26:59.902 algo-1:31 INFO hook.py:591] name:decoder.block.7.layer.2.DenseReluDense.wi_0.weight count_params:524288
[2021-09-16 05:26:59.902 algo-1:31 INFO hook.py:591] name:decoder.block.7.layer.2.DenseReluDense.wi_1.weight count_params:524288
[2021-09-16 05:26:59.902 algo-1:31 INFO hook.py:591] name:decoder.block.7.layer.2.DenseReluDense.wo.weight count_params:524288
[2021-09-16 05:26:59.903 algo-1:31 INFO hook.py:591] name:decoder.block.7.layer.2.layer_norm.weight count_params:512
[2021-09-16 05:26:59.903 algo-1:31 INFO hook.py:591] name:decoder.final_layer_norm.weight count_params:512
[2021-09-16 05:26:59.903 algo-1:31 INFO hook.py:591] name:lm_head.weight count_params:128057344
[2021-09-16 05:26:59.903 algo-1:31 INFO hook.py:593] Total Trainable Params: 300176768
[2021-09-16 05:26:59.903 algo-1:31 INFO hook.py:425] Monitoring the collections: losses
[2021-09-16 05:26:59.906 algo-1:31 INFO hook.py:488] Hook is writing from the hook with pid: 31

{'loss': nan, 'learning_rate': 4.98664e-05, 'epoch': 0.02}
Saving model checkpoint to /opt/ml/model/checkpoint-500
Configuration saved in /opt/ml/model/checkpoint-500/config.json
Model weights saved in /opt/ml/model/checkpoint-500/pytorch_model.bin
tokenizer config file saved in /opt/ml/model/checkpoint-500/tokenizer_config.json
Special tokens file saved in /opt/ml/model/checkpoint-500/special_tokens_map.json
Copy vocab file to /opt/ml/model/checkpoint-500/spiece.model
{'loss': nan, 'learning_rate': 4.96664e-05, 'epoch': 0.04}
Saving model checkpoint to /opt/ml/model/checkpoint-1000
Configuration saved in /opt/ml/model/checkpoint-1000/config.json
Model weights saved in /opt/ml/model/checkpoint-1000/pytorch_model.bin
tokenizer config file saved in /opt/ml/model/checkpoint-1000/tokenizer_config.json
Special tokens file saved in /opt/ml/model/checkpoint-1000/special_tokens_map.json
Copy vocab file to /opt/ml/model/checkpoint-1000/spiece.model
{'loss': nan, 'learning_rate': 4.9466400000000005e-05, 'epoch': 0.06}
Saving model checkpoint to /opt/ml/model/checkpoint-1500
Configuration saved in /opt/ml/model/checkpoint-1500/config.json
Model weights saved in /opt/ml/model/checkpoint-1500/pytorch_model.bin
tokenizer config file saved in /opt/ml/model/checkpoint-1500/tokenizer_config.json
Special tokens file saved in /opt/ml/model/checkpoint-1500/special_tokens_map.json
Copy vocab file to /opt/ml/model/checkpoint-1500/spiece.model
{'loss': nan, 'learning_rate': 4.92664e-05, 'epoch': 0.08}
Saving model checkpoint to /opt/ml/model/checkpoint-2000
Configuration saved in /opt/ml/model/checkpoint-2000/config.json
Model weights saved in /opt/ml/model/checkpoint-2000/pytorch_model.bin
tokenizer config file saved in /opt/ml/model/checkpoint-2000/tokenizer_config.json
Special tokens file saved in /opt/ml/model/checkpoint-2000/special_tokens_map.json
Copy vocab file to /opt/ml/model/checkpoint-2000/spiece.model

Context:
Hey @jackieliu930,
When using run_summarization.py with a T5 like the model you need to add an additional hyperparameter source_prefix: "summarize: ".

Only T5 models t5-small , t5-base , t5-large , t5-3b and t5-11b must use an additional argument: --source_prefix "summarize: " .

You can find more information about the run_summarization.py here:transformers/examples/pytorch/summarization at master · huggingface/transformers · GitHub


Inference Toolkit - Init and default template for custom inference

Question:
Hey,
Had some quick questions regarding the Inference Toolkit. Is there a way to add an init function in the custom inference.py script. I was thinking I could just add what I needed in the model_fn function but when I tried running just the basics, I got an error attached below. This leads into the second question.
Do you have a default template for the custom inference.py script. I saw that you had some documentation on GitHub - aws/sagemaker-huggingface-inference-toolkit 3 but I was wondering if you might have an actual script we could modify to our liking.
Thanks!
# This is the script that will be used in the inference container
import os
import json
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

def model_fn(model_dir):
    """
    Load the model and tokenizer for inference
    """
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    model = AutoModelForSeq2SeqLM.from_pretrained(model_dir).to(device)

    model_dict = {'model':model, 'tokenizer':tokenizer}

    return model_dict

def predict_fn(input_data, model):
    """
    Make a prediction with the model
    """

    text = input_data.pop('inputs')
    parameters = input_data.pop('parameters', None)

    tokenizer = model['tokenizer']
    model = model['model']

    # Parameters may or may not be passed
    input_ids = tokenizer(text, truncation=True, padding='longest', return_tensors="pt").input_ids
    output = model.generate(input_ids, **parameters) if parameters is not None else model.generate(input_ids)

    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

def input_fn(request_body, request_content_type):
    """
    Transform the input request to a dictionary
    """
    request = json.loads(request_body)

    return request

def output_fn(prediction, response_content_type):
    """
    Return model's prediction
    """
    return {'generated_text':prediction}

image3280×900 365 KB

Context:
You can add a requirements.txt into the code/ and the archive and upload it to s3 and provide it as model_data. This should work. You can use my example to test it.
Additionally, does the FrameworkModel class have the attribute dependencies, but it looks way more complex to add your dependencies.

dependencies ( list  [  str  ] ) –A list of paths to directories (absolute or relative) with any additional libraries that will be exported to the container (default: []). The library folders will be copied to SageMaker in the same folder where the entrypoint is copied. If ‘git_config’ is provided, ‘dependencies’ should be a list of relative locations to directories with any additional libraries needed in the Git repo. If the source_dir points to S3, code will be uploaded and the S3 location will be used instead.

Example
The following call

Model(entry_point=‘inference.py’, … dependencies=[‘my/libs/common’, ‘virtual-env’])
results in the following inside the container:

$ ls
opt/ml/code
|------ inference.py
|------ common
|------ virtual-env

This is not supported with “local code” in Local Mode.

If you want to go with source_dir and entry_point I would suggest building a helper function that is executed before all imports like install_dependencies.


TensorFlow Getting Started Demo model outputs the same label for all examples

Question:
Hi,
I’ve been following the SageMaker demo available here 1 - for training a binary sentiment classifier on the IMDB dataset.
I ran the notebook as provided, to get a feel for how to use Hugging Face with SageMaker, and the notebook successfully runs and the model trains. However, upon testing the resulting model I found that the output was the same, regardless of the input sentence. For example:
>>>sentiment_input= {"inputs":"This is the best movie I have ever watched. It is amazing!"}
>>>print(predictor.predict(sentiment_input))
[{'label': 'LABEL_0', 'score': 0.9999932050704956}]

>>>sentiment_input= {"inputs":"This is the worst movie I have ever watched. It is terrible!"}
>>>print(predictor.predict(sentiment_input))
[{'label': 'LABEL_0', 'score': 0.9999932050704956}]

I’ve looked through the code, but I can’t seem to find what might cause this behaviour. Unless the dataset isn’t being loaded properly (eg. if the classifier is only training with one label), or something is going wrong with the tokenization. However, I haven’t made any changes to the notebook and training script provided, so this seems unlikely.
Any help would be much appreciated! Thanks in advance.

Context:
I believe I’ve found the problem. The dataset is never shuffled in the train.py script provided with the demo. As a result the model was learning to assign LABEL_0 to any input. After adding a shuffle to the test set (not strictly necessary) and train set here (after line 45):
    # Load dataset
    train_dataset, test_dataset = load_dataset("imdb", split=["train", "test"])
    train_dataset = train_dataset.shuffle()
    test_dataset = test_dataset.shuffle()

    # Preprocess train dataset

the model trains successfully and returns the following labels as expected:
>>>sentiment_input= {"inputs":"This is the best movie I have ever watched. It is amazing!"}
>>>print(predictor.predict(sentiment_input))
[{'label': 'LABEL_1', 'score': 0.995592474937439}]

>>>sentiment_input= {"inputs":"This is the worst movie I have ever watched. It is terrible!"}
>>>print(predictor.predict(sentiment_input))
[{'label': 'LABEL_0', 'score': 0.9919235110282898}]


Question:
Hello,
I fine tuned 2 BERT models which (1.) classify different customer reviews and tag them with different labels and (2.) detect the sentiment in each text. I process these texts in a weekly batch on AWS Sagemaker. Right now I am writing two different batch transform jobs which (1.) predict the class & (2.) predict the text. My question now if it is possible to integrate both models to one batchtransform job. My fine tuned models are in my S3 bucket in the tar.gz format and my code currently looks like this:
# package the inference scrip and pre-trained classifier model into .tar.gz format
!cd model_token && tar zcvf model.tar.gz *
!mv model_token/model.tar.gz ./model.tar.gz

# upload pre-trained classifier model to s3 bucket
model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

#same procedure for sentiment model
!cd sentiment_token && tar zcvf sentiment.tar.gz *
!mv sentiment_token/sentiment.tar.gz ./sentiment.tar.gz
model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
sentiment_uri = S3Uploader.upload('sentiment.tar.gz',model_url)
print(f"Uploaded model to {sentiment_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class for classifier
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',
    split_type='Line')

#same for sentiment
huggingface_model = HuggingFaceModel(
   model_data=sentiment_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',
    split_type='Line')

Thanks in advance!

Context:
As the errors says you need to adjust AssembleWith to be the same.
=>
# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
   instance_count=1,
   instance_type='ml.g4dn.xlarge',
   output_path=output_s3_path, # we are using the same s3 path to save the output with the input
   accept="application/json",
   assemble_with="Line",
   strategy='SingleRecord')

You can find the documentation here Transformer — sagemaker 2.59.4 documentation

assemble_with ( str ) – How the output is assembled (default: None). Valid values: ‘Line’ or ‘None’.


Rouge eval metrics in summarization not computing

Question:
Hello @philschmid I was trying to fine tune a summarization model with a very small data set (100 samples) and none of the rouge eval metrics are being computed in the training job. Is there size limit in terms of samples that could be causing this?
Thank you.

Context:
Hello @Jorgeutd,
Are you using the examples/ script? and are you passing the hyperparameter → 'predict_with_generate': True,

predict_with_generate ( bool , optional, defaults to False ):
Whether to use generate to calculate generative metrics (ROUGE, BLEU).`


Question:
Hello everyone,
I deployed my BERT classification model for batch jobs on Sagemaker with
create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',
    split_type='Line')

utput_file = f"{dataset_jsonl_file}.out"
output_path = s3_path_join(output_s3_path,output_file)

# download file
S3Downloader.download(output_path,'.')

batch_transform_result = []
with open(output_file) as f:
    for line in f:
        # converts jsonline array to normal array
        line = "[" + line.replace("[","").replace("]",",") + "]"
        batch_transform_result = literal_eval(line)

Anyways, when I want to predict a new batch it feels like I always have to start my Notebook on Sagemaker Studio etc. Is there a way to create an API which I can feed with my data to predict from the outside? Is there any cool tutorial or anything? Thanks in advance

Context:
Hey @marlon89,
I sadly don’t have yet a tutorial or a sample for it. I am looking into creating something like that in the next weeks, with cdk support.
I created an architecture of how this API can look like.

SageMaker-batch-cycle.drawio1522×1602 231 KB

You can basically leverage AWS Lambda with S3 Trigger to start an create you batch transform jobs after a file Is uploaded to a prefix on an s3 bucket.


Question:
Hello,
I am wondering if the “sentence-similarity” pipelines can be used on Sagemaker from Hub with the same ease as popular piplines like “question-answering”?
I was trying to start an endpoint with sentence-similarity, but it gave me this error:
2021-09-16 13:13:49,252 [INFO ] W-sentence-transformers__pa-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: "Unknown task sentence-similarity, available tasks are ['feature-extraction', 'text-classification', 'token-classification', 'question-answering', 'table-question-answering', 'fill-mask', 'summarization', 'translation', 'text2text-generation', 'text-generation', 'zero-shot-classification', 'conversational', 'image-classification', 'translation_XX_to_YY']" : 400

This is how it was deployed inside Sagemaker Studio:

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()

# Hub Model configuration. https://huggingface.co/models
hub = {
    'HF_MODEL_ID':'sentence-transformers/paraphrase-xlm-r-multilingual-v1',
    'HF_TASK':'sentence-similarity' # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.t2.medium"
)

Context:
Hello @pavel-nesterov,
Currently, the SageMaker Inference Toolkit only supports the NLP-based pipelines from the transformers library for zero-code deployments.
But you can easily create your own inference.py which contains the code you would need for that.
Here is Documentation on that: Deploy models to Amazon SageMaker 14
Here the code for the sentence-simliarity pipeline: huggingface_hub/sentence_similarity.py at main · huggingface/huggingface_hub · GitHub 20
Thank you for the request I will add it as potential feature to the roadmap.


Question:
Hi everyone (especially those, who uses a   of HuggingFace+Sagemaker+ServerlessFramework+Lambda).
Since yesterday I tried 20+ options to properly import Sagemaker module to use the HuggingFacePredictor within the Lambda function, but no success. This is why I am asking for help, because I couldn’t find the correct way to make it work (even on serverless.com forums).
Here is what I am trying to do within Lambda:
try:
     print('attempting to unzip_requirements...')
     import unzip_requirements
     print('succesfully imported unzip_requirements to prepare the zipped requirements')
except ImportError:
     print('failed to import unzip_requirements - if you are running locally this is not a problem')
     pass
...
import sagemaker
from sagemaker.huggingface import HuggingFacePredictor
predictor = HuggingFacePredictor(ENDPOINT_NAME)

Here is the log:
attempting to unzip_requirements...
succesfully imported unzip_requirements to prepare the zipped requirements
...

[ERROR] PackageNotFoundError: No package metadata was found for sagemaker
Traceback (most recent call last):
  File "/var/task/serverless_sdk/__init__.py", line 144, in wrapped_handler
    return user_handler(event, context)
  File "/var/task/handler.py", line 184, in ProductMapper
    import sagemaker
  File "/tmp/sls-py-req/sagemaker/__init__.py", line 18, in <module>
    from sagemaker import estimator, parameter, tuner  # noqa: F401
  File "/tmp/sls-py-req/sagemaker/estimator.py", line 28, in <module>
    from sagemaker import git_utils, image_uris
  File "/tmp/sls-py-req/sagemaker/image_uris.py", line 22, in <module>
    from sagemaker.spark import defaults
  File "/tmp/sls-py-req/sagemaker/spark/__init__.py", line 16, in <module>
    from sagemaker.spark.processing import PySparkProcessor, SparkJarProcessor  # noqa: F401
  File "/tmp/sls-py-req/sagemaker/spark/processing.py", line 35, in <module>
    from sagemaker.local.image import _ecr_login_if_needed, _pull_image
  File "/tmp/sls-py-req/sagemaker/local/__init__.py", line 16, in <module>
    from .local_session import (  # noqa: F401
  File "/tmp/sls-py-req/sagemaker/local/local_session.py", line 23, in <module>
    from sagemaker.local.image import _SageMakerContainer
  File "/tmp/sls-py-req/sagemaker/local/image.py", line 39, in <module>
    import sagemaker.local.data
  File "/tmp/sls-py-req/sagemaker/local/data.py", line 27, in <module>
    import sagemaker.local.utils
  File "/tmp/sls-py-req/sagemaker/local/utils.py", line 22, in <module>
    from sagemaker import s3
  File "/tmp/sls-py-req/sagemaker/s3.py", line 20, in <module>
    from sagemaker.session import Session
  File "/tmp/sls-py-req/sagemaker/session.py", line 37, in <module>
    from sagemaker.user_agent import prepend_user_agent
  File "/tmp/sls-py-req/sagemaker/user_agent.py", line 21, in <module>
    SDK_VERSION = importlib_metadata.version("sagemaker")
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 994, in version
    return distribution(distribution_name).version
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 967, in distribution
    return Distribution.from_name(distribution_name)
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 561, in from_name
    raise PackageNotFoundError(name)

Here is serverless.yml:
org: **********
app: rec*************-app
service: re********service

frameworkVersion: '2'

custom:
  bucket: re********bucket
  pythonRequirements:
    dockerizePip: false
    invalidateCaches: true
    zip: true
    slim: true
    strip: false
    noDeploy:
      - docutils
      - jmespath
      - pip
      - python-dateutil
      - setuptools
      - six
      - tensorboard
    useStaticCache: false
    useDownloadCache: false

package:
  include:
    - requirements.txt
  exclude:
    - aws-deployment*
    - .dockerignore
    - Dockerfile
    - README.md
    - .gitignore
    - venv/**
    - test/**

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
  stage: develop-lenovo
  region: eu-central-1
  memorySize: 256
  logRetentionInDays: 30
  iam:
    role:
      statements:
      - Effect: Allow
        Action: s3:*
        Resource: "*"
      - Effect: Allow
        Action: sagemaker:InvokeEndpoint
        Resource: "*"
  s3:
    telegramPuctures:
      name: ***************

functions:
  PictureSaver:
    handler: handler.PictureSaver
    layers:
      arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-python38-requests:20
  TableExtractorWithVeryfi:
    handler: handler.TableExtractor
    timeout: 20 # optional, in seconds, default is 6
    layers:
      arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-python38-requests:20
  ProductMapper:
    handler: handler.ProductMapper
    timeout: 100 # optional, in seconds, default is 6
    layers:
      arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-python38-requests:20

stepFunctions:
  stateMachines:
    re*********step-function-IaC:
      events:
        - http:
            path: /
            method: POST
      definition:
        StartAt: PictureSaverStep
        States:
          PictureSaverStep:
            Type: Task
            Resource:
              Fn::GetAtt: [PictureSaver, Arn]
            Next: TableExtractorStep
          TableExtractorStep:
            Type: Task
            Resource:
                Fn::GetAtt: [TableExtractorWithVeryfi, Arn]
            Next: ProductMapperStep
          ProductMapperStep:
            Type: Task
            Resource:
              Fn::GetAtt: [ProductMapper, Arn]
            End: true
      tracingConfig:
        enabled: true

plugins:
  - serverless-step-functions
  - serverless-python-requirements

And this is what is inside requirements.txt
requests == 2.22.0
sagemaker == 2.59.1.post0

Can maybe some one give me a hint, what could help? It looks like I’m failing to import requirements completely…

Context:
The issue was in this line in serverless.yml:
    slim: true

Changing it to “false” solves the issue, but slightly increases the package size (20% in my case).
    slim: false


Finetuning text summary model support change pretrained model?

Question:
hi,
I am following the tutorial 08 distributed training and i am dealing with text summary in different language(here, chinese),i am wondering, any guidence on how to change pretrained models for multi-language?
best,
jackie

Context:
You can adjust the model, which should be used for training in the hyperparameters. Replace it with the model you want to use.

# hyperparameters, which are passed into the training job
hyperparameters={'per_device_train_batch_size': 4,
                 'per_device_eval_batch_size': 4,
                 'model_name_or_path': 'facebook/bart-large-cnn', # model used for training


Deployed HF model from the hub and got an error: ‘numpy.ndarray’ object has no attribute ‘pop’

Question:
Hi, dear community. I am new member here and I got stuck with the inference with HF model.
Here is what I’m trying to do:

there is a pre-trained HF model deployed as Sagemaker endpoint (code below, #1)
I am trying to access this endpoint from outside Sagemaker - first from Lambda, then from Colab
both cases gave me the same error:

2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - AttributeError: 'numpy.ndarray' object has no attribute 'pop'

Here is the code #1 of endpoint deployment (I deploy it from SageMaker studio):
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()

# Hub Model configuration. https://huggingface.co/models
hub = {
  #'HF_MODEL_ID':'distilbert-base-uncased-distilled-squad', # model_id from hf.co/models
    'HF_MODEL_ID':'distilbert-base-uncased', # model_id from hf.co/models
    'HF_TASK':'question-answering' # NLP task you want to use for predictions
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)

Code#2 for inference from Colab
!pip install boto3
import os
import io
import boto3
from botocore.config import Config
import json
import csv

# grab environment variables
my_config = Config(
    region_name='eu-central-1'
)
ENDPOINT_NAME = 'huggingface-pytorch-inference-2021-09-15-06-35-36-297'

runtime= boto3.client('runtime.sagemaker',
                      region_name='eu-central-1',
                      aws_access_key_id = 'AK************************RW',
                      aws_secret_access_key= 'dZ*************************************o9I')

payload = {

"inputs": {
    "question": "What is used for inference?",
    "context": "My Name is Philipp and I live in Nuremberg. This model is used with sagemaker for inference."
    }
}
print(payload)

response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='text/csv',
                                   Body=json.dumps(payload))
print(response)
result = json.loads(response['Body'].read().decode())
print(result)

Here is a full log of the invocation from Colab.
This is an experimental beta features, which allows downloading model from the Hugging Face Hub on start up. It loads the model defined in the env var `HF_MODEL_ID`
#015Downloading:   0%|          | 0.00/8.54k [00:00<?, ?B/s]#015Downloading: 100%|██████████| 8.54k/8.54k [00:00<00:00, 7.39MB/s]
#015Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 483/483 [00:00<00:00, 498kB/s]
#015Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]#015Downloading:   3%|▎         | 8.57M/268M [00:00<00:03, 85.7MB/s]#015Downloading:   6%|▋         | 17.4M/268M [00:00<00:02, 87.1MB/s]#015Downloading:  10%|▉         | 26.4M/268M [00:00<00:02, 88.5MB/s]#015Downloading:  13%|█▎        | 35.5M/268M [00:00<00:02, 89.6MB/s]#015Downloading:  17%|█▋        | 44.5M/268M [00:00<00:02, 89.2MB/s]#015Downloading:  20%|█▉        | 53.4M/268M [00:00<00:02, 89.2MB/s]#015Downloading:  23%|██▎       | 62.3M/268M [00:00<00:02, 89.0MB/s]#015Downloading:  27%|██▋       | 71.2M/268M [00:00<00:02, 87.6MB/s]#015Downloading:  30%|██▉       | 80.0M/268M [00:00<00:02, 83.0MB/s]#015Downloading:  33%|███▎      | 89.0M/268M [00:01<00:02, 85.1MB/s]#015Downloading:  36%|███▋      | 97.6M/268M [00:01<00:02, 77.0MB/s]#015Downloading:  39%|███▉      | 105M/268M [00:01<00:02, 70.4MB/s] #015Downloading:  42%|████▏     | 113M/268M [00:01<00:02, 65.5MB/s]#015Downloading:  45%|████▌     | 121M/268M [00:01<00:02, 70.0MB/s]#015Downloading:  48%|████▊     | 130M/268M [00:01<00:01, 75.3MB/s]#015Downloading:  52%|█████▏    | 139M/268M [00:01<00:01, 79.6MB/s]#015Downloading:  55%|█████▌    | 147M/268M [00:01<00:01, 81.1MB/s]#015Downloading:  58%|█████▊    | 156M/268M [00:01<00:01, 78.2MB/s]#015Downloading:  61%|██████    | 164M/268M [00:02<00:01, 78.6MB/s]#015Downloading:  64%|██████▍   | 172M/268M [00:02<00:01, 79.1MB/s]#015Downloading:  68%|██████▊   | 181M/268M [00:02<00:01, 82.7MB/s]#015Downloading:  71%|███████   | 190M/268M [00:02<00:00, 85.2MB/s]#015Downloading:  74%|███████▍  | 199M/268M [00:02<00:00, 87.0MB/s]#015Downloading:  78%|███████▊  | 208M/268M [00:02<00:00, 87.6MB/s]#015Downloading:  81%|████████  | 217M/268M [00:02<00:00, 54.2MB/s]#015Downloading:  84%|████████▍ | 226M/268M [00:02<00:00, 61.7MB/s]#015Downloading:  88%|████████▊ | 235M/268M [00:03<00:00, 68.4MB/s]#015Downloading:  91%|█████████ | 244M/268M [00:03<00:00, 74.2MB/s]#015Downloading:  94%|█████████▍| 253M/268M [00:03<00:00, 78.5MB/s]#015Downloading:  98%|█████████▊| 262M/268M [00:03<00:00, 81.8MB/s]#015Downloading: 100%|██████████| 268M/268M [00:03<00:00, 78.5MB/s]
#015Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]#015Downloading:  18%|█▊        | 86.0k/466k [00:00<00:00, 484kB/s]#015Downloading:  96%|█████████▌| 446k/466k [00:00<00:00, 1.39MB/s]#015Downloading: 100%|██████████| 466k/466k [00:00<00:00, 1.30MB/s]
#015Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 16.8kB/s]
#015Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]#015Downloading:  37%|███▋      | 86.0k/232k [00:00<00:00, 484kB/s]#015Downloading: 100%|██████████| 232k/232k [00:00<00:00, 862kB/s]
WARNING - Overwriting /.sagemaker/mms/models/distilbert-base-uncased ...
2021-09-15 06:40:02,907 [INFO ] main com.amazonaws.ml.mms.ModelServer -
MMS Home: /opt/conda/lib/python3.6/site-packages
Current directory: /
Temp directory: /home/model-server/tmp
Number of GPUs: 0
Number of CPUs: 1
Max heap size: 3201 M
Python executable: /opt/conda/bin/python3.6
Config file: /etc/sagemaker-mms.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8080
Model Store: /.sagemaker/mms/models
Initial Models: ALL
Log dir: /logs
Metrics dir: /logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 1
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Preload model: false
Prefer direct buffer: false
2021-09-15 06:40:03,058 [WARN ] W-9000-distilbert-base-uncased com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-distilbert-base-uncased
2021-09-15 06:40:03,161 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model_service_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker_huggingface_inference_toolkit.handler_service --model-path /.sagemaker/mms/models/distilbert-base-uncased --model-name distilbert-base-uncased --preload-model false --tmp-dir /home/model-server/tmp
2021-09-15 06:40:03,162 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000
2021-09-15 06:40:03,162 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 38
2021-09-15 06:40:03,162 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MMS worker started.
2021-09-15 06:40:03,162 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.6.13
2021-09-15 06:40:03,163 [INFO ] main com.amazonaws.ml.mms.wlm.ModelManager - Model distilbert-base-uncased loaded.
2021-09-15 06:40:03,171 [INFO ] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2021-09-15 06:40:03,188 [INFO ] W-9000-distilbert-base-uncased com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000
2021-09-15 06:40:03,278 [INFO ] main com.amazonaws.ml.mms.ModelServer - Inference API bind to: http://0.0.0.0:8080
Model server started.
2021-09-15 06:40:03,281 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000.
2021-09-15 06:40:03,283 [WARN ] pool-2-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet.
2021-09-15 06:40:04,972 [WARN ] W-9000-distilbert-base-uncased-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Some weights of the model checkpoint at /.sagemaker/mms/models/distilbert-base-uncased were not used when initializing DistilBertForQuestionAnswering: ['vocab_projector.bias', 'vocab_layer_norm.weight', 'vocab_transform.weight', 'vocab_transform.bias', 'vocab_projector.weight', 'vocab_layer_norm.bias']
2021-09-15 06:40:04,972 [WARN ] W-9000-distilbert-base-uncased-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - - This IS expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
2021-09-15 06:40:04,972 [WARN ] W-9000-distilbert-base-uncased-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - - This IS NOT expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
2021-09-15 06:40:04,973 [WARN ] W-9000-distilbert-base-uncased-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at /.sagemaker/mms/models/distilbert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']
2021-09-15 06:40:04,973 [WARN ] W-9000-distilbert-base-uncased-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2021-09-15 06:40:05,199 [INFO ] W-9000-distilbert-base-uncased-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Model distilbert-base-uncased loaded io_fd=da001afffe0afdd4-0000001a-00000000-e2ff1d120c2f0e8e-0bb0052c
2021-09-15 06:40:05,206 [INFO ] W-9000-distilbert-base-uncased com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 1871
2021-09-15 06:40:05,209 [WARN ] W-9000-distilbert-base-uncased com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-distilbert-base-uncased-1
2021-09-15 06:40:11,097 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 41
2021-09-15 06:40:15,795 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:20,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:25,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:30,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:35,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:40,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:40:45,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 1
2021-09-15 06:40:50,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 1
2021-09-15 06:40:55,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 1
2021-09-15 06:41:00,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:41:05,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 0
2021-09-15 06:41:10,794 [INFO ] pool-1-thread-3 ACCESS_LOG - /10.32.0.2:37696 "GET /ping HTTP/1.1" 200 1
2021-09-15 06:41:14,478 [WARN ] W-distilbert-base-uncased-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /opt/conda/lib/python3.6/site-packages/sagemaker_inference/decoder.py:58: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2021-09-15 06:41:14,478 [WARN ] W-distilbert-base-uncased-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   return np.genfromtxt(stream, dtype=dtype, delimiter=",")
2021-09-15 06:41:14,479 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error
2021-09-15 06:41:14,480 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-09-15 06:41:14,480 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 222, in handle
2021-09-15 06:41:14,480 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     response = self.transform_fn(self.model, input_data, content_type, accept)
2021-09-15 06:41:14,480 [INFO ] W-9000-distilbert-base-uncased com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 4
2021-09-15 06:41:14,480 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 181, in transform_fn
2021-09-15 06:41:14,480 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     predictions = self.predict(processed_data, model)
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 142, in predict
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     inputs = data.pop("inputs", data)
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - AttributeError: 'numpy.ndarray' object has no attribute 'pop'
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/opt/conda/lib/python3.6/site-packages/mms/service.py", line 108, in predict
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     ret = self._entry_point(input_batch, self.context)
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -   File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 231, in handle
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -     raise PredictionException(str(e), 400)
2021-09-15 06:41:14,482 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: 'numpy.ndarray' object has no attribute 'pop' : 400

Context:
Hello @pavel-nesterov,
Welcome to the Community .
First of all, I saw in Code#1 that you deployed 'distilbert-base-uncased' with question-answering, which is definitely not recommended since it is not fine-tuned for question answering.
The Error:
2021-09-15 06:41:14,481 [INFO ] W-distilbert-base-uncased-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - AttributeError: 'numpy.ndarray' object has no attribute 'pop'

raises because you are forcing ContentType='text/csv' for a JSON input, that’s not going to work.
Change ContentType in code#2 to application/json and it should work. Additionally instead of using boto3 + with runtime.sagemaker you could use the sagemaker sdk which provides a HuggingFacePredictor to invoke your endpoints.Hugging Face — sagemaker 2.59.1.post0 documentation 1
from sagemaker.huggingface import HuggingFacePredictor

predictor = HuggingFacePredictor(ENDPOINT_NAME)
response = predictor.predict(payload)
print(response)


Question:
Hi everyone,
I am trying to train a huggingface model on jupyternotebook on my local machine.
can i get some advice on: how do i pass my IAM role access key ?
i have installed sagemaker on my jupyternotebook, however, how do i connect to my AWS account? i found a line of code:

Blockquote

gets role for executing training job
iam_client = boto3.client(‘iam’)
role = iam_client.get_role(RoleName=’{IAM_ROLE_WITH_SAGEMAKER_PERMISSIONS}’)[‘Role’][‘Arn’]
hyperparameters = {
‘model_name_or_path’:‘finiteautomata/beto-sentiment-analysis’,
‘output_dir’:’/opt/ml/model’
# add your remaining hyperparameters
# more info here transformers/examples/pytorch/text-classification at v4.6.1 · huggingface/transformers · GitHub 3
}

Blockquote

i have set up a new user in my AWS account and have my access keys. just don’t understand how do it pass it on to my local machine?
everytime i only get errors like:

Blockquote
ValueError: Must setup local AWS configuration with a region supported by SageMaker.

NoCredentialsError: Unable to locate credentials

Blockquote

please advise!
thank you in advance.
meng

Context:
Hey @tumon,
You need to create an IAM Role for sagemaker with permission to access all required resources. You can find instructions on how to do this here: SageMaker Roles - Amazon SageMaker 10

To create a new role

Open the IAM console at https://console.aws.amazon.com/iam/ 14.
Select Roles and then select Create role .
Select SageMaker .
Select Next: Permissions .
The IAM managed policy, AmazonSageMakerFullAccess is automatically attached to this role. To see the permissions included in this policy, select the sideways arrow next to the policy name. Select Next: Tags .
(Optional) Add tags and select Next: Review .
Give the role a name in the text field under Role name and select Create role .
On the Roles section of the IAM console, select the role you just created. If needed, use the text box to search for the role using the role name you entered in step 7.
On the role summary page, make note of the ARN.


InternalServer Exception when deploying fine tuned model on Sagemaker

Question:
Hello everyone,
I got a fine tuned german classification model which I want to deploy on Sagemaker. To test my model I want to predict a couple of texts in the json format. Here is my code:
import csv
import json
import sagemaker
from sagemaker.s3 import S3Uploader,s3_path_join

# get the s3 bucket
sess = sagemaker.Session()
role = sagemaker.get_execution_role()
sagemaker_session_bucket = sess.default_bucket()

# uploads a given file to S3.
input_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/input")
output_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/output")
#s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

# datset files
dataset_csv_file="test"
dataset_jsonl_file="test.jsonl"

with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+") as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        # remove @
        json.dump(row, outfile)
        outfile.write('\n')

s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

#package my fine tuned model
!cd model_token && tar zcvf model.tar.gz *
!mv model_token/model.tar.gz ./model.tar.gz

model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',
    split_type='Line')

When I execute that batch tranform job I get the following error:
“code”: 400,
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl:   “type”: “InternalServerException”,
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch_transform/input/test.jsonl:   “message”: “text input must of type str (single example), List[str] (batch or single pretokenized example) or List[List[str]] (batch of pretokenized examples).”
What am I missing?

Context:
Hey @marlon89,
could you share the cloudwatch logs? They should provide more information to why the batch transform job is failing.
Additionally, can you share an example of your test.jsonl?


Question:
Hello everyone,
I fine tuned a pretrained Bert model locally using Tranformers Version 4.9.1. Anyways Sagemaker just allows versions til 4.6 Can this cause problems when I want to deploy my model on Sagemaker?

Context:
Hello @marlon89,
No, your transformers 4.9.1 trained BERT should be compatible with SageMaker and 4.6.1.