Directly load models from a remote storage like S3

Question:
Hi,
Instead of download the transformers model to the local file, could we directly read and write models from S3?
I have tested that we can read csv and txt files directly from S3, but not for models. Is there any solution?

Context:
Hey @leifan,
Is your question related to inference or training?
If it is related to the question you could download a model from s3 when starting the training job the same way as you would do with data.
huggingface_estimator.fit({
    'train': 's3://<my-bucket>/<prefix>/train',  # containing train files
    'test': 's3://<my-bucket>/<prefix>/test',  # containing test files
    'model':  's3://<another-bucket>/<prefix>/model',  # containing model files (config.json, pytroch_model.bin, etc.)
})

SageMaker will then download when starting the training job all of these files into your container.
The Path of the files can either be accessed from the env var SM_CHANNEL_XXXX, e.g. SM_CHANNEL_TRAIN, SM_CHANNEL_MODEL or directly from, e.g. /opt/ml/input/train
And then you can load your model in your training script with
AutoModelForXXX.from_pretrained(os.environ.get('SM_CHANNEL_MODEL',None))


Question: Thanks for putting together this great toolkit. I had a question of how inference batching is handled. I noticed that the examples here 1 all appear to have a single input request. Once deployed, if multiple requests are made to the endpoint of a deployed model at once or in quick succession, are they automatically batched under the hood, or is there something you need to do before hitting the endpoint to feed in a batch of inputs manually?

Context: pinging @philschmid and @jeffboudier in case they hadn’t seen this!


How are the inputs tokenized when model deployment?

Question:
Hi.
I’m working through the series of sagemaker-hugginface notebooks and it is not clear to me how the predict data is preprocess before call the model.
The notebook 01_getting_started_pytorch.ipynb shows these 3 steps:

preprocess datasets
save datsets on s3
train the model using sagemaker Huggingface API
once model trained, deploy model and make predictions from a input data in a dictionary format like: {"inputs": "blablabla"}

my question is: ¿how are these input data being tokenized before to get in the model?

Context:
Hey @Oigres,
Which tokenization step do you mean for training or inference?
For training, the tokenization is done in the preprocessing in the notebook.
For inference, the tokenization is done in the sagemaker-huggingface-inference-toolkit 7 and the toolkit leverages the transformers pipeline.


SageMaker Inference for Model Tuned Elsewhere

Question:
Reading through the documentation for HuggingFace & SageMaker as we are evaluating it and found the following:

Q: Which models can I deploy for Inference?
A: You can deploy

any  Transformers model trained in Amazon SageMaker, or other compatible platforms and that can accomodate the SageMaker Hosting design
any of the 10 000+ publicly available Transformer models from the Hugging Face Model Hub, or
your private models hosted in your Hugging Face premium account!

Is it possible to fine-tune a model elsewhere, outside of SageMaker Training, (for instance, just through a regular PyTorch training loop on a pretrained transformers model), and then deploy it for Inference without hosting it on an account?
Would appreciate any pointers y’all can give on this.

Context:
Hello @charlesatftl,
Yes, you can fine-tune transformers anywhere you want and use it in SageMaker for Inference.
There are currently two options to use your model then:

Push your model to Models - Hugging Face 2 and deploy it directory from there, see: Deploy models to Amazon SageMaker 1

Create a model.tar.gz upload it to s3 and deploy it from there, see: Deploy models to Amazon SageMaker 5
In Addition to 2. here is the documentation on how to create a model.tar.gz Deploy models to Amazon SageMaker 16


Problems in deployment when I configure my own labels

Question:
Hi.
I am training a binary classification model based on this model checkpoint: dccuchile/bert-base-spanish-wwm-cased
When deployment, it uses by default these labels: LABEL_0, LABEL_1.
My goal is to deploy it with my own labels. Browsing the forum I came across this thread, and it seems that resolve a similiar problem.

    Change label names on inference API Beginners

    Hi there,
I recently uploaded my first model to the model hub and I’m wondering how I can change the label names that are returned by the inference API. Right now, the API returns “LABEL_0”, “LABEL_1”, etc. with the predictions and I would like it to be something like “Economy”, “Welfare”, etc.
I looked at the files of other hosted models and I saw that others changed the id2label and label2id in the config.json file, so I also did that here, but the inference API still returns “LABEL_0”. Do I…

After read this thread,  my code looks like:
label2id = {
        "0": "goodUser",
        "1": "badUserFraud"
    }

    id2label = {
        "goodUser": 0,
        "badUserFraud": 1
    }

    # download model from model hub
    config = AutoConfig.from_pretrained(args.model_name, label2id=label2id, id2label=id2label)
    model = AutoModelForSequenceClassification.from_pretrained(args.model_name, config=config)
    tokenizer = AutoTokenizer.from_pretrained(args.model_name)

The training step going well, but when deployment step I find this error:

image1632×552 101 KB

Thanks in advance

Context: I would like to know this aswell


Highest Transformer vers. that works with Sagemaker?

Question:
Hi,
Me and my research team have been wanting to deploy a transformer model on AWS.
Just wanted to know what is the highest version of transformer api that can run with estimator and AWS tuner  ?
Last time i saw was that transformer version 3.5.1 works but can i use 4.10?
Have someone tried it?
Any input is much appreciated.

Context:
Hey @moma1820,
The current latest version of transformers for SageMaker is 4.6.1.
You can find a list of the currently supported versions in our documentation here: Hugging Face on Amazon SageMaker 1
We are keeping this up to date.
Speaking of transformers 4.10.0 we are in the middle of creating a new DLC and support for the Estimator. You can find the PR here [huggingface_tensorflow, huggingface_pytorch] update for Transformers to 4.10 by philschmid · Pull Request #1286 3
But IF you want to use a specific transformers version for either inference and training you can always create an additional requirements.txt including your preferred transformers version and provide it for either training or inference.


InternalServerError after model training finishes, but fails to upload?

Question:
I’m using HuggingFace/SageMaker to fine-tune a distilBert model. Lately, I’ve been running into an issue where model training/evaluation finishes 100% without any errors, gets stuck for a few hours during the model uploading process, and then the job fails with the following error message:

Screen Shot 2021-08-27 at 10.19.51 AM861×103 7.5 KB

Screen Shot 2021-08-27 at 10.39.25 AM810×382 35.2 KB

I didn’t see any indication of why the job would fail in the logs I have access to (e.g. training/eval fully finishes, no issues with CUDA/memory, oddities in the data, etc) and AWS support doesn’t seem to have a clue either.

Screen Shot 2021-08-27 at 10.36.33 AM808×594 172 KB

Screen Shot 2021-08-27 at 10.36.45 AM1015×372 70 KB

This issue doesn’t happen when the model trains on a subset of the available training data (e.g. using 30-50% of the available training data to train) and only seems to occur when training with all the available training data - same model, same config, same instances, etc. So at first, I thought it had to do with S3 checkpoints and distributed training since this only happens when training on our larger dataset.
I’m using 2x of the ml.p4d.24xlarge instances with distributed training for this job.  I did see that AWS had a document on model parallel troubleshooting 1 and have tried their suggestions of removing the debug hook config + disabling checkpointing but no luck either.
Here’s my estimator config, just in case:
huggingface_estimator = HuggingFace(entry_point='train.py',
                            source_dir='./scripts',
                            instance_type=instance_type,
                            instance_count=instance_count,
                            base_job_name='test-run-no-debug-no-checkpoints',
                            # checkpoint_s3_uri=f's3://{sess.default_bucket()}/checkpoints',
                            # use_spot_instances=True,
                            # max_wait=(2*24*60*60),
                            max_run=(2*24*60*60),
                            volume_size=volume_size,
                            role=role,
                            transformers_version='4.6.1',
                            pytorch_version='1.7.1',
                            py_version='py36',
                            hyperparameters = hyperparameters,
                            distribution = distribution,
                            debugger_hook_config=False)
I’m not sure what’s causing this issue and was wondering if anyone has any insight about this?

Context:
Hello @nreamaroon,
thank you for opening the thread! That is indeed strange.
Did you use your own train.py or already created an example? If you wrote your own one can you share your training script?
Especially, which saving strategy you use. And could you also share the size of the dataset and model you use and the hyperparamter?
The only understandable reason for me currently is that during training on the large dataset SageMaker is creating this many checkpoints that it somehow fails uploading at the end. But this wouldn’t make any sense at all.
An easy way to test this would be only saving the last model in opt/ml/model.
In Addition to this, I have shared your issue with the AWS Team directly!


How to improve tqdm log information when training?

Question:
Hi.
The logs of training executions look very bad. Do someone know fix them and get a normal-pretty print?

image1416×763 242 KB

Context:
Hey @Oigres,
The log looks this bad because Training Job is piping the stdout to SageMaker/Cloudwatch and every new “tick” download progress is recorded as separate log.
The logs should look better if you open it in the aws management console in cloudwatch.
Currently there is no way to deactivate the progress bars, when using transformers, but we are on it.


HuggingFace with Sagemaker tutorial doesn’t work

Question:
Hi.
I am going through the HuggingFace - Sagemaker tutorial that you provide in github. I am working with the notebook 1 : 01_getting_started_pytorch. I am using Sagemaker Studio with the image:  Python 3 (PyTorch 1.6 Python 3.6 CPU Optimized)
When I try to download the imdb dataset I have this error:

image1376×808 135 KB

Do someone know how to fix this issue?
Thanks in advance.
Sergio

Context:
Hey @Oigres,
which datasets version have you installed?


Question:
Hi- It’s me again.
I want to get a better sense of the system latency for the speech model I have deployed.
AWS provides Invocation Endpoint Metrics 1 like ModelLatency but that is more end-to-end. I am particularly interested in how much time is spent with the forward pass. I think I have two options:
1- HuggingFace model logs preprocess,predict, and postprocess times here 2 but there is a bug as below where predict time is not captured correctly.

Screen Shot 2021-08-19 at 5.23.12 PM2616×226 70.5 KB

2- Here 1, the metrics Transform Fn is added to the context using the API described here 2.
My question is  where does the Transform Fn go? I have hard time finding it on CloudWatch. I will also submit a PR to fix the predict time logging.
Thanks so much!
Deniz

Context:
Hey @dzorlu,
thank you for finding the bug, please let me know if you not manage to open PR, then I would take care of it.
Maybe @dan21c can tell more about the transform_fn and where is stored?
context.metrics.add_time("Transform Fn", round((predict_end - predict_start) * 1000, 2))


Question:
Hi all,
I’ve been working through adapting the getting started notebook 1 to my particular use case. I wrote out my data to s3, and kicked off .fit(), but am getting this error block:
2021-04-23 04:58:40,552 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 train.py --epochs 1 --model_name distilbert-base-uncased --train_batch_size 32"
Traceback (most recent call last):
  File "train.py", line 42, in <module>
    train_dataset = load_from_disk(args.training_dir)
  File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 781, in load_from_disk
    return Dataset.load_from_disk(dataset_path, fs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in load_from_disk
    state = {k: state[k] for k in dataset.__dict__.keys()}  # in case we add new fields
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in <dictcomp>
    state = {k: state[k] for k in dataset.__dict__.keys()}  # in case we add new fields
KeyError: '_data'

What’s leaving me scratching my head is that when I reference the arrow_dataset.py 1 file, I can’t find lines of this kind, making me think there’s some discrepancy in whatever AWS’ container is and that file.
Regardless, does anyone have any advice/intuition on what may be going on here? I don’t know what the ‘_data’ key would refer to in this case, and am looking for help. Thanks!

Context:
Hey @cccx3,
Thank you for creating this topic! There was an error in the 01_getting_started_pytorch where it installed datasets 1.6.0, which has some changes, and in the DLC currently 1.5.0 is installed.
This was already fixed in a PR 11 yesterday.
To solve this you need to install datasets==1.5.0 in your notebook and pre-process the data again.
!pip install "sagemaker>=2.31.0" "transformers==4.4.2" "datasets[s3]==1.5.0" --upgrade


Is there a way I can use Transformers 3.1.0 in SageMaker?

Question:
I’m trying to use the following package in SageMaker: GitHub - georgian-io/Multimodal-Toolkit: Multimodal model for text and tabular data with HuggingFace t 2
I’m actually not sure if it will be possible unless I make a PR to make it work with Transformers 4.6.1 or I build my own multimodal network.
They used Transformers 3.1.0 and haven’t updated it since. Is there a way around this so that I can use multimodal transformers in SageMaker?
Thanks!

Context:
Hey @JacquesThibs,
Sadly there is no DLC for version 3.1.0 the first DLC contains transformers 4.4.2.
Could elaborate a bit more on what you want to do? Maybe can find a workaround.
Additionally, it is always possible to provide a custom requirements.txt where you could downgrade transformers to 3.1.0.
Do you want to use this for training or inference?


Question:
This is a continuation of my post here 5. I’m trying to deploy BERT for text classification with tensorflow. When I use the model.deploy() method, I can successfully get inferences from BERT. Here’s my problem: I have four different models for classification and I want to run them on the same instance, not multiple instances, to save on cost. So I tried using the MultiDataModel class, but I keep getting the following error:

image979×1189 72.2 KB

The CloudWatch logs don’t add any additional information, unfortunately. Here’s the structure of counterargument.tar.gz in the s3 bucket, which I cloned from my HuggingFace account and zipped.
counterargument.tar.gz
    config.json
    special_tokens_map.json
    tf_model.h5
    tokenizer.json
    tokenizer_config.json
    vocab.txt

The most puzzling thing about this error is that model.deploy() worked fine, but multi_model.deploy() doesn’t! Thanks in advance.

Context: @wsunadawong when using Multi-Model-Endpoints SageMaker stores the models differently. That’s why model.deploy() seems to work and MME does not. We (Amazon & HF) are looking into it. Hopefully, we can come back with a fix as soon as possible!


Question:
Hello-
I am working on deploying a speech-recognition app using HuggingFace following the instructions here 1. My understanding is that the inference toolkit uses pipelines, but the speech-recognition is only introduced with the > 4.9.0 releases, whereas the current AWS images are pointing to 4.6.x.
Is there any way around this? What do you think suggest that I do to make the deployment work? My hunch is that I need to supply a new image_uri.
Thank you!
Deniz
.

Context:
Hello @dzorlu,
Great to hear that you are working on a speech task!! Yes, the inference toolkit uses the pipelines from transformers. The code is open source you want to take a deeper look GitHub - aws/sagemaker-huggingface-inference-toolkit 2.
I am happy to share that we are working on new releases for the DLC, which include 4.9 and higher. Sadly I think it will take around 2 more weeks to be around.
In the meantime, you could use the official DLC and provide as model_data a model.tar.gz which contains a custom module, documented here: Deploy models to Amazon SageMaker 1
With a custom module, you can provide a requirements.txt to upgrade the dependencies and then provide a inference.py with a custom model_fn to load the asr pipeline.


InternalServerException when running a model loaded on S3

Question:
Hi there,
I am trying to deploy a model loaded on S3, following the steps found mainly on this video: [Deploy a Hugging Face Transformers Model from S3 to Amazon SageMaker](https://www.youtube.com/watch?v=pfBGgSGnYLs).
For that I have downloaded a model into a S3 bucket and use this image URI for DLC: image_uri = “763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-pytorch-inference:1.7.1-transformers4.6.1-cpu-py36-ubuntu18.04 1”
When I run the predictor.predict(data) command, I get this error:

Captura de pantalla 2021-08-05 a las 18.13.381168×1127 179 KB

The model I use fot these tests is this one: dccuchile/bert-base-spanish-wwm-uncased, and I could not find the way for letting the model know which action should perform.
I am pretty new with HuggingFace technology, and probably I am missing the point for fixing that.
Please, could you let me know what should I do for informing the model about what to do?
Thank you!

Context:
Hey @vicente-m47 ,
Could you please provide the whole code you executed?
If you want to deploy a model from Hugging Face – The AI community building the future. 1 you can use the “deploy” button on each of the model pages.

image1456×382 55.7 KB

This will generate a code snippet for you

image1455×1159 178 KB

From reading the code you attached you are trying to send an input for question-answering to a model (dccuchile/bert-base-spanish-wwm-uncased), which is not fine-tuned for question-answering, also you are sending an English input to a Spanish model.
A good starting point for new Hugging Facer is our course at Transformer models - Hugging Face Course.
You can find more information about deploying to sagemaker in the documentation here Deploy models to Amazon SageMaker 1


Question:
I am running a Hugging Face training job (which works locally) in a container running in EC2.  When I call fit on the estimator I get the error:
An error occurred (ValidationException) when calling the CreateTrainingJob operation: TrainingImageConfig with TrainingRepositoryAccessMode set to VPC must be provided when using a training image from a private Docker registry. Please provideTrainingImageConfig and TrainingRepositoryAccessMode set to VPC when using a training image from a private Docker registry.
The variables mentioned do not exist in the documentation and I can’t find it in the source code: TrainingImageConfig, TrainingRepositoryAccessMode.
There are some mentions for vpc_config and vpcConfig but there doesn’t seem to be a way to pass these things through to SM from HF.
My code is basically this:
hyperparameters = {'epochs': 1,
                   'per_device_train_batch_size': 32,
                   'model_name_or_path': model_name
                   }
huggingface_estimator = HuggingFace(
    entry_point='train.py',
    source_dir='./scripts',
    instance_type=instance_type,
    instance_count=1,
    role=role,
    image_uri='docker.artifactory.xxxx.com/yyyy/mlaeep/mlaeep.0.0.1-dev',
    transformers_version='4.4',
    # pytorch_version='1.6',
    py_version='py36',
    hyperparameters=hyperparameters
)
huggingface_estimator.fit(
    {'train': training_input_path,
     'test': test_input_path
    },
    job_name='MlAeepTrainer'
)

Context: @philschmid or @OlivierCR may be able to help with this.


Question:
Hey! Sorry to post again. This error is really over my head. The script works when on just one GPU. However, after adding in the argument for distributed training, it results in some really unusual errors. I essentially just added the distribution argument and changed instance type and # of instances. Am I missing something here? This is being run on the run _summarization.py. I also deleted some parts of the log that weren’t relevant because of the space limitations for # of characters in a post. Do I potentially have to use the wrapper script you have in the git examples folder that is meant to be used for distributed training? Thanks a lot!
from sagemaker.huggingface import HuggingFace

hyperparameters={
    'model_name_or_path': 'google/pegasus-large',
    'train_file': "/opt/ml/input/data/train/final_aws_deepgram_train.csv",
    'test_file': "/opt/ml/input/data/test/final_aws_deepgram_test.csv",
    'validation_file': "/opt/ml/input/data/validation/final_aws_deepgram_validation.csv",
    'text_column': 'document',
    'summary_column': 'summary',
    'do_train': True,
    'do_eval': True,
    'fp16': True,
    'per_device_train_batch_size': 2,
    'per_device_eval_batch_size': 2,
    'evaluation_strategy': "steps",
    'eval_steps': 1000,
    'weight_decay': 0.01,
    'learning_rate': 2e-5,
    'max_grad_norm': 1,
    'max_steps': 2000,
    'max_source_length': 500,
    'max_target_length': 100,
    'load_best_model_at_end': True,
    'output_dir': '/opt/ml/model'
}

# configuration for running training on smdistributed Data Parallel
distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git', 'branch': 'v4.6.1'} #'branch': 'v4.6.1'

# instance configurations
instance_type='ml.p3.16xlarge'
instance_count=2
volume_size=200

# estimator
huggingface_estimator = HuggingFace(entry_point='run_summarization_original.py',
                                    source_dir='transformers/examples/pytorch/summarization',
                                    git_config=git_config,
                                    instance_type=instance_type,
                                    instance_count=instance_count,
                                    volume_size=volume_size,
                                    role=role,
                                    transformers_version='4.6.1',
                                    pytorch_version='1.7.1',
                                    py_version='py36',
                                    distribution= distribution,
                                    hyperparameters = hyperparameters)

2021-06-22 21:50:12,132 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command "mpirun --host algo-1:8,algo-2:8 -np 16 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 2 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_HOMOGENEOUS=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.6/site-packages/gethostname.cpython-36m-x86_64-linux-gnu.so -x SMDATAPARALLEL_SERVER_ADDR=algo-1 -x SMDATAPARALLEL_SERVER_PORT=7592 -x SAGEMAKER_INSTANCE_TYPE=ml.p3.16xlarge smddprun /opt/conda/bin/python3.6 -m mpi4py run_summarization_original.py --do_eval True --do_train True --eval_steps 1000 --evaluation_strategy steps --fp16 True --learning_rate 2e-05 --load_best_model_at_end True --max_grad_norm 1 --max_source_length 500 --max_steps 2000 --max_target_length 100 --model_name_or_path google/pegasus-large --output_dir /opt/ml/model --per_device_eval_batch_size 2 --per_device_train_batch_size 2 --summary_column summary --test_file /opt/ml/input/data/test/final_aws_deepgram_test.csv --text_column document --train_file /opt/ml/input/data/train/final_aws_deepgram_train.csv --validation_file /opt/ml/input/data/validation/final_aws_deepgram_validation.csv --weight_decay 0.01"
Warning: Permanently added 'algo-2,10.2.251.196' (ECDSA) to the list of known hosts.#015
[1,13]<stderr>:#0150 tables [00:00, ? tables/s][1,0]<stderr>:#0150 tables [00:00, ? tables/s][1,13]<stderr>:#0151 tables [00:00,  7.31 tables/s][1,13]<stderr>:#015                                #015[1,13]<stderr>:#0150 tables [00:00, ? tables/s][1,13]<stderr>:#015                            #015[1,13]<stderr>:#0150 tables [00:00, ? tables/s][1,13]<stderr>:#015                            #015[1,0]<stderr>:#0151 tables [00:00,  7.17 tables/s][1,0]<stderr>:#015                                #015[1,0]<stderr>:#0150 tables [00:00, ? tables/s][1,0]<stderr>:#015                            #015[1,0]<stderr>:#0150 tables [00:00, ? tables/s][1,13]<stderr>:#015Downloading:   0%|          | 0.00/3.09k [00:00<?, ?B/s][1,13][00:00<00:00, 3.60MB/s]
[1,0]<stderr>:#015                            #015[1,0]<stderr>:https://huggingface.co/google/pegasus-large/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp2ojn0fqy
[1,8]<stderr>:loading configuration file https://huggingface.co/google/pegasus-large/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286

[1,0]<stderr>:#015Downloading:   0%|          | 0.00/3.09k [00:00<?, ?B/s][1,0]<stderr>:#015Downloading: 100%|ââââââââââ| 3.09k/3.09k [00:00<00:00, 2.57MB/s]
[1,0]<stderr>:storing https://huggingface.co/google/pegasus-large/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
[1,0]<stderr>:creating metadata file for /root/.cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
[1,0]<stderr>:loading configuration file https://huggingface.co/google/pegasus-large/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
[1,0]<stderr>:Model config PegasusConfig {
[1,0]<stderr>:  "_name_or_path": "google/pegasus-large",
[1,0]<stderr>:  "activation_dropout": 0.1,
[1,0]<stderr>:  "activation_function": "relu",
[1,0]<stderr>:  "add_bias_logits": false,
[1,0]<stderr>:  "add_final_layer_norm": true,
[1,0]<stderr>:  "architectures": [
[1,0]<stderr>:    "PegasusForConditionalGeneration"
[1,0]<stderr>:  ],
[1,8]<stderr>:loading weights file https://huggingface.co/google/pegasus-large/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/ef3a8274e003ba4d3ae63f2728378e73affec0029e797c0bbb80be8856130c4f.a99cb24bd92c7087e95d96a1c3eb660b51e498705f8bd068a58c69c20616f514
[1,0]<stderr>:loading weights file https://huggingface.co/google/pegasus-large/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/ef3a8274e003ba4d3ae63f2728378e73affec0029e797c0bbb80be8856130c4f.a99cb24bd92c7087e95d96a1c3eb660b51e498705f8bd068a58c69c20616f514
[1,12]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,15]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,9]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,8]<stderr>:All model checkpoint weights were used when initializing PegasusForConditionalGeneration.
[1,8]<stderr>:      "max_position_embeddings": 1024
[1,8]<stderr>:    },
[1,8]<stderr>:    "summarization_reddit_tifu": {
[1,8]<stderr>:      "length_penalty": 0.6,
[1,8]<stderr>:      "max_length": 128,
[1,8]<stderr>:      "max_position_embeddings": 512
[1,8]<stderr>:    },
[1,8]<stderr>:    "summarization_wikihow": {
[1,8]<stderr>:      "length_penalty": 0.6,
[1,8]<stderr>:      "max_length": 256,
[1,8]<stderr>:      "max_position_embeddings": 512
[1,8]<stderr>:    },
[1,8]<stderr>:    "summarization_xsum": {
[1,8]<stderr>:      "length_penalty": 0.8,
[1,8]<stderr>:      "max_length": 64,
[1,8]<stderr>:      "max_position_embeddings": 512
[1,8]<stderr>:    }
[1,8]<stderr>:  },
[1,8]<stderr>:  "transformers_version": "4.6.1",
[1,8]<stderr>:  "use_cache": true,
[1,8]<stderr>:  "vocab_size": 96103
[1,8]<stderr>:
[1,8]<stderr>:All the weights of PegasusForConditionalGeneration were initialized from the model checkpoint at google/pegasus-large.
[1,8]<stderr>:If your task is similar to the task the model of the checkpoint was trained on, you can already use PegasusForConditionalGeneration for predictions without further training.
[1,10]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,8]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,11]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,14]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,13]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,12]<stderr>:#015 50%|█████     | 1/2 [00:01<00:01,  1.94s/ba][1,15]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.35s/ba][1,12]<stderr>:#015100%|██████████| 2/2 [00:02<00:00,  1.56s/ba][1,12]<stderr>:#015100%|██████████| 2/2 [00:02<00:00,  1.30s/ba][1,12]<stderr>:
[1,12]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,12]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.28ba/s][1,12]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.27ba/s]
[1,10]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.75s/ba][1,8]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.83s/ba][1,15]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.84s/ba][1,15]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.51s/ba][1,15]<stderr>:
[1,5]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,6]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,15]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,9]<stderr>:#015 50%|█████     | 1/2 [00:03<00:03,  3.09s/ba][1,4]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,3]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,2]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,1]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,7]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,12]<stderr>:#015Downloading:   0%|          | 0.00/2.17k [00:00<?, ?B/s][1,12]<stderr>:#015Downloading: 5.61kB [00:00, 2.18MB/s]                   [1,12]<stderr>:
[1,11]<stderr>:#015 50%|█████     | 1/2 [00:03<00:03,  3.04s/ba][1,0]<stderr>:All model checkpoint weights were used when initializing PegasusForConditionalGeneration.
[1,0]<stderr>:
[1,0]<stderr>:All the weights of PegasusForConditionalGeneration were initialized from the model checkpoint at google/pegasus-large.
[1,0]<stderr>:If your task is similar to the task the model of the checkpoint was trained on, you can already use PegasusForConditionalGeneration for predictions without further training.
[1,0]<stderr>:#015  0%|          | 0/2 [00:00<?, ?ba/s][1,14]<stderr>:#015 50%|█████     | 1/2 [00:03<00:03,  3.27s/ba][1,15]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  1.96ba/s][1,15]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  1.95ba/s][1,15]<stderr>:
[1,13]<stderr>:#015 50%|█████     | 1/2 [00:03<00:03,  3.48s/ba][1,8]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.24s/ba][1,8]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.85s/ba][1,8]<stderr>:
[1,10]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.22s/ba][1,10]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.87s/ba][1,10]<stderr>:
[1,10]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,8]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,9]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.43s/ba][1,9]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.01s/ba][1,9]<stderr>:
[1,11]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.40s/ba][1,11]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.97s/ba][1,11]<stderr>:
[1,9]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,11]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,10]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.23ba/s][1,10]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.21ba/s][1,10]<stderr>:
[1,8]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.18ba/s][1,8]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.18ba/s]
[1,14]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.55s/ba][1,14]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.07s/ba]
[1,9]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.84ba/s][1,9]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.83ba/s]
[1,14]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,13]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.62s/ba][1,13]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.05s/ba][1,13]<stderr>:
[1,11]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.78ba/s][1,11]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.75ba/s][1,11]<stderr>:
[1,13]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,14]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.90ba/s][1,14]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.89ba/s]
[1,13]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  5.18ba/s][1,13]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  5.17ba/s]
[1,5]<stderr>:#015 50%|█████     | 1/2 [00:01<00:01,  1.97s/ba][1,8]<stderr>:max_steps is given, it will override any value given in num_train_epochs
[1,8]<stderr>:Using amp fp16 backend
[1,5]<stderr>:#015100%|██████████| 2/2 [00:02<00:00,  1.56s/ba][1,5]<stderr>:#015100%|██████████| 2/2 [00:02<00:00,  1.28s/ba]
[1,6]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.55s/ba][1,5]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,2]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.70s/ba][1,1]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.79s/ba][1,3]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.81s/ba][1,5]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.81ba/s][1,5]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.81ba/s][1,5]<stderr>:
[1,4]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.87s/ba][1,7]<stderr>:#015 50%|█████     | 1/2 [00:02<00:02,  2.85s/ba][1,5]<stderr>:#015Downloading:   0%|          | 0.00/2.17k [00:00<?, ?B/s][1,5]<stderr>:#015Downloading: 5.61kB [00:00, 1.62MB/s]
[1,6]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.06s/ba][1,6]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.74s/ba]
[1,2]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.11s/ba][1,2]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.72s/ba]
[1,6]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,2]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,4]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.23s/ba][1,4]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.80s/ba][1,4]<stderr>:
[1,1]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.19s/ba][1,1]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.79s/ba][1,1]<stderr>:
[1,7]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.22s/ba][1,7]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.79s/ba][1,7]<stderr>:
[1,4]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,3]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  2.21s/ba][1,3]<stderr>:#015100%|██████████| 2/2 [00:03<00:00,  1.82s/ba][1,1]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,3]<stderr>:
[1,6]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.08ba/s][1,6]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  4.07ba/s][1,6]<stderr>:
[1,3]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,7]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,0]<stderr>:#015 50%|█████     | 1/2 [00:03<00:03,  3.68s/ba][1,2]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.76ba/s][1,2]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.75ba/s][1,2]<stderr>:
[1,4]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.01ba/s][1,4]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.88ba/s][1,4]<stderr>:
[1,7]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.09ba/s][1,7]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  3.08ba/s]
[1,1]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.31ba/s][1,1]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.27ba/s]
[1,3]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.84ba/s][1,3]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  2.84ba/s]
[1,0]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.73s/ba][1,0]<stderr>:#015100%|██████████| 2/2 [00:04<00:00,  2.09s/ba]
[1,8]<stderr>:}
[1,8]<stderr>:
[1,0]<stderr>:loading configuration file https://huggingface.co/google/pegasus-large/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
[1,0]<stderr>:Model config PegasusConfig {
[1,0]<stderr>:  "_name_or_path": "google/pegasus-large",
[1,0]<stderr>:  "activation_dropout": 0.1,
[1,0]<stderr>:  "activation_function": "relu",
[1,0]<stderr>:  "add_bias_logits": false,
[1,0]<stderr>:  "add_final_layer_norm": true,
[1,0]<stderr>:  "architectures": [
[1,0]<stderr>:    "PegasusForConditionalGeneration"
[1,0]<stderr>:  ],
[1,0]<stderr>:  "attention_dropout": 0.1,
[1,0]<stderr>:  "bos_token_id": 0,
[1,0]<stderr>:  "classif_dropout": 0.0,
[1,0]<stderr>:  "classifier_dropout": 0.0,
[1,0]<stderr>:  "d_model": 1024,
[1,0]<stderr>:  "decoder_attention_heads": 16,
[1,0]<stderr>:  "decoder_ffn_dim": 4096,
[1,0]<stderr>:  "decoder_layerdrop": 0.0,
[1,0]<stderr>:  "decoder_layers": 16,
[1,0]<stderr>:  "decoder_start_token_id": 0,
[1,0]<stderr>:  "dropout": 0.1,
[1,0]<stderr>:  "encoder_attention_heads": 16,
[1,0]<stderr>:  "encoder_ffn_dim": 4096,
[1,0]<stderr>:  "encoder_layerdrop": 0.0,
[1,0]<stderr>:  "encoder_layers": 16,
[1,0]<stderr>:  "eos_token_id": 1,
[1,0]<stderr>:  "extra_pos_embeddings": 1,
[1,0]<stderr>:  "force_bos_token_to_be_generated": false,
[1,0]<stderr>:  "forced_eos_token_id": 1,
[1,0]<stderr>:  "gradient_checkpointing": false,
[1,0]<stderr>:  "id2label": {
[1,0]<stderr>:#015  0%|          | 0/1 [00:00<?, ?ba/s][1,0]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  5.17ba/s][1,0]<stderr>:#015100%|██████████| 1/1 [00:00<00:00,  5.16ba/s]
[1,0]<stderr>:max_steps is given, it will override any value given in num_train_epochs
[1,0]<stderr>:Using amp fp16 backend
[1,0]<stderr>:***** Running training *****
[1,0]<stderr>:  Num examples = 1558
[1,0]<stderr>:  Num Epochs = 41
[1,0]<stderr>:  Instantaneous batch size per device = 2
[1,0]<stderr>:  Total train batch size (w. parallel, distributed & accumulation) = 32
[1,0]<stderr>:  Gradient Accumulation steps = 1
[1,0]<stderr>:  Total optimization steps = 2000
[1,0]<stderr>:#015  0%|          | 0/2000 [00:00<?, ?it/s][1,8]<stderr>:***** Running training *****
[1,8]<stderr>:  Num examples = 1558
[1,8]<stderr>:  Num Epochs = 41
[1,8]<stderr>:  Instantaneous batch size per device = 2
[1,8]<stderr>:  Total train batch size (w. parallel, distributed & accumulation) = 32
[1,8]<stderr>:  Gradient Accumulation steps = 1
[1,0]<stderr>:    "0": "LABEL_0",
[1,0]<stderr>:    "1": "LABEL_1",
[1,0]<stderr>:    "2": "LABEL_2"
[1,0]<stderr>:  },
[1,0]<stderr>:  "init_std": 0.02,
[1,0]<stderr>:  "is_encoder_decoder": true,
[1,0]<stderr>:  "label2id": {
[1,0]<stderr>:    "LABEL_0": 0,
[1,0]<stderr>:    "LABEL_1": 1,
[1,0]<stderr>:    "LABEL_2": 2
[1,0]<stderr>:  },
[1,0]<stderr>:  "length_penalty": 0.8,
[1,0]<stderr>:  "max_length": 256,
[1,0]<stderr>:  "max_position_embeddings": 1024,
[1,0]<stderr>:  "model_type": "pegasus",
[1,0]<stderr>:  "normalize_before": true,
[1,0]<stderr>:  "normalize_embedding": false,
[1,0]<stderr>:  "num_beams": 8,
[1,0]<stderr>:  "num_hidden_layers": 16,
[1,0]<stderr>:  "pad_token_id": 0,
[1,0]<stderr>:  "scale_embedding": true,
[1,0]<stderr>:  "static_position_embeddings": true,
[1,0]<stderr>:  "task_specific_params": {
[1,0]<stderr>:    "summarization_aeslc": {
[1,0]<stderr>:      "length_penalty": 0.6,
[1,0]<stderr>:      "max_length": 32,
[1,0]<stderr>:      "max_position_embeddings": 512
[1,0]<stderr>:    },
[1,0]<stderr>:    "summarization_arxiv": {
[1,0]<stderr>:      "length_penalty": 0.8,
[1,0]<stderr>:      "max_length": 256,
[1,8]<stderr>:  Total optimization steps = 2000
[1,8]<stderr>:#015  0%|          | 0/2000 [00:00<?, ?it/s][1,8]<stderr>:#015  0%|          | 1/2000 [00:07<4:03:57,  7.32s/it][1,0]<stderr>:#015  0%|          | 1/2000 [00:07<4:07:45,  7.44s/it][1,8]<stderr>:#015  0%|          | 2/2000 [00:09<3:17:17,  5.92s/it][1,0]<stderr>:#015  0%|          | 2/2000 [00:10<3:19:39,  6.00s/it][1,0]<stderr>:#015  0%|          | 3/2000 [00:11<2:33:21,  4.61s/it][1,8]<stderr>:#015  0%|          | 3/2000 [00:11<2:33:05,  4.60s/it][1,0]<stderr>:#015  0%|          | 4/2000 [00:12<2:01:18,  3.65s/it][1,8]<stderr>:#015  0%|          | 4/2000 [00:12<2:01:26,  3.65s/it][1,0]<stderr>:#015  0%|          | 5/2000 [00:15<1:47:36,  3.24s/it][1,8]<stderr>:#015  0%|          | 5/2000 [00:15<1:47:43,  3.24s/it]--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 6 in communicator MPI COMMUNICATOR 5 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[algo-1:00046] 13 more processes have sent help message help-mpi-api.txt / mpi-abort
[algo-1:00046] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

MPI_ABORT was invoked on rank 6 in communicator MPI COMMUNICATOR 5 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[algo-1:00046] 13 more processes have sent help message help-mpi-api.txt / mpi-abort
[algo-1:00046] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

2021-06-22 21:50:32 Failed - Training job failed
ProfilerReport-1624398094: Stopping
2021-06-22 21:50:42,166 sagemaker-training-toolkit INFO     MPI process finished.
2021-06-22 21:50:42,166 sagemaker-training-toolkit INFO     Reporting training SUCCESS
---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<ipython-input-10-7e1bcc378f37> in <module>
      3   {'train': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_train.csv',
      4    'test': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_test.csv',
----> 5   'validation': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_validation.csv'}
      6 )

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    680         self.jobs.append(self.latest_training_job)
    681         if wait:
--> 682             self.latest_training_job.wait(logs=logs)
    683
    684     def _compilation_job_name(self):

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
   1623         # If logs are requested, call logs_for_jobs.
   1624         if logs != "None":
-> 1625             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   1626         else:
   1627             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3694
   3695         if wait:
-> 3696             self._check_job_status(job_name, description, "TrainingJobStatus")
   3697             if dot:
   3698                 print()

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3254                 ),
   3255                 allowed_statuses=["Completed", "Stopped"],
-> 3256                 actual_status=status,
   3257             )
   3258

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-06-22-21-41-34-638: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "mpirun --host algo-1:8,algo-2:8 -np 16 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 2 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_HOMOGENEOUS=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.6/site-packages/gethostname.cpython-36m-x86_64-linux-gnu.so -x SMDATAPARALLEL_SERVER_ADDR=algo-1 -x SMDATAPARALLEL_SERVER_PORT=7592 -x SAGEMAKER_INSTANCE_TYPE=ml.p3.16xlarge smddprun /opt/conda/bin/python3.6 -m mpi4py run_summarization_original.py --do_eval True --do_train True --eval_steps 1000 --evaluation_strategy steps --fp16 True --learning_rate 2e-05 --load_best_model_at_end True --max_grad_norm 1 --max_source_length 500 --max_steps 2000 --max_target_length 100 --m

Context:
Hey @ujjirox,
Could you upload the logs as files maybe? When running distributed training it sometimes happen that the real error is way above the exit.
Without seeing the full error. It might be possible that your batch_size is too big. When scaling up from p3.2xlarge to p3.16xlarge (same GPUs) SageMaker might use more of the GPU memory for the distribution.


Question:
Hi,
I cannot for the life of me figure out what is going wrong. I am following the tutorial 08 distributed training and when sending the job to AWS the error continually shows up. I am running it on a local jupyter notebook with these setups:

image1718×1356 272 KB

…
[1,0]:storing https://huggingface.co/facebook/bart-large-cnn/resolve/main/pytorch_model.bin in cache at /root/.cache/huggingface/transformers/4ccdf4cdc01b790f9f9c636c7695b5d443180e8dbd0cbe49e07aa918dda1cef0.fa29468c10a34ef7f6cfceba3b174d3ccc95f8d755c3ca1b829aff41cc92a300
[1,0]:creating metadata file for /root/.cache/huggingface/transformers/4ccdf4cdc01b790f9f9c636c7695b5d443180e8dbd0cbe49e07aa918dda1cef0.fa29468c10a34ef7f6cfceba3b174d3ccc95f8d755c3ca1b829aff41cc92a300
[1,5]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,7]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,3]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,0]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,1]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,4]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
[1,6]:Environment variable SAGEMAKER_INSTANCE_TYPE is not set
…
[1,6]:#015100%|ââââââââââ| 1/1 [00:00<00:00,  8.65ba/s][1,6]:#015100%|ââââââââââ| 1/1 [00:00<00:00,  8.62ba/s]
[1,7]:#015Downloading:   0%|          | 0.00/2.17k [00:00<?, ?B/s][1,7]:#015Downloading: 5.61kB [00:00, 5.02MB/s]
[1,0]:#015Downloading:   0%|          | 0.00/2.17k [00:00<?, ?B/s][1,0]:#015Downloading: 5.61kB [00:00, 4.54MB/s]
[1,0]:Using amp fp16 backend
[1,0]:***** Running training *****
[1,0]:  Num examples = 14732
[1,0]:  Num Epochs = 3
[1,0]:  Instantaneous batch size per device = 4
[1,0]:  Total train batch size (w. parallel, distributed & accumulation) = 32
[1,0]:  Gradient Accumulation steps = 1
[1,0]:  Total optimization steps = 1383
[1,0]:#015  0%|          | 0/1383 [00:00<?, ?it/s][1,0]:#015  0%|          | 1/1383 [00:02<59:19,  2.58s/it][1,0]:#015  0%|          | 2/1383 [00:04<51:42,  2.25s/it][1,0]:#015  0%|          | 3/1383 [00:05<45:46,  1.99s/it][1,0]:#015  0%|          | 4/1383 [00:07<45:13,  1.97s/it][1,0]:#015  0%|          | 5/1383 [00:09<43:12,  1.88s/it][1,0]:#015  0%|          | 6/1383 [00:10<41:53,  1.83s/it][1,0]:#015  1%|          | 7/1383 [00:12<41:48,  1.82s/it][1,0]:#015  1%|          | 8/1383 [00:14<43:16,  1.89s/it][1,0]:#015  1%|          | 9/1383 [00:16<44:01,  1.92s/it][1,0]:#015  1%|          | 10/1383 [00:18<44:03,  1.93s/it][1,0]:#015  1%|          | 11/1383 [00:20<44:13,  1.93s/it][1,0]:#015  1%|          | 12/1383 [00:22<44:31,  1.95s/it][1,0]:#015  1%|          | 13/1383 [00:24<44:06,  1.93s/it][1,0]:#015  1%|          | 14/1383 [00:26<43:58,  1.93s/it][1,0]:#015  1%|          | 15/1383 [00:28<43:42,  1.92s/it][1,0]:#015  1%|          | 16/1383 [00:30<43:46,  1.92s/it][1,0]:#015  1%|          | 17/1383 [00:32<43:47,  1.92s/it]--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 5 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
2021-07-29 04:41:50 Uploading - Uploading generated training model
2021-07-29 04:41:50 Failed - Training job failed
…
UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-07-29-04-30-08-439: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command “mpirun --host algo-1 -np 8 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 1 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_SINGLENODE=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.6/site-packages/gethostname.cpython-36m-x86_64-linux-gnu.so smddprun /opt/conda/bin/python3.6 -m mpi4py run_summarization.py --dataset_name samsum --do_eval True --do_predict True --do_train True --fp16 True --learning_rate 5e-05 --model_name_or_path facebook/bart-large-cnn --num_train_epochs 3 --output_dir /opt/ml/model --per_device_eval_batch_size 4 --per_device_train_batch_size 4 --predict_with_generate True --seed 7”
[1,2]:Environment variable SAGEMAKER_INSTANCE_TYPE is n
Any help would be greatly appreciated. The CloudWatch logs don’t really have anything else to say. Definitely seems like a problem with setting the environment variable SAGEMAKER_INSTANCE_TYPE but I thought I already set it by specifying instance_type when initializing huggingface_estimator??
Thanks

Context:
Hey @cdwyer1bod,
Thanks for opening the thread. Happy to help you.
Could still share the full cloudwatch logs? sometimes the errors are a bit hidden.
I saw you changed the instance ml.p3dn.24xlarge to ml.p3.16xlarge and kept the batch_size this could be the issue. Could reduce the batch_size to 2 or change the instances type?


FP16 doesn’t reduce Trainer Training time

Question:
Hi,
I’m using this SageMaker HF sample notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub 5 adjusted with train_batch_size = 128, tested on both 1 p3.16xlarge and 1 p4d.24xlarge. For each instance I’m doing a job with fp16=True and a job without the flag. GPU usage is a erratic (sawtooth oscillating between 50 and 750%)
The impact of fp16=True is only a 1% training time reduction, on each instance. Is it because:

Not specifying fp16 in the trainer already uses fp16? (seems to be false by default though)
There is a lot of CPU work & I/O in that demo that will not leverage float16?
Transformers don’t benefit from fp16 training?

Context:
Hey @OlivierCR,
can you please share all of your hyperparameters you used? tran_batch_size = 128 seems pretty high to me to work.
Are you using distributed training?
The training time reduction should be way higher more around 20-40%.
When using the example have you adjusted train.py to accept fp16 as hyperparameter or have you defined it directly in the script?


ValueError: Source directory does not exist in the repo. Training causal lm in sagemaker

Question:
ValueError: Source directory does not exist in the repo. (Looks like the link is broken).
I am getting the error above when I am trying to train a text generation model in the sagemaker.
Please see the script and configuration that I am using:
import sagemaker
from sagemaker.huggingface import HuggingFace
git_config = {‘repo’: ‘https://github.com/huggingface/transformers.git’,'branch’: ‘v4.6.1’} # v4.6.1 is referring to the transformers_version you use in the estimator.

gets role for executing training job
role = sagemaker.get_execution_role()
hyperparameters = {
‘model_name_or_path’:‘ktangri/gpt-neo-demo’,
‘output_dir’:’/opt/ml/model’,
‘fp16’: True,
‘train_file’: ‘/opt/ml/input/data/train/train.csv’,
‘validation_file’: ‘/opt/ml/input/data/validation/validation.csv’
# add your remaining hyperparameters
# more info here https://github.com/huggingface/transformers/tree/v4.6.1/examples/language-modeling
}
#configuration for running training on smdistributed Data Parallel
distribution = {‘smdistributed’:{‘dataparallel’:{ ‘enabled’: True}}}

git configuration to download our fine-tuning script
git_config = {‘repo’: ‘https://github.com/huggingface/transformers.git’,'branch’: ‘v4.6.1’}

creates Hugging Face estimator
huggingface_estimator = HuggingFace(
entry_point=‘run_clm.py’,
source_dir=’/examples/language-modeling’,
instance_type=‘ml.p3.16xlarge’,
instance_count=2,
role=role,
git_config=git_config,
transformers_version=‘4.6.1’,
pytorch_version=‘1.7.1’,
py_version=‘py36’,
hyperparameters = hyperparameters
)
huggingface_estimator.fit(
{‘train’: ‘s3://ch-questions-dataset-east1/train/train.csv’,
‘validation’: ‘s3://ch-questions-dataset-east1/validation/validation.csv’}
)
Thank you.

Context:
Thanks for pointing this out with transformers 4.6.1 the examples/ structure changed.
It is for source_dir now →
source_dir=’examples/pytorch/language-modeling’,

We’ll fix the code snippet on the hub.


Running custom data files on run_summarization.py

Question:
Hi there,
I have been running a script to train a pretrained transformer on a summarization task. I am using custom data which I have put into my S3 bucket which is also the default bucket for this job.
I have been getting this error in return, and have not been able to figure out what the solution is. I have run the exact same script on the xsum dataset just to see if it’s the custom dataset that is the issue, and I can indeed confirm that the job works when using the xsum dataset.
from sagemaker.huggingface import HuggingFace

hyperparameters={
    'model_name_or_path': 'google/pegasus-large',
    'train_file': "/opt/ml/input/data/train/final_aws_deepgram_train.csv",
    'test_file': "/opt/ml/input/data/test/final_aws_deepgram_test.csv",
    'validation_file': "/opt/ml/input/data/validation/final_aws_deepgram_validation.csv",
    'text_column': 'document',
    'summary_column': 'summary',
    'do_train': True,
    'do_eval': True,
    'fp16': True,
    'per_device_train_batch_size': 2,
    'per_device_eval_batch_size': 2,
    'evaluation_strategy': "steps",
    'eval_steps': 200,
    'weight_decay': 0.01,
    'learning_rate': 2e-5,
    'max_grad_norm': 1,
    'max_steps': 200,
    'max_source_length': 500,
    'max_target_length': 100,
    'load_best_model_at_end': True,
    'output_dir': '/opt/ml/model'
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git', 'branch': 'v4.6.1'} #'branch': 'v4.6.1'

# instance configurations
instance_type='ml.p3.2xlarge'
instance_count=1
volume_size=200

# metric definition to extract the results
metric_definitions=[
     {"Name": "train_runtime", "Regex": "train_runtime.*=\D*(.*?)$"},
     {'Name': 'train_samples_per_second', 'Regex': "train_samples_per_second.*=\D*(.*?)$"}
]

huggingface_estimator = HuggingFace(entry_point='run_summarization_original.py',
                                    source_dir='transformers/examples/pytorch/summarization',
                                    git_config=git_config,
                                    metric_definitions=metric_definitions,
                                    instance_type=instance_type,
                                    instance_count=instance_count,
                                    volume_size=volume_size,
                                    role=role,
                                    transformers_version='4.6.1',
                                    pytorch_version='1.7.1',
                                    py_version='py36',
                                    hyperparameters = hyperparameters)

# starting the train job
huggingface_estimator.fit(
  {'train': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_train.csv',
   'test': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_test.csv',
  'validation': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_validate.csv'}
)

2021-06-22 15:34:39 Starting - Starting the training job...
2021-06-22 15:35:03 Starting - Launching requested ML instancesProfilerReport-1624376073: InProgress
.........
2021-06-22 15:36:33 Starting - Preparing the instances for training.........
2021-06-22 15:38:06 Downloading - Downloading input data
2021-06-22 15:38:06 Training - Downloading the training image.....................
2021-06-22 15:41:34 Uploading - Uploading generated training model
2021-06-22 15:41:34 Failed - Training job failed
..
---------------------------------------------------------------------------
UnexpectedStatusException                 Traceback (most recent call last)
<ipython-input-7-ca8819244de5> in <module>
      3   {'train': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_train.csv',
      4    'test': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_test.csv',
----> 5   'validation': 's3://qfn-transcription/ujjawal_files/final_aws_deepgram_validate.csv'}
      6 )

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    680         self.jobs.append(self.latest_training_job)
    681         if wait:
--> 682             self.latest_training_job.wait(logs=logs)
    683
    684     def _compilation_job_name(self):

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
   1623         # If logs are requested, call logs_for_jobs.
   1624         if logs != "None":
-> 1625             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
   1626         else:
   1627             self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
   3683
   3684         if wait:
-> 3685             self._check_job_status(job_name, description, "TrainingJobStatus")
   3686             if dot:
   3687                 print()

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
   3243                 ),
   3244                 allowed_statuses=["Completed", "Stopped"],
-> 3245                 actual_status=status,
   3246             )
   3247

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-06-22-15-34-33-634: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 run_summarization_original.py --do_eval True --do_train True --eval_steps 200 --evaluation_strategy steps --fp16 True --learning_rate 2e-05 --load_best_model_at_end True --max_grad_norm 1 --max_source_length 500 --max_steps 200 --max_target_length 100 --model_name_or_path google/pegasus-large --output_dir /opt/ml/model --per_device_eval_batch_size 2 --per_device_train_batch_size 2 --summary_column summary --test_file /opt/ml/input/data/test/final_aws_deepgram_test.csv --text_column document --train_file /opt/ml/input/data/train/final_aws_deepgram_train.csv --validation_file /opt/ml/input/data/validation/final_aws_deepgram_validation.csv --weight_decay 0.01"
Traceback (most recent call last):
  File "run_summarization_original.py", line 606, in <module>
    main()
  File "run_summarization_original.py", line 325, in main
    datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)
  File "/opt/conda/lib/p

Thanks!

Context:
Hey @ujjirox,
Is the error log you attached the whole error log? They should be more, maybe in cloudwatch.
Additionally
2021-06-22 15:34:39 Starting - Starting the training job...
2021-06-22 15:35:03 Starting - Launching requested ML instancesProfilerReport-1624376073: InProgress
.........
2021-06-22 15:36:33 Starting - Preparing the instances for training.........
2021-06-22 15:38:06 Downloading - Downloading input data
2021-06-22 15:38:06 Training - Downloading the training image.....................
2021-06-22 15:41:34 Uploading - Uploading generated training model
2021-06-22 15:41:34 Failed - Training job failed

 → shows that you ran training for 4minutes and also SageMaker tried to upload your model. Maybe there was an issue with uploading the model.


Using custom csv data with run_summarization.py in sagemaker

Question:
Hi -
I am trying to use the sagemaker Huggingface estimator to fine-tune a model for summarization using the run_summarization.py 4 entry_point. I have created a Sagemaker Studio notebook based on the code from the summarization example notebook 3 provided in the Sagemaker examples.
I would like to use my own data to train the model and so have added the following code to make train and validation datasets I have uploaded to s3 storage available to the estimator:
Define s3 locations:
training_input_path = "s3://sagemaker-eu-central-1-88888888888/train_20210607.csv"
test_input_path = "s3://sagemaker-eu-central-1-88888888888/val_20210607.csv"

Define file locations in hyperparameters:
hyperparameters={
    ...,
    'train_file': '/opt/ml/input/data/train_20210607.csv',
    'validation_file': '/opt/ml/input/data/val_20210607.csv',
    'text_column': 'document',
    'summary_column': 'summary',
    ...
}

Ensure data is loaded when starting training job:
huggingface_estimator.fit({'train': training_input_path, 'test': test_input_path})
It looks from the log that the data has been loaded into the expected directory:
SM_HP_VALIDATION_FILE=/opt/ml/input/data/val_20210607.csv
SM_HP_TRAIN_FILE=/opt/ml/input/data/train_20210607.csv

However, after the run_summarization.py script is run, I get the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/train_20210607.csv'
Apologies if I am missing something obvious, but would be great if anyone could let me know how I should be referencing my data so that it can be used by the run_summarization.py script?
Thank you!
Ben

Context:
Hey @benG,
Do you have the full error log? like where the error is thrown in the run_summarization.py.
And which transformers_version are you using and how does your git_config looks like?


‘DistributedDataParallel’ object has no attribute ‘no_sync’

Question:
Hi,
I am trying to fine-tune layoutLM using with the following:
distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}

estimator = HuggingFace(
    entry_point = 'train.py',
    py_version = 'py36',
    transformers_version='4.4.2',
    pytorch_version='1.6.0',
    role = role,
    instance_type='ml.p3.16xlarge',
    instance_count=1,
    checkpoint_s3_uri=checkpoint_dir,
    checkpoint_local_path='/opt/ml/checkpoints',
    hyperparameters = {'epochs': 3,
                       'batch-size': 16,
                       'learning-rate': 5e-5,
                       'use-cuda': True,
                       'model-name':'microsoft/layoutlm-base-uncased'
                      },
    debugger_hook_config=False,
    volume_size = 40,
    distribution = distribution,
    source_dir = source_dir)

estimator.fit({'input_data_dir': data_uri}, wait = True)

Relevant code in train.py file:
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased',num_labels = len(labels))

training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=4,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    warmup_ratio=0.1,
    weight_decay=0.01,
    report_to='wandb',
    run_name = 'test_run',
    logging_steps = 500,
    fp16 = True,
    load_best_model_at_end = True,
    evaluation_strategy = 'steps',
    gradient_accumulation_steps = 1,
    save_steps = 500,
    save_total_limit = 5,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    data_collator = data_collator,
    compute_metrics=compute_metrics,
    callbacks = [EarlyStoppingCallback]
)

trainer.train()

Unfortunately I keep getting the following error. Tried tracking down the problem but cant seem to figure it out.
[1,7]<stdout>:Traceback (most recent call last):
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,7]<stdout>:    "__main__", mod_spec)
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,7]<stdout>:    exec(code, run_globals)
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,7]<stdout>:    main()
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,7]<stdout>:    run_command_line(args)
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,7]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,7]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,7]<stdout>:    mod_name, mod_spec, pkg_name, script_name)
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,7]<stdout>:    exec(code, run_globals)
[1,7]<stdout>:  File "train.py", line 619, in <module>
[1,7]<stdout>:    trainer.train()
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1050, in train
[1,7]<stdout>:    with model.no_sync():
[1,7]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 800, in __getattr__
[1,7]<stdout>:    type(self).__name__, name))
[1,7]<stdout>:torch.nn.modules.module.ModuleAttributeError: 'DistributedDataParallel' object has no attribute 'no_sync'

Any help would be appreciated!

Context: Oh and running the same code without the ddp and using a 1 GPU instance works just fine but obviously takes much longer to complete


Where is SageMaker Distributed configured in HF Trainer?

Question:
I see that the HF Trainer run_qa script 1 is compatible with SageMaker Distributed Data Parallel, but I don’t see where is it configured?
In particular, I can see in the training_args 1 that smdist gets imported and configured, but where is the model wrapped with smdist DDP?
According to the smdist doc 1 the below snippet is a required step ; I’d like to understand where it’s done with HF Trainer
from smdistributed.dataparallel.torch.parallel.distributed import DistributedDataParallel as DDP
model = DDP(Net().to(device))

Context:
Hey @OlivierCR,
both the SageMaker Distributed Data-Parallel and the Model-Parallel library are directly integrated into the Trainer API, which uses and initializes both libraries automatically.
For SMD:

The library is first imported with an alias for the default PyTorch DDP library here 5

and then wrapps the model here 3

P.S. The _wrap_model() 1 function also handles SMP


“No space left on device” when using HuggingFace + SageMaker

Question:
Hi, I’m trying to train a model using a HuggingFace estimator in SageMaker but I keep getting this error after a few minutes:
[1,15]:  File “pyarrow/ipc.pxi”, line 365, in pyarrow.lib._CRecordBatchWriter.write_batch
[1,15]:  File “pyarrow/error.pxi”, line 97, in pyarrow.lib.check_status
[1,15]:OSError: [Errno 28] Error writing bytes to file. Detail: [errno 28] No space left on device
[1,15]:
I’m not sure what is triggering this problem because the volume size is high     (volume_size=1024)
My hyperparameters are:
{‘per_device_train_batch_size’: 4,
‘per_device_eval_batch_size’: 4,
‘model_name_or_path’: ‘google/mt5-small’,
‘dataset_name’: ‘mlsum’,
‘dataset_config’: ‘es’,
‘text_column’: ‘text’,
‘summary_column’: ‘summary’,
‘max_target_length’: 64,
‘do_train’: True,
‘do_eval’: True,
‘do_predict’: True,
‘predict_with_generate’: True,
‘output_dir’: ‘/opt/ml/model’,
‘num_train_epochs’: 3,
‘seed’: 7,
‘fp16’: True,
‘save_strategy’: ‘no’}
And my estimator is:

create the Estimator
huggingface_estimator = HuggingFace(
entry_point=‘run_summarization.py’, # script
source_dir=’./examples/seq2seq’, # relative path to example
git_config=git_config,
instance_type=‘ml.p3.16xlarge’,
instance_count=2,
volume_size=1024,
transformers_version=‘4.4.2’,
pytorch_version=‘1.6.0’,
py_version=‘py36’,
role=role,
hyperparameters = hyperparameters,
distribution = distribution
)
Any help would be very much appreciated!

Some more details:

I’m calling fit without extra params, just like this:
huggingface_estimator.fit()
The entry point is this public script:
transformers/run_summarization.py at master · huggingface/transformers · GitHub 11

From traceback I see that the error is happening on line 433:
load_from_cache_file=not data_args.overwrite_cache,
(I guess something is happening here but not totally sure what)
At the moment I’m not saving checkpoints (to prevent that causing the error), using the param  ‘save_strategy’: ‘no’
The dataset isn’t that big, 1.7 GB.
The model is quite big, but less than 3 GB
My volume is 1024 GB

Context: Could you also include your .fit() call so that the example can be reproduced? And a link to the run_summarization.py if public? Do you have a sense of what could take up a lot of storage? do you checkpoint a large model very frequently? or do you read a large dataset?


NER on SageMaker Ground Truth annotations

Question: Anybody has a public sample showing how to run NER on annotations coming from SageMaker Ground Truth NER 9?

Context:
Hey @OlivierCR,
Sorry, I don’t have an example using NER annotations coming from SageMaker Ground Truth NER.
Using the example you have sent me

HF Datasets (conll2003)
{‘chunk_tags’: [11, 21, 11, 12, 21, 22, 11, 12, 0],
‘id’: ‘0’,
‘ner_tags’: [3, 0, 7, 0, 0, 0, 7, 0, 0],
‘pos_tags’: [22, 42, 16, 21, 35, 37, 16, 21, 7],
‘tokens’: [‘EU’,
‘rejects’,
‘German’,
‘call’,
‘to’,
‘boycott’,
‘British’,
‘lamb’,
‘.’]}
SM Ground Truth NER 1 (doc)
{ “crowd-entity-annotation”: { “entities”: [ { “endOffset”: 26, “label”: “software”, “startOffset”: 0 }, { “endOffset”: 38, “label”: “version”, “startOffset”: 35 }, { “endOffset”: 88, “label”: “software”, “startOffset”: 84 }, { “endOffset”: 90, “label”: “version”, “startOffset”: 89 }, { “endOffset”: 93, “label”: “version”, “startOffset”: 92 }, { “endOffset”: 100, “label”: “version”, “startOffset”: 98 } ] } }

You could use the load_dataset 1 to load your JSON files coming from SM Ground Truth and then use dataset.map() 1 to iterate through it and adjust it to the datasets format


Question:
Hey Everyone!
Thanks for the great application!
I was wondering, are there plans to upgrade the streamlit library to a more recent verison e.g. 1.4?
Many thanks

Context:
Yes, we’ll be doing this and it’s in the roadmap.
cc @cbensimon


Space stopped working due to uvicorn reload issue

Question:
Hello I have a space that is running TTS demo using gradio.
Spaces Link 1
I didn’t do anything to the space but on restarting it is giving me the below error:

Screenshot 2022-01-26 at 7.04.10 PM1802×314 44.7 KB

Context: The space is up now. We got some issues yesterday but restarted all broken spaces.


Build error: failed to create endpoint sleepy_shirley on network bridge

Question:
I’m struggling to understand this error, any assistance?

Build error

failed to create endpoint sleepy_shirley on network bridge: adding interface vethb79c97b to bridge docker0 failed: exchange full
Here’s my Spaces:

      huggingface.co

Recommendations_Metacritic_Scores - a Hugging Face Space by seyia92coding 5

  Discover amazing ML apps made by the community

Context: The space seems to be working now after being restarted


Question:
Hey guys,
How does one record audio from the browser in streamlit? Nothing fancy, just audio capture with a push of a button similar to how every speech-to-text model on HF Hub has it, for example, facebook/wav2vec2-base-960h · Hugging Face 1
TIA,
Vladimir

Context:
Hey there! This question might be a better fit in the Streamlit forums or Discord. I found this dicussions from some time ago which I hope helps!

      Streamlit – 15 Sep 20

Record sound from the user's microphone with streamlit 1

  Hello. If audio input is possible with stremlit,  the scalability of machine learning project will be much wider.  I would like to know if this feature is currently in progress as a development project  inside streamlit to make it possible for...

    Reading time: 1 mins 🕑
      Likes: 5 ❤


Question:
Hi,
There’s a build error when I accessing my space app. It seems the error caused by the streamlit installation.  It’s working fine beforehand, until today this error popped out.

      huggingface.co

Language Detection Xlm Roberta Base - a Hugging Face Space by ivanlau 2

  Discover amazing ML apps made by the community

error log:

image1493×270 18.6 KB

Any thoughts on fixing this?
Thanks

Context: I restarted it in “Settings” and it’s working now


Question:
Hello
I have some troubles with installing dependencies, the Space can’t resolve the correct versions from the requirements.txt and I think this is due to missing wheels for certain OS.
It’d be great to know on which OS the apps are run.
Thanks and kind regards!

Context:
Hi @edichief! @cbensimon might be able to help you a bit here, but let me provide some tips. You can click in “build error” to see all the logs, although in this case it might not be super useful. I’m not sure which is the exact OS being used, but it’s linux based.
In my local Ubuntu computer, doing pip install -r requirements.txt in a new environment with your requirements give me exactly the same error.
Looking in indexes: https://pypi.org/simple, https://packagecloud.io/github/git-lfs/pypi/simple
Collecting spacy==3.1.4
  Downloading spacy-3.1.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.1 MB)
     |████████████████████████████████| 6.1 MB 4.4 MB/s
ERROR: Could not find a version that satisfies the requirement holmes-extractor==3.0.0 (from -r requirements.txt (line 2)) (from versions: 2.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.1.0, 2.2.0, 2.2.1)
ERROR: No matching distribution found for holmes-extractor==3.0.0 (from -r requirements.txt (line 2))


Build error : failed to create endpoint hungry_leavitt on network bridge: adding interface veth1291603 to bridge docker0 failed: exchange full

Question:
huggingface.co

ohayo_face2 - a Hugging Face Space by Reeve 2

  Discover amazing ML apps made by the community

This is my space.
A build error occurred. But I don’t know how to solve this error.

      huggingface.co

ohayo_face - a Hugging Face Space by Reeve 3

  Discover amazing ML apps made by the community

And this is my other space,
There was no error in this space, but the error occurred as soon as I changed the readme.
commit/3f1b52788af3f3d06698c453f4567a96b955f733
I created a test space, wrote a very simple code, and built it.

      huggingface.co

Test - a Hugging Face Space by Reeve 2

  Discover amazing ML apps made by the community

However, the test space did not work either.
I don’t think this is my problem.

Context:
It may be a server-wide problem.
The same happens to mine radiobee aligner - a Hugging Face Space by mikeee 2 which ran Okay previously.
Logs say Successfully built 989912a9c4b6 Successfully tagged space-cpu-mikeee/radiobee-aligner-8dd688d44271d792c7b84f717e8e285aae89f07c:latest
Yet the app does not run –  a Build error shows up.


Question:
Hello!
I tried updating our gadio implementation in our HF Space, to better handle the increased traffic we have had the last few days. The build seems to be successful but we get a connection error when loading the tokenizer (although I guess that part is anecdotal and the connection error might be more general?). Here is the full traceback:
Traceback (most recent call last):
  File "app.py", line 19, in <module>
    tokenizer = AutoTokenizer.from_pretrained('gpt2')
  File "/home/user/.local/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 464, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/home/user/.local/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 330, in get_tokenizer_config
    resolved_config_file = cached_path(
  File "/home/user/.local/lib/python3.8/site-packages/transformers/file_utils.py", line 1491, in cached_path
    output_path = get_from_cache(
  File "/home/user/.local/lib/python3.8/site-packages/transformers/file_utils.py", line 1715, in get_from_cache
    raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

Any ideas what we can do to fix this?
Thanks in advance,
Theodore.

Context:
Hi Theodore!
This same as the same issue as the one discussed in Build error : failed to create endpoint hungry_leavitt on network bridge: adding interface veth1291603 to bridge docker0 failed: exchange full - #2 by mikeee 8. Looking into it!


Question:
I tried my times checked everything but still I getting this error please someone help to fix it

20220103_2133511080×532 148 KB

Context: These errors are for GPU-related things. Your space is running under CPU, so that would explain the errors


Plot Graphs on Gradio Partially Cut off on X- Axis

Question:
Hi,
The boxplot on the output of my gradio interface cuts off just below the x-axis so you can’t see the titles.

image1233×491 32.2 KB

See the code here - app.py · seyia92coding/Popular_Spotify_Albums at main 2
Has anyone had much luck with seaborn plot outputs with Gradio?
Thanks

Context: Can you try plt.tight_layout() and let me know if it fixes things?


How to install a specific version of gradio in Spaces?

Question:
Hi,
The last version of gradio (2.7.0) comes with a display bug (launch this notebook diff_texts.ipynb 1 from gradio at HighlightedText 1).

image1827×921 52.3 KB

With the version 2.6.4 of gradio, there was no display bug:

image1815×787 39.6 KB

However, when you create an App with gradio, Hugging Face Spaces says that the gradio package comes pre-installed at version 2.2.6. I think this text needs an update. (cc @sgugger)

image1914×337 36.1 KB

More, I did try to create a new Spaces App with gradio. In the requirements.txt file, I added gradio==2.6.4 but it did not work.

image1166×550 29.4 KB

How can I install the gradio version of my choice for a Spaces App? Thank you.
Note: see this issue on gradio github about the version 2.7.0 and HighlightedText 2.

Context:
Sorry about that! We’re fixing the bug but for now you can install a specific version of Gradio by adding lines like this at the top of your app.py file:
import os
os.system("pip uninstall -y gradio")
os.system("pip install gradio==2.6.4")


Question:
Hi,
I am getting very weird behaviour in spaces.
The same code is working in colab but not in spaces due to numpy version mismatch.
Error:

Screenshot 2022-01-04 at 11.14.48 PM2284×866 127 KB

Colab Link where code is working: Google Colab
Spaces Link

Context:
os.system('pip uninstall -y numpy')
os.system('pip install numpy')
import gradio as grd

Using the above code in the exact order solved my problem.


Can I create a confidential app using Spaces?

Question:
Hello there!
I am intrigued by spaces but I am wondering if I can create an app that is only accessible to me (and not to Huggingface’s staff for instance) and to others with a password. Is this level of privacy possible?
Thanks!

Context:
You can create a private space. Private spaces are visible to only you (personal space) or members of your organization (organization space) as per HF.
Maybe they are visible to HF staff too, I am not sure about it.
If you don’t want to create space on HF, you can do it on gradio.app 1 or streamlit.io 2


Question: I have a model/app deployment that works perfectly on my local machine and errors out after 1-2 seconds upon pressing “submit” on the hosted Gradio app. Where can I check a log of what is going on? I have no clue… and the documentation for spaces seems incomplete, at least from what I found.

Context:
Hello @pszemraj
Can you send me the link to your Space?
If you’re using it with gradio try setting debug = True when launching (one thing I can think of).


Gradio fn function errors right after 60 seconds

Question:
Always exactly after 60 seconds since execution, the function I pass to gradio.interface errors, and in the web console I get a JSON parsing error. I set enable_queue to True but that didn’t seem to change anything.
Here’s how I launch:
iface = gr.Interface(f, [
    "text",
    temperature,
    top_p,
    gr.inputs.Slider(
        minimum=20, maximum=512, default=30, label="max length"),
    gr.inputs.Dropdown(["GPT-J-6B", "GPT-2"], type="index", label="model"),
    gr.inputs.Textbox(lines=1, placeholder="xxxxxxxx", label="space verification key")

], outputs="text", title=title, examples=examples)
iface.launch(enable_queue=True)

How can I prevent it timing out after 60 seconds every time the function takes long?

^as soon as that reaches 60, it errors

Context:
You can see Is there a timeout (max runtime) for spaces? - #2 by Epoching or the Gradio docs. You need to set enable_queue to True for longer inference

enable_queue (bool) - if True, inference requests will be served through a queue instead of with parallel threads. Required for longer inference times (> 1min) to prevent timeout.


Question:
hi all,
hf spaces gradio iframe embedding appears useful, seems to work fine but does not seem to be documented well.
are there plans to expland the gradio documentation for hf spaces?
if yes, is there a place to make suggestions?
in good spirits,
luis

Context:
Hi @luisoala, one of the developers of Gradio here. Thanks for your question - the Gradio website has some documentation on hosting on Spaces (Gradio 1), but we can definitely improve it.
What improvements would you like to see?


Question:
Hey All, I wonder if there is any way to show python output
—i.e.logs or print(‘Using device:’, device)—
in Spaces.
Thanks in advance

Context:
Hi,
You can check the output by clicking on the “building” or “running” button next to the name of the Space.

Schermafbeelding 2021-11-10 om 11.05.471164×196 27.2 KB


Access a folder in my spaces thru app.py

Question:
hi … im not sure if this is considered a beginners question or not but
im trying to load a model into my app.py which exists in files sections …
folder name containing the model is  “spacy.aravec.model”… i tried runing it using :
nlp = spacy.load(“spacy.aravec.model”)
but got an error :
ValueError: Expected object or value

Screen Shot 2021-12-22 at 10.07.15 PM710×573 72.6 KB

Context:
Hi @Ralfouzan! It seems like you uploaded meta.json with GIT LFS so the call to read_json is causing some issues here.
Would you be able to delete that file, and re-upload it without using git lfs to see if that alleviates the issue?


Question:
I need to include my API key to use the inference API, but how do I do that without making my key publicly visible in my space’s files?
I don’t want to make it private.

Context:
You can add secrets to have access to variables such as API keys in your Space

If your app requires secret keys or tokens, don’t hard-code them inside your app! Instead, go to the Settings page of your Space repository and enter your secrets there. The secrets will be exposed to your app with Streamlit Secrets Management 1 if you use Streamlit, and as environment variables in other cases.


Gradio save_state problem and introducing a basic chatbot

Question:
Hello all,
I have uploaded a more basic version of my chatbot to https://huggingface.co/spaces/gorkemgoknar/moviechatbot 2 .
If you checkout app.py code you can see how you can use a transformers model types which is not included in standart interface (do not forget requirements.txt to import modules)
I also shared my model and information to use  https://huggingface.co/gorkemgoknar/gpt2chatbotenglish
A more feature rich version (including movie and character selection) is on :  metayazar.com/chatbot  with Turkish version on https://www.metayazar.com/chatbot_tr
While building gradio app I noticed that save_state(history) get_state() does not work in embedded application. While gradio app is full screen it works but in HF spaces information is lost (likely due to Flask session not being statefull inside embedded fram).
To overcome this I use a global history but likely this would make models history being shared between different users (not a problem for a demo and it may be fun), yet for information there should be a way to save state if HF Spaces would be a different product.

Context: Thanks for the detailed feedback @gorkemgoknar! I one of the Gradio developers and I’ll share this feedback with my team so that we can fix it. (We are working on a better Chatbot output component altogether!)


Is there a timeout (max runtime) for spaces?

Question:
I’ve been trying to make a 3d photo inpainting project work on huggingface spaces. After finally getting PyQt5 working with a headless display (lots of fun debugging via subprocess calls via python in app.py, since we don’t have access to the shell 0.0), it turns out spaces automatically timesout at around ~60 seconds?
The documentation said to use enable_queue=True in the iface.launch(enable_queue=True) function for gradio, but this doesn’t seem to help. Does HF Spaces automatically drop requests that span longer than 60 seconds? (the gradio timer on the spaces page counts up to 60, then prints error afterwards).
Checking the logs does show that the inference is able to finish running, but won’t be able to return anything back to the front-end via gradio since the connection gets cut.
Spaces currently set to private, but if any HF devs want to take a look & have access to private instances, here’s the link  3D_Photo_Inpainting - a Hugging Face Space by Classified
Error on request:
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 319, in run_wsgi
    execute(self.server.app)
  File "/home/user/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 311, in execute
    write(data)
  File "/home/user/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 290, in write
    self.wfile.write(data)
  File "/usr/local/lib/python3.8/socketserver.py", line 826, in write
    self._sock.sendall(b)
TimeoutError: [Errno 110] Connection timed out

Context:
Update 2 / Solution:
So turns out using enable_queue=True in launch() doesn’t work, e.g:
iface = gr.Interface(fn=..., inputs=..., outputs=...)
iface.launch(enable_queue=True)

After looking at the source code, I saw that enable_queue is a deprecated argument in Inference()
Doing the following fixed the 60 second timeout for me:
iface = gr.Interface(fn=..., inputs=..., outputs=..., enable_queue=True)
iface.launch()


Question:
I was wondering if there are any additional resources aside from the hf docs here for setting up an automatic pipeline between github and spaces? I keep running into a bug and I am not sure how to resolve it so I was hoping someone might be able to point me in the right direction.
I have been trying to sync my github repo and my hf space using github actions but unfortunately keep getting the following error:
<i>Run git push ***[huggingface.co/spaces/pleonova/multi-label-long-text](http://huggingface.co/spaces/pleonova/multi-label-long-text) main
remote: Invalid username or password.
fatal: Authentication failed for 'https://huggingface.co/spaces/pleonova/multi-label-long-text/'
Error: Process completed with exit code 128.</i>

I added the actions to my workflow. I have also tried removing pleonova in the url but that does not work either. (same error as above). I added the hf token to my github repo environment as well as enabled all actions in the actions permissions.
Any advice would be greatly appreciated!

Context: This was solved in (Spaces) Unable to sync with github actions · Issue #534 · huggingface/huggingface_hub · GitHub 8


Hosting a HF Space for Ultra-Large Language Models

Question:
Hi all!
My collaborators and I would like to host a research project demo on spaces. The challenge is that we operate on ultra-large language models such as GPT-2 XL (requires ~6GB VRAM) and GPT-J-6B (~24GB VRAM). Our code itself does not use much VRAM outside of the loading the model (it basically makes some user-specified changes to model and lets users generate text using the new model).
It seems like we can fit GPT-2 XL into our 16 GB allowance for T4s, but what about GPT-J-6B? How is this even hosted at EleutherAI/gpt-j-6B · Hugging Face 3?
Thanks for your insight

Context: Discussed over email but let’s also try to paste relevant discussion items here in the future so that interesting discussions can be publicly available


Question:
I tried using import cv2 and various combinations of opencv and cv2 codes in the requirements.txt as well as the app.py.
I get this ImportError:
  File "/home/user/.local/lib/python3.8/site-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

I’ve seen a couple of Spaces that have the same error, and have not been able to see a Space that does deploy a program with a cv2 successfully.  I would suggest leads on how to address this Import Error, or some samples of working code.
Thank you!
Maria

Context:
Hey there!
You can create a packages.txt file and use python3-opencv there. I think this worked for me last time I checked. Let me know if that works please
Thanks,
Omar


Widespread server errors on various Spaces? Site-wide?

Question:
Tried all the Spaces of the Week, they all give some form of error. So far today I’ve seen mostly 500, but also a few others including 504.
I’m guessing this is known already, but I haven’t seen an announcement here or on Twitter. Where would I go to check for the latest?

Context: resolved, apparently.


Question:
Hey,
I have no technical knowledge about all this stuff, just a “consumer”
So when i go to the spaces tab (Spaces - Hugging Face 1) it seems like the list only goes back to Nov. 25. Is that the whole list of spaces or does it cut off from there? Sadly there seems to be no ability to search via tags (like video/photo/audio/etc). How can i discover some more cool spaces that are created before Nov. 25?
thanks in advance!

Context:
Agree. I hope there’s a  search or filter widget above the space (circle region) so that we can search what we want easily (such as which demo is built from streamlit/gradio, tags etc.).

image1904×616 149 KB

Thanks


Uploading large files (>5GB) to HF Spaces

Question:
Hi everyone,
I need to use an 8.5 GB faiss index for my HF Spaces app. Although I’ve successfully uploaded large files to datasets and model hub before, I am now getting an error when pushing to HF Spaces repo. I read all the issues and troubleshooting others had with git lfs, but this one I really couldn’t solve after trying for some time.
Is this the recommended approach for large faiss files (pushing with git lfs)? I am using a cloud server so upload speed shouldn’t be an issue. It seems like the first 5 GB are transferred ok, and then towards the end of the transfer, I notice “400 Bad request” error logs. Here are my full git logs.
TIA, Vladimir

Context: i don’t remember, did we implement lfs-largefiles for Spaces or no? @cbensimon @pierric


Copy to Clipboard doesn’t work in Spaces

Question:
We are trying to use copy to clipboard functionality in HF Spaces, following is the code:
import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS
from streamlit_bokeh_events import streamlit_bokeh_events
from datetime import datetime

current_time = datetime.now().strftime("%H:%M:%S")
text = f"## Current time: {current_time}"
st.write(text)
copy_button = Button(label="Copy Text")
copy_button.js_on_event("button_click", CustomJS(args=dict(text=text), code="""
    navigator.clipboard.writeText(text);
    """))

no_event = streamlit_bokeh_events(
    copy_button,
    events="GET_TEXT",
    key="get_text",
    refresh_on_update=True,
    override_height=75,
    debounce_time=0)

It works well locally, but it doesn’t work on Spaces. It seems copy to clipboard is blocked in an iframe (errror message in the browser console: “Uncaught (in promise) DOMException: The Clipboard API has been blocked because of a permissions policy applied to the current document. See Deprecating Permissions in Cross-Origin Iframes - The Chromium Projects for more details.”), it needs to add specific allowance in iframe, which I don’t know how to do it. Maybe someone has a solution?

Context: maybe this discussion is relevant Feature-Policy: clipboard-read and clipboard-write · Issue #322 · w3c/webappsec-permissions-policy · GitHub 1


Can one allow Spaces-hosted apps to take up the entire screen?

Question:
This is a bit of a cheeky feature request maybe…
Is it possible/planned to allow streamlit apps hosted on Spaces to take the entirety of the screen i.e. removing the components on the top of the page (see what I show below)?

image708×215 14.5 KB

Ideally, I would a URL that allows to access the app without the elements above. It’d be great to reduce “clutter” when demoing things (maybe at the price of paying for the premium or having some small HF logo a la “Made with Streamlit” “Hosted on Hugging Face Spaces” on the bottom of the page).
Thanks for the great work with Spaces!

Context: @cbensimon @victor for info


Any way to install transformers in a PR branch?

Question:
Currently, my PR on FlaxVisionEncoderDecoderModel  is not merged to transformers master yet. But I can’t wait to update my poor-french-image-captioning Spaces demo app.
So I need to
git clone https://github.com/ydshieh/transformers.git
cd transformers
git checkout flax_vision_encoder_decoder
pip install --upgrade .[dev]

Is this doable in Spaces currently? I can probably upload a copy of transformers to avoid git  part, but still need  pip install.
Otherwise, I think I can still figure out a hard way to make the demo work.

Context: I saw that we can specify the branch/fork in the requirements.txt, I will give it a try.


What stack is used to create chat boxes in conversational model cards?

Question:
Hello!
I was wondering if the frontend components used in conversational model cards on the HF website are open-sourced or not.
Take e.g. microsoft/DialoGPT-small · Hugging Face page (shown below) it looks great! It’d be nice to reuse it to show chatbot prototypes, as it doesn’t include distracting entities (e.g. the name of the sender, timestamp, sent/delivered/read marks, etc.).

image863×902 74.4 KB

Streamlit doesn’t have a dialog component AFAIK and I was wondering if the underlying code could be adapted to create custom components a la Create a Component - Streamlit Docs 1, e.g. to power chatbots on HF Spaces. I have seen bots implementations in streamlit, but the ones I saw are way less polished than this.
Not being a frontend developer myself, I’d have a hard time creating a chat box component from scratch, so it’d be cool to reuse yours somehow!
Thanks!

Context:
Hi,
Yes the code for the widgets is open-source (and part of the huggingface_hub library), you can view the code of the conversational widget here: huggingface_hub/ConversationalWidget.svelte at main · huggingface/huggingface_hub · GitHub 6


Question: Hi @cahya , can you please add a dataset card with explanation for id_personachat 2?

Context: Ok, I’ll have a look on it. Are you working on a chatbot project btw?


[Announcement] Model cards metadata automatic cleaning on the hub

Question:
Hi everyone,
HuggingFace’s team started a model metadata automatic correction project !
This comes just after deploying a mode card metadata changes validation on the git server side.
Starting with licenses and invalid “null” value : and the licenses sidebar is much better organized now : Models - Hugging Face 2
Model repositories concerned present a commit authored from me (“elishowk”) with “Automatic correction” message.
Now every license in the hub fits into the documented list
Coming-soon : automatic correction for languages and model-index.
Feel free to ask for any question.
Regards

Context: That’s awesome!


Question:
Not sure where to report this…
For this model: google/byt5-large · Hugging Face 1
The paper link is not correct. Instead it should be this: [2105.13626] ByT5: Towards a token-free future with pre-trained byte-to-byte models

Context:
mrdrozdov:

google/byt5-large · Hugging Face

Thanks, I’ve fixed the paper link for all byT5 models on the hub.


Model card for microsoft/xprophetnet-large-wiki100-cased-xglue-qg is inccorect

Question:
Hi,
the model card for microsoft/xprophetnet-large-wiki100-cased-xglue-qg seems to be incorrect (I think it’s the one from the xglue-ntg model).
Just try out the example – it generates to ‘PAD’ tokens, and that’s it. It’s also unclear on how to use it for question generation.
I’m not sure who the creators are, so I can’t tag they here.
Best,
Niklas

Context: Maybe @patrickvonplaten could help here since he reviewed the original PR (I don’t think Weizhen - the creator - is in the forum).


How to set minimum length of generated text by widget?

Question:
Hi,
I have a T5 finetuned model I want to set min length and max length for the widget output but unfortunately  I have not found any kind of info for this on similar models or documentation. Can anyone help?

Context: Does anyone face something same?


ValueError: Tokenizer class ByT5Tokenizer does not exist or is not currently imported

Question:
Docs here suggest to use tokenizer for padding, and i really want so, but cannot.

      huggingface.co

google/byt5-xl · Hugging Face 29

Context:
Hi,
for me it works fine:

      colab.research.google.com

Google Colaboratory 77

Make sure to install Transformers from source (i.e. from Github).