Modern NLP for “Economics of Innovation” (Open Research Project using Patent Data)

Question:
Hi all,
Suraj and I started discussing a potential research project and he suggested I make a thread here to discuss. As a quick intro, I am an NLP hobbyist and consumer of NLP research, and Suraj is a software developer with a keen interest in NLP.
From my perspective, here are a few goals of the project:

Upgrade our NLP skills in general
Make an immediate contribution to an applied field by introducing modern NLP methods
Dig deeper into NLP research and potentially make a minor advancement

My idea was to introduce the Innovation Studies (or Economics of Innovation) field to modern NLP methods. I suggested this for a few reasons. First, it is generally accepted that the long-run economic growth rate, and standard of living, is driven by innovation. And second, there are about 8 million published US patents - that are freely available - that we can use as data.
I am open to any directions to take, but here are a few starting points:

Patent Classification
I can see two reasons for improving patent classifications. One is for Innovation researchers to use the improved patent classes for their research - rather than relying on officially listed patent classes. And two, would be for actual innovation policy. One consensus in the field is that basic research is drastically under-invested in, since companies do not directly benefit from the large spillovers of basic research. So the rate of return on basic research is much higher for society than for any single company. However, when governments try to encourage basic research through incentivizing these types of patents, inventors can try to “cheat the system” by re-labeling their patent. Economists Ufuk Akcigit and Stefanie Stantcheva [1] say “Going forward, finding a feasible way to differentiate between basic and applied research is essential to better innovation tax policies.”

Estimating the “Impact” of a Patent
As far as I know, the vast majority of innovation studies, that use patent data, use the number of citations as a proxy for the impact of a patent. So improving the “impact score” of a patent might help many innovation researchers. Professor Bryan Kelly et al [2] use a very clever modification of TF-IDF to find similarity scores between patents. A patent’s impact is then estimated by finding the difference in similarity scores between the target patent and all previous patents, and the target patent and all future patents. This makes sense to me, and is well explained in their paper. However, I do think that using other methods of finding patent embeddings may be worth investigating - like using AllenAI’s SPECTER document embedding approach. I’d also like to look into deep graph networks to see if they can help produce an estimate of the impact of a patent, without using citations.

Patent Idea Generation
I think it would be cool to generate a patent abstract (or idea) either unconditionally, or conditioned on a sentence that would guide the generation. There are lots of directions we could pursue with this.

Anyway, sorry for the long post. Please let us know if you have ideas, suggestions, would like to participate, etc.
[1] https://www.nber.org/chapters/c14428.pdf 3
[2] https://www.nber.org/papers/w25266.pdf 4

Context: @VictorSanh, @joeddav, @yjernite we would love to hear your thoughts on this


ACL 2020 - Some personal highlights - Victor

Question:
Hey!
I had such a blast at ACL2020 this week! So many cool works, and lots of very interesting discussions both in the chat and in the zoom Q&A sessions!
Here’s a pick of 3 of my highlights (there are extremely biased towards what I’m currently interested in):
(1) Inherent Disagreements in Human Textual Inferences 8
by Ellie Pavlick, Tom Kwiatkowski
Natural Language Inference (sometimes referred to as textual entailment) has become fundamental in evaluating language understanding and semantics. The central question of this paper is “what should we use as ground truth labels for textual inference?” The authors show that the apparent “annotation noise” often results from a multi-modality among the annotators’ labels. They discuss the implication of this uncertainty and argue for a refined evaluation that better captures the diversity of human judgments.
(2) Unsupervised Domain Clusters in Pretrained Language Models 9
by Roee Aharoni, Yoav Goldberg
The authors propose a “data-driven” approach to define what a domain is in NLP and to select in-domain data. They show that large pre-trained language models are able to capture these domains in an unsupervised way and leverage this insight to select in-domain data to train neural machine translation models.
(3) Syntactic Data Augmentation Increases Robustness to Inference Heuristics 14
by Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen
Natural Language Inference models fine-tuned on top of models like BERT show high accuracy on standard test datasets but fail on challenge sets. The authors propose a simple syntactic data augmentation procedure to augment the standard training set to up to a thousand examples. Results show great improvement (and generalization) by just exposing the model to these controlled syntactic examples supporting that hypothesis that BERT contains knowledge that simply needs to be “activated”. Cases failures (like passive) support that the idea there is also knowledge pre-trained BERT is not aware of.
How about you? Did any work change your perspective?

Context:
Hi @VictorSanh, thanks so much for your list. As the conference is overwhelming with contents, I did not see these papers at all.
In paper (3) , syntactic augmentation is very interesting since
(a) Augmentation is very successful in Computer Vision (CV), but in NLP, augmentation is much more non-obvious (regarding how to do it) and maybe sensitive to downstream tasks (more robust in CV)
(b) In the paper Section 3, author stated that the augmented examples are noisy

We did not attempt to ensure the naturalness of
the generated examples; e.g., in the INVERSION
transformation, The carriage made a lot of noise
was transformed into A lot of noise made the carriage. In addition, the labels of the augmentation
dataset were somewhat noisy; e.g., we assumed
that INVERSION changed the correct label from entailment to neutral, but this is not necessarily the
case (if The buyer met the seller, it is likely that
The seller met the buyer). As we show below, this
noise did not hurt accuracy on MNLI.

This is very interesting to me (in CV it’s often intuitively clear which augmentation is noiseless / noisy), so I assume that the ‘noisy-ratio’ is minimum since too much noise should degrade the overall performance …
Further, in CV, we also have soft-labels augmentation like MixUp and CutMix, so maybe this similar area in NLP also has more potential.
(on Kaggle we also tried our own (non-published) augmentation to NLP with this similar ideas –
e.g. In the recent Jigsaw Toxic classification competition where a paragraph of comment texts are given as an example. We can combine two paragraphs together [with Toxic + Neutral = Toxic label Formula) , or dynamic random shuffling sentences within the given paragraph where toxicity degree should be invariant with this operation.)


Question:
I took some notes on some ICLR2020 papers that seemed most relevants to my research topics: information retrieval for QA, model architectures and analysis, and text generations. You can find them here!

      docs.google.com

ICLR papers 168

Transformer architectures / pretraining losses Lite Transformer with Long-Short Range Attention Long Short Range Attention uses smaller dimension global attention in parallel with convolutions to capture local context. The approach is more...

Context:
Thanks for the great summary Yacine @yjernite .
It’s a pity that there’s no ICLR video presentation now on slidelive (maybe they deleted 1-2 weeks after the conference end ?) … Some of them can still be found on Youtube though


Question: Use this category for any research question or to coordinate on a project with other users.

Context: Thanks so much to have this category! Love it.


Question:
The original Twitter thread here 4.
The selecting criterion here is being interesting. Not an exhaustive list.
Let Me Choose: From Verbal Context to Font Selection

      aclweb.org

2020.acl-main.762.pdf 5

687.19 KB

Bridging text with its font! Very interesting application paper from Adobe. They even have emojis play a role in it! Even fonts have their semantics and sentiments.
Contextualized Weak Supervision for Text Classification

      aclweb.org

2020.acl-main.30.pdf 6

1819.53 KB

This paper cleverly introduces word disambiguation into weakly supervised text classification and the method for data augmentation is also great!
Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?

      aclweb.org

2020.acl-main.419.pdf 12

813.57 KB

This paper answered the interesting question that if machines read text just like us humans? Though the conclusion may not be surprising, it opens a new path to understand attention.
Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders

      aclweb.org

2020.acl-main.23.pdf 10

912.50 KB

Sorry for self-promoting but this paper is actually very interesting. The flexible framework can be extended to many more fields including text style transfer, image generation, voice conversion, etc.

Context: I like the human attention maps one. It’s interesting that humans have much more peaked distributions, focusing in a few key words where as the ML system attends to a larger sweet of words with varying weights.


Wav2Vec2 WER remains 1.00 and return blank transcriptions

Question:
Hello everyone. I faced a strange problem and not sure about how i can resolve the problem myself.
I was fine tuning wav2vec2 on two custom dataset and although previously it used to work fine now I am not sure what changed.
Issue#1 : The WER does not decrease, it always remains constant at 1.00 WER .
Issue#2 : When you make predictions you get blank transciptions.It happened twice and then i decided to run exactly on English Fine tuning blog post.
Fine-Tune Wav2Vec2 for English ASR with  Transformers 2
I am attaching an image below, the only change i made is on train and eval steps to experiment faster.
Please run the following colab notebook to reproduce results : Google Colaboratory 4

Context:
Hey there
I was experiencing the same problem on my own dataset. I managed to solve it by using librosa to load the raw data. librosa.load(filename, sr = 16000)
Previously i was using pydub.AudioSegment


Wav2Vec2: fix growing training and validation loss after few epochs

Question:
Hi,
I’m using Wav2Vec2ForCTC.from_pretrained(“facebook/wav2vec2-large-xlsr-53”) to fine-tune on the Lithuanian language dataset. I’ve limited the dataset to 100 hours of records in the range of 1 to 15 seconds.
I’m following the example from this notebook: Fine-Tune Wav2Vec2 for English ASR in Hugging Face with  Transformers 2 by  @patrickvonplaten.
My issue is that that the training loss and validation loss steadily decrease first few epochs and then all metrics start to worsen.
Eval loss

image1218×730 30.2 KB

Wer

image1230×680 44.9 KB

Train loss (although not so clearly visible)

image1238×740 48.7 KB

My configuration is:
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-large-xlsr-53",
    activation_dropout=0.055,
    attention_dropout=0.094,
    hidden_dropout=0.047,
    feat_proj_dropout=0.04,
    mask_time_prob=0.082,
    layerdrop=0.041,
    gradient_checkpointing=True,
    ctc_loss_reduction="mean",
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer),
)

model.freeze_feature_extractor()

training_args = TrainingArguments(
    output_dir="/workspace/models/wav2vec-lt",
    group_by_length=True,
    per_device_train_batch_size=24,
    gradient_accumulation_steps=2,
    evaluation_strategy="steps",
    num_train_epochs=30,
    fp16=True,
    save_steps=1000,
    eval_steps=1000,
    logging_steps=1000,
    learning_rate=2.34e-4,
    warmup_steps=500,
    save_total_limit=20,
    load_best_model_at_end=True,
    greater_is_better=False,
    log_level='debug',
    dataloader_num_workers=6,
    metric_for_best_model="wer",
)

trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=dataset_prepared['train'],
    eval_dataset=dataset_prepared['valid'],
    tokenizer=processor.feature_extractor,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=5, early_stopping_threshold=0.0001)],
)

Model params were taken from another wav2vec finetuning example.
At first, I had a higher learning rate, later reduced to the current one.
I was thinking of using HPT to get the params to get the best results, but I’d like to resolve this problem first.
Any advice?

Context: Have you been able to resolve the issue? Facing the same problem


Error for Training job huggingface-sdk-extension-2022-01-24-16-31-30-883: Failed. Reason: AlgorithmError: ExecuteUserScriptError:

Question:
Hi All I am getting the below error msg when trying to train Bert , any help will be great. Bit urgent.
error:-
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
Error for Training job huggingface-sdk-extension-2022-01-24-16-47-13-971: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
LOG:-
2022-01-24 16:53:12,880 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2022-01-24 16:53:12,904 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2022-01-24 16:53:15,927 sagemaker_pytorch_container.training INFO     Invoking user training script.
2022-01-24 16:53:16,395 sagemaker-training-toolkit INFO     Invoking user script
Training Env:
{
“additional_framework_parameters”: {},
“channel_input_dirs”: {
“test”: “/opt/ml/input/data/test”,
“train”: “/opt/ml/input/data/train”
},
“current_host”: “algo-1”,
“framework_module”: “sagemaker_pytorch_container.training:main”,
“hosts”: [
“algo-1”
],
“hyperparameters”: {
“train_batch_size”: 32,
“model_name”: “distilbert-base-uncased”,
“epochs”: 2
},
“input_config_dir”: “/opt/ml/input/config”,
“input_data_config”: {
“test”: {
“TrainingInputMode”: “File”,
“S3DistributionType”: “FullyReplicated”,
“RecordWrapperType”: “None”
},
“train”: {
“TrainingInputMode”: “File”,
“S3DistributionType”: “FullyReplicated”,
“RecordWrapperType”: “None”
}
},
“input_dir”: “/opt/ml/input”,
“is_master”: true,
“job_name”: “huggingface-sdk-extension-2022-01-24-16-47-13-971”,
“log_level”: 20,
“master_hostname”: “algo-1”,
“model_dir”: “/opt/ml/model”,
“module_dir”: “s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz”,
“module_name”: “train”,
“network_interface_name”: “eth0”,
“num_cpus”: 8,
“num_gpus”: 1,
“output_data_dir”: “/opt/ml/output/data”,
“output_dir”: “/opt/ml/output”,
“output_intermediate_dir”: “/opt/ml/output/intermediate”,
“resource_config”: {
“current_host”: “algo-1”,
“hosts”: [
“algo-1”
],
“network_interface_name”: “eth0”
},
“user_entry_point”: “train.py”
}
Environment variables:
SM_HOSTS=[“algo-1”]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={“epochs”:2,“model_name”:“distilbert-base-uncased”,“train_batch_size”:32}
SM_USER_ENTRY_POINT=train.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={“current_host”:“algo-1”,“hosts”:[“algo-1”],“network_interface_name”:“eth0”}
SM_INPUT_DATA_CONFIG={“test”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”},“train”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=[“test”,“train”]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=train
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz
SM_TRAINING_ENV={“additional_framework_parameters”:{},“channel_input_dirs”:{“test”:"/opt/ml/input/data/test",“train”:"/opt/ml/input/data/train"},“current_host”:“algo-1”,“framework_module”:“sagemaker_pytorch_container.training:main”,“hosts”:[“algo-1”],“hyperparameters”:{“epochs”:2,“model_name”:“distilbert-base-uncased”,“train_batch_size”:32},“input_config_dir”:"/opt/ml/input/config",“input_data_config”:{“test”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”},“train”:{“RecordWrapperType”:“None”,“S3DistributionType”:“FullyReplicated”,“TrainingInputMode”:“File”}},“input_dir”:"/opt/ml/input",“is_master”:true,“job_name”:“huggingface-sdk-extension-2022-01-24-16-47-13-971”,“log_level”:20,“master_hostname”:“algo-1”,“model_dir”:"/opt/ml/model",“module_dir”:“s3://sagemaker-eu-west-2-352316401451/huggingface-sdk-extension-2022-01-24-16-47-13-971/source/sourcedir.tar.gz”,“module_name”:“train”,“network_interface_name”:“eth0”,“num_cpus”:8,“num_gpus”:1,“output_data_dir”:"/opt/ml/output/data",“output_dir”:"/opt/ml/output",“output_intermediate_dir”:"/opt/ml/output/intermediate",“resource_config”:{“current_host”:“algo-1”,“hosts”:[“algo-1”],“network_interface_name”:“eth0”},“user_entry_point”:“train.py”}
SM_USER_ARGS=["–epochs",“2”,"–model_name",“distilbert-base-uncased”,"–train_batch_size",“32”]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TEST=/opt/ml/input/data/test
SM_CHANNEL_TRAIN=/opt/ml/input/data/train
SM_HP_TRAIN_BATCH_SIZE=32
SM_HP_MODEL_NAME=distilbert-base-uncased
SM_HP_EPOCHS=2
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages
Invoking script with the following command:
/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32
2022-01-24 16:53:21,122 - main - INFO -  loaded train_dataset length is: 572
2022-01-24 16:53:21,122 - main - INFO -  loaded test_dataset length is: 144
2022-01-24 16:53:21,457 - filelock - INFO - Lock 140311318218288 acquired on /root/.cache/huggingface/transformers/23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.91b885ab15d631bf9cee9dc9d25ece0afd932f2f5130eba28f2055b2220c0333.lock
2022-01-24 16:53:21,790 - filelock - INFO - Lock 140311318218288 released on /root/.cache/huggingface/transformers/23454919702d26495337f3da04d1655c7ee010d5ec9d77bdb9e399e00302c0a1.91b885ab15d631bf9cee9dc9d25ece0afd932f2f5130eba28f2055b2220c0333.lock
2022-01-24 16:53:22,163 - filelock - INFO - Lock 140311212156688 acquired on /root/.cache/huggingface/transformers/9c169103d7e5a73936dd2b627e42851bec0831212b677c637033ee4bce9ab5ee.126183e36667471617ae2f0835fab707baa54b731f991507ebbb55ea85adb12a.lock
2022-01-24 16:53:27,475 - filelock - INFO - Lock 140311212156688 released on /root/.cache/huggingface/transformers/9c169103d7e5a73936dd2b627e42851bec0831212b677c637033ee4bce9ab5ee.126183e36667471617ae2f0835fab707baa54b731f991507ebbb55ea85adb12a.lock
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]

This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2022-01-24 16:53:28,847 - filelock - INFO - Lock 140311146924352 acquired on /root/.cache/huggingface/transformers/0e1bbfda7f63a99bb52e3915dcf10c3c92122b827d92eb2d34ce94ee79ba486c.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock
2022-01-24 16:53:29,484 - filelock - INFO - Lock 140311146924352 released on /root/.cache/huggingface/transformers/0e1bbfda7f63a99bb52e3915dcf10c3c92122b827d92eb2d34ce94ee79ba486c.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99.lock
2022-01-24 16:53:29,811 - filelock - INFO - Lock 140311211720320 acquired on /root/.cache/huggingface/transformers/75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4.lock
2022-01-24 16:53:30,528 - filelock - INFO - Lock 140311211720320 released on /root/.cache/huggingface/transformers/75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4.lock
2022-01-24 16:53:31,518 - filelock - INFO - Lock 140311211719928 acquired on /root/.cache/huggingface/transformers/8c8624b8ac8aa99c60c912161f8332de003484428c47906d7ff7eb7f73eecdbb.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79.lock

2022-01-24 16:53:38 Uploading - Uploading generated training model2022-01-24 16:53:31,850 - filelock - INFO - Lock 140311211719928 released on /root/.cache/huggingface/transformers/8c8624b8ac8aa99c60c912161f8332de003484428c47906d7ff7eb7f73eecdbb.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79.lock
[2022-01-24 16:53:36.715 algo-1:26 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2022-01-24 16:53:36.867 algo-1:26 INFO profiler_config_parser.py:102] User has disabled profiler.
[2022-01-24 16:53:36.868 algo-1:26 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.
[2022-01-24 16:53:36.869 algo-1:26 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.
[2022-01-24 16:53:36.870 algo-1:26 INFO hook.py:255] Saving to /opt/ml/output/tensors
[2022-01-24 16:53:36.871 algo-1:26 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.
#015Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 483/483 [00:00<00:00, 460kB/s]
#015Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]#015Downloading:   2%|▏         | 4.08M/268M [00:00<00:06, 40.8MB/s]#015Downloading:   3%|▎         | 8.71M/268M [00:00<00:06, 42.3MB/s]#015Downloading:   5%|▌         | 13.5M/268M [00:00<00:05, 43.8MB/s]#015Downloading:   7%|▋         | 18.4M/268M [00:00<00:05, 45.2MB/s]#015Downloading:   9%|▊         | 23.3M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  10%|█         | 28.0M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  12%|█▏        | 32.6M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  14%|█▍        | 37.7M/268M [00:00<00:04, 47.5MB/s]#015Downloading:  16%|█▌        | 42.7M/268M [00:00<00:04, 48.4MB/s]#015Downloading:  18%|█▊        | 47.8M/268M [00:01<00:04, 49.1MB/s]#015Downloading:  20%|█▉        | 52.8M/268M [00:01<00:04, 49.2MB/s]#015Downloading:  22%|██▏       | 57.8M/268M [00:01<00:04, 49.6MB/s]#015Downloading:  23%|██▎       | 62.9M/268M [00:01<00:04, 50.0MB/s]#015Downloading:  25%|██▌       | 68.0M/268M [00:01<00:03, 50.3MB/s]#015Downloading:  27%|██▋       | 73.2M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  29%|██▉       | 78.2M/268M [00:01<00:03, 50.5MB/s]#015Downloading:  31%|███       | 83.3M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  33%|███▎      | 88.5M/268M [00:01<00:03, 50.9MB/s]#015Downloading:  35%|███▍      | 93.6M/268M [00:01<00:03, 51.1MB/s]#015Downloading:  37%|███▋      | 98.8M/268M [00:02<00:03, 51.3MB/s]#015Downloading:  39%|███▉      | 104M/268M [00:02<00:03, 50.5MB/s] #015Downloading:  41%|████      | 109M/268M [00:02<00:03, 50.5MB/s]#015Downloading:  43%|████▎     | 114M/268M [00:02<00:03, 50.7MB/s]#015Downloading:  44%|████▍     | 119M/268M [00:02<00:02, 50.8MB/s]#015Downloading:  46%|████▋     | 124M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  48%|████▊     | 130M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  50%|█████     | 135M/268M [00:02<00:02, 51.2MB/s]#015Downloading:  52%|█████▏    | 140M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  54%|█████▍    | 145M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  56%|█████▌    | 150M/268M [00:03<00:02, 51.4MB/s]#015Downloading:  58%|█████▊    | 155M/268M [00:03<00:02, 50.9MB/s]#015Downloading:  60%|█████▉    | 160M/268M [00:03<00:02, 51.0MB/s]#015Downloading:  62%|██████▏   | 166M/268M [00:03<00:02, 51.2MB/s]#015Downloading:  64%|██████▎   | 171M/268M [00:03<00:01, 51.5MB/s]#015Downloading:  66%|██████▌   | 176M/268M [00:03<00:01, 52.1MB/s]#015Downloading:  68%|██████▊   | 181M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  70%|██████▉   | 187M/268M [00:03<00:01, 52.9MB/s]#015Downloading:  72%|███████▏  | 192M/268M [00:03<00:01, 52.0MB/s]#015Downloading:  74%|███████▎  | 198M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  76%|███████▌  | 203M/268M [00:04<00:01, 52.9MB/s]#015Downloading:  78%|███████▊  | 208M/268M [00:04<00:01, 49.9MB/s]#015Downloading:  80%|███████▉  | 213M/268M [00:04<00:01, 50.8MB/s]#015Downloading:  82%|████████▏ | 219M/268M [00:04<00:00, 51.4MB/s]#015Downloading:  84%|████████▎ | 224M/268M [00:04<00:00, 52.0MB/s]#015Downloading:  86%|████████▌ | 229M/268M [00:04<00:00, 52.4MB/s]#015Downloading:  88%|████████▊ | 235M/268M [00:04<00:00, 52.7MB/s]#015Downloading:  90%|████████▉ | 240M/268M [00:04<00:00, 52.8MB/s]#015Downloading:  92%|█████████▏| 245M/268M [00:04<00:00, 52.9MB/s]#015Downloading:  94%|█████████▎| 251M/268M [00:04<00:00, 53.0MB/s]#015Downloading:  96%|█████████▌| 256M/268M [00:05<00:00, 53.1MB/s]#015Downloading:  98%|█████████▊| 261M/268M [00:05<00:00, 52.7MB/s]#015Downloading: 100%|█████████▉| 267M/268M [00:05<00:00, 51.6MB/s]#015Downloading: 100%|██████████| 268M/268M [00:05<00:00, 50.9MB/s]
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]

This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
#015Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]#015Downloading:  16%|█▌        | 36.9k/232k [00:00<00:00, 242kB/s]#015Downloading:  82%|████████▏ | 190k/232k [00:00<00:00, 321kB/s] #015Downloading: 100%|██████████| 232k/232k [00:00<00:00, 755kB/s]
#015Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]#015Downloading:  20%|█▉        | 92.2k/466k [00:00<00:00, 610kB/s]#015Downloading:  73%|███████▎  | 338k/466k [00:00<00:00, 750kB/s] #015Downloading: 100%|██████████| 466k/466k [00:00<00:00, 1.52MB/s]
#015Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]#015Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 24.3kB/s]
#015  0%|          | 0/36 [00:00<?, ?it/s]Traceback (most recent call last):
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 699, in convert_to_tensors
tensor = as_tensor(value)
ValueError: too many dimensions ‘str’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “train.py”, line 83, in
trainer.train()
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1246, in train
for step, inputs in enumerate(epoch_iterator):
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 444, in next
(data, worker_id) = self._next_data()
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 526, in _next_data
2022-01-24 16:53:37,616 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
#015Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]#015Downloading: 100%|ââââââââââ| 483/483 [00:00<00:00, 460kB/s]
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
#015Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]#015Downloading:   2%|â         | 4.08M/268M [00:00<00:06, 40.8MB/s]#015Downloading:   3%|â         | 8.71M/268M [00:00<00:06, 42.3MB/s]#015Downloading:   5%|â         | 13.5M/268M [00:00<00:05, 43.8MB/s]#015Downloading:   7%|â         | 18.4M/268M [00:00<00:05, 45.2MB/s]#015Downloading:   9%|â         | 23.3M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  10%|â         | 28.0M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  12%|ââ        | 32.6M/268M [00:00<00:05, 46.5MB/s]#015Downloading:  14%|ââ        | 37.7M/268M [00:00<00:04, 47.5MB/s]#015Downloading:  16%|ââ        | 42.7M/268M [00:00<00:04, 48.4MB/s]#015Downloading:  18%|ââ        | 47.8M/268M [00:01<00:04, 49.1MB/s]#015Downloading:  20%|ââ        | 52.8M/268M [00:01<00:04, 49.2MB/s]#015Downloading:  22%|âââ       | 57.8M/268M [00:01<00:04, 49.6MB/s]#015Downloading:  23%|âââ       | 62.9M/268M [00:01<00:04, 50.0MB/s]#015Downloading:  25%|âââ       | 68.0M/268M [00:01<00:03, 50.3MB/s]#015Downloading:  27%|âââ       | 73.2M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  29%|âââ       | 78.2M/268M [00:01<00:03, 50.5MB/s]#015Downloading:  31%|âââ       | 83.3M/268M [00:01<00:03, 50.7MB/s]#015Downloading:  33%|ââââ      | 88.5M/268M [00:01<00:03, 50.9MB/s]#015Downloading:  35%|ââââ      | 93.6M/268M [00:01<00:03, 51.1MB/s]#015Downloading:  37%|ââââ      | 98.8M/268M [00:02<00:03, 51.3MB/s]#015Downloading:  39%|ââââ      | 104M/268M [00:02<00:03, 50.5MB/s] #015Downloading:  41%|ââââ      | 109M/268M [00:02<00:03, 50.5MB/s]#015Downloading:  43%|âââââ     | 114M/268M [00:02<00:03, 50.7MB/s]#015Downloading:  44%|âââââ     | 119M/268M [00:02<00:02, 50.8MB/s]#015Downloading:  46%|âââââ     | 124M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  48%|âââââ     | 130M/268M [00:02<00:02, 51.1MB/s]#015Downloading:  50%|âââââ     | 135M/268M [00:02<00:02, 51.2MB/s]#015Downloading:  52%|ââââââ    | 140M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  54%|ââââââ    | 145M/268M [00:02<00:02, 51.3MB/s]#015Downloading:  56%|ââââââ    | 150M/268M [00:03<00:02, 51.4MB/s]#015Downloading:  58%|ââââââ    | 155M/268M [00:03<00:02, 50.9MB/s]#015Downloading:  60%|ââââââ    | 160M/268M [00:03<00:02, 51.0MB/s]#015Downloading:  62%|âââââââ   | 166M/268M [00:03<00:02, 51.2MB/s]#015Downloading:  64%|âââââââ   | 171M/268M [00:03<00:01, 51.5MB/s]#015Downloading:  66%|âââââââ   | 176M/268M [00:03<00:01, 52.1MB/s]#015Downloading:  68%|âââââââ   | 181M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  70%|âââââââ   | 187M/268M [00:03<00:01, 52.9MB/s]#015Downloading:  72%|ââââââââ  | 192M/268M [00:03<00:01, 52.0MB/s]#015Downloading:  74%|ââââââââ  | 198M/268M [00:03<00:01, 52.5MB/s]#015Downloading:  76%|ââââââââ  | 203M/268M [00:04<00:01, 52.9MB/s]#015Downloading:  78%|ââââââââ  | 208M/268M [00:04<00:01, 49.9MB/s]#015Downloading:  80%|ââââââââ  | 213M/268M [00:04<00:01, 50.8MB/s]#015Downloading:  82%|âââââââââ | 219M/268M [00:04<00:00, 51.4MB/s]#015Downloading:  84%|âââââââââ | 224M/268M [00:04<00:00, 52.0MB/s]#015Downloading:  86%|âââââââââ | 229M/268M [00:04<00:00, 52.4MB/s]#015Downloading:  88%|âââââââââ | 235M/268M [00:04<00:00, 52.7MB/s]#015Downloading:  90%|âââââââââ | 240M/268M [00:04<00:00, 52.8MB/s]#015Downloading:  92%|ââââââââââ| 245M/268M [00:04<00:00, 52.9MB/s]#015Downloading:  94%|ââââââââââ| 251M/268M [00:04<00:00, 53.0MB/s]#015Downloading:  96%|ââââââââââ| 256M/268M [00:05<00:00, 53.1MB/s]#015Downloading:  98%|ââââââââââ| 261M/268M [00:05<00:00, 52.7MB/s]#015Downloading: 100%|ââââââââââ| 267M/268M [00:05<00:00, 51.6MB/s]#015Downloading: 100%|ââââââââââ| 268M/268M [00:05<00:00, 50.9MB/s]
Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: [‘vocab_layer_norm.bias’, ‘vocab_projector.weight’, ‘vocab_projector.bias’, ‘vocab_transform.bias’, ‘vocab_layer_norm.weight’, ‘vocab_transform.weight’]
This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: [‘classifier.weight’, ‘pre_classifier.weight’, ‘pre_classifier.bias’, ‘classifier.bias’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
#015Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]#015Downloading:  16%|ââ        | 36.9k/232k [00:00<00:00, 242kB/s]#015Downloading:  82%|âââââââââ | 190k/232k [00:00<00:00, 321kB/s] #015Downloading: 100%|ââââââââââ| 232k/232k [00:00<00:00, 755kB/s]
#015Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]#015Downloading:  20%|ââ        | 92.2k/466k [00:00<00:00, 610kB/s]#015Downloading:  73%|ââââââââ  | 338k/466k [00:00<00:00, 750kB/s] #015Downloading: 100%|ââââââââââ| 466k/466k [00:00<00:00, 1.52MB/s]
#015Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]#015Downloading: 100%|ââââââââââ| 28.0/28.0 [00:00<00:00, 24.3kB/s]
#015  0%|          | 0/36 [00:00<?, ?it/s]Traceback (most recent call last):
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 699, in convert_to_tensors
tensor = as_tensor(value)
ValueError: too many dimensions ‘str’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “train.py”, line 83, in
trainer.train()
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1246, in train
for step, inputs in enumerate(epoch_iterator):
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 444, in next
(data, worker_id) = self._next_data()
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 526, in _next_data
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch
return self.collate_fn(data)
File “/opt/conda/lib/python3.6/site-packages/transformers/data/data_collator.py”, line 123, in call
return_tensors=“pt”,
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 2680, in pad
return BatchEncoding(batch_outputs, tensor_type=return_tensors)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 204, in init
self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch
return self.collate_fn(data)
File “/opt/conda/lib/python3.6/site-packages/transformers/data/data_collator.py”, line 123, in call
return_tensors=“pt”,
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 2680, in pad
return BatchEncoding(batch_outputs, tensor_type=return_tensors)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 204, in init
self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 716, in convert_to_tensors
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
#015  0%|          | 0/36 [00:00<?, ?it/s]
File “/opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py”, line 716, in convert_to_tensors
"Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.
#015  0%|          | 0/36 [00:00<?, ?it/s]

2022-01-24 16:54:35 Failed - Training job failed
ProfilerReport-1643042834: Stopping
UnexpectedStatusException                 Traceback (most recent call last)
 in
10
11 # starting the train job with our uploaded datasets as input
—> 12 huggingface_estimator.fit({‘train’: training_input_path, ‘test’: test_input_path})
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
690         self.jobs.append(self.latest_training_job)
691         if wait:
 → 692             self.latest_training_job.wait(logs=logs)
693
694     def _compilation_job_name(self):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1665         # If logs are requested, call logs_for_jobs.
1666         if logs != “None”:
 → 1667             self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1668         else:
1669             self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3783
3784         if wait:
 → 3785             self._check_job_status(job_name, description, “TrainingJobStatus”)
3786             if dot:
3787                 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
3341                 ),
3342                 allowed_statuses=[“Completed”, “Stopped”],
 → 3343                 actual_status=status,
3344             )
3345
UnexpectedStatusException: Error for Training job huggingface-sdk-extension-2022-01-24-16-47-13-971: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command “/opt/conda/bin/python3.6 train.py --epochs 2 --model_name distilbert-base-uncased --train_batch_size 32”
Downloading: 100%|ââââââââââ| 483/483 [00:00<00:00, 460kB/s]
Downloading:  18%|ââ        | 47.8M/268M [00:01<00:04, 49.1MB/s]

Context: Hi All any updates?


How calculate loss.backward() in MLM by PyTorch

Question:
I am going to train an MLM model by pytorch, but in the training part, I do not know how to calculate the
loss.backward.
model_name = "distilbert-base-uncased"

model = AutoModelForMaskedLM.from_pretrained(model_name, max_length=256,  return_dict=True)

model.train()
device = torch.device("cpu")

for epoch in range(epochs):
    loop = tqdm(dataloader)
    for batch in loop:
        optimizer.zero_grad()

        input_ids = batch['input_ids'].to(device)
        labels = batch['labels'].to(device)
        attention_mask = batch['attention_mask'].to(device)

        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

        loss = outputs.loss
        loss.backwards()
        optimizer.step()

@lewtun

Context:
Hey @MahdiA what you have looks pretty close - I think you just need loss.backward() instead of loss.backwards()
Does that solve your issue?


Question:
I’m trying to reproduce the codet5 fine-tuning results (GitHub - salesforce/CodeT5: Code for CodeT5: a new code-aware pre-trained encoder-decoder model.)
The script being used is:
python3 /home/ubuntu/CodeT5/run_gen.py \
--task summarize \
--sub_task python \
--summary_dir /home/ubuntu/CodeT5/summary \
--cache_path /home/ubuntu/CodeT5/cache \
--data_dir /home/ubuntu/CodeT5/data \
--res_dir /home/ubuntu/CodeT5/res \
--output_dir /home/ubuntu/CodeT5/output \
--save_last_checkpoints \
--always_save_model \
--do_eval_bleu \
--model_name_or_path='Salesforce/codet5-base-multi-sum' \
--tokenizer_name='Salesforce/codet5-base-multi-sum' \
--train_filename /home/ubuntu/CodeT5/data/summarize/python/train.jsonl \
--dev_filename /home/ubuntu/CodeT5/data/summarize/python/valid.jsonl \
--test_filename /home/ubuntu/CodeT5/data/summarize/python/test.jsonl \
--do_train \
--do_eval \
--do_test \
--save_steps=500 \
--log_steps=100 \
--local_rank=-1

Running it leads to the following error:
Traceback (most recent call last):
  File "/home/ubuntu/CodeT5/run_gen.py", line 387, in <module>
    main()
  File "/home/ubuntu/CodeT5/run_gen.py", line 234, in main
    outputs = model(input_ids=source_ids, attention_mask=source_mask,
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1561, in forward
    encoder_outputs = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 998, in forward
    layer_outputs = layer_module(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 639, in forward
    self_attention_outputs = self.layer[0](
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
    attention_output = self.SelfAttention(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 483, in forward
    scores = torch.matmul(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

When run with an extra --no_cuda flag, ithe training script produces this error:
Traceback (most recent call last):
  File "/home/ubuntu/CodeT5/run_gen.py", line 394, in <module>
    main()
  File "/home/ubuntu/CodeT5/run_gen.py", line 241, in main
    outputs = model(input_ids=source_ids, attention_mask=source_mask,
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1561, in forward
    encoder_outputs = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 898, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py", line 2044, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

My guess is something fishy is going on with the source_ids, but I haven’t been able to figure it out.
A simple test shows that source_ids has shape torch.Size([8, 64]) , while target_ids have torch.Size([8, 64]) .
I wonder:

whether (and how) this can be debugged and fixed?
whether a Trainer can be adapted to this fine-tuning task? The main things I expect the Trainer to provide are deepspeed support and wandb reporting integration

UPD: an example summarization script seems to work fine. Perhpas I’ll just have to tweak it a bit so that it can summarize code instead of natural language

Context:
Ok, to the best of my understanding at the moment, it goes like this:
the example summarization script uses certain (I presume standard/based on the original paper) values for max_source_length and max_target_length, which are 1024 and 128, respectively.
But codet5 uses 64 and 32 respectively, for some reason I was unable to discern.
Anyway, once those were changed, it … didn’t work, since it didn’t fit into a single a100 memory,
but then I changed the batch_size from 8 to 4, and now it does work.
Hooray!


Question:
Hi,
Sorry if this is a duplicate thread.
Thanks for this beautiful platform for sharing our models and datasets.
My question is, how can I delete a model repository from huggingface model hub?
Thanks in advance.
Sagor

Context:
To delete a model repository, right now you need to do it in code from the transformers library and do something like:
from transformers.hf_api import HfApi

HfApi().delete_repo(...)

We’ll ship a UI to do it directly from the website in the coming days, though. (cc @pierric @thomwolf)


Hubert ASR Fine Tuning giving weird results

Question:
Hi
I have taken hubert-large-ls960-ft model and fine tuned it for my dataset. I can see the loss and wer decreasing at each step and the best model is saved. Now when i try to do inference i get repeated characters like ‘zhzhzhzzhzhz’. I am not sure where I am going wrong because I used the same script (Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers) for my other wav2vec2 models and they work fine. I have just replaced Wav2Vec2ForCTC with HubertForCTC and the entire script is same. Can anyone please help me
Training progression is as follows:-

image501×778 34.9 KB

And the code snippet where I made the change is as follows:-

from transformers import HubertForCTC

model = HubertForCTC.from_pretrained(

    "facebook/hubert-large-ls960-ft",

    attention_dropout=0.01,

    feat_proj_dropout= 0.05,

    activation_dropout=0.05,

    hidden_dropout=0.05,

    hidden_act= "gelu",

    # feat_proj_dropout=0.0,

    mask_time_prob=0.1,

    layerdrop=0.05,

    gradient_checkpointing=True,

    ctc_loss_reduction="mean",

    pad_token_id=processor.tokenizer.pad_token_id,

    vocab_size=len(processor.tokenizer)

)

Inference script is as follows:-

from transformers import HubertForCTC,Wav2Vec2Processor

model = HubertForCTC.from_pretrained('/content/drive/MyDrive/finalmodel/checkpoint-9500').to("cuda")

processor = Wav2Vec2Processor.from_pretrained("/content/drive/MyDrive/finalmodel")

test_df['audio_path'] = '/content/test_audio/'+test_df['Clip_ID']+'.mp3'

test_df1 = test_df[['audio_path']]

test_data  = Dataset.from_pandas(test_df1)

def speech_file_to_array_fn(batch):

    speech_array, sampling_rate = torchaudio.load(batch["audio_path"])

    batch["speech"] = librosa.resample(np.asarray(speech_array[0].numpy()), 32_000, 16_000)

    batch["sampling_rate"] = 16_000

    return batch

test_data = test_data.map(speech_file_to_array_fn, remove_columns=test_data.column_names)

def evaluate(batch):

    inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

    with torch.no_grad():

        logits = model(inputs.input_values.to("cuda")).logits

    pred_ids = torch.argmax(logits, dim=-1)

    batch["pred_strings"] = processor.batch_decode(pred_ids)

    return batch

result = test_data.map(evaluate, batched=True, batch_size=8)

result["pred_strings"]

Context:
Hey @sammy786,
Could you please upload all relevant files that are created during training to the hub so that I can take a look. I especially need to take a look at the tokenizer that was created when running the script.
So please upload all files that are required to run the model in inference and all bash and Python scripts that you used for training to a repository to the Hub. There is no way that I could find out from just looking at screenshots what might be wrong here. Thank you!


Finetuning wav2vec2-large-xlsr-53 only outputs blank labels

Question:
Hi,
When I try to finetune wav2vec2-large-xlsr-53 1 with FSC 1 dataset for ASR using built-in class Wav2VecForCTC, the CTC loss is not converging, and the system only outputs blank labels even for training instances.
Here is the log of training (overfiting) with only 8 training instances in total:
Epoch 60/1000, Batch 1/1, Total Step = 60, Loss = 26.296, CER = 100.000,
Gold: ['SWITCH OFF THE LIGHTS', 'TURN THE VOLUME UP']
Pred: ['', '']
Epoch 1000/1000, Batch 1/1, Total Step = 1000, Loss = 2.663, CER = 100.000
Gold: ['SWITCH OFF THE LIGHTS', 'TURN THE VOLUME UP']
Pred: ['', '']

We can see that even at Epcoh 60, the CTC loss is ~26 and the system only outputs blank labels for training instances. Continuing training to 1000 epoch will reduce the CTC loss, but still the system outputs blanks.
However, if I use an ASR finetuned model 3 (the model is finetuned even on Chinese corpora), with exactly the same code, continuing finetuning on FSC can quickly overfit the training instances, And we can see now the CTC loss is small and we can reproduce input pretty well:
Epoch 37/1000, Batch 1/1, Total Step = 37, Loss = 0.681, CER = 21.818,
['CHANGE LANGUAGE', 'TURN THE LIGHTS ON']
['CHANE LANGUAEEI', 'TURN THE LITSH ONT']
Epoch 100/1000, Batch 1/1, Total Step = 100, Loss = 0.028, CER = 2.727,
['SWITCH OFF THE LIGHTS', 'SWITCH ON THE LIGHTS']
['SWITCH OFF THE LIGHTS', 'SWIITCH ON THE LIGHTSWW']

Here is the code to create new model loading different pretrained models:
model = Wav2Vec2ForCTC.from_pretrained(
            args.audio_model,
            gradient_checkpointing=True,
            apply_spec_augment=False,
            vocab_size=processor.tokenizer.vocab_size,
            hidden_dropout=0.05,
            activation_dropout=0.05,
            feat_proj_dropout=0.05,
            layerdrop=0.05,
            final_dropout=0.05,
            mask_time_prob=0.05,
            ctc_loss_reduction='mean',
            ctc_zero_infinity=True,
        )

I am using Adam with 1e-4 as learning rate. Both models use the same vocabulary of size ~3k (with Chinese chars). This configuration is exactly the same for both pretrained models, but still yields different behaviors. Note that I also tried sum for ctc_loss_reduction in xlsr but also got the same blanks.
Could anybody help me on that? Thank you very much!

Context: I meet the same issue. Do you solve it???


Question:
I am doing the following to finetune CLIPMode further on my own dataset.
But with no luck.
Please advise.
Model load
from transformers import CLIPProcessor, CLIPModel, CLIPConfig
config = CLIPConfig.from_pretrained("openai/clip-vit-base-patch32")
model = CLIPModel(config)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

Data loader
import torch
from torch.utils.data import Dataset
from PIL import Image

class CLIPDataset(Dataset):
    def __init__(self, root_dir, df,processor, max_target_length=32):
        self.root_dir = root_dir
        self.df = df
        self.processor = processor
        self.max_target_length = max_target_length

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        file_name = self.df['image'][idx]
        image = Image.open(self.root_dir + "/" + file_name).convert("RGB")
        text = self.df['title'][idx]

        pixel_values = self.processor.feature_extractor(image, return_tensors="pt").pixel_values
        labels = self.processor.tokenizer(text,
                                padding="max_length",
                                max_length=self.max_target_length,
                                truncation=True).input_ids
        labels = [label if label != self.processor.tokenizer.pad_token_id else -100 for label in labels]
        return {"input_ids":torch.tensor(labels), "pixel_values":pixel_values.squeeze()}

Using default data_collator with Trainer
trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=default_data_collator,
    tokenizer=processor.feature_extractor
)

Context: @nielsr / @valhalla / @patrickvonplaten can I seek your advice ?


Question:
Hello, I implemented wav2vec2.0 code 69 and a language model is not used for decoding. How can I add a language model (let’s say a language model which is trained with KenLM) for decoding @patrickvonplaten ?
thanks in advance.
Note: I also opened an issue 65, but redirected here.

Context:
Hey Emre!
Yeah good question - we currently don’t support evaluating with a language model, but we plan on adding this functionality soon! It’s sadly not that trivial to decode a CTC model with a language model. I’ll try to keep you posted for updates here!


Question:
I am working on a task in which I should modify the encoding results.
what I would like to do is generally like this:
input_ids = tokenizer(“i am trying hard!”, return_tensors=‘pt’).input_ids
last_hidden_state=model.encoder(input_ids=input_ids).last_hidden_state
modified_last_hidden_state = modify(last_hidden_state)
outputs = model.decoder(modified_last_hidden_state)
output_sequence = tokenizer.decode(outputs)
I think this model.decoder() actually doesn’t work as I want.

Context:
reply myself:
I think this is a good try since the loss and hidden states are totally the same as the standard training process, and I will test the training process later.
the separate process:

Screenshot from 2022-01-10 12-27-07939×495 74.9 KB


Question:
Hello,
I’m working on using T5LM (small) for semantic parsing tasks. However, there seems to be discrepancies between the model properties as listed on the T5 documentation page: https://huggingface.co/docs/transformers/model_doc/t5
versus the error outputs I get from trying to run the Supervised Training example. More specifically, when running the last line:
 loss = model(input_ids=input_ids, labels=labels).loss
I get the error saying the model needs either  decoder_inputs_embeds  or  decoder_input_ids  in place of the labels parameter. Additionally, the tutorial text states:

The model will automatically create the decoder_input_ids based on the labels ,…

Is there source code for this model implementation that I can reference? I don’t understand the difference between the embeds vs decoder input ids for a start. And I can’t tell if the model I imported using
 model = transformers.AutoModel.from_pretrained(“google/t5-small-lm-adapt”)
is correct because in addition to the above discrepancy, when I run the original loss command, I got the error that
 Seq2SeqModelOutput  does not have a  .loss  function.
I thought the LM adaptation of T5 would be 1-to-1 to the original T5 architecture, but that clearly doesn’t seem to be the case. Any help or direction would be greatly appreciated.
Thanks,
Selma

Context:
I now understand the difference between decoder_inputs_embeds and the decoder_input_ids; however I’m still confused about the difference between the
google/t5-small-lm-adapt that is being loaded from AutoModel
vs
The T5ConditionalGeneration model supplied by hugging face


How to use the inference api on tts model?

Question:
Hi, how can I use the inference api on this model: espnet/kan-bayashi_ljspeech_vits?
It receives text and should return audio file, since it is text to speech model.

Context:
This looks like a cache issue (There’s a cache in front of the API to prevent calculating things over and over).
You can try adding {"inputs": "....", {"parameters": {"use_cache": False}} to your input to force the output to be calculated.
The caching mechanism should be upgraded at some point so you don’t have to do this.


Increasing loss during LM fine-tuning on custom dataset

Question:
Hello,
During the fine-tuning of camembert-base on a custom dataset using the Trainer API, the loss starts to increase quickly (compared to its value at step 0) and then decreases slowly, without ever reaching its value before training. What’s stranger is that lowering the learning rate results in a higher peak, whereas I expected the opposite to happen. Has anyone experienced a similar issue?
For information, the dataset is quite small (35k items of 15-300 words each for training, 6k for validation).
I’ve attached a screenshot of the metrics for different learning rate values (I’ve early stopped most of them).
Capture d’écran 2020-09-03 à 13.51.302172×622 62.9 KB
Here is the code I used for training:
import json
import logging
import os
import math

import torch
from torch.utils.data.dataset import Dataset
from transformers import (
    AutoTokenizer,
    CamembertForMaskedLM,
    DataCollatorForLanguageModeling,
    HfArgumentParser,
    PreTrainedTokenizer,
    Trainer,
    TrainingArguments,
)

logger = logging.getLogger()

class LMJsonDataset(Dataset):
    def __init__(
        self, tokenizer: PreTrainedTokenizer, file_path: str, field_name: str, block_size: int
    ):
        assert os.path.isfile(file_path), f"Input file path {file_path} not found"
        logger.info("Creating features from dataset file at %s", file_path)

        with open(file_path, encoding="utf-8") as f:
            texts = [t for t in (json.loads(item)[field_name] for item in f if item) if t]

        batch_encoding = tokenizer(
            texts, add_special_tokens=True, truncation=True, max_length=block_size
        )
        self.examples = batch_encoding["input_ids"]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i: int) -> torch.Tensor:
        return torch.tensor(self.examples[i], dtype=torch.long)

parser = HfArgumentParser(TrainingArguments)
parser.add_argument("--train_dataset", required=True)
parser.add_argument("--eval_dataset", required=True)
parser.add_argument("--mlm_probability", type=float, default=0.15)
parser.add_argument("--base_model", default="camembert-base")
parser.add_argument("--dataset_json_field_name", default="text")
parser.add_argument("--block_size", type=int, default=128)
training_args, remaining_args = parser.parse_args_into_dataclasses()

model = CamembertForMaskedLM.from_pretrained(remaining_args.base_model)
tokenizer = AutoTokenizer.from_pretrained(remaining_args.base_model)

train_dataset = LMJsonDataset(
    tokenizer=tokenizer,
    file_path=remaining_args.train_dataset,
    field_name=remaining_args.dataset_json_field_name,
    block_size=remaining_args.block_size,
)
eval_dataset = LMJsonDataset(
    tokenizer=tokenizer,
    file_path=remaining_args.eval_dataset,
    field_name=remaining_args.dataset_json_field_name,
    block_size=remaining_args.block_size,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=remaining_args.mlm_probability
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    prediction_loss_only=True,
)

trainer.train()

Thank you for your help!

Context: I have a question do you work with french data ?


Fionetune model always predicts same output class for new data

Question:
I have a problem, trained a model with bert which give around 0.90% on test data and I decide to use it on new data which were not annotated. When running the model, I keep getting the same class output. And I would like to know why ?
Can you help me ?

Context: Hey! Just wanted to state that I am facing the same problem and still don’t know how this is happening. I was working with RoBERTaForSequenceClassification with a binary classification problem and it gave 50% ACC. after 5 epochs. It did not learn anything at all.


Unispeech-sat-base-plus-sv evalution runs out of VRAM

Question:
Hi! Currently I’m working on some utterance classification problem. Previously I was using Wav2Vec2 (facebook/wav2vec2-base) model and everything was fine. Now I tried to use this new model using the same code. The training process goes fine, but when it comes to evaluation, it runs out of CUDA memory:
 43%|████▎     | 55/129 [00:41<01:09,  1.07it/s]e[ATraceback (most recent call last):
  File "/a2e_workspace/train.py", line 331, in <module>
    trainer.train()
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1399, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 1521, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2158, in evaluate
    output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2341, in evaluation_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 106, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 106, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 106, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 106, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 108, in nested_concat
    return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 76, in torch_pad_and_concatenate
    result = tensor1.new_full(new_shape, padding_index)
RuntimeError: CUDA out of memory. Tried to allocate 480.00 MiB (GPU 0; 15.78 GiB total capacity; 13.21 GiB already allocated; 415.75 MiB free; 13.99 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The GPU has 16 GB VRAM.
Batch size parameters:
  per_device_train_batch_size: 4
  per_device_eval_batch_size: 4
  gradient_accumulation_steps: 4

The code is pretty much default, it’s based on this guide
The strange thing is that the size of the model is approximately the same as Wav2Vec2, and also memory always runs out exactly on evalution. So maybe there is some memory leak bug in the code.
I’m gonna test this on 32 GB GPU and report the results.

Context:
I figured out why this happens. There is a line 1783 in transformers/models/unispeech_sat/modeling_unispeech_sat.py:
output_hidden_states = True if self.config.use_weighted_layer_sum else output_hidden_states
So even if you pass output_hidden_states=None in forward pass, the output will anyways contain hidden states (because config.use_weighted_layer_sum=True by default). And if you are using output in a form such like that:
@dataclass
class SpeechClassifierOutput(ModelOutput):
    loss: Optional[torch.FloatTensor] = None
    logits: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
    attentions: Optional[Tuple[torch.FloatTensor]] = None

This will actually contain all the hidden states.
Evaluation round will accumulate all the hidden states for all samples, therefore you will run out of memory. So I just manually set hidden_states=None in the output.
Hopefully this will help someone.


Use one-hot encoding as input for T5 and GPT

Question:
Hi,
Is it possible to train the T5 model by using onehot encoding input and integers as target?
something like that loss = model(onehot, attn, targetInList , attn)
As it’s a translation problem, the onehot input will be [  [0,0,0,1…],[0,1,0,0,…]…]
Any help would be appretiated!

Context: If cannot, is there any other way to convert that one-hot encoding input to normal integer list? because torch.argmax is not differentiable


Models for Multilingual Classification Tasks

Question:
Hello,
I want to build a multilingual classification model (english, hindi, malayalam…) and wanted to ask if anyone has some suggentions which models to use. I would like to explore the performance of different models.
So which models are good in general for the usecase of a text classification with different languages?
Thanks in advance!

Context:
Some great multilingual models:

multilingual BERT 1/DistilBERT 1

XLM-RoBERTa
RemBERT 1
CANINE

These are all encoder-only Transformer models (great for classification, question-answering, NER,…).
CANINE is a relatively new model that is tokenizer-free, meaning it’s a character-level model and does not require an explicit tokenization step.
For summarization/translation/etc. (seq2seq tasks), mT5 is a great model.


Question:
Hello,
I am having some issues to run tokenization using the XLSUM tokenizer.
It seems to be an issue related to Huggingface as the file in the error message changes every time I run my script.
The error message:
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/csebuetnlp/mT5_multilingual_XLSum/resolve/main/tokenizer_config.json

Here is my code:
from datasets import Dataset, concatenate_datasets, load_from_disk
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from sklearn.model_selection import train_test_split
import pandas as pd
import glob
import os
import re

# PARAMETERS
encoder_max_length = 512
decoder_max_length = 150
batch_size = 100
model_name = "csebuetnlp/mT5_multilingual_XLSum"
num_processes_to_tokenize = 12
WHITESPACE_HANDLER = lambda k: re.sub('\s+', ' ', re.sub('\n+', ' ', k.strip()))

def load_description_and_transcript_data(lang):
    # concat all CSVs and load them as one pandas dataframe
    all_files = glob.glob('data/{}/description_and_transcript*.csv'.format(lang))
    li = []

    for filename in all_files:
        df = pd.read_csv(filename, index_col=None, header=0)
        li.append(df)

    df = pd.concat(li, axis=0, ignore_index=True)

    # keep only 3 columns: episode_uri, episode_description_cleaned, transcript_text
    df = df[['episode_uri', 'episode_description_cleaned', 'transcript_text']]
    df = df.dropna()

    # clean text (source https://huggingface.co/csebuetnlp/mT5_multilingual_XLSum)
    df["transcript_text"] = df["transcript_text"].apply(WHITESPACE_HANDLER)
    df["episode_description_cleaned"] = df["episode_description_cleaned"].apply(WHITESPACE_HANDLER)

    return df

def load_train_dev_splits_data(lang):
    dev = pd.read_csv('data/{}/out.final.filtered.dev.tsv'.format(lang), index_col=None, header=0, sep='\t')
    train = pd.read_csv('data/{}/out.final.filtered.train.tsv'.format(lang), index_col=None, header=0, sep='\t')

    return train, dev

# Copied from https://github.com/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb
def process_data_to_model_inputs(batch, lang):
    # load tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    inputs = tokenizer(
        batch["transcript_text"],
        # return_tensors="pt",
        padding="max_length",
        truncation=True,
        max_length=encoder_max_length
    )

    outputs = tokenizer(
        batch["episode_description_cleaned"],
        # return_tensors="pt",
        padding="max_length",
        truncation=True,
        max_length=decoder_max_length,
    )

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["labels"] = outputs.input_ids

    return batch

def tokenize_data(hf_dataset, lang):
    processed_data = hf_dataset.map(
        lambda batch: process_data_to_model_inputs(batch, lang),
        num_proc=num_processes_to_tokenize,
        batched=True,
        batch_size=batch_size,
        remove_columns=["transcript_text", "episode_description_cleaned"],
    )

    # set Python list to PyTorch tensor
    processed_data.set_format(
        type="torch",
        columns=["input_ids", "attention_mask", "labels"],
    )

    return processed_data

def preprocess_data(lang):
    """
    Steps:
    - load splits out.final.filtered.dev.tsv and out.final.filtered.train.tsv
    - load the dataset containing both the episode description and the transcript text
    - split the dataset from the previous step into train and dev
    - tokenize the data
    - save tokenized
    :param df:
    :return:
    """
    print("START preprocessing language:", lang)

    desc_transc = load_description_and_transcript_data(lang)
    print('desc_transc:', len(desc_transc))

    train, dev = load_train_dev_splits_data(lang)
    print('train/dev:', len(train), len(dev))

    desc_and_transc_train = desc_transc[desc_transc['episode_uri'].isin(train['episode_uri'].tolist())]
    desc_and_transc_dev = desc_transc[desc_transc['episode_uri'].isin(dev['episode_uri'].tolist())]

    # Load as HF dataset
    print('Loading as HF dataset...')
    podcast_train = Dataset.from_pandas(desc_and_transc_train)
    podcast_dev = Dataset.from_pandas(desc_and_transc_dev)

    # remove column __index_level_0__
    podcast_train = podcast_train.map(lambda x: x, remove_columns=['__index_level_0__'])
    podcast_dev = podcast_dev.map(lambda x: x, remove_columns=['__index_level_0__'])

    # Tokenize and save data
    print('Processing TRAIN data...')
    tokenized_data = tokenize_data(podcast_train, lang)
    tokenized_data.save_to_disk(f'data/preprocessed/{lang}/train/')
    print('Processing TRAIN data FINISHED')

    # Tokenize and save data
    print('Processing DEV data...')
    tokenized_data = tokenize_data(podcast_dev, lang)
    tokenized_data.save_to_disk(f'data/preprocessed/{lang}/dev/')
    print('Processing DEV data FINISHED')

def read_lines(file_path):
    with open(file_path, "r") as f:
        return f.readlines()

def merge_and_shuffle():
    en_data = load_from_disk('data/preprocessed/en_XX/train/')
    pt_data = load_from_disk('data/preprocessed/pt_XX/train/')
    all_train = concatenate_datasets(dsets=[en_data, pt_data])
    all_train = all_train.shuffle()
    all_train.save_to_disk('data/preprocessed/all/train/')

    en_data = load_from_disk('data/preprocessed/en_XX/dev/')
    pt_data = load_from_disk('data/preprocessed/pt_XX/dev/')
    all_dev = concatenate_datasets(dsets=[en_data, pt_data])
    all_dev = all_dev.shuffle()
    all_dev.save_to_disk('data/preprocessed/all/dev/')

def main():
    preprocess_data('pt_XX')
    preprocess_data('en_XX')
    merge_and_shuffle()

main()

Context:
I am getting the same error since yesterday. Below is my instance:
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/api/models/roberta-base


How is the AdafactorScheluder suppose to be used?

Question:
Hi,
I recently saw my transformer model having divergence issues and I saw a paper that uses Adafactor and wanted to try it out. The docs are fantastic but they don’t mention how often or how the Adafactor scheduler actually works. How is that suppose to be used? When do I call the scheduler in my code?
from transformers.optimization import Adafactor, AdafactorSchedule
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
lr_scheduler = AdafactorSchedule(optimizer)

all my code is my custom training so idk when lr_scheduler is suppose to be called. Usually it depends on the model or I try to start of with the convenient ReduceLROnPlateau.
When using lr=None with [ Trainer ] you will most likely need to use AdafactorSchedule scheduler as following:
from transformers.optimization import Adafactor, AdafactorSchedule optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) lr_scheduler = AdafactorSchedule(optimizer) trainer = Trainer(..., optimizers=(optimizer, lr_scheduler))

related:

Optimization — transformers 4.7.0 documentation 6
T5 Finetuning Tips - #34 by brando 3

Context:
Apologies for the direct ping, but can you help me get the right person to help me with this? @sgugger?
Thank you in advance!


Tensorflow h5 file doesn’t contain network, it only include weighs?

Question:
I have downloaded bert-base-uncased tf_model.h5 form bert-base-uncased at main 1
Inspect it with netron, only weight can be found, how can I save network in h5?

image632×1314 27.1 KB

Context:
Hello,
Sorry for late reply, you might’ve already figured it out.
TF models can be saved in two ways, one is .h5 file which contains weights and other is SavedModel protobuf file. SavedModel has everything you need (graphs, variables etc) so it’s more platform agnostic. When you save as SavedModel you can use it everywhere (android/flutter apps with TF lite, browser with tf.js, or REST API with tensorflow serving).
See more here 1 on how to save as SavedModel and h5.


Difference between language modeling scripts

Question:
Hi,
I would like to fine-tune a language model (Tensorflow-based) and see that you have two scripts pertaining to that:

https://github.com/huggingface/notebooks/blob/master/examples/language_modeling-tf.ipynb

https://github.com/huggingface/transformers/blob/master/examples/tensorflow/language-modeling/run_mlm.py

I followed the first script and am quite satisfied with the result. However, I am curious if the second one can be also an option for me.
Could you, please, outline what are the differences between these two scripts and under which circumstances each one should be used? Or, alternatively, could you, please, provide a link to documentation, if such exists, that can answer my question?
I apologize in advance, if my question sounds ignorant, I am quite a newbie in machine learning and mastering it now with HuggingFace ^^
Thank you!

Context:
Hello Lenn,
First one is an end-to-end tutorial on how to train a mask language model, and it shows what you need to do (preprocessing etc) if you want to train an MLM, helps you understand what’s going on in the background, the problem itself etc.
Second one is a more production-y script with no EDA. I used to use those scripts like run_glue.py in my former job to train/fine-tune language model directly, like this:
!python run_glue.py \
  --model_name_or_path "model_checkpoint" \
  --task_name cola \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --per_gpu_eval_batch_size 32 \
  --learning_rate 2e-5 \
  --save_steps 2000 \
  --num_train_epochs 50 \
  --output_dir 'output_dir_here'


Question:
Hi @patrickvonplaten, I am trying to fine-tune XLSR-Wav2Vec2. Data contains more than 900k sound, it is huge. In this case, I always receive out of memory, even batch size is 2 (gpu = 24gb). When I take a subset (100 sound) and fine-tune on this subset, everything is fine. What could be the problem? Is there any issue which is related to loading data to memory?
I think it should not depend how much bigger data is when batch size is same.

Context:
Hey @EmreOzkose do you train locally or in a google colab? Also do you get hard disk out-of-memory errors or RAM out-of-memory?
Feel free to share your fine-tuning script here so that I can take a look


Fine-Tune for MultiClass or MultiLabel-MultiClass

Question:
Hi,
I want to build a:

MultiClass Label
(eg: Sentiment with VeryPositiv, Positiv, No_Opinion, Mixed_Opinion, Negativ, VeryNegativ)

and a MultiLabel-MultiClass model to detect 10 topics in phrases
(eg: Science, Business, Religion …etc)

and I am not sure where to find the best model for these types of tasks?
I understand this refers to the Sequence Classification Task. So, I could search for a model tagged with that task on your model repository site - but not all models are tagged like that and the transformers API seems to provide much more task applications beyond the original training.
I found with the code below that I can have a model that supports originally 5 labels but load it into a ConvBertForSequenceClassification model to support, for example 25 labels. Would this (plus softmax or sigmoid and fine-tuning) be the correct way to pick up an existing model and implement 1. or 2. or is there a different more effective way to choose a model and fine tune it?
Thanks dirk
from transformers import pipeline
nlp = pipeline("sentiment-analysis", 'bert-base-multilingual-uncased-sentiment')
result = nlp("I hate you")[0]
print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
result = nlp("I love you")[0]
print(f"label: {result['label']}, with score: {round(result['score'], 4)}")

#label: 1 star, with score: 0.6346
#label: 5 stars, with score: 0.8547

from transformers import ConvBertForSequenceClassification, ConvBertTokenizer

convBertModel = ConvBertForSequenceClassification.from_pretrained('bert-base-multilingual-uncased-sentiment', num_labels=25)
convBerttokenizer = ConvBertTokenizer.from_pretrained('bert-base-multilingual-uncased-sentiment')

print ( f"                              num_labels: {model.num_labels}")
print ( f"                              classifier: {model.classifier}")

# num_labels: 25
# classifier: ConvBertClassificationHead(
#   (dense): Linear(in_features=768, out_features=768, bias=True)
#   (dropout): Dropout(p=0.1, inplace=False)
#   (out_proj): Linear(in_features=768, out_features=25, bias=True)
# )

Context:
Hi @dikster99,
The way I usually search for models on the Hub 184 is by selecting the task in the sidebar, followed by applying a filter on the target dataset (or querying with the search bar if I know the exact name). In both your cases, you’re interested in the Text Classification tags, which is a specific example of sequence classification:

Screen Shot 2021-02-27 at 4.00.33 pm942×1346 132 KB

However, this assumes that someone has already fine-tuned a model that satisfies your needs. If not, there are two main options:

If you have your own labelled dataset, fine-tune a pretrained language model like distilbert-base-uncased (a faster variant of BERT). You can find a nice example for text classification here 271 and see here 493 for the multi-label case. In general this is similar to your second example with ConvBertForSequenceClassification and you were correct to specify num_labels in the from_pretrained function

If you have no labelled dataset, then you could try out one of the Zero-Shot Classification models (e.g. here 43). See this 32 blog post for an explanation on how zero-shot works.

Hope that helps!
PS. the Pipeline classes are typically used for generating predictions from a fine-tuned model, so your example with bert-base-multilingual-uncased-sentiment wouldn’t work because that model was trained on 5 labels and does not know about the 25 labels you are interested in.


Question:
I want to add a “portion” operator that will calculate the count of the selected cells divided by the number of rows in the table (count \ total). To answer questions like “what is the portion of players with more than 10 points?” and get “0.32” as an answer.
Count result is already calculated in the _calculate_expected_result method in the modeling_tapas.py:
scaled_probability_per_cell = (scaled_probability_per_cell / numeric_values_scale) * input_mask_float count_result = torch.sum(scaled_probability_per_cell, dim=1)
However, I’m not sure how to continue from here. How to get the number of cells (i thought it is the size of scaled_probability_per_cell, but I’m not sure it is correct in small tables)? How to add the result to the loss? , etc.

Context:
So you want to know the number of rows for every example in the batch? You can probably take the max of the unique row IDs in the token type ids created by TapasTokenizer. Small example:
from transformers import TapasTokenizer
import pandas as pd

model_name = 'google/tapas-base-finetuned-wtq'
tokenizer = TapasTokenizer.from_pretrained(model_name)

data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], 'Number of movies': ["87", "53", "69"]}
queries = ["What is the name of the first actor?", "How many movies has George Clooney played in?", "What is the total number of movies?"]
table = pd.DataFrame.from_dict(data)
inputs = tokenizer(table=table, queries=queries, padding='max_length', return_tensors="pt")

row_ids = inputs["token_type_ids"][:,:,2]

You can easily get the unique row IDs for every example in the batch, and then take the max:
import torch

nrows = [torch.max(torch.unique(example)).item() for example in row_ids]

You can pass this variable to the _calculate_expected_result method, such that you can compute the portion result as:
portion_result = count_result / nrows


Question:
Starting this for results, sharing + tips and tricks, and results. This is my first attempt at this kind of thread so it may completely fail.
Some things I’ve found

Apparently if you copy AdaFactor from fairseq, as recommended by t5 authors, you can fit batch size = 2 for t5-large lm finetuning
fp16 rarely works.
for most tasks, you need to manually add </s> to the end of your sequence.

Thing’s I’ve read

task specific prefix doesn’t matter much.

cc @mrm8488 @valhalla @patrickvonplaten who have all tried different experiments.

Context:
Things I’ve found

task prefixes matter when
1. When doing multi-task training
2. When your task similar or related to one of the supervised tasks used in T5 pre-training mixture.
Needs slightly higher LR than the default one set in Trainer, in my experiments  1e-4 and 3e-4 worked for almost all problems (classification, QA, que-gen, summ)
no need to pass decoder_input_ids  to T5 yourself, just pass labels and the T5Model will prepare them for you. labels should end with eos_token. (important! This is where most of the mistakes are happening).
T5 uses pad_token as the decoder_start_token_id so when doing generation without the generate function make sure you start it with pad token.
trimming batches when training on TPU leads to very slower training.
apparently, because of sentencepiece and some possible leakage of other languages in C4 data, T5 gives somewhat sensible results for french lang. fine-tuned it on FQuAD (french version of SQuAD) for que gen and BLUE-4 against dev set was 15.

Not sure if it’s an issue or not but in some cases using label_smoothing in T5 resulted in nan loss


Question:
hi guys, I have this problem using paraphrase-distilroberta-base-v2 model.
once I call the model through in my python code:
model = SentenceTransformer(‘sentence-transformers/paraphrase-distilroberta-base-v2’)
this error appear:
403 Client Error: Forbidden for url: https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2/resolve/486c69f0e5395a86ef58883e1c18e475cc7b8aba/.gitattributes 1
same thing if I use  Hosted inference API in model main page.
Some advise about this problem?

Context:
Hi @giovanni94s,
I guess there is today a problem in the Hugging Face hub as I even can not edit a model or a datasets card in my profile. @julien-c can you help us? Thank you.


Question:
hi @nielsr nielsr, not sure if the docs for SegformerConfig default image_size=512 is wrong… I suppose it should be 224 according to the source code?
master link: SegFormer — transformers 4.13.0.dev0 documentation
v4.12.5 link: SegFormer — transformers 4.12.5 documentation

截屏2021-11-24 上午11.51.491156×391 73.8 KB

Context:
Hi,
Thanks for reporting, this has been fixed.


Train Bart for Conditional Generation (e.g. Summarization)

Question:
Hi everybody
I ran into some issues when trying to fine-tune bart for summarization using the BartForConditionalGeneration model. The issue evolved around properly masking and ignoring the padding tokens when training. Without the following fix the loss went down but the model produced bad summaries. I post the solution here in case anyone else runs into similar problems.
from transformers import BartTokenizer, BartForConditionalGeneration
from transformers import Trainer, TrainingArguments
from transformers.modeling_bart import shift_tokens_right

dataset = ... # some Datasets object with train/validation split and columns 'text' and 'summary'
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')

def convert_to_features(example_batch):
    input_encodings = tokenizer.batch_encode_plus(example_batch['text'], pad_to_max_length=True, max_length=1024, truncation=True))
    target_encodings = tokenizer.batch_encode_plus(example_batch['summary'], pad_to_max_length=True, max_length=1024, truncation=True))

    labels = target_encodings['input_ids']
    decoder_input_ids = shift_tokens_right(labels, model.config.pad_token_id)
    labels[labels[:, :] == model.config.pad_token_id] = -100

    encodings = {
        'input_ids': input_encodings['input_ids'],
        'attention_mask': input_encodings['attention_mask'],
        'decoder_input_ids': decoder_input_ids,
        'labels': labels,
    }

    return encodings

dataset = dataset.map(convert_to_features, batched=True)
columns = ['input_ids', 'labels', 'decoder_input_ids','attention_mask',]
dataset.set_format(type='torch', columns=columns)

training_args = TrainingArguments(
    output_dir='./models/bart-summarizer',
    num_train_epochs=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['validation']
)

The convert_to_features function makes sure that the decoder inputs are correctly shifted and still include the padding tokens while in the labels the padding tokens are replaced by -100 such they are ignored in the model loss.

Context:
Hi,
I tried to use your fix but I’m wondering whether this is still up-to-date:
transformers.modeling_bart doesn’t exist. There is only transformers.models.bart.modeling_bart but its shift_tokens_right function requires a torch.Tensor object while huggingface’s datasets object only consists of lists (plus it needs an additional decoder_start_token_id). This also leads to an error in labels[labels[:, :] == model.config.pad_token_id] = -100 because this is numpy syntax.
Also, batch_encode_plus is deprecated.
It would be nice if there was a statement from huggingface if this is still necessary and if not, could it please be removed from the BART page?


Question:
I have a wrapper for a huggingface model. In this wrapper I have some encoders, which are mainly a series of embeddings. In forward of the wrapped model, I want to call forward of each of encoders in a loop, but I get the error:
Traceback (most recent call last):
  File "/home/pouramini/mt5-comet/comet/train/train.py", line 1275, in <module>
    run()
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/pouramini/mt5-comet/comet/train/train.py", line 1069, in train
    result = wrapped_model(**batch)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pouramini/mt5-comet/comet/transformers_ptuning/ptuning_wrapper.py", line 135, in forward
    prompt_embeds = encoder(prompt_input_ids,\
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pouramini/mt5-comet/comet/transformers_ptuning/ptuning_wrapper.py", line 238, in forward
    return self.embedding(prompt_token_ids)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/home/pouramini/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 2043, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument index in method wrapper_index_select)

This the code that results the error:
def forward(self,input_ids, labels, decoder_input_ids=None,prompt_ids=None,**kwargs):
        # find masks based on the range of prompt ids (offset_id < X < offset_id + prompt_length)
        #Because this wrapper only deals with a single prompt, the length should be the same, you can use masked_select to reshape
        prompt_masks = self.prompt_token_fn(input_ids)
        if prompt_masks.any():
            input_ids_ = input_ids.clone()
            wlog.info("inpu ids :{}".format(input_ids))
            if self.replacing_token_id is not None:
                # replace prompt ids in input_ids with replacing token
                input_ids_[prompt_masks]=self.replacing_token_id
            # find the model embeddings of input ids except for prompt tokens
            inputs_embeds = self.model_embeddings(input_ids_)

           for encoder in self.prompt_encoders:
                #encoder = self.prompt_encoders[0]
                wlog.info("********** offset: %s, length: %s", encoder.id_offset, encoder.length)
                prompt_token_fn = encoder.get_prompt_token_fn()
                encoder_masks = prompt_token_fn(input_ids)
                wlog.info("Encoder masks: %s", encoder_masks)
                if encoder_masks.any():
                    #find input ids for prompt tokens
                    prompt_input_ids = input_ids[encoder_masks]
                    wlog.info("Prompt Input ids: %s", prompt_input_ids)
                    # call forwards on prompt encoder whose outputs are prompt embeddings
                    prompt_embeds = encoder(prompt_input_ids,\
                        prompt_ids).to(device=inputs_embeds.device)

The code however runs if I just use cpu as device. Also if I have one encoder, the code is run with cuda, but when there are multiple encoders, it seems it expects that all of them are transfered to device, which I don’t know how to do that.

Context:
It seems I need to put all encoders to device.
      wrapped_model.to(device=device)
      for encoder in wrapped_model.prompt_encoders:
            encoder.to(device=device)

However, when there was a single encoder or a list of encoders but including one encoder, I didn’t need to explicitly put it on device, but for the list of encoders i must!


Question:
Hi there,
I’m experiencing an unexpected behaviour working on a summarization model:
As I increase my training samples, as expected, my validation loss decreases BUT the ROUGE metrics do not improve (do not increase), so the best model based on validation loss is not the best model based on ROUGE, they are not even close: a model with a quite larger validation loss is the one with the best ROUGE scores.
Can anyone explaina bit this? How normal is it?
Thank you guys

Context: What is your objective function for validation loss? I think the answer to your question depends on several factors, the biggest of which are: the kind of rouge you are using (R1 vs. R2 vs. RL) and the score range (high vs. low).


Trouble saving/loading fine-tuned BART model

Question:
Hello! I am fine-tuning the ‘sshleifer/distilbart-cnn-12-6’ model using the following code.
model = AutoModelForSeq2SeqLM.from_pretrained(‘sshleifer/distilbart-cnn-12-6’)
optim = AdamW(model.parameters(), lr= params_layered.lr)
model.train()
for epoch in range(params_layered.epochs):
with tqdm(train_loader_l, unit=“batch”) as tepoch:
for inputs,masks,labels in tepoch:
tepoch.set_description(f"Epoch {epoch}")
optim.zero_grad()
outputs = model(inputs, attention_mask=masks, labels=labels)
loss = outputs[0]
loss.backward()
optim.step()
This produces great results. Then, I’d like to save this model and load it back. However, I cannot seem to do this correctly. First, it’s curious to note that when I print out the weights of this fine-tuned model, they are identical to the original pre-trained (before fine tuning) model weights. And indeed, when I save and load the model back, the results are the same as the model results before fine-tuning.
I am testing this using the code below:
(fine-tuned model)
print(dict(model.named_parameters()))
Output: "tensor([[-0.0369,  0.0782,  0.1621,  …,  0.1831,  0.0589, -0.0659], …
test = AutoModelForSeq2SeqLM.from_pretrained(‘sshleifer/distilbart-cnn-12-6’)
print(dict(test.named_parameters()))
Output: "tensor([[-0.0369,  0.0782,  0.1621,  …,  0.1831,  0.0589, -0.0659], … output matches w/ above print statement for all weights printed
I am attempting to save the weights using:
torch.save(model.state_dict(),’/dbfs/FileStore/…/summary_model.pt’)
net = AutoModelForSeq2SeqLM.from_pretrained(‘sshleifer/distilbart-cnn-12-6’)
arxiv = torch.load(’/dbfs/FileStore/…/summary_model.pt’)
net.load_state_dict(arxiv)
(All keys do match successfully)
I have also tried:
model.save_pretrained("/dbfs/FileStore/…/summary-model/")
test = AutoModelForSeq2SeqLM.from_pretrained("/dbfs/FileStore/…/summary-model/")
And both of these have identical weights to the above as well. Not sure what I’m doing wrong, would appreciate any advice!

Context: This actually resolved itself!


Re-training NLP model with training AND validation dataset after validation has been done

Question:
Hello,
I have a question about methodology. I have a good dataset (about 8000 samples) which I can use to train a model on an NLP task. As usual, I would normally divide the dataset into training and validation (90% - 10%), or perhaps even training, validation and test datasets (80% - 10% - 10%), and perform training as usual on the training set, then do a hyper parameter search using the performance on the validation set, and once I’m happy with my model parameters, I would run it once on the test set to get the final performance metrics.
Now, assuming (seems a reasonable assumption) that the model will not get worse if we add more training data to it, I was wondering if there is anything wrong with re-training the model with the full dataset once the steps above (validation and testing) have been completed. Obviously you can’t then re-test the model (because the newly trained model will have already seen all the samples in your dataset), but it seems reasonable to say that the performance metric you got on the test set when you only used 80% of the samples for training will not get any worse if you add the remaining 20% of the samples to the training set, hence you can keep the performance you got as a baseline, and then train the model on the full dataset knowing that the performance, if anything, is only going to get better than that.
Is there anything fundamentally wrong with this approach from a scientific or philosophical perspective? Any reason why it shouldn’t be done?

Context: It might work well, it might not. When you add data to your training run, the model will converge differently because it sees different data and optimizes accordingly. This might be good or bad (it is not a given that it will deterministically end up being better simply because you gave it more data), but the problem is that you simply cannot tell because you have no held-out set anymore. If you do wish to squeeze everything out of your data, cross validation is recommended.


Fine-tuning GPT2 for bidirectional training

Question:
Hallo,
we all know, that GPT2 is widely used in text generation task as autoregressive model. May I ask, could I further train GPT2 in a bidirectional attention, like BERT as auto-encoding model? If yes, how could I flexible control the uni- or bidirectional training in GPT2? Thanks a lot!!

Context:
I don’t think this is possible at all due to structural differences in the model’s design, but I may be wrong.
Reference this if you want more clarity on what I am saying - Comparison between BERT, GPT-2 and ELMo | by Gaurav Ghati | Medium
" Drawbacks: GPT is its uni-directional nature — the model is only trained to predict the future left-to-right context ." - from the article
Hope this answers your question