noamaanMulla-03/transformers-issues huggingface dataset:
Add AzureML in integrations via dedicated callback 
Author: davidefiocco
Pull request: yes

# What does this PR do?

I propose this PR to allow transformers to call AzureML logging using https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run(class)?view=azure-ml-py

The intended behaviour is to enable `transformers` users to track metrics in the AzureML UI in this fashion

![image](https://user-images.githubusercontent.com/4547987/97200283-4f2e0b80-17b1-11eb-8928-20d6062c69a4.png)

Contributors to https://github.com/microsoft/AzureML-BERT and folks @microsoft may well come up with a better implementation though!

I am glad to improve the following if reviewers like the idea, and update docs and tests if needed.
@reviewers feel free to add any suggestions as my contributions to transformers have been very limited so far :)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Yes, check for detais https://discuss.huggingface.co/t/how-to-integrate-an-azuremlcallback-for-logging-in-azure/1713/4
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@julien-c is aware of this and @sgugger participated in the thread on forums above and implemented callbacks with https://github.com/huggingface/transformers/pull/7596


Comments:
Code changes pass tests @sgugger ! Thanks @alvarobartt for having a look at those also.

I guess I need to take care of  docs/source/main_classes/callback.rst as well to complete the checklist though?

Yes, if you could add a line to the rst file, that would be great!

You might need to rebase to have the latest script for doc formatting, which should do everything to make the CI happy with just `make style`. Let me know if you need help.

Sorry for the somewhat messy hacktoberfest @sgugger ! 
I wasn't so aware of rst idiosyncrasies, and ways to diagnose issues, so I struggled a bit. Should be in order now (bonus, I fixed a typo on the way https://github.com/huggingface/transformers/pull/8062/commits/286f20c0594c3d16c824963c24c8fb1bc1d43bc6)
infer entailment label id on zero shot pipeline
Author: joeddav
Pull request: yes

# What does this PR do?

Adds an optional argument to the zero shot pipeline constructor to specify the label id of the NLI model that corresponds to "entailment", which it needs to calculate each candidate label's score. Most models in the hub use the last last label id, but some differ (e.g. the recent [ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli](https://huggingface.co/ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli)).

If the argument is not passed, the pipeline will attempt to look up the entailment dimension in the model config's id2label mapping. If the config does not specify the entailment dimension, the value will be set to `-1`, indicating the last dimension of the model output.

With this logic in place, the arg only needs to be passed when both (1) the model's entailment label id is not the last id and (2) when the model config's `label2id` doesn't specify the entailment id.

Comments:
Wouldn't it be better to:
- ask the model authors for the relevant models to update their config.json to include a `id2label`
- and/or to modify them automatically for them on the hub?

I feel this PR is adding a feature that works around an issue when we should be fixing the root issue (also fixes the displayed labels, other future features, etc.). Wdyt?

@julien-c Well, to be clear this PR ~~does~~ did two things:

1. Switches from always using the last index to using the index determined by looking in the model config if present
2. Gives the user the option to manually override the index in case the information isn't present in the config

If I understand correctly, your issue is just with (2). I think it's a fair point. I don't think we'll be able to ensure that [all NLI models](https://huggingface.co/models?search=nli) have a clearly defined label mapping though. But instead of an override arg, I think it might be better to just add a warning if the entailment label ID can't be found in the config.

@joeddav Ok yes 1/ is great.

> I don't think we'll be able to ensure that [all NLI models](https://huggingface.co/models?search=nli) have a clearly defined label mapping though.

Why not?

@julien-c Just because there are almost 100 results for ["NLI"](https://huggingface.co/models?search=nli) on the model hub and I'd guess from a quick sampling that the majority don't have a label mapping defined. For each model we'd have to figure out which label is which, which would mean either getting the author to look it up and tell us or else running tests on the correct dataset to figure it ourselves.

Do you think it'd be worthwhile to warn the user when uploading or creating configs with generic/missing label mappings (or with any other important fields missing) going forward? Defining a label2id seems like a rather obscure property that I would assume is purely cosmetic if I were uploading a model, i.e. I wouldn't expect it to actually impact code behavior for someone using my model.
run_language_modeling crashes with import cannot import name 'DataCollatorForWholeWordMask' from 'transformers'
Author: spacemanidol
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: linux
- Python version: 3.8
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No


## Information
I am trying to run the example for [language modeling](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) but can't get it to start. Import fails
Traceback (most recent call last):
  File "run_language_modeling.py", line 32, in <module>
    from transformers import (
ImportError: cannot import name 'DataCollatorForWholeWordMask' from 'transformers' (/home/spacemanidol/miniconda3/envs/prunetransformer/lib/python3.8/site-packages/transformers/__init__.py)
Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ X] the official example scripts: (give details below)
running the language modeling example script

The tasks I am working on is:
* [ X] an official GLUE/SQUaD task: (give the name)
Language modeling
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create conda enviorment. install transformers from the source. 
2. Run language modeling example script


## Expected behavior
Example runs. 


Comments:
As explained in the README of the examples, you need an [installation from source](https://huggingface.co/transformers/installation.html#installing-from-source) to run the examples, which you don't have, otherwise you would have this object.

Alternatively, you can run the examples associated to your current version by using the files on the [last release tag](https://github.com/huggingface/transformers/releases/tag/v3.4.0).
Documentation code snippet has extra ) after model code 
Author: bhadreshpsavani
Pull request: no

Documentation at https://huggingface.co/transformers/model_doc/roberta.html#tfrobertaforsequenceclassification has code snippet

```
>> from transformers import RobertaTokenizer, TFRobertaForSequenceClassification
>> import tensorflow as tf
>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
>> model = TFRobertaForSequenceClassification.from_pretrained('roberta-base', return_dict=True))
>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
>> inputs["labels"] = tf.reshape(tf.constant(1), (-1, 1)) # Batch size 1
>> outputs = model(inputs)
>> loss = outputs.loss
>> logits = outputs.logits
```

in the fourth line there is extra `)` at the end.

This is issue is also for other model code snippet as well

Comments:
Hello, this has been fixed on: https://github.com/huggingface/transformers/pull/8082 and is now available in the `master` documentation and will be updated in the next version. Thanks for opening an issue!
BertTokenizer's add_token won't add token
Author: HenryPaik1
Pull request: no

`add_token` actually won't add token. Please refer to the code below:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
tokenizer.vocab_size
>>30522

tokenizer.add_tokens(new_tokens=['[SUBJ]', '[OBJ]'], special_tokens=True)
>>2

tokenizer.vocab['[OBJ]']
>> KeyError: '[OBJ]'

tokenizer.vocab_size
>>30522 # not changed

tokenizer.tokenize('[OBJ]')
>>['[', 'ob', '##j', ']'] # expected: '[OBJ]'
```

Comments:
It seems not work until I save the new tokenizer. I close the issue.
```
tokenizer.save_pretrained('/path/to/tokenizer')
tokenizer = BertTokenizer.from_pretrained('/path/to/tokenizer')
```

looks like tokenizer.vocab_size does not update after add tokens. but len(tokenizer) shows correct number 

Yes, the `vocab_size` only contains the information relative to the initial vocabulary. You can find the added tokens either in `tokenizer.get_added_vocab()`, which returns the dictionary, or `tokenizer.added_tokens_encoder`, which returns the amount of added tokens.
[testing] port test_trainer_distributed to distributed pytest + TestCasePlus enhancements
Author: stas00
Pull request: yes

This PR:

* [x] ports `test_trainer_distributed` to run with pytest - it will skip if gpus < 2.

* [x] includes various improvements via refactoring now 3 use cases of distributed testing by extending `TestCasePlus` with a whole set of convenient features:

Feature 1: A set of fully resolved important file and dir path accessors.

In tests often we need to know where things are relative to the current test file, and it's not trivial since the test could be invoked from more than one directory or could reside in different sub-directories. This class solves this problem by sorting out all the basic paths and provides easy accessors to them:

* ``pathlib`` objects (all fully resolved):

   - ``test_file_path`` - the current test file path (=``__file__``)
   - ``test_file_dir`` - the directory containing the current test file
   - ``tests_dir`` - the directory of the ``tests`` test suite
   - ``examples_dir`` - the directory of the ``examples`` test suite
   - ``repo_root_dir`` - the directory of the repository
   - ``src_dir`` - the directory of ``src`` (i.e. where the ``transformers`` sub-dir resides)

* stringified paths - same as above but these return a string, rather than a ``pathlib`` object

   - ``test_file_path_str``
   - ``test_file_dir_str``
   - ``tests_dir_str``
   - ``examples_dir_str``
   - ``repo_root_dir_str``
   - ``src_dir_str``

Feature 2: Get a copy of the ``os.environ`` object that sets up ``PYTHONPATH``  correctly, depending on the test suite it's invoked from. This is useful for invoking external programs from the test suite - e.g. distributed training.

```
    def test_whatever(self):
        env = self.get_env()
        # now call the external program, passing ``env`` to it
```

All these are also documented in `testing.rst`.

Fixes: #8058

@sgugger, @LysandreJik, @sshleifer 


Comments:
LGTM! cc @patrickvonplaten for awareness!
[testing] port test_trainer_distributed to run with pytest
Author: stas00
Pull request: no

Extracting the request from https://github.com/huggingface/transformers/pull/7993#issuecomment-716508513 to this issue to make it easier to track.

Now that we have a framework to run distributed under `pytest` `test_trainer_distributed` needs to be ported there. 

I will work on that.


Comments:
Ooops, closed it by mistake too early. But it has been resolved here:
https://github.com/huggingface/transformers/pull/8107
[seq2seq distributed] child process stuck on error
Author: stas00
Pull request: no


Filing an issue for myself to help move along https://github.com/huggingface/transformers/pull/7281
```
pytest -sv examples/seq2seq/test_seq2seq_examples_multi_gpu.py
```
gets stuck if one of the child subprocesses throws an error.

Remove `skip_output_dir_check=True,` to reproduce the problem.

It could be related to realtime async read/write pipes getting into a deadlock - see the XXX warning in that test, need to first try whether the problem goes away if I switch to non-real time pipes.

I will get back to it later.



Comments:
OK, this is the problem with PL if anything, as the command:
```
PYTHONPATH="src" /home/stas/anaconda3/envs/main-38/bin/python /mnt/nvme1/code/huggingface/transformers-distr-train-test/examples/seq2seq/distillation.py --supervise_forward --normalize_hidden --label_smoothing=0.0 --eval_beams=1 --val_metric=loss --save_top_k=1 --adafactor --early_stopping_patience=-1 --logger_name=default --length_penalty=0.5 --cache_dir= --task=summarization --num_workers=2 --alpha_hid=0 --freeze_embeds --sortish_sampler --student_decoder_layers=1 --val_check_interval=0.5 --output_dir=/tmp/tmpqajqhzwo --no_teacher --fp16_opt_level=O1 --gpus=2 --max_grad_norm=1.0 --do_train --do_predict --accumulate_grad_batches=1 --seed=42 --model_name_or_path=sshleifer/tinier_bart --config_name= --tokenizer_name=facebook/bart-large --learning_rate=0.3 --lr_scheduler=linear --weight_decay=0.0 --adam_epsilon=1e-08 --warmup_steps=0 --max_epochs=2 --train_batch_size=1 --eval_batch_size=2 --max_source_length=12 --max_target_length=12 --val_max_target_length=12 --test_max_target_length=12 --n_train=-1 --n_val=-1 --n_test=-1 --student_encoder_layers=1 --freeze_encoder --data_dir=/tmp/tmpo_9t6k7e --alpha_mlm=0.2 --alpha_ce=0.8 --teacher=sshleifer/bart-tiny-random
```
hangs on its own, so this is definitely not an issue of the newly added distributed support for `pytest`. 

I think that when one of the workers fails (e.g. asserts on output directory already exists) and the coordinating process gets stuck waiting for the worker to send something back.
fix(trainer_callback]: typo
Author: borisdayma
Pull request: yes

# What does this PR do?

Fix a typo in `trainer_callback`
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR. @sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Thanks a ton!
How to pass tokenized hypotheses to TFRobertaForSequenceClassification model directly for faster inference?
Author: akshatapatel
Pull request: no

Hi, 
I have been running the zero shot classification pipeline for my use case by passing each text and it’s corresponding list of hypotheses labels, however it takes around 3 hours on a 32B GPU to classify ~22000 sentences(each sentence can have a varying number of labels ranging from 70 to 140 labels).

Is there a way to reduce the computation time by passing the embeddings to the sequence classification model directly rather than the raw list of labels?

Comments:
Hello @akshatapatel!

Pinging @joeddav as he is the person who takes care of the zero-shot classification in the lib.

Can you share with us a piece of code on how you are trying to do your sequence classification?

Question moved to [the forums](https://discuss.huggingface.co/t/new-pipeline-for-zero-shot-text-classification/681/30?u=joeddav). Other discussions on speeding up zero shot classification [here](https://discuss.huggingface.co/t/speeding-up-zero-shot-classification-solved/692) and [here](https://discuss.huggingface.co/t/way-to-make-inference-zero-shot-pipeline-faster/1384)
Document the various LM Auto models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the documentation for the three classes of mdoels for LM (`AutoModelForCausalLM`, `AutoModelForMaskedLM` and `AutoModelForSeq2SeqLM`) and their TF equivalent. It also removes the documentation of `AutoModelWithLMHead` which is deprecated.

Comments:
[DOC] Improve pipeline() docstrings for config and tokenizer
Author: BramVanroy
Pull request: yes

As currently written, it was not clear to me which arguments were needed when using a non-default model in `pipeline()`. It seemed that when you provided a non-default `model`, that you still needed to manually change the `config` and `tokenizer` because otherwise the "task's default will be used". In practice, though, the pipeline is smart enough to automatically choose the right config/tokenizer for the given model. This PR clarifies that a bit in the docstrings/documentation, by explaining exactly which priorities are used when loading the tokenizer. A small change was made for `config`, too.

Admittedly, the wording for the tokenizer part is a bit off (programmatical, even), but I think it should make clear how the right tokenizer is loaded.

cc @sgugger 

Comments:
@sgugger I made the change as you requested. Not sure why CI is failing on build_doc. Seems to have to do with some env installation.

The failure is spurious (basically the new version of pytorch is not cached on the CI and it fails to download it sometimes). Thanks for th fix!
Name or path should be added on configuration as well
Author: LysandreJik
Pull request: yes

Close https://github.com/huggingface/transformers/issues/8035

Currently a configuration initialized with 
```
config = BertConfig.from_pretrained(model_name)
```
does not have the `_model_name_or_path` attribute, whereas a configuration intialized from a model with 
```py
model = BertModel.from_pretrained(model_name)
``` 
does.

This fixes the discrepancy and fixes the failing test in the process.

Comments:
Added an Adapter training example
Author: hmohebbi
Pull request: yes
Labels: name: wontfix

Contributed an example of adding and training adapters among Hugging's BERT layers according to the paper of [Houlsby et al. (2019)](https://arxiv.org/abs/1902.00751).
GLUE test results have also provided for this implementation where BERT(base, uncased) was used as the pre-trained model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.