noamaanMulla-03/transformers-issues huggingface dataset:
Model card for German BERT fine-tuned for LER/NER
Author: mrm8488
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik 
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Create README.md
Author: hardyqr
Pull request: yes
Labels: name: model card


# What does this PR do?

Creare model card for `cambridgeltl/BioRedditBERT-uncased`.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik 
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Add AI-SOCO models
Author: AliOsm
Pull request: yes
Labels: name: model card

# What does this PR do?

Adding AI-SOCO language and classification models.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 

## Who can review?

@julien-c.

Comments:
TensorBoard/Wandb/optuna/raytune integration improvements.
Author: madlag
Pull request: yes

Improves TensorBoard logging by grouping train / eval metrics as it is usually done in TensorBoard.
Improves TensorBoard/optuna model hyper-parameters logging.
Improves optuna and Ray/tune integration, and provides model hyper-parameter naming.

Test (and sample code) is provided in test_trainer.TrainerHyperParameterIntegrationTest .

Some more work may be need to harmonize metrics naming for eval / train, as the "eval_" prefix used is not very convenient, using a "eval/" prefix would be more foolproof, and consistent with TensorBoard usage, but it would break quite some code, and so may be done in a separate PR.


Comments:
[multiple models] skip saving/loading deterministic state_dict keys
Author: stas00
Pull request: yes

Implementing https://github.com/huggingface/transformers/issues/7296

* [x] make testing save/load of `keys_to_never_save` a common test by moving it from `fsmt` modeling test
* [x] implement skipping for all models with static positional embeddings: mbart, pegasus, marian (fsmt had this feature in first place)

Fixes: #7296

@LysandreJik, @sshleifer, @sgugger 

Comments:
OK, so how would you recommend we run a specific common test for a model whose test module isn't running common tests at all? 

Please have a look at the approach I used for `mbart`  https://github.com/huggingface/transformers/pull/7878/files#diff-f3b9af7b4721bdf972a72e35a35e8742ddc986f12218e728cd2b6ad30a156abb

I basically made a very stripped down `ModelTester` and then used an alias hack to run just a specific common test via:
```
class SelectiveCommonTest(unittest.TestCase):
    test_save_load_keys_to_never_save = ModelTesterMixin.test_save_load_keys_to_never_save
```
Is this any good, or would you recommend a different approach - the idea is not to copy the same test to each test module.


I like how you did it.
Adding @LysandreJik and @sgugger to see if they agree before you do it 5 times.


> I imagine you double-checked the test was indeed executed.

Yes. 

Indeed, my initial test that I wrote for fsmt wasn't executing, since I renamed the keys in the code and forgot the test and the test was no-op if it didn't find the right keys. So some print()s helped to verify it indeed worked in this new incarnation.

I think  `test_save_load_missing_keys` should be handled separately since it can add a significant overhead if run by all models, since I noticed tf models are **very** slow at save/load tests.. I reverted moving it from fsmt to common tests for now - back to just fsmt - will do a separate PR about it.


Anybody with a good knowledge of XLM? @sshleifer suggested to add xlm to this PR, but I'm not sure whether it's safe to not save `self.position_embeddings`, since as you can see here:
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlm.py#L439
the deterministic non-trained weights happen only if `config.sinusoidal_embeddings` is True. Otherwise it gets trained like any other embed layer.

Which leaves just: `position_ids` which is tiny a set of 512-2048 floats, which is not worth bothering for IMHO.



wrt t5 - I don't see anything positional or deterministic for that matter, but this is quite a different model - are there any state_dict keys here that would be a good candidate for not saving?

I implemented this for: mbart, pegasus, marian - all subclasses of `BartForConditionalGeneration` w/ `static_position_embeddings=True`. `fsmt` already had it.

Anything else that I may have missed?

If not, this PR is good to go.
fix 'encode_plus' docstring for 'special_tokens_mask' (0s and 1s were reversed)
Author: epwalsh
Pull request: yes

# What does this PR do?

Fixes the docstring for `encode_plus`.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger or anyone really.


Comments:
[utils/check_copies.py] fix DeprecationWarning
Author: stas00
Pull request: yes

in `tests/test_utils_check_copies.py` I was getting intermittently:
```
utils/check_copies.py:52
  /mnt/nvme1/code/transformers-comet/utils/check_copies.py:52: DeprecationWarning: invalid escape sequence \s
    while line_index < len(lines) and re.search(f"^{indent}(class|def)\s+{name}", lines[line_index]) is None:
```
So this should fix it. Not sure why it wasn't showing up all the time.

@sgugger 

Comments:
Thanks!
Bert Fine-Tuning on SQuAD with native TF2
Author: srcarroll
Pull request: no

I'm trying to run Bert fine tuning on SQuAD with Tensorflow's model.fit function.  I was able to run the tutorial on sequence classification from here https://huggingface.co/transformers/training.html#tensorflow, but a similar set up does not seem to work for question answering.  I am able to get it working with synthetic data, but not real data.  I can never seem to get the data in the right format.  Here is the script I am trying to use

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForQuestionAnswering.from_pretrained('bert-base-uncased')

data = tfds.load('squad')
examples = SquadV1Processor().get_examples_from_dataset(data)
train_dataset = squad_convert_examples_to_features(examples, tokenizer, max_seq_length=328, doc_stride=128, max_query_length=32, is_training=True, return_dataset='tf')

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

model.compile(optimizer=optimizer, loss=loss)
model.fit(train_dataset, epochs=2, steps_per_epoch=1)

I've tried many other things, but this is the closest I got to something working.  The problem is that the dataset has keys that aren't recognized.  Specifically, I get this error

ValueError: Found unexpected keys that do not correspond to any Model output: dict_keys(['start_positions', 'end_positions', 'cls_index', 'p_mask', 'is_impossible']). Expected: ['output_1', 'output_2'] 

I might be able to find someway to manually remove these keys, but I feel like there should be some canonical way of getting this to work correctly.  I would appreciate any suggestions.  Thanks in advance.

Comments:
Hello !

You have an example here https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb

Perfect! Thank you, I did not see that.  I think I'm all good now.

I want to have a fine tuning on my own dataset,  but I don't know how to deal with the data format. Any suggestion will be appreciated. 

@mug2mag this post is closed, please open another one with more detail on what you would like to do. Thanks.
[s2s testing] turn all to unittests, use auto-delete temp dirs
Author: stas00
Pull request: yes

Currently in `examples/seq2seq/test_seq2seq_examples.py` many tmp dirs remain uncleaned by the end of the test:

This PR does:
* Now that we have `parameterized`, convert all tests into unittests
* Now that we have unittests, we can use auto-delete tmp dirs easily using `get_auto_remove_tmp_dir()`
* convert 4 test files
* moved out the broken multi_gpu test - working on it separately here https://github.com/huggingface/transformers/pull/7281 - will re-add when it's complete.

@sshleifer 

Comments:
@sshleifer, one thing I wasn't sure about - since now the temp dir creation/destruction is automated and it's easy to hardwire those by just passing the explicit path `self.get_auto_remove_tmp_dir("/tmp/favoritedir")` when debugging - I removed the prefices that were used in `tempfile.mkdtemp`. Do you still want to have the option of having a specific prefix? With prefix you still have to hunt for the unique ID which changes on every test rerun. This is a much simpler option, IMHO.

To me it sounds it'd be useful during a debug to have one common path prefix for all tmp dirs of the same test and then add suffices instead for the different specific tmp dir of the test. 

Whatever you think is best! 

I think the prefixes are no longer needed. So it's good to go.
Herbert tokenizer auto load
Author: rmroczkowski
Pull request: yes

Adding HerbertTokenizer imports  for autoloading proper tokenizer.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik.
@julien-c

Comments:
support relative path for best_model_checkpoint
Author: HaebinShin
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #7431

When I give a relative path to `output_dir`, it raises some error at line 1222.
https://github.com/huggingface/transformers/blob/83481056921296fadbdc86cd51c157a9a9327946/src/transformers/trainer.py#L1205-L1227

If I give './path1/path2' to `output_dir`, `checkpoints_sorted` becomes 'path1/path2' by Path library on line 1208.
But `self.state.best_model_checkpoint` is still './path1/path2'.
So it will raise error as below.
```
Traceback (most recent call last):
  File "../finetune.py", line 369, in <module>
    main(**vars(args))
  File "../finetune.py", line 315, in main
    device)
  File "../finetune.py", line 120, in train
    trainer.train(optim_pretrained_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 803, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 860, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 918, in _save_checkpoint
    self._rotate_checkpoints(use_mtime=True)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1235, in _rotate_checkpoints
    checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1223, in _sorted_checkpoints
    best_model_index = checkpoints_sorted.index(self.state.best_model_checkpoint)
ValueError: './results/use_pretrained_test/checkpoint-1162' is not in list
```

So I resolve this error by using Path for `self.state.best_model_checkpoint`.
Any other idea is also welcome. please check it


Comments:
How to load a pretrained TF model using AutoModel?
Author: erikchwang
Pull request: no

Run the following code:
```
import tensorflow as tf
from transformers import AutoModel, TFBertModel

auto_model = AutoModel.from_pretrained("bert-base-uncased")
tfbert_model = TFBertModel.from_pretrained("bert-base-uncased")

print(auto_model.__class__)
print(tfbert_model.__class__)
```
Then the output is:
```
<class 'transformers.modeling_bert.BertModel'>
<class 'transformers.modeling_tf_bert.TFBertModel'>
```
It seems that AutoModel defaultly loads the pretrained PyTorch models, but how can I use it to load a pretrained TF model?

Comments:
Hi @erikchwang, you should use `TFAutoModel` instead

Is this TFAutoModel mentioned in the document? I cannot find it...

I'll add it to the model pages soon. Thanks!

when I load TF model by use AutoModel from your document, there are many errors, like this 
`model = AutoModel.from_pretrained(r'/Users/maxiong/Workpace/Code/transformers/pre_model/bert_model.ckpt.index', from_tf=True,  config=config)
`
![image](https://user-images.githubusercontent.com/30991932/76698572-99112800-66df-11ea-9dd7-5e382abda388.png)

when I used TFAutoModel to load a model, there is like this
`model = TFAutoModel.from_pretrained(r'/Users/maxiong/Workpace/Code/transformers/pre_model/bert_model.ckpt.index', config=config)
`
![image](https://user-images.githubusercontent.com/30991932/76698602-f73e0b00-66df-11ea-9121-67a663b4a8ee.png)

I tried many functions to load TF Pretraining model in your document, most of them appeared errors


I can't able to load model for  model = TFAutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
and
TFAutoModel
    .from_pretrained('microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext')
anyone ?

> I can't able to load model for model = TFAutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
> and
> TFAutoModel
> .from_pretrained('microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext')
> anyone ?

HI @blmali, I had the same issue when trying to load "emilyalsentzer/Bio_Discharge_Summary_BERT". I solved it by passing `from_pt` argument as `True`:
`model = TFAutoModel.from_pretrained("emilyalsentzer/Bio_Discharge_Summary_BERT", from_pt=True)`.

I hope this helps.
GPT2 - Remove else branch adding 0 to the hidden state if token_type_embeds is None.
Author: mfuntowicz
Pull request: yes

Currently, when `token_type_embeds` is `None` we set its value to `0` and add it to the triplet `inputs_embeds + position_embeds + token_type_embeds`.

This can be simplified to: 
- avoid summing 0 over many elements
- avoid using raw Python scalar value which cannot be traced by TorchScript / ONNX when exporting.

Leading to: 

> [ONNXRuntimeError] : 1 : FAIL : TensorRT input: 200 has no shape specified. Please run shape inference on the onnx model first.

Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

Comments: