noamaanMulla-03/transformers-issues huggingface dataset:
ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING'
Author: iBrushC
Pull request: no

ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING' from 'transformers' (C:\Users\<username>\anaconda3\envs\tensorflow\lib\site-packages\transformers\__init__.py)

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
[x] the official example scripts: (give details below)
a necessary package does not exist for import

The tasks I am working on is:
[x] my own task or dataset: (give details below)
I am trying to finetune the GPT2 library to work with recipes

## To reproduce

Steps to reproduce the behavior:

1. create conda virtualenv
2. install latest version of Tensorflow GPU
3. install transformers library
4. navigate to the 'language-modeling' folder
5. run 'python run_language_modeling.py' in conda virtualenv

LOG:
```
python run_language_modeling.py
2020-06-01 23:53:43.603759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "run_language_modeling.py", line 29, in <module>
    from transformers import (
ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING' from 'transformers' (C:\Users\<name>\anaconda3\envs\tensorflow\lib\site-packages\transformers\__init__.py)
```

## Expected behavior

There should be an error with not receiving variables or no output at all. If you were to insert data it would train an instance of GPT2 and save it to the output directory

## Environment info

- `transformers` version: 2.10.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.7
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


It would be nice to figure out what the issue is. I am aware that there are 3 questions similar to this, but they all use torch while I am using tensorflow. All of the solutions I have tried (reinstalling tensorflow, reinstalling transformers, updating packages, reformatting code, etc.) have not worked. Thanks.


Comments:
This might help: #3444

`MODEL_WITH_LM_HEAD_MAPPING` is a mapping containing all the *Pytorch* models that have an LM head. Since you don't have PyTorch installed, you can't import them.

With TensorFlow you're probably looking for `TF_MODEL_WITH_LM_HEAD_MAPPING`.

Please note that the `run_language_modeling.py` script is currently PyTorch only. A TensorFlow version will be available in the future.
When using the Hugging Face Transformer, if I set my pad_token to be a different token then the default, do I need to train my model on that new pad_token as well?
Author: h56cho
Pull request: no

Hello,

For using the GPT2DoubleHeadsModel, I used ``<eos>`` as the last token of my text sequence, which my model will use to make predictions for multiple-choice questions.
I also set ``<pad>`` as the padding token in my tokenizer, which is different than the default.

When using the Hugging Face pre-trained GPT2DoubleHeadsModel, do I need to train the already pre-trained Transformer because the two tokens I mentioned above are new?

Thank you,


Comments:
Hi @h56cho, if you add new tokens then yes, you'll need to train the model. If you just want `eos`  and `pad` token then its a good idea to use the model defaults. `eos` and  `pad`  tokens are already available in GPT2 model 

Hi,

Thank you for your reply.
So what I am getting is that if I add any new "special token" onto the existing pre-trained tokenizer, I will need to re-train the pre-trained Transformer to make it learn that new special token.

 But what if I just add extra non-special tokens? for example, a word "paradox" is not included in the existing GPT-2 tokenizer, so say I add the word "paradox" to the existing set of GPT-2 vocabulary. If I didn't make any changes to the special tokens in the GPT-2 tokenizer, do I still need to train the pre-trained GPT-2 because I added a new word to a set of vocabulary?

Thanks,
TFTrainer: Checkpoints not getting saved in `output_dir` but in {cwd}/checkpoint
Author: 0dust
Pull request: no

I am using TFTrainer for the SQuAD task.
Checkpoints are being created in cwd/checkpoint insted of output_dir.

**Potential Cause:**
https://github.com/huggingface/transformers/blob/9ca485734aea269961d63a040ff194365d151fd1/src/transformers/trainer_tf.py#L156
Instead of PREFIX_CHECKPOINT_DIR we need to have
```python
os.path.join(self.args.output_dir, PREFIX_CHECKPOINT_DIR)
```

Comments:
Pinging @jplu as it might be of interest.

@0dust This is the intent behavior :)

A solution would be to add a parameter to the arguments to select the checkpoint folder location you want

@jplu Sorry if i am missing something but isn't 'output_dir' the folder to save the checkpoint?
https://github.com/huggingface/transformers/blob/ed4df85572924871758ca32133b46116121c706f/src/transformers/training_args.py#L41-L43

Not for the TF one, it is one of the few difference between the both trainers.

Ohh,  I see!  Thanks for the clarification. Just a quick question before i close the issue,  Is there any specific reason for this?  Or it's just a matter of time before it starts to behave similar to pytorch trainer. 

It is just matter of time :)
Error on dtype in modeling_bertabs.py file
Author: pn12
Pull request: no

# 🐛 Bug

Traceback (most recent call last):
  File "run_summarizationn.py", line 324, in <module>
    main()
  File "run_summarizationn.py", line 309, in main
    evaluate(args)
  File "run_summarizationn.py", line 84, in evaluate
    batch_data = predictor.translate_batch(batch)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 797, in translate_batch
    return self._fast_translate_batch(batch, self.max_length, min_length=self.min_length)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 844, in _fast_translate_batch
    dec_out, dec_states = self.model.decoder(decoder_input, src_features, dec_states, step=step)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 231, in forward
    step=step,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/kaggle/working/transformers/examples/summarization/bertabs/modeling_bertabs.py", line 328, in forward
    dec_mask = torch.gt(tgt_pad_mask + self.mask[:, : tgt_pad_mask.size(1), : tgt_pad_mask.size(1)], 0)
RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

## Information

Model I am using (Bert, XLNet ...):bertabs

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [ ] the official example scripts: (give details below) - Yes
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below) - my own task



Comments:
feat(trainer): log epoch and final metrics
Author: borisdayma
Pull request: yes

This PR adds new logging capabilities to `Trainer`:

* logs epoch -> this let us compare runs with different batch sizes, using "epoch" as the x-axis instead of "global_step"
* logs final metrics -> metrics are also logged by `Trainer.evaluate` at the end of the script (or at the start if we just use `--do_eval` and not train).

In below example, we see `epoch` has been used for x-axis and `eval_loss` has one extra point corresponding to the end of the script.
![image](https://user-images.githubusercontent.com/715491/81350982-d57e5600-9088-11ea-8ce6-400bda09bea7.png)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=h1) Report
> Merging [#4221](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c99fe0386be118bceaab1c85cdb8309eb8cb8208&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `39.39%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4221/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4221      +/-   ##
==========================================
+ Coverage   78.31%   78.34%   +0.02%     
==========================================
  Files         120      120              
  Lines       19854    19864      +10     
==========================================
+ Hits        15549    15562      +13     
+ Misses       4305     4302       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4221/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `42.73% <39.39%> (+2.55%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4221/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=footer). Last update [c99fe03...f220d8f](https://codecov.io/gh/huggingface/transformers/pull/4221?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Already merged through PR #4324
Bart model for textinfilling
Author: andompesta
Pull request: no
Labels: name: seq2seq

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
@sshleifer
I was wandering why you used a registered_buffer to define the ``final_logits_bias`` in the BartForConditionalGeneration. In my understanding a registered_buffer would have not gradient so if I fine-tune the a Bart model for generation these biases would not be updated (remain to 0). Should we use  register_parameter or remove the bias completely or there is something I haven't understand ? 


<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
it is a hack to make `MarianMTModel`, which inherits from Bart work. For the Bart models that parameter does nothing, as you suggest. You can remove or ignore.

Thanks got it
[cleanup] PretrainedModel.generate: remove unused kwargs
Author: sshleifer
Pull request: yes

`_generate_beam_search` and `_generate_no_beam_search` do not use `bos_token_id` or `decoder_start_token_id`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=h1) Report
> Merging [#4761](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5bf9afbf351f9419505eb1c9e0c5ab78883c3caf&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4761/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4761      +/-   ##
==========================================
- Coverage   77.41%   77.31%   -0.10%     
==========================================
  Files         128      128              
  Lines       21059    21059              
==========================================
- Hits        16302    16282      -20     
- Misses       4757     4777      +20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4761/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.15% <ø> (ø)` | |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4761/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.66% <0.00%> (-6.37%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4761/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.80% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4761/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.79% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=footer). Last update [5bf9afb...95a6ef1](https://codecov.io/gh/huggingface/transformers/pull/4761?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Multilingual BART -
Author: sshleifer
Pull request: yes

This adds `mbart-en-ro` model, a BART variant finetuned on english-romanian translation.
### TODO
- [x] (docs) pretrained_model.rst
- [ ] (docs) README.md
- [ ] (docs) bart.rst


### Differences with Bart
`config.normalize_before`: all the `LayerNorm` calls happen before attention calls
`config.add_final_layer_norm`: There is one extra layer_norm in the decoder
`config.scale_embedding`: embeddings are multiplied by 32 (`sqrt(d_model=1024)`)

### Future PRs
- The model returns the same variables as fairseq, but the tokenizer is not yet at parity with fairseq. This is the next PR in the pipeline.
- the `mbart-large-cc25` (no finetuning) model has a very different state dict. Also WIP.

### Misc
- the link_tester got angry about me not typing out URLs in this PR. Unclear why it didn't happen earlier.
Needs documentation but unclear where to put it.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=h1) Report
> Merging [#3602](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1a16d9d94a81e95463b166adfce4a8e02cdc47eb&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3602/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3602   +/-   ##
=======================================
  Coverage   78.06%   78.06%           
=======================================
  Files         100      100           
  Lines       17181    17181           
=======================================
  Hits        13413    13413           
  Misses       3768     3768           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=footer). Last update [1a16d9d...1a16d9d](https://codecov.io/gh/huggingface/transformers/pull/3602?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, is there any reason that cc-25 was removed and only the fine-tuned one is kept? any way I can quickly enable that? thanks

When the authors released the CC25 checkpoint, it was shaped differently than `mbart-large-en-ro` and I am not clear on whether that is fixed yet.
See https://github.com/pytorch/fairseq/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+mbart
bart-large-cnn model weights updated?
Author: lzmax888
Pull request: no

Hi,

Today, I noticed that the bart model path has been changed to 'facebook/bart-large-cnn' rather than 'bart-large-cnn'. And I run the demo below, but the result changed since then. (seems worse) 

https://colab.research.google.com/drive/11hKBPfsfBXPKo-dK_gHsPklF4PcNflQZ#scrollTo=dyTJ_ZavDp1q

So, is that weights updated?

Thanks,
Max

Comments:
Hello! The weights have not been updated. The path has been changed in the last version, as you can see in the [release notes](https://github.com/huggingface/transformers/releases/tag/v2.11.0) (cc @julien-c), but no change was done to these weights.
Codecov setup
Author: LysandreJik
Pull request: yes

Setup codecov

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=h1) Report
> Merging [#4768](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2b8b6c929e282958a920ba2aa26ee59106986ec3&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4768/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4768      +/-   ##
==========================================
+ Coverage   77.31%   77.33%   +0.01%     
==========================================
  Files         128      128              
  Lines       21059    21059              
==========================================
+ Hits        16282    16285       +3     
+ Misses       4777     4774       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4768/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4768/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.03% <0.00%> (+1.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=footer). Last update [2b8b6c9...fc6d7f5](https://codecov.io/gh/huggingface/transformers/pull/4768?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Remove unnecessary model_type arg in example
Author: zphang
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=h1) Report
> Merging [#4771](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e645b9ab9407e1c1b2c168317dc79fe13fc6e0b4&el=desc) will **decrease** coverage by `0.81%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4771/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4771      +/-   ##
==========================================
- Coverage   77.31%   76.49%   -0.82%     
==========================================
  Files         128      128              
  Lines       21059    21059              
==========================================
- Hits        16281    16110     -171     
- Misses       4778     4949     +171     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/4771/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19ncHQyLnB5) | `31.32% <0.00%> (-55.07%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4771/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `89.91% <0.00%> (-0.25%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4771/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.27% <0.00%> (+0.41%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4771/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.03% <0.00%> (+1.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=footer). Last update [e645b9a...7ad3a66](https://codecov.io/gh/huggingface/transformers/pull/4771?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).