noamaanMulla-03/transformers-issues huggingface dataset:
can anyone tell me how to do the pretraining of Reformer model on my text data?
Author: doppler21
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
```python
from transformers import ReformerModelWithLMHead, ReformerConfig
config = ReformerConfig()  # define the config as you like
model = ReformerModelWithLMHead(config)
loss = model(input_ids, labels=input_ids)  # input_ids are automatically shifted for labels
=> train
```

All this can also be done using the trainer. See this notebook for example:
https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb
Training TFBertForSequenceClassification with custom X and Y data
Author: rahulg963
Pull request: no
Labels: name: wontfix

name: Usage

I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library.

I followed the example given on their github page, I am able to run the sample code with given sample data using tensorflow_datasets.load('glue/mrpc'). However, I am unable to find an example on how to load my own custom data and pass it in  model.fit(train_dataset, epochs=2, steps_per_epoch=115, validation_data=valid_dataset, validation_steps=7).

How can I define my own X, do tokenization of my X and prepare train_dataset with my X and Y. Where X represents my input text and Y represents classification category of given X.

Sample Training dataframe :
```
    text    category_index
0   Assorted Print Joggers - Pack of 2 ,/ Gray Pri...   0
1   "Buckle" ( Matt ) for 35 mm Width Belt  0
2   (Gagam 07) Barcelona Football Jersey Home 17 1...   2
3   (Pack of 3 Pair) Flocklined Reusable Rubber Ha...   1
4   (Summer special Offer)Firststep new born baby ...   0
```
```
Question already asked on SO : 
https://stackoverflow.com/questions/60463829/training-tfbertforsequenceclassification-with-custom-x-and-y-data
```

Comments:
Maybe this is a little late but you could take a look in both `examples/run_tf_glue.py` and [this function](https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/glue.py#L31-L168) from`src/transformers/data/processors/glue.py` and write a custom training script based from those.

To make things a little more concrete, I've written and annotated [an end-to-end example](https://gist.github.com/papapabi/124c6ac406e6bbd1f28df732e953ac6d) of how to fine-tune a `bert-base-cased` model from your `DataFrame`'s spec. Do comment if it helps you out!

@papapabi Thank you for your inputs. I will check this out.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
After enable fp16, torch.save model has error
Author: Charonnnnn
Pull request: no
Labels: name: wontfix

# 🐛 Bug

After complete training, the model cannot be saved.
## Information
![image](https://user-images.githubusercontent.com/37766299/78298025-7b5b1400-7563-11ea-8928-aa1dae6fb0eb.png)
![image](https://user-images.githubusercontent.com/37766299/78298046-8746d600-7563-11ea-955e-49b1e6673771.png)



Comments:
Please use the template in the future. It is there for a reason. As mentioned in the template, don't post a screenshot. Use code blocks, post your code or the example script that you used, and the error trace. Also provide your version of PyTorch and Python.

https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
parse arguments from dict
Author: patil-suraj
Pull request: yes

This PR adds `parse_dict` method to `HfArgumentParser`  to allow parsing arguments from `dict`.  

I find this necessary for notebook workflows where I'm not using `Trainer` from command line. Otherwise I need to write arguments to json file and  use that path with `parse_json_file` or pass a list of strings to `parse_args_into_dataclasses`. 

@julien-c @patrickvonplaten 

Comments:
Hey @patil-suraj :-),

Hmm, not sure we would definitely need that...let's say you have a dict of arguments that you want to parse it into a dataclass like `TrainingArguments`, you could just do 

```python
training_args = TrainingArguments(**your_dict)
```

like it is done in the Reformer Colab for example: https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb

Can you give me an example, where going over the `HfArgumentParser` instead of directly instantiating the dataclass makes more sense? 

@patrickvonplaten 
wouldn't this 
```
train_args, model_args, data_args = parser.parse_dict(your_dict)
```
be better than this  
```
training_args = TrainingArguments(**your_dict)
model_args = ModelArguments(**your_dict)
data_args = DataArguments(**your_dict)
```

Anyway, its just a small utility, so if not needed by lot of people we can close this.

I'm ok with merging this!

Always nice to add a unit test though:)

closing this, accidentally merged upstream into this. Will open a new one
Supported language information by model
Author: alexcombessie
Pull request: no
Labels: name: wontfix

Hi there,

Another feature/documentation request. I am evaluating the language support of all pre-trained models distributed via huggingface :)

I have quickly looked into the code and happily found that XLNet and FlauBERT models have that information: https://github.com/huggingface/transformers/search?q=lang&unscoped_q=lang

Do you plan in the short term to add an `available_languages` attribute to all pre-trained models?

If not, happy to do that investigation and share results.

Cheers,

Alex



Comments:
Hi Alex, language support is described in the metadata in the models' [model cards](https://github.com/huggingface/transformers/tree/master/model_cards) and then rendered in a searchable way on the https://huggingface.co/models website.

The mapping is not exhaustive right now because a lot of the canonical/historical models do not have a model card yet. Feel free to create them for the models you're researching. cc @thomwolf @LysandreJik @clmnt 

Thanks, that's a good place to start. I am more interested in the canonical/historical models as you say. I see that some README.md in model cards have a 
```
---
language:
- bulgarian
- czech
- polish
- russian
- ...
---
```
Are you OK for me to do a pull request to add these?

I would also like to standardize languages using the ISO 693-1 two-letter code (https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes). Do you agree on this standard? 

Sounds good to me!

Super, I have forked the repo and created a branch. Expect a pull request in the new few weeks :) so we can close this.

FYI @MobiusLooper

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[readability] consolidate examples/summarization/bart  and examples/summarization/t5
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: Summarization

name: Examples

This involves 

- [ ]  consolidating README.md 
- [ ]  consolidating evaluate_cnn.py scripts.
- [ ]  evaluate_wmt.py should also work
- [ ]  consolidating unittests
- [ ]  updating bash scripts
- [ ]  checking that consolidated stuff works, adding appropriate test coverage.

the resulting code should probably all be in `summarization/` with only a `bertabs/` subdirectory.



Comments:
Hey @sshleifer !
I would like to work on the issue. It's my first issue so I appreciate any help!

Awesome, helpful command to get quick feedback on whether your change is working:

```bash
pytest --tb=short -p no:warnings examples/summarization/bart
```

Make sure you tag @sshleifer when you send a PR and I will give a careful review :)
pip install -e does not always install the correct isort version
Author: sshleifer
Pull request: no

```bash
pip install -e .["dev"]
make quality
```
### Output
```bash
black --check --line-length 119 --target-version py35 examples templates tests src utils
All done! ✨ 🍰 ✨
306 files would be left unchanged.
isort --check-only --recursive examples templates tests src utils
ERROR: /Users/shleifer/transformers_fork/examples/benchmarking/plot_csv_file.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/templates/adding_a_new_example_script/run_xxx.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/templates/adding_a_new_example_script/utils_xxx.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/src/transformers/__init__.py Imports are incorrectly sorted.
make: *** [quality] Error 1
```
relevant packages: 
```python
flake8==3.8.1
isort==4.3.21
black==19.10b0
```

Env:
```
- `transformers` version: 2.11.0
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
```

Would also be good to add more verbose error messages if possible


Comments:
@LysandreJik @julien-c any ideas?

Works for me, no issues. But `isort==4.3.21` is not precise enough, you need to have the actual precise commit.

Can you try `pip uninstall isort && pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort`
?

That worked! 
but my pip freeze still says 
```
isort==4.3.21
```

which seems like the reason that 
```bash
pip install -e .["dev"]
```
didn't work.

pip is confusing, but you technically have version "4.3.21" of isort if you install from the specified commit – but not **the** version "4.3.21".

i.e. the version number in the setup.py of the package that you install from git is still the string "4.3.21".

Do you see what I mean?

Yes I think I do, rephrase: there are multiple different versions of isort called 4.3.21 and pip install -e . will be satisfied if you have any of them, so if you have the wrong one you have to manually run
```bash
pip uninstall isort
pip install git+git://github.com/timothycrosley/isort.git@e63ae06ec7d70b06df9e528357650281a3d3ec22#egg=isort
```

Yep
Fix the __getattr__ method in BatchEncoding
Author: jplu
Pull request: yes

Fix the issue where the `__getattr__` method in `BatchEncoding` was raising a `KeyError` instead of an `AttributeError` when the attribute was accessed with `getattr()`.

Example:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
features = tokenizer.encode_plus("Hello here")
getattr(features, "attr", False)
```

Previous output:
```
/home/jplu/transformers/src/transformers/tokenization_utils.py:204 __getattr__
        return self.data[item]
    KeyError: 'attr'
```

New output:
```
False
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=h1) Report
> Merging [#4772](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5856999a9f2926923f037ecd8d27b8058bcf9dae&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4772/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4772      +/-   ##
==========================================
- Coverage   77.98%   77.96%   -0.03%     
==========================================
  Files         123      123              
  Lines       20436    20437       +1     
==========================================
- Hits        15938    15933       -5     
- Misses       4498     4504       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4772/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.47% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/hf\_api.py](https://codecov.io/gh/huggingface/transformers/pull/4772/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9oZl9hcGkucHk=) | `93.06% <0.00%> (-4.96%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4772/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.50% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=footer). Last update [5856999...fa0abc6](https://codecov.io/gh/huggingface/transformers/pull/4772?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[examples] consolidate summarization examples
Author: aretius
Pull request: yes

Consolidating summarization examples of T5 & Bertabs models into one - [#3826](https://github.com/huggingface/transformers/issues/3826)
@sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=h1) Report
> Merging [#4837](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e33fdc93b4ecb571dd7a8002a74789ec8bfffc09&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4837/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4837   +/-   ##
=======================================
  Coverage   76.17%   76.18%           
=======================================
  Files         128      128           
  Lines       21497    21497           
=======================================
+ Hits        16375    16377    +2     
+ Misses       5122     5120    -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4837/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.27% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4837/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.20% <0.00%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=footer). Last update [e33fdc9...ed4de25](https://codecov.io/gh/huggingface/transformers/pull/4837?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


That's really cool, thanks for working on this @aretius!

@sshleifer Thanks for approving the PR!
Also, I am really interested to contribute more. It would be really great for me to be offered more chance to contribute to the repo :)
Choosing between adding frequent out of vocab words and doing further pretraining.
Author: PieterDujardin
Pull request: no
Labels: name: wontfix

I have a dutch medical dataset (for Namen Entity Recognition) which contains a lot of domain-specific words. The dutch BERT tokenizer therefor outputs a lot of [UNK] tokens when it tokenizes.
Given that I dispose over a corpus of 60k labelled tokens, and right now I have also a relatively small annotated corpus of 185k tokens, would it be best to:

- just add the most frequent out of vocab words to the vocab of the tokenizer
- start from a BERT checkpoint and do further pretraining on the unlabeled dataset (which is now of size 185k which is pretty small I assume..). There might be a possibility for me to obtain a much larger unannotated dataset of potentially millions of (unlabelled) tokens, but I was wondering if even millions of tokens is enough to do some meaningful further pretraining?
Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Wrong tokenization for distilbert-base-multilingual-cased
Author: ricardorei
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (DistillBert):

The problem arises when using:
* my own modified scripts: (give details below)

The tasks I am working on is:
*  my own task or dataset: (give details below)

## To reproduce
with transformers 2.3.0:

```python
import torch
from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
result = torch.tensor(tokenizer.encode("Hello, my dog is cute"))
print (result)

itos = tokenizer.ids_to_tokens
print (itos[61694])
print (itos[10133])

# The original token for 'Hello' exits but for some reason it's not used?
print (itos[31178])
```

Output:
```bash
[101, 61694, 10133, 117, 15127, 17835, 10124, 21610, 10112, 102]
'hell'
'##o'
'Hello'
```

## Expected behavior
```python
import torch
from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
result = torch.tensor(tokenizer.encode("Hello, my dog is cute"))
print (result)
```

Output:
```bash
[101, 31178, 117, 15127, 17835, 10124, 21610, 10112, 102]
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.3.0
- Python version: >3.6


Comments:
Can you show how you initialize `tokenizer`? Which vocab are you using?

> Can you show how you initialize `tokenizer`? Which vocab are you using?

sorry I forgot that... I updated the code in the issue already.

I was using `distilbert-base-multilingual-cased`

`tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")`



This behaviour seems to have been solved in v2.7.0 as running your code yields the correct result in my environment.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How i take an OpenAIGPTDoubleHeadsModel from run_language_modeling.py script? 
Author: nikkon3
Pull request: no
Labels: name: wontfix

I train a gpt2 type model from scratch with run_language_modeling.py .
But i want to take an OpenAIGPTDoubleHeadsModel as my model.
My config.json is below. What i should change?

config = {
    "architectures": [
		"gpt2"
	],
    "model_name_or_path": None , 
    "model_type": "gpt2",
    "vocab_size":5000, 
    "n_positions":1024,
    "n_ctx":1024,
    "n_embd":768,
    "n_layer":6, 
    "n_head":12,
    "activation_function":'gelu_new',
    "resid_pdrop":0.1,
    "embd_pdrop":0.1,
    "attn_pdrop":0.1,
    "layer_norm_epsilon":1e-05,
    "initializer_range": 0.02,
    "summary_type":"cls_index",
    "summary_use_proj":True,
    "summary_activation":None,
    "summary_proj_to_labels":True,
    "summary_first_dropout":0.1,
    "bos_token_id":4999,
    "eos_token_id":4999,
}

Comments:
What are you trying to do exactly? If using our scripts, you would usually pre-train the transformer model (without the heads, so it doesn't make sense to use a double heads model), and you would then fine-tune the model with the heads on a specific dataset. 

If you can explain a bit more what you're trying to do then I could guide you better towards the appropriate scripts.  

@LysandreJik I am new to all these , just i try to do things to learn. I saw this script:  https://github.com/huggingface/transfer-learning-conv-ai/blob/master/train.py 
and i try to do something similar for another language(non english).  
So my idea was, to start with pre-training a gpt2 with run_language_modeling, from scratch in a new language and after fine-tune it in dialogue.
From your answer, i think if i understood, first i must pretrain the gpt2 and after to fine-tune it with the heads in a spesific dataset as dialogue in my case.
But how after the pre-training i use the DoubleHeadsModel?


That's right, that's how I would do it. I would use the `run_language_modeling.py` script with GPT-2 on your data (beware that pre-training requires a lot of data, and a lot of compute).

Once your model is trained on your *big* dataset, then you can use the double heads model and fine-tune it to dialog. We don't have a script for this now, but the link you shared alongside the [blog post](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) detail how to do it!

In order to do what I just mentioned, you would therefore need a large corpus for pre-training, and a dialog dataset in the specific language for fine-tuning.

thanks @LysandreJik. You cleared a litlle my mind at least for the pre-train. 
A small blind spot i have only in how i will pass to the double heads after the pre-training(in blog i don't think they say something for that).
Hope i will find an answer and to that  when i ll be in that spot.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.