noamaanMulla-03/transformers-issues huggingface dataset:
weird resize during the initialization in the PreTrainedModel
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I am using BertForMaskedLM in the run_lm_finetuning.py code. This module call the module of BertLMPredictionHead, in which there is a decoder layer which is of the size of hidden_size*vocab_size. I would like to change the dimension of this layer. when I change it, I realize that during the call to  elf.init_weights() inside the  BertForMaskedLM, the function resize the weights for the decoder layer. I cannot track where is this exactly happening, thanks for your advice on how I can resolve this issue.

Comments:
This layer is resized in `self.init_weights` because it is sharing weights with the embedding layer. They need to be the same size.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
unable to use distilbert multilingual model
Author: nikhar2008
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I'm trying to use the distilbert-base-multilingual-cased model but have been unable to do so.

I have checked and I am using transformers version 2.3.0. I have already tried these things:
1)     tokenizer = AutoTokenizer.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json")
    model = AutoModel.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-pytorch_model.bin")

Gives following error message:
OSError: Model name 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

2)     tokenizer = AutoTokenizer.from_pretrained("distilbert-base-multilingual-cased")
    model = AutoModel.from_pretrained("distilbert-base-multilingual-cased")

Gives following error message: OSError: Model name 'distilbert-base-multilingual-cased' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'distilbert-base-multilingual-cased' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

3) Same as (2) but with DistilBertTokenizer and DistilBertModel.

Can I please get some help in fixing this issue?


Comments:
Hi
I have verified that it is working. Could you please share your environment details etc.

https://colab.research.google.com/drive/1Bo0luU5q7bztalw5-trWsvl7G0J6zE10

It seems you're not actually running on transformers 2.3.0. Could you please tell me the output of this code in your environment?

```py
from transformers import AutoModel, AutoTokenizer, __version__

print(__version__)
model = AutoModel.from_pretrained("distilbert-base-multilingual-cased")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-multilingual-cased")
```

Thank you.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
It occurs error when python run_lm_finetuning.py 
Author: ARDUJS
Pull request: no
Labels: name: wontfix


- Environment
> - python 3.6.9
> - torch 1.1.0
> - have installed transformers
- Command
> python run_lm_finetuning.py     --output_dir=output     --model_type=bert     --model_name_or_path=bert-base-uncased     --do_train     --train_data_file=$TRAIN_FILE     --do_eval     --eval_data_file=$TEST_FILE     --mlm
- Error
> Traceback (most recent call last):
  File "run_lm_finetuning.py", line 498, in <module>
    main()
  File "run_lm_finetuning.py", line 447, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 96, in load_and_cache_examples
    dataset = TextDataset(tokenizer, file_path=args.eval_data_file if evaluate else args.train_data_file, block_size=args.block_size)
  File "run_lm_finetuning.py", line 78, in __init__
    self.examples.append(tokenizer.add_special_tokens_single_sequence(tokenized_text[:block_size]))
**AttributeError: 'BertTokenizer' object has no attribute 'add_special_tokens_single_sequence'**
- Help
> How to deal it?


Comments:
What is your version of transformers?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
loss_fct = CrossEntropyLoss(ignore_index=-1) for BERT/RoBERTa MaksedLM
Author: Sylar257
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

The models I am using (Bert, RoBERTa....):

Language I am using the model on (English):

The problem arise when I tried to fine-tune the model using `MaskedLM` given the `masked_lm_labels`:

It seems that the model forward loop specifies that `loss_fct = CrossEntropyLoss(ignore_index=-1)` where the instructions previously stated masked ids are -100. This gives a "device-side assert triggered " error for GPU training and "Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97" for CPU training.


* [modeling_bert.py / modeling_roberta.py ] the official example scripts: for `RobertaForMaskedLM` / `BertForMaskedLM` we have `loss_fct = CrossEntropyLoss(ignore_index=-1)`
* [ ] my own modified scripts: set ignore_index = -100 or simply remove it

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. run the "run_lm_finetuning.py" file in the examples
2. It seems that if we use `pip install transformers` and get transformers 2.3.0  We would have this error. If installing from source code, the error is gone
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Linux 18.04.3
* Python version:  3.6
* PyTorch version:  1.3.1
* PyTorch Transformers version (or branch):  2.3.0
* Using GPU ? yes
* Distributed or parallel setup ? nope
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hello! This is due to the pull request #2130. I believe you're running the examples with transformers 2.3.0 whereas they're maintained to work with the current master branch. Please install the library from master:

```pip install git+https://github.com/huggingface/transformers```

in order to get the examples working with the source code.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to run the "run_lm_finetuning.py" with my own corpus?
Author: JiangYanting
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
 Hello! I am a phD student in Beijing Normal University. I'm trying to further pre-train the model "bert-base-chinese" with my own corpus using the "run_lm_finetuning.py". However in the Examples, there is only an example using WikiText-2. If I use my corpus, what format should my data file has ? Thank you !

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
PreTrainedEncoderDecoder on tensorflow
Author: FrancescoSaverioZuppichini
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Hi, would it be possible to create a tensorflow version of `PreTrainedEncoderDecoder`?
## Motivation

The main motivation is that I would like to use `PreTrainedEncoderDecoder` in TensorFlow. Yeah, I got it, PyTorch is better and I totally agree but unfortunately, I have to use TensorFlow.

## Additional context

Looking at the code it does not seem too hard to create a `TFPreTrainedEncoderDecoder`

Thank you guys 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


We are still settling on the proper API for the pytorch version, so it will probably be awhile (months) before we make a tensorflow version. Feel free to take a stab at it, of course!

Thank you for the reply :) I guess I will use PyTorch then.
On 11 Feb 2020, 16:28 +0100, Sam Shleifer <notifications@github.com>, wrote:

We are still settling on the proper API for the pytorch version, so it will probably be awhile (months) before we make a tensorflow version. Feel free to take a stab at it, of course!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/2163?email_source=notifications&email_token=ADZLZXGTOQOXZ37FREVFBZLRCK723A5CNFSM4J2MJBQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELM262Q#issuecomment-584691562>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADZLZXDXDM3BSWXJF5D375TRCK723ANCNFSM4J2MJBQA>.
Improved Error message when loading config/model with .from_pretrained()
Author: patrickvonplaten
Pull request: yes

Given the previous error message, it can be quite time-consuming to find out that the only problem was that the /path/to/model/dir was incorrect :D 

Comments:
Thanks to @mariamabarham for pointing this out :-) 

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=h1) Report
> Merging [#3247](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/dc848c29944265e04f1473cd0312eeffc1842276&el=desc) will **decrease** coverage by `0.36%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3247/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3247      +/-   ##
==========================================
- Coverage   78.32%   77.95%   -0.37%     
==========================================
  Files          98       98              
  Lines       16665    16665              
==========================================
- Hits        13053    12992      -61     
- Misses       3612     3673      +61     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3247/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.82% <ø> (ø)` | |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/3247/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `28.89% <0.00%> (-27.60%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3247/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.84% <0.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3247/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.40% <0.00%> (+4.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=footer). Last update [dc848c2...cd5998e](https://codecov.io/gh/huggingface/transformers/pull/3247?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create CodeBERTaJS model card
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=h1) Report
> Merging [#3295](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/af471ce5e8ca7c19183e70bb998561170addc276?src=pr&el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3295/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3295      +/-   ##
==========================================
+ Coverage   77.82%   78.02%   +0.19%     
==========================================
  Files          98       98              
  Lines       16666    16666              
==========================================
+ Hits        12970    13003      +33     
+ Misses       3696     3663      -33
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3295/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.58% <0%> (-0.14%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3295/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `68.4% <0%> (+0.4%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3295/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.37% <0%> (+5.9%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=footer). Last update [af471ce...62cb24d](https://codecov.io/gh/huggingface/transformers/pull/3295?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Installation error: can not find Rust compiler
Author: jiyanloveyou
Pull request: no

I used pip to install transformers like this:
pip install transformers
in the end, I got the error:
   Can not find Rust compiler
while, I have installed rust on my apple computer, please tell me how to deal with the problem, thank you!


Comments:
Can you please open an issue on https://github.com/huggingface/tokenizers?

Thanks!
cc @n1t0 @mfuntowicz
ImportError: cannot import name 'BartForConditionalGeneration'
Author: NinaHristozovaTR
Pull request: no



## Information

Hi, I am trying to use the BART model to sumamrize a text snippet.

The problem arises when using:
* from transformers import BartTokenizer, BartConfig, BartForConditionalGeneration

## To reproduce

Steps to reproduce the behavior:

1. Installed Tensorflow 2.0 and Pytorch
2. pip install transformers
3. from transformers import BartTokenizer, BartConfig, BartForConditionalGeneration


<!-- ImportError: cannot import name 'BartForConditionalGeneration'-->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: (2.5.1)
- Platform:
- Python version: Python 3.6.5 :: Anaconda, Inc.
- PyTorch version (GPU?): (1.3.1)
- Tensorflow version (GPU?): (2.0.0)


Comments:
I saw that to use the examples, it has to be installed from source.
DistilBertTokenizer defaults to tokenize_chinese_chars=True
Author: Bidek56
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using DistilBert:
Language I am using the model on English

The problem arise when using:
* [ ] the official example [run_tf_ner.py](https://github.com/huggingface/transformers/blob/master/examples/run_tf_ner.py)  scripts

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SST-2

## To Reproduce
Steps to reproduce the behavior:

1. run run_tf_ner.py

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
I am expecting DistilBertTokenizer to have a tokenize_chinese_chars=False but because it extends BertTokenizer, the default is set to be tokenize_chinese_chars=True

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04
* Python version: 3.7.6


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ALBERT tokenizer : local variable 'tokenizer' referenced before assignment
Author: rdisipio
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using BERT and ALBERT

Language I am using the model on: English

The problem arise when using:
* my own script

The tasks I am working on is:
* my own task or dataset: text classification

## To Reproduce

Steps to reproduce the behavior:

```$ pip install transformers```
This installs version 2.3.0

```
>>> from transformers import AlbertTokenizer
>>> tokenizer = AlbertTokenizer.from_pretrained("albert-base")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/Users/Riccardo/development/ideal/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 444, in _from_pretrained
    tokenizer.init_inputs = init_inputs
UnboundLocalError: local variable 'tokenizer' referenced before assignment
```

## Expected behavior

It works perfectly with Bert, e.g.:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
```

## Environment

* OS: MacOsX 10.14
* Python version: 3.7.5
* TensorFlow version: 2.0
* Using GPU ? No
* Distributed or parallel setup ? None


Comments:
The error is misleading, I’ll fix that. Your error stems from the tokenizer initialization: there is no pretrained checkpoint called `albert-base`, only `albert-base-v1` or `albert-base-v2`.

You can check the list of pretrained checkpoints [here](https://huggingface.co/transformers/pretrained_models.html).

Oh, I see. I confirm it works correctly if I load the model `albert-base-v2`. Thanks for taking care of improving the error message!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
model.generate should support past as an input
Author: zaksemenov
Pull request: no
Labels: name: wontfix

## 🚀 Feature

the `model.generate` method should support `past` as an input (and return the hidden states so that the next time it can inject past)




Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fine-tuning pretrained BERT model using own dataset but with same training task
Author: stefanknegt
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I would like to finetune a pretrained model using the same task as the original model was trained on, so this means that I want the model to predict masked words and do next sentence prediction. Is there anywhere some code snippet that achieves this or gives an idea on how I can implement this?


Comments:
Here is very barebone but working example. It does not have next sentence prediction code but it will work for masked language model:

```python
import numpy as np
import tensorflow as tf
from transformers import *

MODEL = 'distilbert-base-uncased'
model = TFDistilBertForMaskedLM.from_pretrained(MODEL)
tokenizer = DistilBertTokenizer.from_pretrained(MODEL)

sent = tokenizer.encode('people lost their jobs to ai')
sent = np.array([sent])
inpx = sent.copy()
inpx[0][1] = tokenizer.vocab['[MASK]']  # Replace people with mask token

loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.Adam()

# Try to overfit model for single example
for _ in range(10):
    with tf.GradientTape() as g:
        out, = model(inpx)
        loss_value = loss_object(y_true=sent, y_pred=out)
    gradients = g.gradient(loss_value, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    print(loss_value.numpy())
    print('>', tokenizer.decode(model(inpx)[0].numpy()[0].argmax(-1)))
```

You will have to handle proper loss masking and other things like warmup etc.

@stefanknegt  I have the same question...Now I am trying to implement this according to the tutorial "Language model fine-tuning" based on `run_lm_finetuning.py` in https://github.com/huggingface/transformers/blob/master/examples/README.md. Maybe it works......



@JiangYanting   哈哈别的问题里看到过你，老哥考试考完了啊，这模型能直接做NSP和MLM么

@TLCFYBJJHYYSND  哈哈哈幸会！好像进一步pre training还是不行……用run_lm_finetuning.py，照着example里的例子做，还是要报错“ValueError: num_samples should be a positive integeral value, but got num_samples=0”

@JiangYanting 我这一直报这个错，老哥有没有遇到过呀
RuntimeError: CUDA error: device-side assert triggered


@TLCFYBJJHYYSND  这个error倒是没遇到过，不过可以看一看这篇博客，不知有无帮助？ https://blog.csdn.net/Geek_of_CSDN/article/details/86527107

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Missing `do_sample` argument for run_generation example
Author: jzhoubu
Pull request: no

# ❓ Questions & Help

It seems the arguments `k`, `p`, `temperature` are disabled because `do_sample` is set to False by default. Thus, [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py)  will always use greedy decoding no matter how the `k`, `p`, `temperature` are set, which is kind of misleading. 

I think `do_sample` argument should be included in the code.

Comments:
You're absolutely correct, I pushed a fix with 7365f01!

Ran into this issue myself by accident.

IMO, `do_sample=True` should be the default behavior for `generate()` since that's more in line with user expectations.

I agree with you, the default should be set to `True`. I've changed the default in 6c1b235.

Decision reverted in #3298 (see the PR for discussion and details).
New default to `do_sample==False`.