noamaanMulla-03/transformers-issues huggingface dataset:
Fix architectures count
Author: dataista0
Pull request: yes


Comments:
Great, thanks!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=h1) Report
> Merging [#1602](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1cfd9748683db43af2c98da1a19d39f0efc8cc3b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1602/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1602   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=footer). Last update [1cfd974...25d32f4](https://codecov.io/gh/huggingface/transformers/pull/1602?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Issue in Cost Function
Author: anandhperumal
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

And in cost function, in logits we are ignoring the last element, why is that? though we're not using any padding
```
shift_logits = lm_logits[..., :-1, :].contiguous() 
shift_labels = labels[..., 1:].contiguous()
```
 
and for labels we're dropping the first token why is that  ?

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Loss function shouldn't drop the last and first element in logits and labels unless it is padded, correct me if I'm wrong.



Comments:
Hi @anandhperumal. Remember that you train GPT-2 by doing next-token prediction, therefore you need to compare the i-th input label--the truth--with what the model predicted: the (i-1)th output. Hence the indices shift.

@rlouf oh yeah. Thanks for the input.
if you don't mind can you answer this question as well [transformers](https://github.com/huggingface/transfer-learning-conv-ai/issues/43) it's not directly related to transformers.
Thanks again.

You're welcome. I haven't worked on the other codebase, but I'll try to help if I can.
Sequence to sequence with GPT model 
Author: ghost
Pull request: no

Hi, I really appreciate if you could tell me if I can build a seq2seq model with gpt2 like this:
I am getting GPT2 run_generation codes, and I want to finetune it in a way, that I give a 
sequence as a context, and then generate another sequence with gpt2, and then I minimize
the cross-entropy loss between the generated sequence and the expected out, and I want to
modify the run_finetune_lm in a way to do it, I was wondering if this way I can make a seq2seq
model with GPT, thank you.

Comments:
We are currently working on implementing seq2seq for most models in the library (see https://github.com/huggingface/transformers/pull/1455). I won't be ready before a week or two.

I'm closing this issue, but feel free to reply in #1506 that we leave open for comments on this implementation.
Whole Word Masking Models update
Author: frankxu2004
Pull request: no
Labels: name: wontfix

Recently Google updated their TF implementation (`https://github.com/google-research/bert`) with Whole Word Masking Models that masks whole random word instead of just random wordpieces, which results in a performance gain. 

Just wondering if this will be implemented here?

Thanks.

Comments:
It's not yet but thanks for the pointer, we can probably add it fairly easily. I'll have a look.

+10000 This would be very helpful!

Hi, 

I converted the cased and uncased whole-word-masking models using the command line tool. If you're interested in adding these to the repository, I've uploaded them to [this](https://www.kaggle.com/bkkaggle/bert-large-whole-word-masking) kaggle dataset. 

Is this resolved? These seem to be available at head, and I don't see anything immediately wrong when I try them...

Yes they are working fine, I've added them to master last week.
They will be advertised in the next release.
When fine-tuned with run_squad they give pretty nice results: `exact_match: 86.91, f1: 93.15`.
I've included a version fine-tuned on SQuAD as well.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Does Whole Word Masking support bert base as well?
failed to download pretrained weights
Author: minapril
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):BERT

Language I am using the model on (English, Chinese....):English

During downloading  pretrained weights with code of modeling_bert.BertForMaskedLM.from_pretrained('bert-base-uncased'),another exception occurred:
![image](https://user-images.githubusercontent.com/52790610/67380476-faafe900-f5bc-11e9-84a1-bd0c8eeb7515.png)




Comments:
Hi, this seems to be a network error. Are you sure you have access to the internet on this machine, or is it behind a firewall?

I had exactly the same problem Yesterday and s3.amazonaws.com just was not reachable. We also had the same problem with another service as well. After trying for some time it just started working again.
ALBERT: will it be supported?
Author: xinqipony
Pull request: no

 will you release an ALBERT model?
it sets the new state of art;

# 🌟New model addition
ALBERT: A LITE BERT FOR SELF-SUPERVISED
LEARNING OF LANGUAGE REPRESENTATIONS
https://arxiv.org/pdf/1909.11942.pdf

## Model description
![image](https://user-images.githubusercontent.com/42603620/67136720-eedec280-f25c-11e9-8681-383c5e3d27a0.png)

<!-- Important information -->

## Open Source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
https://github.com/brightmart/albert_zh

Please direct all your questions to the main albert topic. https://github.com/huggingface/transformers/issues/1370 

Please close this current topic. It does not add anything.

... We should extend the issue template and redirect all ALBERT questions to #1370 😂
How can I get the probability of a word which fits the masked place?
Author: traymihael
Pull request: no

## ❓ Questions & Help

I want to get the probability of a word which fits the masked place.
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

text = '[CLS] I want to [MASK] the car because it is cheap . [SEP]'
tokenized_text = tokenizer.tokenize(text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0] * len(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

with torch.no_grad():
    predictions = model(tokens_tensor, segments_tensors)

masked_index = tokenized_text.index('[MASK]')
predicted_score, predicted_indexes = torch.topk(predictions[0][0, masked_index], k=5)
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_indexes.tolist())
```
`predicted_tokens`
> `['buy', 'sell', 'rent', 'take', 'drive']`

'predicted_score'
> 'tensor([10.9675, 10.4480,  9.5352,  9.5170,  9.3046])'

`predicted_score` is not a probability. I want a pair of the word and its probability (the total score for all words is 1) which goes in the [MASK].

Comments:
why the acc of chinese model(bert) is just 0.438
Author: zsk423200
Pull request: no
Labels: name: wontfix

dataset: XNLI-1.0
i run the dataset of xnli-1.0, but the result is `acc = 0.43855421686746987`, and i run on google bert in tf, the result is `eval_accuracy = 0.7674699`. i use the same epochs and lr, i really don't know why.

i add the dataprocess of xnli ,the same with the version of tf bert:
```
class XnliProcessor(DataProcessor):
  """Processor for the XNLI data set."""

  def __init__(self):
    self.language = "zh"

  def get_train_examples(self, data_dir):
    """See base class."""
    lines = self._read_tsv(
        os.path.join(data_dir, "multinli",
                     "multinli.train.%s.tsv" % self.language))
    examples = []
    for (i, line) in enumerate(lines):
      if i == 0:
        continue
      guid = "train-%d" % (i)
      text_a = line[0]
      text_b = line[1]
      label = line[2]
      if label == "contradictory":
        label = "contradiction"
      examples.append(
          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
    return examples

  def get_dev_examples(self, data_dir):
    """See base class."""
    lines = self._read_tsv(os.path.join(data_dir, "xnli.dev.tsv"))
    examples = []
    for (i, line) in enumerate(lines):
      if i == 0:
        continue
      guid = "dev-%d" % (i)
      language = line[0]
      if language != self.language:
        continue
      text_a = line[6]
      text_b = line[7]
      label = line[1]
      examples.append(
          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
    return examples

  def get_labels(self):
    """See base class."""
    return ["contradiction", "entailment", "neutral"]
```

 and my pytorch command is :
```
python run_glue.py --model_type bert --model_name_or_path bert-base-chinese --task_name XNLI --do_train --do_eval --do_lower_case --data_dir $XNLI_DIR --max_seq_length 128 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --learning_rate 5e-5 --num_train_epochs 2.0 --output_dir /tmp/MRPC4/ --overwrite_output_dir --save_steps=1000
```

and my tf common is:
```
python run_classifier.py   --task_name=XNLI   --do_train=true   --do_eval=true   --data_dir=$XNLI_DIR   --vocab_file=$BERT_BASE_DIR/vocab.txt   --bert_config_file=$BERT_BASE_DIR/bert_config.json   --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt   --max_seq_length=128   --train_batch_size=32   --learning_rate=5e-5   --num_train_epochs=2.0   --output_dir=/tmp/xnli_output
```


Comments:
i have the same issue as you.  Do you have a good solution for it？

I met the same problem in multi-labels classification task, have no idea about the problem!

@zsk423200  Maybe you can try the "bert-base-multilingual-cased-pytorch_model" , it's performance seems better a lot  than the pure Chinese ver in my task, just a temporal solu.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Does the function of  'evaluate()' change the result?
Author: lceustc
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
when i run RTE task , and logging steps=50: the result  is:
    gloabl_step 50:  0.8953 
    global_step 100:  0.8953 
    gloabl_step 150:  0.8916
    global_step 200:  0.8736
but when logging steps =100:
    global_step 100:  0.8953  
    global_step 200:  0.8880
when global_step is 200,what cause the difference? i didn't  modify any code





Comments:
Could you specify what script you're running, with which parameters? Did you set a random seed?

> Could you specify what script you're running, with which parameters? Did you set a random seed?

for SEEDS in  99

do
CUDA_VISIBLE_DEVICES=2 python run_glue.py \
        --data_dir '/data/transformers/data/RTE/' \
        --model_type 'roberta' \
        --model_name_or_path '/data/transformers/examples/pretrained_model/roberta-mnli/' \
        --task_name 'rte' \
        --output_dir ./$SEEDS \
        --overwrite_output_dir \
        --max_seq_length 128 \
        --do_train \
        --do_eval \
        --evaluate_during_training \
        --per_gpu_train_batch_size 8 \
        --per_gpu_eval_batch_size 8 \
        --gradient_accumulation_steps 2 \
        --learning_rate 1e-5 \
        --num_train_epochs 10 \
        --logging_steps 50 \
        --save_steps -1 \
        --seed $SEEDS \

done




> Could you specify what script you're running, with which parameters? Did you set a random seed?

i change 'roberta' to 'bert' and set the same seed, the result is also different,  Is there any wrong with my shell script?
Fine-tuning BERT using Next sentence prediction loss
Author: jasonmusespresso
Pull request: no

In `pytorch_pretrained_bert`, there is an example for fine-tuning BERT using next sentence prediction loss. In the new version, how shall we fine-tune BERT on the next sentence prediction task?

Thank you.

Comments:
We do not have any scripts that display how to do next sentence prediction as it was shown with RoBERTa to be of little importance during training.

We had some scripts until version 1.1.0 that allowed this, you can find them [here](https://github.com/huggingface/transformers/tree/1.1.0/examples/lm_finetuning). They are deprecated but can give you an idea of the process.

Ah, gotcha. Thanks!
Error in converting tensorflow checkpoints to pytorch 
Author: ghost
Pull request: no
Labels: name: wontfix

@thomwolf 
 I downloaded tensorflow checkpoints  for domain specific bert model and extracted the zip file into the folder  **pretrained_bert** which contains the following the three files

model.ckpt.data-00000-of-00001
model.ckpt.index
model.ckpt.meta

I used the following code to convert tensorflow checkpoints to pytorch

```
import torch

from pytorch_transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert


tf_checkpoint_path="pretrained_bert/model.ckpt"
bert_config_file = "bert-base-cased-config.json"
pytorch_dump_path="pytorch_bert"

config = BertConfig.from_json_file(bert_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForPreTraining(config)

# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
```

I got this error when ran the above code

**NotFoundError: Unsuccessful TensorSliceReader constructor:** Failed to find any matching files for pretrained_bert/model.ckpt

Any help is really appreciated............

Comments:
For me it worked to convert checkpoints without specifying the exact checkpoint. So only pointing to the folder of the checkpoint:
`tf_checkpoint_path="pretrained_bert"`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
reproducing bert results on snli and mnli 
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I have finetuned bert for snli and mnli for 6 epochs for none of them I could reproduce bert results on these datasets. I also encountered degenerate solution which get around 47 accuracy, could you assist me how I can avoid this issue? so when there are several checkpoints, I always evaluate the last one after 6 epochs, thanks. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using hidden states from BERT (Similar to using precomputed hidden states in GPT2 model "past" argument)
Author: PExplorer
Pull request: no
Labels: name: wontfix

Hi,

In question and answer model using BERT, I will be querying context and around 50 questions on the same context. To reduce latency for obtaining results, I would like to cache hidden states of context before prediction.
For each question answer prediction on the same context, can the model use precomputed hidden states of context and get answer predictions?
Similar to GPT2, past can be used to reuse precomputed hidden state in a subsequent predictions.

Thank you,



Comments:
Hi, no, Bert doesn't have a cached hidden-states option.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can't GPT-2 set special_tokens? (or unk tokens)
Author: jeonggwanlee
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
In GPT, we can set special tokens. (I also did it branch 0.6.2)
https://github.com/huggingface/pytorch-transformers/blob/v1.0.0/pytorch_transformers/modeling_openai.py

But, in GPT-2, It seems like no way to add special tokens.
https://github.com/huggingface/pytorch-transformers/blob/b33a385091de604afb566155ec03329b84c96926/pytorch_transformers/modeling_gpt2.py#L619

I also saw #994. They said it's impossible. 
It it true? and do you have any plan to add it?

Comments:
I also saw #468. It will be probably added soon.
But, If someone informs new information about this, I'll thank for that.

Passing them like so works for me:

`GPT2Tokenizer.from_pretrained(args.model_name, unk_token="
")`

You can all pass a list to `tokenizer.add_tokens`, then call `model.resize_token_embeddings(len(tokenizer))`.

@aburkard  Thank you so much!!!! I'll try it.


Hi, in GPT-2 there wasn't the option at first but we've added it down the line. It is available if you compile this repo from source from the master branch, or you can wait for the version 1.1 which should drop sometimes this week. In this version you're able to add special tokens to GPT-2.

Release 1.1.0 is here :-)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_ner.py file with Distill Bert
Author: amankedia
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I wish to use the Distill Bert model for NER. I am not sure if it will work with it directly. Any suggestions on that end would be great.
Also, what values should the parameters **--model_type** and **--model_name_or_path** take for Distill Bert?
Other parameters per my understanding would be the same.

Comments:
I have the same issue. Did you end up using Distilbert?

Not sure how well it will perform. Casing is an important feature used in many NER tasks. So I would say it _could_ work, but ymmv. For reference: https://stackoverflow.com/questions/56384231/case-sensitive-entity-recognition

RoBERTa is cased so you guys can try using DistilRoBERTa, released today by @VictorSanh:

```
--model_name_or_path distilroberta-base
```

You'll probably need to adapt run_ner.py (PR welcome)

Actually working on that today so I'll let you know how it goes. 

This is actually really cool. I was looking today at the models and had no idea DistilRoBERTa was released today. Awesome @VictorSanh!

@amankedia I think this issue is resolved? If so we can resolve it :)

Solved indeed. Thanks everyone for contributing!
When I set fp16_opt_level == O2 or O3,  I can not use multiple GPU
Author: liuyukid
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->


Comments:
We need more information, like a full error log and the detailed command line you used for instance.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Tokenizer not found after conversion from TF checkpoint to PyTorch
Author: HansBambel
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: run_generation.py, convert_tf_checkpoint_to_pytorch.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: Text generation. I finetuned a gpt2 model using Tensorflow and I converted the checkpoint using the `convert_tf_checkpoint_to_pytorch.py` script to PyTorch. Running `run_generation.py` from the examples folder results in an error. It seems like the tokenizer is not loaded from the converted model. (Maybe it is not saved?)

## To Reproduce

Steps to reproduce the behavior:

1. Have a tensorflow checkpoint.
2. Convert it with `python pytorch_transformers gpt2 path/to/checkpoint path/to/save/model`
3. Run `python run_generation.py --model_type gpt2 --model_name_or_path path/to/saved/model --top_p 0.9 --prompt "Hello Huggingface"`

This results in the following error:
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
`Traceback (most recent call last):
  File "run_generation.py", line 195, in <module>
    main()
  File "run_generation.py", line 175, in main
    context_tokens = tokenizer.encode(raw_text)
AttributeError: 'NoneType' object has no attribute 'encode'`

## Expected behavior
Text generation like using "gpt2" as `model_name_or_path`.
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows 10
* Python version: 3.7
* PyTorch version: 1.1
* PyTorch Transformers version (or branch): 1.0
* Using GPU ? Yes, but doesn't work with CPU either
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context
I manged to get it working by substituting the loading of the tokenizer with "gpt2", that way the tokenizer is loaded not from my fine-tuned model, but from the cache of the 117M version. Is the tokenizer actually trained?
Right now I have 3 files in the models folder: `config.json`, `pytorch_model.bin` and `vocab.bpe`. Am I missing a file?
<!-- Add any other context about the problem here. -->

Comments:
Hi, no the tokenizer is not trained. You can just load the original `gpt2` one.

Shouldn't the tokenizer then be loaded from `args.model_type` and not `args.model_name_or_path`? Or do they differ from `gpt2` to `gpt2-medium`?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.