noamaanMulla-03/transformers-issues huggingface dataset:
lm_fine-tuning on small dataset of 3 documents
Author: vr25
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

I am trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) on a sample dataset [here](https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/sample_text.txt).

I am running the script with following arguments but I get the exact identical pytorch_model.bin [440.5 MB] saved in the output_dir=op:
python run_lm_finetuning.py --train_data_file=sample_text.txt --output_dir=op --mlm --do_train --overwrite_output_dir --do_lower_case --save_steps=50

I was wondering if this dataset of 3 documents is too small to fine-tune on or if I can modify some arguments to get a domain-fine-tuned model.

Thanks!

Comments:
How do you know you have exact identical `pytorch_model.bin` files? Do you just compare file sizes? IF so, it is not a qualified method just because weights usually are just float numbers and they (almost) always occupy same size on the disk. You can compare the hashes of files to make sure.

Yes, I just thought of comparing the files naively by comparing their sizes. 

I see, yes, "hashes" sounds a much better way of comparing files, thanks. I'll post here if that works.

Also, do you have any beginner suggestions on generating the hashes quickly and efficiently?

I used md5sum pytorch_model.bin to generate the hashes of the files and both are different. Anyway, thanks, again!
How to use pytorch-transformers for transfer learning?
Author: vasilynikita
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

```python
pretrained_weights='bert-base-uncased'
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

features = []
input_ids = torch.tensor([self.tokenizer.encode(phrase, add_special_tokens=True)])
with torch.no_grad():
    # Models outputs are tuples
    outputs = self.model(input_ids)
    last_hidden_states = outputs[0]
    avg_pool_hidden_states = np.average(last_hidden_states[0], axis=0))
    return avg_pool_hidden_states
```

I am working on Sentence Similarity problem, given sentence S, find similar sentences W. So I want to encode sentence S and find the closest top-K sentences from W.

I have read the documentations, but I want to confirm few things how do I do. the following:
- How do I get the second to last layer?
- Am I average pooling the last hidden states correctly?

Comments:
Hey, so not exactly a direct answer to your question, but bert outright doesn't do amazing on sentence similarity. This repo here should help with your question and their paper does a great job at explaining how their method works https://github.com/UKPLab/sentence-transformers. I think you will find better results with that. Hope it helps.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_squad.py not running
Author: maxmatical
Pull request: no

## ❓ Questions & Help

When I try to run the script to fine-tune BERT on squad using the code from the examples:

```
python ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
```
But my terminal just gets stuck on this stage

<img width="518" alt="image" src="https://user-images.githubusercontent.com/8890262/69446607-c39b3780-0d22-11ea-8bea-6135f05640da.png">

When I run the pytests, everything has passed


Comments:
How long does it take? (BERT Model Finetuning using Masked ML objective)
Author: echan00
Pull request: no
Labels: name: wontfix

I am about to finetune a multilingual BERT model using English and Chinese text from the legal domain.  

My corpus is around 27GB, how long should I expect to train 3 epochs (default parameters) using a Google TPU?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Has anyone reproduced RoBERTa scores on Squad dataset?
Author: Morizeyao
Pull request: no
Labels: name: wontfix

I have been working on and made some modifications to run_squad.py in examples folder and is currently having problem reproduce the scores. 

If we can have help (or even a PR) on RoBERTa in run_squad.py that would be great.

Comments:
@Morizeyao Were you able to find any answers to this?

Can you give us more info on what you tried and which results you obtained?

Sorry I was no longer working with the RoBERTa solution and switched to XLNet. Sadly the RoBERTa tries are overwritten. :(

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ModuleNotFoundError: No module named 'pytorch_transformers.modeling' using convert_pytorch_checkpoint_to_tf.py
Author: yangyaofei
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
BERT
Language I am using the model on (English, Chinese....):
Chinese
The problem arise when using:
* [✔️ ] the official example scripts: (give details)
run convert_pytorch_checkpoint_to_tf.py to generate the tf check point

## To Reproduce

Steps to reproduce the behavior:
    
```
python3  /Users/xxx/py3ml/lib/python3.6/site-packages/pytorch_transformers/convert_pytorch_checkpoint_to_tf.py
    Traceback (most recent call last):
  File "/Users/xxx/py3ml/lib/python3.6/site-packages/pytorch_transformers/convert_pytorch_checkpoint_to_tf.py", line 23, in <module>
    from pytorch_transformers.modeling import BertModel
ModuleNotFoundError: No module named 'pytorch_transformers.modeling'
```

## Expected behavior
convert it
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: macOS 10.14
* Python version: 3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch):1.2.0
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information:
I change the code to 
`from pytorch_transformer import BertModel`
It works fine


## Additional context

<!-- Add any other context about the problem here. -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Step restarts from step 0 when reload from an existing checkpoint?
Author: chuanmingliu
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, everyone

I am totally new to Transformers, it's really a good solution. :-)

Question:

When I reload my program from a break point (existing checkpoint, say checkpoint-30000),
what I expect is that, from Tensorboard, I can see the program restarting from 30001.

However, it restarts from step 0, although parameters are up to date.

Does any config point to this problem?

Or any easy solutions?

Thanks.


Comments:
Methods get_input_embeddings() and set_input_embeddings() appear in documentation but not available.
Author: avacaondata
Pull request: no

## ❓ Questions & Help
Hi, I'm trying to access the word embeddings layer of Bert multilingual, as I want to take out all tokens not belonging to spanish and add some tokens which are part of this language, with the objective of adapting BERT multilingual to spanish. The thing is, in your documentation you claim that there are 2 functions for this purpose: get_input_embeddings() and set_input_embeddings() (https://huggingface.co/transformers/model_doc/bert.html); in fact there is a link to the source code and they appear to be there. However, once I try to do this in version 2.1.1. (the one documentation refers to), none of this methods are part of BertModel class, which is astonishing to me. Please tell me what's wrong!! Did you remove these methods from current versions but have a unique documentation for all versions? Which versions can I find these methods in? Is the source code showed in the documentation different from the actual source code of the library? 
<!-- A clear and concise description of the question. -->


Comments:
Did you check if the methods are in `modeling_utils.py` in your local package?
See #1837 . As @julien-c said, the version of the lib you use may not be in sync with the scripts you run.
Try to install the lib from `master`:
`pip install git+https://github.com/huggingface/transformers`


The doc you linked to is for the `master` version, if I'm not mistaken. Maybe we should make that clearer, cc @LysandreJik 

Yes, @loveritsu929 is right – install from source if you want to use those methods right now. Thanks!
Mask probability in run_lm_finetuning.py
Author: leuchine
Pull request: no

   Hi:

 I don't understand why 0.5 is used when replacing masked input tokens with a random word. I think the probability should be 0.1? Or, the positions replaced with [MASK] shall already be stripped out before using 0.5. I think it is a small bug here? Thanks!

     # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.convert_tokens_to_ids(tokenizer.mask_token)

    # 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]

    # The rest of the time (10% of the time) we keep the masked input tokens unchanged
    return inputs, labels

Best Regards,
Qi


Comments:
The lines:

```
indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
```
and
```
indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
```
make it that `indices_random` has a 50% chance of being true when `indices_replaced`, which has a 80% chance of being active, is not active, which is: 100% - 80% = 20%. 50% of 20% is 10%, so the chance that indices_random is true \*is\* 10%

Thanks for the reply. Yes. You are right! I missed the  ~indices_replaced when reading the code. Thanks!
connection limit of pregenerate_training_data.py
Author: ntubertchen
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Inside pregenerate_training_data.py, one can use multiprocessing to process each epoch in parallel.

However, since the communication between process is limited by pickle limit size. We can only transfer arguments less than 1GB and argument: docs are very likely to exceed this limit.

I save the docs as pickle file and ask every process to read on their own.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
R-BERT implementation
Author: RichJackson
Pull request: no
Labels: name: wontfix

## 🚀 Feature

An implementation of the R-BERT architecture for relationship classification

## Motivation

Hi @Huggingface. A recent paper describes an architecture for relationship classification called [R-BERT](https://arxiv.org/pdf/1905.08284.pdf), which claims SOTA performance on the Semeval 2010 Task 8 challenge. However, no code was provided with the paper. [I’ve written an implementation of this](https://github.com/azdatascience/pytorch-transformers/blob/rbert/examples/run_semeval.py) and can confirm it produces very good results (F1=89.07 using the official Semeval scoring script). Is this suitable for merging into the pytorch-transformers repo, or should it exist as a separate package?

Comments:
This is mostly a new head for Bert, right?
If so, yes I think it could be a nice addition.
Is the tokenizer different as well?

That's correct. The new head is [here](https://github.com/azdatascience/pytorch-transformers/blob/rbert/pytorch_transformers/modeling_bert.py#L832) (Think that's in the right place). A new [tokenizer](https://github.com/azdatascience/pytorch-transformers/blob/rbert/pytorch_transformers/tokenization_rbert.py) is required as well, as it needs to insert some special characters surrounding the entities of interest. 

Ok, I think we can accept a PR for that if you want to submit one.

Two notes on that:
- the tokenizer can inherit from `BertTokenizer` and you can probably mostly override the methods called `add_special_tokens_single_sentence` and `add_special_tokens_sentences_pair` to insert the special characters.
- the model and tokenizer should have tests (check how we test the other models, it's pretty simple) and docstring.
- adding an example similar to `run_glue` would be nice also I think or just a usage example in the docstring.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Offsets in original text from tokenizers
Author: michaelrglass
Pull request: no
Labels: name: wontfix

## 🚀 Feature

A new method for tokenizers: tokenize_with_offsets. In addition to returning the tokens, it returns the spans in the original text that the tokens correspond to.
After
tokens, offsets = tokenizer.tokenize_with_offsets(text)
then
tokens[i] maps to text[offsets[i, 0]:offsets[i, 1]]

## Motivation

I find it useful to be able to get the spans in the original text where the tokens come from. This is useful for example in extractive question answering, where the model predicts a sequence of tokens, but the user would like to see a highlighted passage.

## Additional context

I have a version of this in a fork: https://github.com/michaelrglass/pytorch-transformers
There is a test (regression_test_tokenization_with_offsets) that verifies the tokenization with offsets gives the same tokenization for many models - still working on XLM.
The test data I used is available from https://ibm.box.com/s/228183fe95ptn8eb9n0zq4i2y7picq4r

Since this touches several files, and works for the majority (but not all) of the tokenizers, I thought creating an issue for discussion would be better than an immediate pull request.

Comments:
I am glad to see someone is working on this and really appreciate your work.
Currently I'm using LCS the original xlnet also uses to align the token and raw input to extract the answer highlighting.
This is really painful as it's slow and may fail at some corner case.
Can't wait to see your pulled feature merged!

Commenting in #1274 thread

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
breaking change
Author: akosenkov
Pull request: no
Labels: name: wontfix

Great job, just in case it went unnoticed:
from revision 995e38b7af1aa325b994246e1bfcc7bf7c9b6b4f
to revision 2c177a87eb5faab8a0abee907ff75898b4886689

examples are broken due to changed orders of parameters in 
pytorch_transformers/modeling_bert.py

```
<     def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None,
<                 position_ids=None, head_mask=None):
<         outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,
<                             attention_mask=attention_mask, head_mask=head_mask)
---
>     def forward(self, input_ids, attention_mask=None, token_type_ids=None,
>                 position_ids=None, head_mask=None, labels=None):
>
```

Comments:
Indeed. What do you mean by "examples"? The docstrings examples?

Most of examples folder.
In particular run_swag.py and lm finetuning scripts.

just came here to say the same. 

Indeed, I've fixed and cleaned up the examples in 8334993 (the lm finetuning examples are now replaced by `run_lm_finetuning`). Also indicated more clearly which examples are not actively maintained and tested.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Any plan to include BART and T5? 
Author: leuchine
Pull request: no

# 🌟New model addition

## Model description

<!-- Important information -->

## Open Source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them)

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
please search for issues beforehand (and fill the template if there's not already a relevant issue)
SequenceSummary / quenstion regarding summary types
Author: cherepanovic
Pull request: no
Labels: name: wontfix

in the class SequenceSummary(nn.Module) which is a part of {BERT, XLNet}ForSequenceClassification:

https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L643-L644

in case of XLNet we can see that the last token will be taken, for BERT it is otherwise the first one.
https://github.com/huggingface/pytorch-transformers/blob/32e1332acf6fd1ad372b81c296d43be441d3b0b1/pytorch_transformers/modeling_utils.py#L692

Why only the first one or the last one? Why not to apply the max or average pooling over all token from the output? 

Thanks a lot!

Comments:
That's how it's done in the respective original implementations.

it is very interesting. All other tokens will not be considered. 

Do you know whether other architectures has been tried out? Initially, when I had taken the transformer-model I created another output architecture. It is very interesting, whether this "end architecture" affects the performance significantly

Thanks a lot!

I found this project doc [1], there are four variants of the end architecture, their performance is almost equal. 

I will be glad if you can provide other papers regarding this issue. 

[1] - http://web.stanford.edu/class/cs224n/reports/custom/15785631.pdf

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Running XLNet on Squad
Author: LeonCrashCode
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

This is the padding problem. In GLUE codes in the examples, the padding for XLNet is on the left of the input, but in Squad codes, the padding is on right. I was wondering which one is correct. 

Also, the inputs of `convert_examples_to_features` are different in Glue and Squad, where Squad uses most of default values like `pad_token, sep_token, pad_token_segment_id and cis_token_segment_id`, but Glue use the value of `tokenizer`. which one is correct?

Or the example codes are out-of-date? Thanks 

Comments:
Seems like the run_squad script is in bad shape now. It just doesn't work.

same question.. also running this script with XLNet on Squad, is ~10 F1 points below BERT-Large-WWM. The difference in preprocessing as pointed out above could be one of the reasons.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rebase and merge Louismartin/camembert
Author: julien-c
Pull request: yes


Comments:
Please, continue the comments [here](https://github.com/huggingface/transformers/pull/1822#issuecomment-558095282)

> Great work!
> 
> Could you show how to get the embedding vector of a sentence please?
> 
> ```python
> from transformers import CamembertTokenizer
> import torch
> 
> camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
> 
> camembert_tokenizer.encode("Salut, ça va ?") # How to get embedding of this sentence not just the ids of tokens ? 
> ```
Write with Transformer: Please, add an autosave to browser cache!
Author: varkarrus
Pull request: no
Labels: name: wontfix

name: Write With Transformer

## 🚀 Feature

In Write with Transformer, the writing should be periodically saved to the browser cache, so that if the user accidentally refreshes the page, their work that they may have spent hours on won't be lost.

## Motivation

I just lost several hours worth of writing because I accidentally refreshed the page.

## Additional context

:(

Comments:
Hi @varkarrus, thank you for your feature request. There is a "save & publish" button on the top right-hand side, which saves your document on a specific URL. Does this fit your needs?

Does it begin autosaving after you do that? If so, then probably!

Nope, it does not currently auto-save 🙁

One quick fix I've thought about would be to pop a window alert if closing a tab that's got non-saved changes.

Oh, even that alone would be a lifesaver!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Write With Transformer adding spaces?
Author: zacharymacleod
Pull request: no
Labels: name: wontfix

name: Write With Transformer

This issue didn't happen before, but now whenever you use the autocomplete it always adds a space to the beginning, even when a space is not needed. I.E: when adding a comma / period to the end of a sentence, when starting a new line, or (most egregiously) when finishing a word that was only partially written.

As an aside, I wouldn't mind waiting a few extra seconds to get auto-fill suggestions that are longer than two words.

Comments:
Commenting to say I have also noticed this. Also I would assume the small amount of tokens being generated per autocomplete lately is because of compute concerns, not time concerns. It is a bit limiting.

Also having this problem!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
`overwrite_cache` argument in `run_lm_finetuning.py` not used at all
Author: iedmrc
Pull request: no

There is an unused argument in `run_lm_finetuning.py`

https://github.com/huggingface/transformers/blob/e70cdf083ddb8bfe298d43e6d70d698a3a2f56d3/examples/run_lm_finetuning.py#L418

Is it forgotten?

Comments:
Indeed, thank you!