noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizing/Loading Data for GPT-2 (1 example per line)
Author: varunnambikrishnan
Pull request: no
Labels: name: wontfix

I want to finetune GPT-2 on a dataset that has one example per line, and the examples all have different length. What is the best way to alter the TextDataset class to allow for this? I understand GPT-2 requires fixed length inputs but I'm not sure how to apply the attention mask to achieve this? Also do I need to add a bos/eos token to each line?

Appreciate the help

Comments:
Please check this out: https://github.com/huggingface/transformers/issues/1816#issuecomment-554759751
This finetuning script https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py and the library do almost everything for you. Thus you don't need to split the text into fixed blocksize etc. Just give it one datafile for training and one for evaluating.

I was using the script but when generating samples it seemed like it was mixing information from multiple examples in the training set, when they should be treated as individual examples. Is there a way to avoid this?

Like in this thread it mentions not concatenating different articles in the same input: https://github.com/huggingface/transformers/issues/1145#issuecomment-527225854? Is there a way to do that with single examples (if they are not correlated)

Language models look for entire corpus. You can seperate them via `
` special token. The model learns (and already learned if you based it on a pretrained one while finetuning) to separate contexts by `
` token. This is a common case for language models. For example, XLNet uses `<eod>` as separator. This is also implied at the original GPT paper while covering how to train GPT for document classification etc. See figure one: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
But, still if you want to make sure that different rows in the data are not even connected while training, then you can pad each line until fill `block_size`.  For example, GPT2 has 1024 input size and you have a data long 1540 (=1024+516) as tokens. Then you could have: 
<1024 tokens><516 tokens + 508 padding><nextdata..>
Thus you can make sure that GPT2 doesn't get mixed data as input.

Also, please check this tutorial http://jalammar.github.io/illustrated-gpt2/ by Jay Alammar

I thought GPT-2 didn't use padding tokens https://github.com/huggingface/transformers/issues/1435#issuecomment-538779523 and it's unclear to me how to use the attention mask. I'll try just using the 
 token however

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Some Questions about XLNet 
Author: qlwang25
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

1, XLNet is a model with relative position embeddings,  so  can either pad the inputs on the right or on the left ?

2, If i pad the inputs on the left, the length of input is 100 and the max length is 128, i don't consider the PAD token in NER task, i.e.  just use the hidden state of input token,  use  [-100:, :] over the output of model  ? 
For length dimension, the last 100 are taken; for hidden state  dimension, the values of all dimensions are taken.

3, CLS token must be at end ?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNet run_squad.py IndexError: tuple index out of range
Author: phuongpm241
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English (xlnet-base-cased)

The problem arise when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: run_squad.py
* [ ] my own task or dataset: (give details)

## To Reproduce

CUDA_VISIBLE_DEVICES=0,1,2,3 python run_squad.py \
  --model_type xlnet \
  --model_name_or_path xlnet-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --version_2_with_negative \
  --train_file /data/medg/misc/phuongpm/squadv2/train-v2.0.json \
  --predict_file /data/medg/misc/phuongpm/squadv2/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 10000 \
  --output_dir /scratch/phuongpm/tuned/squad_xlnet/

## Expected behavior

Epoch:   0%|                                                                        | 0/2 [00:00<?, ?it/sTraceback (most recent call last):                                                | 0/2791 [00:00<?, ?it/s]
  File "run_squad.py", line 837, in <module>
    main()
  File "run_squad.py", line 776, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 221, in train
    inputs.update({"is_impossible": batch[7]})
IndexError: tuple index out of range
Epoch:   0%|                                                                        | 0/2 [00:00<?, ?it/s]
Iteration:   0%|                                                                 | 0/2791 [00:00<?, ?it/s]

## Environment

* OS: Linux
* Python version: 3.6 
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 
commit babd41e7fa07bdd764f8fe91c33469046ab7dbd1
Author: Lysandre <lysandre.debut@reseau.eseo.fr>
Date:   Fri Jan 24 17:06:55 2020 -0500
* Using GPU ? Yes
* Distributed or parallel setup ?
* Any other relevant information:


Comments:
Hi, are you sure you're running on commit babd41e, and that you didn't take the script from this version without updating the library itself? I believe this was patched in 073219b.

Could you try to install from source `pip install git+https://github.com/huggingface/transformers` and let me know if it fixes this issue?

It works after updating reinstalling the library. I think I might forget to install after git pull. 

Thank you!
How to perform common sense reasoning task with GPT-2?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

I am new to NLP so I have lots of questions.
I am interested in carrying out common sense reasoning task with GPT-2, for example, with Winograd Schema Challenge dataset. 

 Q1. How should I tokenize the Winograd Schema Challenge dataset to process it with GPT-2 (with the double heads model, for instance)? Can someone please give me an example?

 Q2. Can GPT2DoubleHeadsModel be used to conduct common sense reasoning task with Winograd Schema Challenge dataset?

Thank you,  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bert perform way worse than simple LSTM+Glove
Author: zhaoxy92
Pull request: no

Hi, I am doing a very straightforward entity classification task, but Bert is not giving a good result. I am wondering if there is something wrong with my code. 

My task is give a sentence and an entity boundary in that sentence, I predict entity type.

Here is my code to prepare input data. The basic idea is I have a sentence batch `batch_data`, then  I use tokenizer to encode that sentence batch to `s_encoded`. Then to pad the encoded ids so that each encoded id list can have same length, I pad them with 0's according the longest sentence in that batch (`max_len`).  Because I need to extract feature for the entity, so I use `entity_idx` to keep track of the entity boundary after it's tokenized.

Later, I just use `s_tensor` and `attn_mask_list` as input for BERT, and use `entity_idx` list to extract BERT feature for the entity.

Is there anything wrong with this part? Thank you!

```
s_bat = [x.tokens for x in batch_data]
s_encoded = tokenizer.batch_encode_plus([' '.join(x) for x in s_bat], add_special_tokens=True)
max_len = max([len(x) for x in s_encoded['input_ids']])

start_ids = [x.mention_start for x in batch_data]
ids_list = []
entity_idx = []
attn_mask_list = []
for i in range(len(s_bat)):
    ids = s_encoded['input_ids'][i]
    ids.extend([0 for _ in range(max_len-len(ids))])
    attn = s_encoded['attention_mask'][i]
    attn.extend([0 for _ in range(max_len-len(attn))])
    ids_list.append(ids)
    attn_mask_list.append(attn)

    m_start_idx = len(self.tokenizer.encode(' '.join(s_bat[i][:start_ids[i]]))) if start_ids[i]>0 else 0
    entity_idx.append([1+m_start_idx, 1+m_start_idx+len(m_encoded['input_ids'][i])]) #consider start special token
s_tensor = torch.LongTensor(ids_list)
attn_mask_list = torch.Tensor(attn_mask_list)

```


Comments:
Well, how are you actually using it? Are you actually fine-tuning the model? What's your train loop?

I actually just solved the issue. It seems that the code I posted was
correct, but it has to do with where I placed the scheduler.step().

On Mon, Jan 20, 2020 at 6:57 AM Bram Vanroy <notifications@github.com>
wrote:

> Well, how are you actually using it? Are you actually fine-tuning the
> model? What's your train loop?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2567?email_source=notifications&email_token=AFB56KPQC3V5YOYAOQN5YRLQ6WGRXA5CNFSM4KIOCMLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJMMIKY#issuecomment-576242731>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFB56KMFFHCCUDF7SFZNR63Q6WGRXANCNFSM4KIOCMLA>
> .
>


Glad to hear that you solved the issue. Please close the question.
How to process ARC dataset with HuggingFace GPT2
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

I am interested in processing ARC dataset with HuggingFace GPT2.
The ARC dataset (http://nlpprogress.com/english/question_answering.html) is a question answering, which contains 7,787 genuine grade-school level, multiple-choice science questions. The dataset also comes with the full corpus of texts extracted from various articles that explains various scientific concepts that can be used to solve these 7,787 multiple choice questions (i.e. this full-corpus is not in the multiple choice format; it's just a series of excerpts from various articles)

I am assuming I'd have to use the GPT2DoubleHeadsModel to process this ARC dataset, since it is a set of multiple-choice questions. However, I also need to somehow train my GPT2DoubleHeadsModel based on the contents of the full corpus of texts that contains excerpts from various scientific articles, since GPT2DoubleHeadsModel wouldn't have acquired any scientific knowledge prior to processing this dataset. 

But the thing is, the corpus of series of scientific articles on which I am interested in training my GPT2DoubleHeadsModel with is not written in a multiple-choice format -- is it possible to train only those parts of the GPT2DoubleHeadsModel that are responsible for language modelling with the series of scientific articles, and then fine-tune the entire component of the GPT2DoubleHeadsModel with the training data from the multiple choice questions? 

If it is possible, how can I do it?

Thank you,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Installation example #2 fails: cannot import name 'glue_compute_metrics'
Author: ghost
Pull request: no

## 🐛 Bug

<!-- Important information -->

I am having issues with the official installation procedure, where running `python -m pytest -sv ./examples` fails with an opaque error message (below).

## To Reproduce

Steps to reproduce the behavior:

1. Create virtualenv
2. Install Pytorch (`pip install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html`)
3. Install transformers (`pip install transformers`)
4. Install pytest (`pip install pytest`)
5. Run `python -m pytest -sv ./transformers/tests/`; no tests fail
6. Run `python -m pytest -sv ./examples/`; fails requiring tensorboardX
7. Install tensorboardX (`pip install tensorboardX`)
8. Run `python -m pytest -sv ./examples/`; fails with message:

```
==================================== ERRORS ====================================
__________________ ERROR collecting examples/test_examples.py __________________
ImportError while importing test module '/home/evancw/Projects/transformers/examples/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
examples/test_examples.py:30: in <module>
    import run_glue
examples/run_glue.py:49: in <module>
    from transformers import glue_compute_metrics as compute_metrics
E   ImportError: cannot import name 'glue_compute_metrics'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.64s ===============================
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

I would expect the tests to pass.

## Environment

* OS: Ubuntu 18.04.3 LTS
* Python version: 3.6.8
* PyTorch version: 1.2.0 + CPU
* PyTorch Transformers version (or branch): 2.0.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information: System is completely clean before pip installs

## Additional context

Please let me know if there is any more information I can provide.


Comments:
 @evanweissburg  Hi, i got the same error..have you found any solution???
 

Hello! I believe you must have sklearn installed in order to pass these tests. Please let me know if it doesn't work while having sklearn installed.

yeah got it .... i guess we need to run pip install -r ./examples/requirements.txt

Indeed!

Could we add this to the getting started documentation?

pip install -r ./examples/requirements.txt

Had the same issue. Probably a problem with sklearn. Installed with conda and it was fixed.
Remove unnecessary `del` in run_tf_glue.py example
Author: jarednielsen
Pull request: yes

Platform: Ubuntu 18.04 (Linux-4.15.0-1054-aws-x86_64-with-Ubuntu-18.04-bionic)
Python: 3.6.9
PyTorch: 1.4.0
TensorFlow: 2.0.0

Running `./examples/run_tf_glue.py` gives `KeyError: 'special_tokens_mask'`.

Diving into the code, it looks like there's an optional keyword argument in [`encode_plus()`](https://github.com/huggingface/transformers/blob/9d87eafd118739a4c121d69d7cff425264f01e1c/src/transformers/tokenization_utils.py#L834) named `return_special_tokens_mask` that defaults to False. I'm guessing that this argument was added recently and the example just needs to be updated?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=h1) Report
> Merging [#2670](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9d87eafd118739a4c121d69d7cff425264f01e1c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2670/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2670   +/-   ##
=======================================
  Coverage   74.51%   74.51%           
=======================================
  Files          87       87           
  Lines       14920    14920           
=======================================
  Hits        11117    11117           
  Misses       3803     3803
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=footer). Last update [9d87eaf...ff1a4b3](https://codecov.io/gh/huggingface/transformers/pull/2670?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, thanks for catching that and removing it!
Add FlauBERT: Unsupervised Language Model Pre-training for French
Author: formiel
Pull request: yes

This PR adds [FlauBERT](https://github.com/getalp/Flaubert). Most of the code is derived from XLM (there are some new features in FlauBERT such as `pre_norm` and `layerdrop`).

`make test` had 1 failure related to BERT and not to FlauBERT:

> [gw0] FAILED tests/test_configuration_auto.py::AutoConfigTest::test_pattern_matching_fallback

`make style` passed.

`make quality` passed.

Comments:
Hi, I don't really know how it happened but I was denied push access on your repository while patching the failing FlauBERT bug. Instead I pushed to a new branch `flaubert` on this remote (huggingface/transformers), and I'm opening a pull request with your changes.

You're still the author of the commit.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=h1) Report
> Merging [#2632](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/adb8c93134f02fd0eac2b52189364af21977004c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2632/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2632   +/-   ##
=======================================
  Coverage   74.59%   74.59%           
=======================================
  Files          89       89           
  Lines       14971    14971           
=======================================
  Hits        11168    11168           
  Misses       3803     3803
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=footer). Last update [adb8c93...adb8c93](https://codecov.io/gh/huggingface/transformers/pull/2632?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


The PR is #2677. I'm updating the documentation directly on this PR.
Confused about the prune heads operation.
Author: sjcfr
Pull request: no

In codes there are a 'prune_heads' method for the  'BertAttention'  class, which refers to the 'prune_linear_layer' operation.  Not understanding the meaning of such operation. The codes of 'prune_linear_layer' is listed below. Thanks for any help!

def prune_linear_layer(layer, index, dim=0):
    """ Prune a linear layer (a model parameters) to keep only entries in index.
        Return the pruned layer as a new layer with requires_grad=True.
        Used to remove heads.
    """
    index = index.to(layer.weight.device)
    W = layer.weight.index_select(dim, index).clone().detach()
    if layer.bias is not None:
        if dim == 1:
            b = layer.bias.clone().detach()
        else:
            b = layer.bias[index].clone().detach()
    new_size = list(layer.weight.size())
    new_size[dim] = len(index)
    new_layer = nn.Linear(new_size[1], new_size[0], bias=layer.bias is not None).to(layer.weight.device)
    new_layer.weight.requires_grad = False
    new_layer.weight.copy_(W.contiguous())
    new_layer.weight.requires_grad = True
    if layer.bias is not None:
        new_layer.bias.requires_grad = False
        new_layer.bias.copy_(b.contiguous())
        new_layer.bias.requires_grad = True
    return new_layer

Comments:
Yes, I'll add a detailed example for this method in the coming weeks (update of the bertology script).

This can be used to remove heads in the model following the work of [Michel et al. (Are Sixteen Heads Really Better than One?)](http://arxiv.org/abs/1905.10650) among others.

Thanks a lot！

Hi @thomwolf, would it be possible to provide an example on how to prune or select some heads for a layer? when i just change the config file by setting 
config.pruned_heads = {11:[1,2,3]} and use it in initializing the model, it throws an error.
```
size mismatch for bert.encoder.layer.11.attention.self.query.weight: copying a param with shape torch.Size([768
urrent model is torch.Size([576, 768]). and more. 
```
so, the default query,key and vaule are set with 768 dim. 
I assume we can not just prune heads and still load the pre-trained model because the word embedding and layer norm was setup up with 768 dim. 

meanwhile i came across bertology.py script and realize that we can save a model after pruning. that works fine for me. now, i'm trying to load the saved model, and I get the opposite error. 
```
size mismatch for bert.encoder.layer.11.attention.self.query.weight: copying a param with shape torch.Size([576, 768]) from checkpoint, the sh
ape in current model is torch.Size([768, 768]).
```
the error wouldn't go away after even changing the config file.