noamaanMulla-03/transformers-issues huggingface dataset:
Why is the weight of linear layer tied to the input embeddings in OpenAIGPTLMHeadModel?
Author: KaitoHH
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Yes the original GPT paper also uses same `W_e` as both token embedding matrix and linear weight, and seems that many succeeding models like GPT-2, XLNet also use the same matrix. In my perspective, the token embedding matrix and the weight in linear layer have nothing related (though they have both the same shape). Could you please explain a bit of that?


Comments:
The token embedding matrix and the linear layer of the **language modeling head** are indeed tied. The embedding matrix is used to map the vocabulary to vectors of last dimension `hidden_size`. 

The linear layer is used to do the exact same thing, just the other way around -> mapping the model output of last dimension `hidden_size` to the vocabulary, so that the output may be converted into vocabulary tokens.

First of all, thanks for the replay!

I know that the last linear layer is to mapping `hidden_state` with the size `hidden_size` to the size of the vocabulary, but the linear layer does not need to output concrete tokens, right? It just needs to output a group of probabilities (with the size of vocabulary) with softmax, and these probabilities seem to have nothing to do with the token embedding matrix?

I have read some other papers, like the CBOW model in word2vec, which uses a linear layer with separate parameters before softmax to train the language model. As a result, the way that GPT does makes me feel confused.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to initialize AdamW optimizer in HuggingFace transformers?
Author: h56cho
Pull request: no

Hello,
I am new to Python and NLP and so I have some questions that may sound a bit funny to the experts.

I had been trying to set my optimizer by setting
`optimizer = AdamW()`

but of course it failed, because I did not specify the required parameter `'param'` (for lr, betas, eps, weight_decay, and correct_bias, I am just going to use the default values).

As a beginner, I am not so clear on what `'param'` stands for in this case. What kind of input should I provide for `'param'`?

Thank you,


Comments:
I had the same issue, apparently it should be model.params()

Thank you! This is helpful

You have to tell the optimizer which parameters it should optimize. Theoretically you could use multiple optimizers for different parameters. This is useful if you want to use different learning rates or different weight decays.

If your question is answered, please close the question. 

> You have to tell the optimizer which parameters it should optimize. Theoretically you could use multiple optimizers for different parameters. This is useful if you want to use different learning rates or different weight decays.
> 
> If your question is answered, please close the question.

I'm guessing this may have something to do with how the params are set in this code from the squad example

```
    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": args.weight_decay,
        },
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
    )
```

https://github.com/huggingface/transformers/blob/master/examples/run_squad.py
GPT2 not in modeltype
Author: tuhinjubcse
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): ENGLISH

The problem arise when using:
* [ ] the official example scripts: (give details) run_glue.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) MRPC GLUE
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

python ./examples/run_glue.py --model_type gpt2 --model_name_or_path gp
t2 --task_name MRPC --do_train --do_eval --do_lower_case --data_dir ./fake --max_seq_length 512 --per_gpu_eval_batc
h_size=8 --per_gpu_train_batch_size=8 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir /tmp/bot/
10/18/2019 00:22:28 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False,
 16-bits training: False
Traceback (most recent call last):
  File "./examples/run_glue.py", line 541, in <module>
    main()
  File "./examples/run_glue.py", line 476, in main
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
KeyError: 'gpt2'
## Environment

* OS: UBUNTU LINUX
* Python version: 3.7
* PyTorch version: LATEST
* PyTorch Transformers version (or branch): LATEST
* Using GPU ? YES
* Distributed of parallel setup ? NO
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hey @tuhinjubcse gpt2 is a text generation model. If you look in the run_glue.py file you will see your options for model selection for using the run_glue.py script.
```
MODEL_CLASSES = {
    'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
    'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
    'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer)
}
```

gpt2 is a transformer model. Why would it be limited to only generation ??

It's not that you can't use it for classification, etc. It's that you would need to make a few changes to the code and model. #1248 . Right now the changes are not made for gpt2. Generally speaking, people use gpt2 for text generation.

Autoregressive models are not as good as mlms, at classification tasks. You should check masked language models (mlm) or similars. But it's not impossible, Open AI has shown some example use cases on original [GPT paper](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf). Check the figure 1 and page 6 for more details.
Also, this is not a bug but just not implemented. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
computing self-attention for tokens in a sentence
Author: vr25
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi,

Please refer to this [issue](https://github.com/google-research/bert/issues/914).

Thanks!


Comments:
To get the attention score of each heads, refer to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel) : see `outputs`.

---

> how do I know which heads give the best scores ?

You can't really know. Each heads will compute some kind of attention. It can be attention for `this token refer to that token`, or it can be attention for `this token is similar to that token` or even if can be any kind of relation, even something, we human, can't interpret.  

If you want to extract some specific kind of attention, I see no other solution than manually inspect all the heads' attention and empirically choose one (or several).  

Some papers did it : [Contrastive Attention Mechanism for Abstractive Sentence Summarization](https://arxiv.org/abs/1910.13114)

Thanks for your reply.

Please refer to [here](https://github.com/huggingface/transformers/issues/2054) for more details.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is it possible/is there a plan to enable continued pretraining?
Author: oligiles0
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Standardised interface to pretrain various Transformers with standardised expectations with regards to formatting training data. 

## Motivation

To achieve state of the art within a given domain it is not sufficient to take models pretrained on nonspecific literature (wikipedia/books/etc). The ideal situation would be able to leverage all the compute put into this training and then further train on domain literature before fine tuning on a specific task. The great strength of this library is having a standard interface to use new SOTA models and it would be very helpful if this was extended to include further pretraining to help rapidly push domain SOTAs.

Comments:
Hi @oligiles0, you can actually use ```run_lm_finetuning.py``` for this. You can find more details in the **RoBERTa/BERT and masked language modeling** section in the README

> Hi @oligiles0, you can actually use `run_lm_finetuning.py` for this. You can find more details in the **RoBERTa/BERT and masked language modeling** section in the README

Thanks very much @enzoampil . Is there a reason this uses a single text file as opposed to taking a folder of text files? I wouldn't want to combine multiple documents because some chunks will then cross documents and interfere with training, but I also wouldn't want to rerun the script for individual documents. 

> Thanks very much @enzoampil . Is there a reason this uses a single text file as opposed to taking a folder of text files? I wouldn't want to combine multiple documents because some chunks will then cross documents and interfere with training, but I also wouldn't want to rerun the script for individual documents.

Please check https://github.com/huggingface/transformers/issues/1896#issuecomment-557222822



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ERROR:CUDA out of memory when using GPT2 tour
Author: papermannnn
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

I follow the tour in the program.Everything is OK . When i start the train,an error occurred.

RuntimeError:
CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 6.00 GiB total capacity; 4.44 GiB already allocated; 3.06 MiB free; 4.57 GiB reserved in total by PyTorch)
I'm using GTX2060(6GB) i'd like to know whether this GPU is quite qualified for the work.
Thanks

Comments:
I've already tried to change batch_size to 1. This doesn't seem to be effective
Problem loading trained keras model
Author: johnwu0604
Pull request: no
Labels: name: wontfix

I'm running the following line of code:
```
model = TFBertForSequenceClassification.from_pretrained(model_dir, num_labels=len(labels))
```
where model_dir is a directory containing a tf_model.h5 and a config.json file that was exported using the .save_pretrained() method.

However I get the following error shown below:

![image](https://user-images.githubusercontent.com/44329080/66222304-bb5d3f00-e685-11e9-9017-6d7733bd5d15.png)

Could someone help here?


Comments:
It seems to me like your file is corrupted 😕

You can refresh the file in the cache with the `force_download` option (`model.from_pretrained(shortcut_name, force_download=True)`)

I think this worked @thomwolf, thanks!

Bringing this back up because it seems like the corrupted file actually happens even when it isn't in the cache. It seems like every 2/3 runs using .save_pretrained() results in a corrupted file for some reason.

Not sure we can do much about this here, we are just calling `tf.keras.Model.save_weights()` for that. Maybe ask upstream in the TensorFlow issues?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
error on bert.fit for Squad dataset
Author: stardoxx
Pull request: no
Labels: name: Need more information

name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The size of tensor a (384) must match the size of tensor b (12) at non-singleton dimension 1.
On passing values to fit function in modelling/bert.py facing above error.


Comments:
We can't help you without more information.
Please fill in the issues templates required information.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Camenbert length Tokenizer not equal config vocab_size
Author: Keisn1
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi there,
when I load the pretrained Camenbert model and tokenizer via 

`model = CamembertForMaskedLM.from_pretrained('camembert-base')
tokenizer = CamembertTokenizer.from_pretrained('camembert-base')`

the length of the tokenizer is 32004 but the vocab_size of the model is 32005.
`print(len(tokenizer))`
'print(model.config.vocab_size'

This throws me an error
> Index out of range

when I try to adapt the lm_finetuning example because of 
`model.resize_token_embeddings(len(tokenizer))`

It runs when I comment out this line. So my question is, is this the intended behaviour resp. what's the reason for the unevenness between the tokenizer and the model vocab_size?

Comments:
Indeed, upon deeper investigation, it appears that the original fairseq model has a bunch of duplicate tokens in the dictionary:
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['
', '<pad>', '
', '<unk>', '<unk>', '
', '
', ',', '▁de', '.']
```

I'm cleaning and updating for this in #2065

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
CPU RAM out of memory when detach from GPU
Author: brucenguyen1110
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am using the following code to get embedding layer from BERT:
```
class BertEmbedding():
    def __init__(self, load_model=None, load_config=None, model='bert-base-uncased', max_len=512, batch_size=6):
        self.pre_trained_model = model
        self.max_len = max_len
        self.batch_size = batch_size
        self.model = BertModel.from_pretrained(self.pre_trained_model)
        self.tokenizer = BertTokenizer.from_pretrained(self.pre_trained_model)
        #self.optimizer = AdamW(params = self.model.parameters(), lr=1e-5)
    
    def create_ids(self, sentences):
        logging.getLogger("transformers.tokenization_utils").setLevel(logging.ERROR) #Disable tokenizer logs, it's really annoy
        input_ids = []
        for sen in tqdm_notebook(sentences, desc="Create Ids"):
            tmp = self.tokenizer.encode(sen)
            input_ids.append(tmp)
        input_ids = pad_sequences(input_ids, 
                                  maxlen=self.max_len, 
                                  dtype='int64', 
                                  truncating='post', 
                                  padding='post')
        return input_ids
    
    def generate(self, inputs):
        test_ids = self.create_ids(inputs)
        test_dataloader = DataLoader(torch.tensor(test_ids), batch_size=self.batch_size)
        
        embedding = []
        self.model.to(device)
        self.model.eval()
        for input_ids in tqdm_notebook(test_dataloader, desc="Generating"):
            with torch.no_grad():
                last_state = self.model(input_ids.to(device))[0]
            last_state = last_state.detach().cpu().numpy()
            embedding.extend(last_state)
        return embedding

bert_embedding = BertEmbedding(batch_size=100)

embedding = bert_embedding.generate(train.sentence.values)
```

The problem is when it generate embedding layer from model (train on GPU and detach to CPU), RAM is increasing significantly (1GB --> 30GB for a list of 25,000 arrays (512,768)). While I checked with `sys.getsizeof(embedding)` = `224208` and size of `bert_embedding` is `56` only.

If I delete both `embedding` and `bert_embedding`, RAM ~ 20GB. I think that the model is still existing. How can I optimize this for CPU RAM?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Missing module "startlette" when calling transformers-cli
Author: tailaiw
Pull request: no
Labels: name: Core: Pipeline

Calling `transformers-cli` in terminal returns error ``ModuleNotFoundError: No module named 'starlette'``.

I assume starlette should be added into dependencies in setup.py and it will be a quick fix. 

Comments:
Hi @tailaiw, thanks for reporting the issue.

Can you try to update to the latest version of transformers ? it should have been fixed in 5004d5af42c61c91d5df07aa139d37599ceb6215.

Feel free to reopen if its not the case !
Any workaround to extend the embeddings on TFGPT2DoubleHeadsModel?
Author: alexorona
Pull request: no
Labels: name: wontfix

Getting access to Keras' `model.fit()` method makes life so much easier for transfer learning/fine-tuning, but TFGPT2DoubleHeadsModel doesn't currently support extending embeddings, so it really restricts practical applications. You almost always have to add something to the vocabulary / generate special tokens. Does anyone know of a workaround?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
configuration from custom config file not working 
Author: mainulquraishi
Pull request: no
Labels: name: Usage

I am trying to get the configuration from a custom config file by the following line : 
`config = GPT2Config.from_pretrained("./lm/gpt2-xl/lm/my_config.json")`

This is similar to the example of this [page]. But I am getting the following error:
```
OSError: Model name './lm/gpt2-xl/lm/my_config.json' was not found in model name list. We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/./lm/gpt2-xl/lm/my_config.json/config.json' was a path, a model identifier, or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.
(https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig)
```
Am I missing something?

Comments:
Did you try using `GPT2Config.from_json_file('xxx.json')`? `from_pretrained` should be used when pointing to a directory containing a `config.json` file.

yes it is working now 

Great to hear!
XLMWithLMHeadModel forwarding questions
Author: y0mingzhang
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1. Why is the labels argument named 'labels' instead of 'masked_lm_labels' like in BertForMaskedLM?

2. When I change labels for masked tokens to -1 as suggested in documentation, I got an error from NLLLoss for label being outside valid num_classes range. When I instead change labels for masked tokens to -100 (default ignore_index), it seems to work. Why is this happening?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Meaning of run_lm_finetuning.py output
Author: aclifton314
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Is there documentation somewhere about what the various output files that get created when running `run_lm_finetuning.py` are and what the meaning of their contents is? Concretely, what are the files and directories:
```
added_tokens.json  
checkpoint-50/
checkpoint-100/    
checkpoint-150/  
checkpoint-200/  
checkpoint-250/
checkpoint-300/  
checkpoint-350/  
checkpoint-400/  
config.json    
eval_results.txt  
merges.txt        
pytorch_model.bin     
runs/   
special_tokens_map.json  
tokenizer_config.json  
training_args.bin
vocab.json
```
and what is the meaning of their contents? the `checkpoint` directories contain:
```
config.json  
pytorch_model.bin  
training_args.bin
```

and `runs/Dec03_09-15-51_MACHINENAME` contains:
```
events.out.tfevents.20414.0
```

## Finetuning Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.1, I believe
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```

Thanks in advance!


Comments:
I have a similar question. When using default settings, does anything change in the tokenizer? Is the tokenizer fine-tuned in anyway (or is any vocabulary added)? In other words, is the vocab.txt of use in any way, when using the default tokenizer? If not, I assume that you only need the `pytorch_model.bin` file and you're good to go?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.