noamaanMulla-03/transformers-issues huggingface dataset:
Error when using np.where() during squad tokenization 
Author: Adawindcatcher
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):Bert

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [x] the official example scripts: squad1.1
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD task

* [ ] my own task or dataset: (give details below)

## To reproduce
 I'm confused about this line:
https://github.com/huggingface/transformers/blob/896300177bf9f35feac4698370212a80a5ab6138/src/transformers/data/processors/squad.py#L230

Should this be  
```
pad_token_indices = np.where(np.array(span["input_ids"]) == tokenizer.pad_token_id)
```
Cause "p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)", and the padding positions should not be answers

Steps to reproduce the behavior:
```python
span["input_ids"] = [1,1,3,0,0,0]
np.where([1,1,3,0,0,0] == tokenizer.pad_token_id)
# return (array([], dtype=int64),)

np.where(np.array([1,1,3,0,0,0]) == tokenizer.pad_token_id)
# return (array([3, 4, 5]),)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
As described above.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 7a68d401388bc68f10dfeb591709352736a6c0b6
- Platform:
- Python version: python3.6.8
- PyTorch version (GPU?):1.5.0 CPU
- Tensorflow version (GPU?):
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?:NO


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[s2s] add config params like Dropout in Seq2SeqTrainingArguments
Author: patil-suraj
Pull request: yes

# What does this PR do?

1. Adds `config` params (`encoder_layerdrop`, `decoder_layerdrop`, `dropout`, `attention_dropout`) in `Seq2SeqTrainingArguments`
2. Fix T5 warnings (don't pass src_lang, tgt_lang args to `T5Tokenizer`)
3. Correct `vocab_size` for `FSMT`.
4. Fix `test_finetune_trainer_slow`
5. minor code cleanup in `Seq2SeqTrainer`

@sshleifer 

Comments:
Yes, will add lr scheduler in separate PR 

@sshleifer  anything missing in this PR ?
Using control codes for finetuning
Author: tuhinjubcse
Pull request: no
Labels: name: wontfix

Hi
I have a use case of style conditioned generation, where I ask the LM to generate a sentence based on the control code I provide. CTRL is pretty suitable for that task.
Can you tell me how to use control codes for fine-tuning as well as inference?  It should be the same as any CLM like GPT2 but I want to specifically know about the style and control code conditioning. How should be the data format and other stuffs

Comments:
@julien-c 

It'd be best to re-read the paper and original implem, but I think you just prepend a control code to each of your samples.

Cc'ing @keskarnitish for information.

PS/ for general questions, please use https://discuss.huggingface.co!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Don't see how to use correct padding with QA pipeline
Author: nathan-chappell
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details

In some cases, when using a **QuestionAnsweringPipeline**, I get an error similar to the following:

    *** ValueError: expected sequence of length 384 at dim 1 (got 260)

I've traced the problem to line 1496 of pipelines.py, and the change to [this commit](https://github.com/huggingface/transformers/commit/896300177bf9f35feac4698370212a80a5ab6138).  The troubled line is:

    fw_args = {k: torch.tensor(v, device=self.device) for (k, v) in fw_args.items()}

Basically, when trying to convert this **v** to a tensor an error is thrown because not every array in **v** has the same length - because the padding strategy has been changed (if I comment out the padding strategy in the call to **squad_convert_examples_to_features** then the default value `"max_length"` takes effect and there is no problem).  I guess this was done as some sort of optimization, but I'm not really sure how to use it.  Every other argument to **squad_convert_examples_to_features** will use a *kwarg*, but this one does not.  Maybe it should use a *kwarg* like everything else, so if you need the padding (or don't want to have to deal with it) you can set the **padding_strategy** as you like?  Or am I missing something?

### Minimal code to reproduce:

    from transformers import QuestionAnsweringPipeline, AutoModelForQuestionAnswering, AutoTokenizer
    model = AutoModelForQuestionAnswering.from_pretrained('twmkn9/distilbert-base-uncased-squad2')
    tokenizer = AutoTokenizer.from_pretrained('twmkn9/distilbert-base-uncased-squad2')
    
    # I've omitted the context for brevity.  You can, for example, take the **plot** section from [the matrix](https://en.wikipedia.org/wiki/The_Matrix)
    context = """..."""
    
    pipeline({"question":"what was Neo's job?", "context": context}) # error as described above

<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
🐛 Empty TypeError on BartTokenizerFast.decode(tensor)
Author: astariul
Pull request: no
Labels: name: wontfix

## Environment info
`transformers` `3.0.2`

### Who can help
 Summarization: @sshleifer 

## To reproduce

```python
import torch
from transformers import BartTokenizerFast

t = BartTokenizerFast.from_pretrained('facebook/bart-large')

x = torch.tensor([0, 34, 45, 23, 54, 65, 765, 2])
t.decode(x)
```

will throw an empty `TypeError` :

```
File "/home/me/.venv/summarization/lib/python3.6/site-packages/tokenizers/implementations/base_tokenizer.py", line 267, in decode
    return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)
TypeError
```

To reproduce : [Colab Notebook](https://colab.research.google.com/drive/1bnP8TvmRrHrMD-7H2MOQQSE8QrhCb7SC?usp=sharing)

## Expected behavior

No error thrown, like with regular `Tokenizer` :

```python
import torch
from transformers import BartTokenizer

t = BartTokenizer.from_pretrained('facebook/bart-large')

x = torch.tensor([0, 34, 45, 23, 54, 65, 765, 2])
t.decode(x)
```

> `
 has not at who one short
`

Comments:
I noticed this is due to `x` being a tensor. It works fine with a list :

```python
import torch
from transformers import BartTokenizerFast

t = BartTokenizerFast.from_pretrained('facebook/bart-large')

x = [0, 34, 45, 23, 54, 65, 765, 2]
t.decode(x)
```

> `
 has not at who one short
`

---

So the current work-around is to first convert to a list :

```python
import torch
from transformers import BartTokenizerFast

t = BartTokenizerFast.from_pretrained('facebook/bart-large')

x = torch.tensor([0, 34, 45, 23, 54, 65, 765, 2])
t.decode(x.tolist())
```

Tokenizer bug. @mfuntowicz is this expected behavior?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RAG: error in outputs = model(input_ids=input_ids, labels=input_dict["labels"])
Author: shamanez
Pull request: no

I tried the following code given in the documentation. 

```
from transformers import RagTokenizer, RagRetriever, RagModel
import torch
tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-base")
retriever = RagRetriever.from_pretrained("facebook/rag-token-base", index_name="exact", use_dummy_dataset=True)
# initialize with RagRetriever to do everything in one forward call
model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever)
input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", "In Paris, there are 10 million people.", return_tensors="pt")
input_ids = input_dict["input_ids"]
outputs = model(input_ids=input_ids, labels=input_dict["labels"])

```
In the last step, it gives the following error. 


**/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'labels'**


Comments:
Yes, the `RagModel` just contains the bare model and has no training objective (like all HF `XxxModel`). It doesn't take a `labels` argument, the example in the documentation is wrong.
Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained?
Author: h56cho
Pull request: no

Hello,
Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained as well?
I am asking because like for the pre-trained `GPT2DoubleHeadsModel`, the main body of the model is trained but its multiple-choice head is not.

Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model untrained, just like the case for the `GPT2DoubleHeadsModel`?

Thank you,

Comments:
Hi! This depends on the checkpoint. If you're using a checkpoint pre-trained on multiple-choice, then it's very possible that it is pre-trained. If you're using a checkpoint pre-trained on another task, it might not be pre-trained.

You should be wary of the task on which the model was trained when leveraging a model with a pre-trained head, as it might not overlap with your current task.
Cleanup documentation for BART, Marian, MBART and Pegasus
Author: sgugger
Pull request: yes

# What does this PR do?

This is a follow-up from #7345 to finishing cleaning up the documentation for all models. Nothing of importance apart from the configurations of the 4 classes (BART, Marian, MBART and Pegasus) that can't have the same docstrings with a choose your own adventure default. They all need their own docstrings, since the defaults are different for a lot of values (which also means they all need their own implementation and not subclass the same config or at least pass the arguments with the proper defaults to the superclass).

I tried my best to document the actual default for each, but may missed a few fields.

Comments:
ELECTRA - some weights are not loaded
Author: mdocekal
Pull request: no

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-118-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
 albert, bert, GPT2, XLM: @LysandreJik 
 Model Cards: @julien-c 

## Information

Model I am using: ELECTRA
I am getting a warning:

> Some weights of the model checkpoint at google/electra-large-discriminator were not used when initializing ElectraModel: ['electra.embeddings_project.weight', 'electra.embeddings_project.bias']
> - This IS expected if you are initializing ElectraModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
> - This IS NOT expected if you are initializing ElectraModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

when using the AutoModel.from_pretrained for google/electra-base-discriminator or google/electra-large-discriminator. There is no warning for google/electra-small-discriminator.

The problem remains the same when directly using the ElectraModel.from_pretrained method.


## To reproduce
```
import transformers
m=transformers.AutoModel.from_pretrained("google/electra-large-discriminator")
# or
m=transformers.AutoModel.from_pretrained("google/electra-base-discriminator")
```


## Expected behavior
no warning

Comments:
Hi, indeed, this can scare users even if there is no actual problem. #7569 will fix this.
Problem with Finetuned GPT-2
Author: 51naa
Pull request: no

Hi,
I have written this code to finetune gpt-2 on a new corpus.

```
from transformers import (
    AutoModelWithLMHead,
    AutoConfig,
    Trainer,
    AutoTokenizer,
    TextDataset,
    DataCollatorForLanguageModeling,
    TrainingArguments)

def modelTrainer(text_path, output_dir, batch_size=2, conf='gpt2', cache_dir='./Cache'):

    config = AutoConfig.from_pretrained(conf)
    model = AutoModelWithLMHead.from_config(config)
    tokenizer = AutoTokenizer.from_pretrained(conf)
    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
    train_dataset = TextDataset(
        tokenizer=tokenizer,
        file_path=text_path,
        block_size=128,
        cache_dir=cache_dir
    )
    training_args =TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=1,
        per_device_train_batch_size=batch_size,
        warmup_steps=500,
        save_steps=500,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        prediction_loss_only=True
    )

    trainer.train()
    trainer.save_model()

```

And then I use this to generate text from the finetuned model:

```
from transformers import pipeline, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
def textGenerator(model_dir):
    gen = pipeline('text-generation', model=model_dir, tokenizer=tokenizer)
    return gen
```

Now my problem is that even with 1 epoch a training, the quality of generated text deteriorates drastically and I get some unknown tokens in the output, like: 'Hello,�\n\n,�\n”,\n and\n,,� the\n,\n\n the,\n� Alice\n\n,“\n the\n,,,“ on”, to� she the�\n'. 
I'm guessing there is a problem with tokenizer. Can anybody help me with this?

Comments:
This problem was resolved with 10 epochs of training.
Sequence Classification One-Hot Encoded Data
Author: moritzblum
Pull request: no

## Environment info     
- `transformers` version: 3.3.1

### Who can help 
 albert, bert, GPT2, XLM: @LysandreJik 
 
## Information
I am using Bert and Roberta.

The tasks I am working on is:
Sequence Classification

## Problem
The model does not work with "One-Hot Encoded" data. The model only accepts a list of integers as labels which is feed into the MSELoss. This is undesired in a multi-label classification task with categorical data, because an order of the classes is induced.



Comments:
Hi, if you want to implement a custom loss, you should not pass `labels` to the model, but instead retrieve the `hidden_states` and compute them as you would with any other PyTorch model.

Thanks for your fast answer. I found out that nn.CrossEntropyLoss expects class indices and does not take one-hot encoded tensors as target labels. So there is no difference between your implementation and using one-hot encoded labels.
Is training distilbert with TPU supported yet?
Author: xinyiz1019
Pull request: no

# 🚀 Feature request

Hi! I tried training my own distilbert model with [this code](https://github.com/huggingface/transformers/blob/master/examples/distillation/train.py) using GPU, and it was a success.
I'm wondering if training distilbert model with TPU is supported yet, or if there's any plan to release a new version in which TPU is supported?



Comments:
Distillation is not supported (yet) on TPU. You can check the status of different example scripts [here](https://github.com/huggingface/transformers/tree/master/examples). Those supported by `Trainer` or `TFTrainer` or `pytorch-lightning` can be run on TPU, others cannot.
[model_card] bert-base-5lang-cased
Author: amineabdaoui
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
Hosted Inference API: Error loading tokenizer Can't load config
Author: sampathkethineedi
Pull request: no

Hi,

[https://huggingface.co/sampathkethineedi/industry-classification-api](https://huggingface.co/sampathkethineedi/industry-classification-api)

I uploaded my classification model fine tuned on BERT. There is no issue running the model from the hub and using the ‘sentiment-analysis’ pipeline. But there seems to be some problem when it comes to Hosted Inference API.

```
Error loading tokenizer Can't load config for 'sampathkethineedi/industry-classification-api classification'. Make sure that: - 'sampathkethineedi/industry-classification-api' is a correct model identifier listed on 'https://huggingface.co/models' - or 'sampathkethineedi/industry-classification-api' is the correct path to a directory containing a config.json file OSError("Can't load config for 'sampathkethineedi/industry-classification-api'. Make sure that:\n\n- 'sampathkethineedi/industry-classification-api' is a correct model identifier listed on 'https://huggingface.co/models'\n\n- or 'sampathkethineedi/industry-classification-api' is the correct path to a directory containing a config.json file\n\n")
```

Can someone help me with this?

Thanks!

Comments:
I can load the pipeline locally so I am not sure what's happening here, indeed. Will post here when we know more.

Hey @julien-c, thanks for the quick response!

I'm assuming the inference API is using the `transformers-cli serve` ?
When I do the same locally, I'm able to get results at **localhost:8888/forward** using `POST -d {"inputs": ["< Text >"]}` 
But when I try it on the https://api-inference.huggingface.co/models/sampathkethineedi/industry-classification-api, it doesn't work.

The Hosted Inference API has the same structure as `transformers-cli serve` or is it any different?

[https://huggingface.co/sampathkethineedi/industry-classification-api](https://huggingface.co/sampathkethineedi/industry-classification-api)

I just checked and the API is working. Would love to know what was causing the problem. Closing the issue.

@sampathkethineedi can i get dataset u trained on
[GPT-2] How many columns in LM model wte layer are positional embeddings?
Author: changyeli
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Hello everyone, I have a quick question about the token weight matrix from the GPT-2 model.

The transformers documentation for GPT-2 indicates that 

> GPT-2 is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather than the left.

How many columns in the ```transformer.wte.weight``` are linked to the positional embeddings? For the GPT-2 small model, the size of embedding matrix is (50257, 768), in those 768 columns, how many of them are linked to the positional embeddings?

Many thanks!

Comments:
I would say that none are. The positions are managed by the position embedding: `transformer.wpe.weight`.