noamaanMulla-03/transformers-issues huggingface dataset:
ForTokenClassification head on BART
Author: jonatasgrosman
Pull request: no

# 🚀 Feature request

Hello guys! I'm trying to reproduce the token classification experiments from the [BART paper](https://arxiv.org/abs/1910.13461) using the HF/Transformers and found that a token classification head is missing on the current BART model HF implementation. 

The current BART implementation only has the "BartForConditionalGeneration" and "BartForSequenceClassification". Are there any plans to add a "BartForTokenClassification" head too? 


Comments:
It would be great to have a `BartForTokenClassification`. Does it use the same head as `BertForTokenClassification`, etc.? 

Feel free to open a PR :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
pretraining objective of T5 model 
Author: dorooddorood606
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Hi, it would be great to have pretraining of T5 model implemented. Currently, run_mlm.py script does not support it.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

T5 is the SOTA model and having pertaining would be very helpful to the community.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Tensorflow not found but i can import it
Author: fedor360139
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: macOS-11.2.1-arm64-arm-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


@jplu, @patrickvonplaten, @LysandreJik 

## Description
When i import transformers i get message bellow.

## To reproduce

Steps to reproduce the behavior:

1. install tf for mac m1(https://github.com/apple/tensorflow_macos)
2. install transformers
3. import transformers


Message when i import transformers:
```
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
```

## Expected behavior
I'd like to make transformers find tensorflow.


Comments:
What is the output of:
```
import tensorflow
print(tensorflow.__version__)
```
?

> What is the output of:
> 
> ```
> import tensorflow
> print(tensorflow.__version__)
> ```
> 
> ?

'2.4.0-rc0'

Hello!

We currently don't support other implementations than the Google's pypi versions. The reason is because we don't tests on other versions and then we cannot guarantee it will works on those "extra" versions.

To make `transformers` works on Mac I suggest you to use the official version of TensorFlow as proposed in their documentation https://www.tensorflow.org/install/pip

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Masking issues with GPT2LMHeadModel.generate()
Author: xxbidiao
Pull request: no

Is this intended behavior, that padding a sentence and attention_mask it will not give the exact same generation result comparing to the same sentence unpadded?

Edit: [This notebook](https://colab.research.google.com/drive/1oyFRFigtSNUYwKO1EQPRHfEqke0-F6_N?usp=sharing) demonstrates this, with the newest version available on colab. I realized that I didn't turn sampling off with the example below but the colab one has sampling off.

```
>>> gpt2_model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
>>> gpt2_model.generate(torch.tensor([[100,200,300]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[100, 200, 300,  84,  12,  75,  84,  12,  75,  84,  12,  75,  84,  12,
          75,  84,  12,  75,  84,  12]])
>>> gpt2_model.generate(torch.tensor([[100,200,300,50256]]),attention_mask=torch.tensor([[1,1,1,0]]))
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[  100,   200,   300, 50256,   198,   198,     7,    16,     8,   383,
          3381,   366,    75,     1,  1724,   262,  4129,   286,   262,  4731]])
```

Comments:
Hey @xxbidiao, 

For batched generation GPT2 has to be used in quite a special way... -> could you check out [this](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/2) forum post to see whether this makes sense for you?

```
import torch,transformers
gpt2_model = transformers.GPT2LMHeadModel.from_pretrained("gpt2")
print(gpt2_model.generate(torch.tensor([[100,200,300]]),do_sample=False))
print(gpt2_model.generate(torch.tensor([[100,200,300,50256]]),attention_mask=torch.tensor([[1,1,1,0]]),do_sample=False))
print(gpt2_model.generate(torch.tensor([[50256,100,200,300]]),attention_mask=torch.tensor([[0,1,1,1]]),do_sample=False))
```

```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[100, 200, 300,  84,  12,  75,  84,  12,  75,  84,  12,  75,  84,  12,
          75,  84,  12,  75,  84,  12]])
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
tensor([[  100,   200,   300, 50256,   198,   198,     7,    16,     8,   383,
          3381,   366,    75,     1,  1724,   262,  4129,   286,   262,  4731]])
tensor([[50256,   100,   200,   300,    84,    12,    75,    84,    12,    75,
            84,    12,    75,    84,    12,    75,    84,    12,    75,    84]])
```

Looks like it works! Will double check. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Integrating GPT-2 model with Web page
Author: states786
Pull request: no

Hi, 

I would like to integrate GPT-2 model with web technology such as html, and javascript in order to build a similar editor https://transformer.huggingface.co/doc/gpt2-large


Can you please guide me how can I achieve it?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Tokenizer return offsets
Author: borsork377
Pull request: no

# 🚀 Feature request

Request for the feature raised in [Issue #1263](https://github.com/huggingface/transformers/issues/1263).

Previous PRs have attempted to address this but none of them were merged - https://github.com/huggingface/transformers/pull/1274 and https://github.com/huggingface/transformers/pull/2178.

## Motivation

Refer [Issue #1263](https://github.com/huggingface/transformers/issues/1263)

## Your contribution

Please guide me on how to submit a PR.

Comments:
All fast tokenizers have this feature, just pass along `return_offsets_mapping=True` in your call to the tokenizer. Also note that fast tokenizers are used by default when `AutoTokenizer` is called.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
PPLM example - AttributeError issue
Author: OlegDurandin
Pull request: no

Hi all,
Thank you for great library.
Now I try to understand PPLM model (see reference https://eng.uber.com/pplm/ ), and when I try to start example from HuggingFace repository https://github.com/huggingface/transformers/tree/master/examples/research_projects/pplm (run_pplm.py) - I faced with the next issue:


```
Traceback (most recent call last):
  File "run_pplm.py", line 820, in <module>
    run_pplm_example(**vars(args))
  File "run_pplm.py", line 678, in run_pplm_example
    repetition_penalty=repetition_penalty,
  File "run_pplm.py", line 405, in full_text_generation
    repetition_penalty=repetition_penalty,
  File "run_pplm.py", line 511, in generate_text_pplm
    device=device,
  File "run_pplm.py", line 115, in perturb_past
    grad_accumulator = [(np.zeros(p.shape).astype("float32")) for p in past]
  File "run_pplm.py", line 115, in <listcomp>
    grad_accumulator = [(np.zeros(p.shape).astype("float32")) for p in past]
AttributeError: 'tuple' object has no attribute 'shape'

```
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: 4.3.0.dev0
- Python version: 3.8
- PyTorch version (GPU?): 1.7.1

I started this example on my own laptop and in Google Colab environment

## To reproduce

Steps to reproduce the behavior:

1. Follow to https://github.com/huggingface/transformers/tree/master/examples/research_projects/pplm and do Setup steps
2. Do command: `python run_pplm.py -B military --cond_text "The potato" --length 50 --gamma 1.5 --num_iterations 3 --num_samples 10 --stepsize 0.03 --window_length 5 --kl_scale 0.01 --gm_scale 0.99 --colorama --sample
`

## Expected behavior
This script should work without an error :)


Comments:
Hi! Could you install an earlier version of `transformers` to see if it works? I believe it was tested with `transformers==3.0.1`

It works fine up to and including version v4.2.2 but is broken in versions above that

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Benchmark]onnx-export
Author: Zjq9409
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark
I  follow 04-onnx-export.ipynb this guidance on CPU, and my CPU model is:
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
NUMA node0 CPU(s):     0-9,20-29
NUMA node1 CPU(s):     10-19,30-39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

## Set-up
I set the environment is:
export OMP_NUM_THREADS = 8
export OMP_WAIT_POLICY= 'ACTIVE'
then I run all the test program py taskset -c 0-7 python test_.py

## Results
when I finish the test program, I print all result,like this:
dict_keys(['PyTorch CPU', 'ONNX CPU', 'PyTorch CPU Quantized', 'ONNX CPU Quantized'])
dict_values([94.01082992553711, 96.25397443771362, 82.11332082748413, 71.06868505477905])
so, In my operation “PyTorch CPU”:“ONNX CPU Quantized" promote 1.32X
but in the guidance  “PyTorch CPU”:“ONNX CPU Quantized" promote 5.78X
Why didn't it increase so many times confused me?


Comments:
Hi @jianqianzhou,

Thanks for raising this issue.

I would remove the `OMP_NUM_THREADS` environment variable to fully exploit all the cores/threads you have on your machine. Also, tests were run on a machine with 56 cores so it might impact final performances. 

Also, it might be possible to further optimize the model / quantized model through the ONNX Runtime optimizer tool.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
bug in distributed codes AssertionError: Default process group is not initialized
Author: ghost
Pull request: no

Hi
I am using transformers 3.5.1, on distributed fashion on multiple gpus with pytorch 1.6 and python=3.7 
I am running:

python -m torch.distributed.launch  --nproc_per_node=1 --nnodes=2         --node_rank=1 --master_addr=$host --master_port=$port finetune_trainer.py config.json 

Huggingface codes only work wiht distributed fashion when all gpus are relying on one machine.
If user wants to run two copy of codes on two machines, since in the codes it decides based on local_rank not rank, the local_rank for bot copies would be zero. 

could you have a look please?
thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Question About Attention Score Computation & Intuition
Author: rezhv
Pull request: no

When it comes to transformers, the Query and Key matrices are what determine the attention scores. Here is a nice visual taken from Jay [Alammar's blog post ](http://jalammar.github.io/illustrated-transformer/)on transformers that illustrates how attention scores are computed:
![self-attention_softmax](https://user-images.githubusercontent.com/56566565/104143061-e64b3e00-5372-11eb-8b0f-2c9568988aaa.png)

As you can see the attention score depends solely on qi and kj vectors multiplied with no additional parameters. However each of these two vectors are calculated through a linear layer **which had the word embedding (+positional) of just 1 word as input.** My question is: how can the network assign attention scores meaningfully if q and k are computed without looking at different parts of the sentence other than their corresponding word? **How can the network produce k and q vectors that when multiplied represent a meaningful attention score if k and q are computed based on a single word embedding?** 

lets say I want to process this sentence:
The man ate the apple; It didn't taste good. 

When calculating the attention scores for the word 'it', how would the model know to assign a higher attention score to 'apple'  (it refers to the apple) than to 'man' or basically any other word?  The model had no way of understanding the context of the sentence because q and k are calculated solely based on the embedding of one word and not the sentence as a whole. q for 'it' is computed from the apple's embedding and the same goes for k for 'apple'. The two vectors are then multiplied to get the attention score. wouldn't this mean that if the two words are present in a different sentence but with the same distance the attention score between the two would be identical in the second sentence?   

What makes sense to me is the classic approach to attention models. Look at the following visual from Andrew NG's deep learning specialization.

![eac4f222d9d468a0c29a71a3830a5c60-c5w3l08attentionmodel-3-638](https://user-images.githubusercontent.com/56566565/104143423-3971c080-5374-11eb-88e0-78454c3b795b.jpg)

Here the attention scores are calculated using the hidden states at that timestamp. The hidden states are calculated with FC layers in a bidirectional RNN. In other words a hidden state at a certain timestamp is influenced by the words that come after and before it, So it makes sense that the model is able to calculate attention scores there.

Comments:
HI @rezhv , that's a great question, I would suggest you ask such general questions on the forum https://discuss.huggingface.co/ and use issues to report bugs and to discuss new features :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Question on the example script run_glue.py for text classification
Author: xiaolin-cheng
Pull request: no

When we run this script to train a text classification, are the weights of the underlying language model frozen and not updated? Whether they are fixed or trainable, is there any config to change the behavior of the training process? Thanks!

Comments:
hi @xiaolin-cheng 

`run_glue.py` fine-tunes the whole model, it doesn't freeze anything. You would need to manually freeze the base model, you could do this after loading the `ForSequenceClassification` and then freeze the base model. For example for `BertForSequenceClassification` you can access the base model using `model.bert`.

```python
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
request for run_text_classification.py
Author: max-yue
Pull request: no

# 🚀 Feature request

There is a run_tf_text_classification.py file under text_classification examples, but no run_text_classification.py.  



Comments:
Hi! The `run_glue.py` script does what you're looking for.

> Hi! The `run_glue.py` script does what you're looking for.

Let's say I have train.csv, dev.csv and test.csv, how can I do it without modify the code? Thank you for your patience.

Please use the [forums](https://discuss.huggingface.co/) for questions around the script. Running it with the -h option will give you the list of arguments it accepts, in particular `--train_file` and `--validation_file`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
rename "gpu" --> "device"
Author: yuchenlin
Pull request: yes

# What does this PR do?

Rename the arg names from "per_gpu" to "per_device" such that it aligns with the instruction in the readme  https://github.com/huggingface/transformers/tree/master/examples/text-classification#xnli

Comments:
Could you run the code quality scripts for the code quality test? `make style && make quality`, after installing the latest code quality versions: `pip install -U .[quality]`

Looks like there is some styling issue. Could you run `make style` on your branch?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to implement seq2seq attention mask conviniently?
Author: zhizeng8
Pull request: no

BERT's attention mask is square, GPT's attention mask is triangular. How to implement seq2seq attention mask with transformers package conviniently? like the one appears in UniLM, a triangle concatenates a rectangle.
![unilm](https://user-images.githubusercontent.com/49787234/103397155-ff354180-4b71-11eb-8283-1c0f50f5b462.jpg)



Comments:
The image is also on this link.
https://img-blog.csdnimg.cn/20191025102941935.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9qYWNrY3VpLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70

Hey @zhizeng8, 

In the future, it would be nice if such questions are posted in the forum: https://discuss.huggingface.co/ as it is not about a bug. 
To answer your question I'd use something like the following

```python
    import torch

    tgt_len = 5
    # make causal mask
    mask = torch.full((tgt_len, tgt_len), float("-inf"))
    mask_cond = torch.arange(mask.size(-1))
    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
    # attend to encoder part
    mask[:, :3] = 0
```

This mask can however not just be input as an `attention_mask` to transformer models. Because BERT accepts 3d masks however with the 0-th index being the batch_size the above mask could be extended for all batches and input to BERT:

```python
3d_attention_mask = mask[None, :, :]

bert = BertModel.from_pretrained(...)
bert(input_ids, attention_mask=3d_attention_mask)
```

Thank you very much!
I think in my case the 3d_attention_mask is different for each instance in a batch, due to the different length of source and target sequence.

I have been reading this article recently. BertModel accepts 3d masks of dimensions [batch_size, from_seq_length, to_seq_length]. 
```
text='I love you' 
attention_mask = [[[1,0,0],[1,1,0],[1,1,1]]]
```
Maybe can help you.
Also I have a question.
```
text='I love you' 
attention_mask = [[[1,1,1],[1,1,1],[1,1,1]]]
```
 I found tensor of the last word 'you' are not same, I don't know the reason.

I know. Last word's tensor in first attention layer is same, but there are 12 attention layer. Hiddenstate may change.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.