noamaanMulla-03/transformers-issues huggingface dataset:
possible issues with run_summarization_finetuning.py
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi,
thanks for pushing summarization codes, here are my comments on this file:
 
- line 473: checkpoints = [] is empty and will not be evaluated. Also, the evaluation script is not called.

- line 482: results = "placeholder" is set to the placeholder, I was wondering if the function could return the generated text,so user could see the performance of the method visually. 

- line 463: the model is only saved after the training, would be great to have the saving option like run_glue, "eval_during_training" also during the training activated.

- line 272 /transformers/modeling_encoder_decoder.py, the tie of weights is not done, this is a part of the model, would be great to have it implemented

- line 139 of transformers/modeling_auto.py, here you check the "path" if this starts with the name of bert, ..., you load the relevant one, but in run_summarization_finetuning, the user does not need to save the model in args.output_dir which starts with the name of model, so the codes wont work if the model is not saved with the path starting with the name of the model

- line 152 of /run_summarization_finetuning.py for param_group in optimizer.param_groups: I think this should be optimizer[stack] not optimizer alone 

- utils_summarization.py: line 180, the comparison should not work, since s is a sequence, and then it is compared with special token, 

- utils_summarization.py:  line 182: embeddings.append(sentence_num % 2), to me you need to add sentence_sum%2 for the length of the sentence, but not 1 for each sentence.

thanks. 

Comments:
Hi!

Thanks for pointing these out. The summarization is still work in progress and should be included in the next release. Latest changes are in the `example-summarization` branch.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add PRETRAINED_INIT_CONFIGURATION to DistilBERT tokenizer
Author: LysandreJik
Pull request: yes

The DistilBERT tokenizer does not make use of `PRETRAINED_INIT_CONFIGURATION`, instead loading BERT's.

This PR fixes this, fixing the issue detailed in #2423.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=h1) Report
> Merging [#2469](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f599623a99b808e3d5926d89cd13237457b9eeba?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2469/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2469      +/-   ##
==========================================
+ Coverage   73.23%   73.24%   +<.01%     
==========================================
  Files          87       87              
  Lines       15003    15005       +2     
==========================================
+ Hits        10988    10990       +2     
  Misses       4015     4015
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/2469/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZGlzdGlsYmVydC5weQ==) | `100% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=footer). Last update [f599623...89df3b4](https://codecov.io/gh/huggingface/transformers/pull/2469?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
New SQuAD API for distillation script
Author: LysandreJik
Pull request: yes

The squad distillation script is still using methods from files that do not exist anymore (utils_squad and utils_squad_evaluate).

I updated the script to use the newer API.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=h1) Report
> Merging [#2457](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/16ce15ed4bd0865d24a94aa839a44cf0f400ef50?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2457/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2457   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       15001    15001           
=======================================
  Hits        10988    10988           
  Misses       4013     4013
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=footer). Last update [16ce15e...8eaea4e](https://codecov.io/gh/huggingface/transformers/pull/2457?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Masked tokens are -1 not -100?
Author: emillykkejensen
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

#2130

After haveing updated run_lm_finetuning.py to the newest git branch, I have incountered an error in train(). Haveing spent some time trying to figure it out, I realized, that masked tokens have been changed from -1 to -100. If I change it back to -1 it all works again.

https://github.com/huggingface/transformers/blob/f599623a99b808e3d5926d89cd13237457b9eeba/examples/run_lm_finetuning.py#L179

Won't work:
`
labels[~masked_indices] = -100  # We only compute loss on masked tokens
`

Works:
`
labels[~masked_indices] = -1  # We only compute loss on masked tokens
`

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
To fast... Sorry :)
https://github.com/huggingface/transformers/issues/2442
No way to control ID of special chars e.g. mask IDs
Author: DomHudson
Pull request: no
Labels: name: wontfix

## Summary

Hi, many thanks for the library - this is a fantastic tool for the NLP community!

I notice there are a number of constants defined in code that the user cannot inject whilst initialising. 
Some examples are:
- `padding_idx = 1` in `RobertaEmbeddings`
- `CrossEntropyLoss(ignore_index=-1)` in `RobertaForMaskedLM`. 
- `padding_idx=0` in `BertEmbeddings`

`RobertaModel` also raises a warning if there are no tokens with index `0`, but it is not clear which control character this corresponds to.

1. Would it be a good idea to allow these parameters to be injectable so a user can control the ID of the special tokens?
2. Is it possible to provide a list of what the expected indices for special characters are?
I think for example:
```
    -1 => Ignore target during loss
    0 => `[CLS]`
    1 => `[PAD]`
```
but 0 could also be `[SEP]` as I believe both are always used in roBERTa.
Is there an index I must respect other than these? E.g. does `[SEP]` need a specific index?
3. Why is the ignore index -1? Is this just to stay true to the original papers? Wouldn't the index of the `[PAD]` token make sense? I notice this index is different in the different embedding classes.

Many thanks for your thoughts,
Dom


Comments:
To answer question 2:
You can see the assumed indices for special characters by loading a tokenizer and inspecting the `vocab` attribute.

For example:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print(tokenizer.vocab)
```
This shows that for the `bert-base-uncased` model it is assumed that:

| Special Token | Index |
| --- | --- |
| [PAD] | 0 |
| [UNK] | 100 |
| [CLS] | 101 |
| [SEP] | 102 |
| [MASK] | 103 |

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Slight different output between transformers and pytorch-transformers
Author: tcqiuyu
Pull request: no
Labels: name: wontfix

I am now working on a Chinese NER tagging task. I applied BertForTokenClassification. The original library I used is pytorch-transformers 1.2.0. Then I migrated to Tranformers 2.1.1. But I found the output is slightly different between two versions. See the pics below

![Screen Shot 2019-10-24 at 14 46 00](https://user-images.githubusercontent.com/6031166/67460665-2b9c2680-f66e-11e9-9371-d29246dedc9f.jpg)
![Screen Shot 2019-10-24 at 14 44 47](https://user-images.githubusercontent.com/6031166/67460670-2d65ea00-f66e-11e9-8728-d615247107b6.jpg)

I wondered what potentially caused this difference?


## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
BertForTokenClassification

Language I am using the model on (English, Chinese....):
Chinese

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Maybe you didn't put the model in evaluation mode in one of the tests and the DropOut modules were not deactivated as such.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add Transformer-XL fine-tuning support.
Author: torshie
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Add Transformer-XL fine-tuning support.

## Motivation

This model archieves good language modeling result while having a "saner" number of parameters compared with GPT-2 or other language modeling.



Comments:
We don't have the bandwidth for that at the moment. But if somebody in the community is interested in working on that, happy to welcome a PR.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Undefined behavior
Author: konarkcher
Pull request: no
Labels: name: wontfix

## 🐛 Bug

There is an undefined behavior in `get_from_cache()` method in `transformers/transformers/file_utils.py`:
```python3
if not os.path.exists(cache_path) and etag is None:
    matching_files = fnmatch.filter(os.listdir(cache_dir), filename + '.*')
    matching_files = list(filter(lambda s: not s.endswith('.json'), matching_files))
    if matching_files:
        cache_path = os.path.join(cache_dir, matching_files[-1])
``` 

According to [docs](https://docs.python.org/3/library/os.html)  `os.listdir()`
> Return a list containing the names of the entries in the directory given by path. The list is in **arbitrary order**, ...

so taking last element from list returned by `os.listdir()` in the last row of snippet doesn't make sense because of arbitrary order. A possible solution is to add `sorted()`:
```python3
cache_path = os.path.join(cache_dir, sorted(matching_files)[-1])
```
I can make a PR if you agree.

Comments:
Yes thanks would be happy to welcome a PR.
Thanks for ca(t😂)ching that

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Changing LM loss function
Author: alecalma
Pull request: no
Labels: name: wontfix

Hi all,

I am modifying gpt2 loss function. My new code looks like that:


        lm_logits = self.lm_head(hidden_states)

        outputs = (lm_logits,) + transformer_outputs[1:]
        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = lm_logits[..., :-1, :].contiguous()
            shift_labels = labels[..., 1:].contiguous()
            all_logits = shift_logits[0].cpu().data.numpy()
            tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
            sampled_words = []
            for elem in all_logits:
                logits = elem.reshape(-1)
                exps = np.exp(numbers - np.max(logits))
                output_logits_normalized = exps / np.sum(exps)
                sampled_word = np.array(np.argmax(output_logits_normalized)).reshape([1,1])
                sampled_words.append(sampled_word)
            text = tokenizer.decode(np.array(sampled_words).reshape(-1))

            # Flatten the tokens
            loss_fct = CrossEntropyLoss(ignore_index=-1)
            loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
                            shift_labels.view(-1))
            print("CE Loss:", loss.cpu().data.numpy())
            l = Lyrics(text=text, language='en-us', lookback=15)
            rl = l.get_avg_rhyme_length()
            beta = 1
            rl_loss = rl * beta
            print("RL loss: ", rl_loss)
            total_loss = loss * 1/rl_loss
            print("Total loss: ", total_loss.cpu().data.numpy())
            outputs = (total_loss,) + outputs 

        return outputs  # (loss), lm_logits, presents, (all hidden_states), (attentions)


But after evaluation, every model checkpoint returns the same loss on the test set, so it seems that parameters are never updated. Could you please tell me why and how I could solve this?

Thank you a lot.

Comments:
This is quite a general question. Perhaps it's more useful to put this on Stack Overflow. 

We plan to have a forum associated to the repo to discuss these types of general questions.
In the meantime, we are still happy to welcome them in the PR but the visibility is limited indeed.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Missing required argument 'mode' in run_ner.
Author: apohllo
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using: BERT

Language I am using the model on: Polish (irrelevant for the error)

The problem arise when using:
* [x] the official example scripts: run_ner.py

The tasks I am working on is:
* [x] my own task or dataset: token classification (aka NER)

## To Reproduce

Steps to reproduce the behavior:

1. Start run_ner.py with --evaluate_during_training
2. During evaluation the error will happen


## Expected behavior

Evaluation should run fine

## Additional context

There is a missing argument in line 156 `mode`, which (I belive) should be `"dev"`.
I can provide a PR if the above solution is confirmed.

Comments:
yes sure, happy to welcome a PR on this

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Quick Tour TF2.0 Training Script has Control Flow Error when Replacing TFBERT with TFRoberta
Author: ryanrgarland
Pull request: no
Labels: name: wontfix

## 📚 Migration

<!-- Important information -->

Model I am using (Bert, XLNet....): TFRoberta.

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: Quick Tour TF2.0 Training Script.
* [ ] my own modified scripts: 

Details of the issue:

When replacing TF BERT with TF Roberta (and the relevant tokenizer) in the quick tour script, I get the following error:

```
TypeError: You are attempting to use Python control flow in a layer that was not declared to be dynamic. Pass `dynamic=True` to the class constructor.
Encountered error:
"""
using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph is disabled in this function. Try decorating it directly with @tf.function.
"""
```

I suspect this extends to all models, though I haven't verified this. Any thoughts?

## Environment

* OS: Catalina
* Python version: 3.7.14
* PyTorch version: NA
* PyTorch Transformers version (or branch): Transformers
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Checklist

- [x] I have read the migration guide in the readme.
- [x] I checked if a related official extension example runs on my machine.

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi, I think this was fixed by #1601, could you try now by cloning and installing from master?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
T5: move rp_bucket to relative_attention_bias' device
Author: mschrimpf
Pull request: yes

otherwise, `rp_bucket` will always be on cpu and fail if `self.relative_attention_bias` is on cuda

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=h1) Report
> Merging [#2495](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/331065e62d11d5c26642cb92a597904eee4c159b?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2495/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2495      +/-   ##
==========================================
- Coverage   73.24%   73.06%   -0.18%     
==========================================
  Files          87       87              
  Lines       15005    15006       +1     
==========================================
- Hits        10990    10964      -26     
- Misses       4015     4042      +27
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/2495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `81.09% <100%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `25% <0%> (-7.15%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/2495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `66.37% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `91.53% <0%> (-1.59%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2495/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `86.91% <0%> (-0.65%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=footer). Last update [331065e...90d3b78](https://codecov.io/gh/huggingface/transformers/pull/2495?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks Martin!
glue_convert_examples_to_features not working if no task is provided
Author: oliviernguyenquoc
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

The problem arise when using:
* [X] my own modified scripts: (give details)

The tasks I am working on is:
* [X] my own task or dataset: (give details)

In these lines:
https://github.com/huggingface/transformers/blob/04c69db399b2ab9e3af872ce46730fbd9f17aec3/transformers/data/processors/glue.py#L66-L69

If no task is provided (even if it's a optional parameter), no processor is selected, label_list is still none.

At least an error should be thrown.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how can I pre-training my own model from the existed model or from scratch
Author: hischen
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I want to load the pre-training model like bert offered by google,and train language model on more corpus,how can I do it ?thanks

Comments:
Hi, you can see how to use the library in the [documentation](https://huggingface.co/transformers/). You might be interested in the library philosophy and the way to load pre-trained models, which is [described here](https://huggingface.co/transformers/quickstart.html). You might also be interested in the [examples](https://huggingface.co/transformers/examples.html), which showcase [how to fine-tune a language model](https://huggingface.co/transformers/examples.html#language-model-fine-tuning).

@hischen did you find solution for pre-training BERT on your corpus?
@LysandreJik fine tuning is different from pre-training. I could not find documentation about pre-training the model on a corpus. Can you please help me with that.

Regards,
D. Ravi Theja.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.