noamaanMulla-03/transformers-issues huggingface dataset:
Valohai integration
Author: JuhaKiili
Pull request: yes

Huggingface / Transformers Valohai integration

Changes to existing Transformers code:
- Prints Valohai-styled logs (JSON)

Additional info:
- valohai.yaml has most (but not all) parameters used by run_glue.py
- Valohai execution downloads all glue datas by default (still pretty fast). Download script placed in `utils/download_glue_data.py`.
- Valohai execution only saves model/checkpoint at the end by default (adjust with Valohai UI)
- Valohai execution logs every 25 steps (adjust with Valohai UI)


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=h1) Report
> Merging [#1903](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0cdfcca24b5739e25b584c05b866baa19ea382ef?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1903/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1903   +/-   ##
=======================================
  Coverage   84.04%   84.04%           
=======================================
  Files          97       97           
  Lines       14333    14333           
=======================================
  Hits        12046    12046           
  Misses       2287     2287
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=footer). Last update [0cdfcca...66fc8d2](https://codecov.io/gh/huggingface/transformers/pull/1903?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> * Loss logging no longer accumulates (why was that?) but shows raw loss

With the revised commits, this is no longer true (but I'm not able to edit @JuhaKiili's original message, and he's AFK at Slush :) )
Always use SequentialSampler during evaluation
Author: ethanjperez
Pull request: yes

When evaluating, shouldn't we always use the SequentialSampler instead of DistributedSampler? Evaluation only runs on 1 GPU no matter what, so if you use the DistributedSampler with N GPUs, I think you'll only evaluate on 1/N of the evaluation set. That's at least what I'm finding when I run an older/modified version of this repo.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=h1) Report
> Merging [#2002](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b0ee7c7df3d49a819c4d6cef977214bd91f5c075?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2002/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2002   +/-   ##
=======================================
  Coverage   84.05%   84.05%           
=======================================
  Files         105      105           
  Lines       15555    15555           
=======================================
  Hits        13075    13075           
  Misses       2480     2480
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=footer). Last update [b0ee7c7...508f939](https://codecov.io/gh/huggingface/transformers/pull/2002?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, this works! Thank you @ethanjperez.
How to fine-tune xlnet on SQuAD with the parameter setting provided in the paper?
Author: mralexis1
Pull request: no
Labels: name: wontfix

From [here](https://arxiv.org/pdf/1906.08237.pdf) on page 16, it seems we should set Layer-wise lr decay to 0.75. However, I didn't find a way to do so in `run_squad.py`. Could someone provide a sample command line that could run this fine-tune task with the given parameters?

Thanks!

Comments:
Here is my attempt to do layer-wise lr decay. It didn't help with the model performance though. Fixing the preprocessing code helped a lot, but still a few points lower than what they reported in the paper and lower than BERT large WWM model. See my comment in #947 
```
lr_layer_decay = 0.75
n_layers = 24
no_lr_layer_decay_group = []
lr_layer_decay_groups = {k:[] for k in range(n_layers)}
for n, p in model.named_parameters():
	name_split = n.split(".")
	if name_split[1] == "layer":
		lr_layer_decay_groups[int(name_split[2])].append(p) 
	else:
		no_lr_layer_decay_group.append(p)

optimizer_grouped_parameters = [{"params": no_lr_layer_decay_group, "lr": learning_rate}]
for i in range(n_layers):
	parameters_group = {"params": lr_layer_decay_groups[i], "lr": learning_rate * (lr_layer_decay ** (n_layers - i - 1))}
	optimizer_grouped_parameters.append(parameters_group)

optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=1e-6)
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
A Micro BERT
Author: aditya-malte
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello,
Has anyone solved a problem like this, or knows of a solution:
I want to pre-train BERT on a custom dataset, but this data is much smaller than the one used by Google.
So is it possible to train it on a "micro" bert with much lesser layers, etc.
Thanks in advance

Comments:
I am using a much smaller dataset with my project, but it doesn't mean I need a bert with lesser layers. Otherwise, I have no way to utilize the pre-trained model.

What is the problem you have with the smaller dataset?

My dataset is very esoteric, in the sense that BERTs pretrained weights will almost be like noise.

YOU NEED ALBERT

Einstein?

They are referring to this new [ALBERT paper](https://old.reddit.com/r/MachineLearning/comments/d9tdfo/albert_a_lite_bert_for_selfsupervised_learning_of/). No weights are available however so give it a few months.

Definitely try fine-tuning a pre-trained BERT first, you can also just edit the BertConfig class to get a smaller network, but you probably can't train it from scratch on a small amount of data.

Interesting. Can't I train a very small BERT as you said(maybe 2 layers) on like 4million tokens

I'm not sure what the minimum tokens and layers are, I'm not sure anyone has published that. Best to try it out.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Schedulers cause memory accumulation across folds in cross-validation?
Author: JohnGiorgi
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am facing a strange issue when using the schedulers available in this library within a cross-validation loop. Basically, in each fold, I initialize a new model, optimizer, and scheduler. GPU memory accumulates until I eventually get a CUDA out of memory issue. 

The simplest example I could come up with to reproduce the error is:

```python
import torch
from pytorch_transformers import WarmupConstantSchedule, WarmupCosineSchedule, WarmupLinearSchedule, WarmupCosineWithHardRestartsSchedule

# In my actual project, this is a for loop over the k-folds of k-fold cross-validation.
# In this example I use a while just to demonstrate the OOM error.
while True:
    net = torch.nn.Linear(10000, 10000)
    net = net.cuda()

    optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
    scheduler = WarmupCosineWithHardRestartsSchedule(optimizer, 1, 1000)

    # I also tried all the other schedulers. Same issue.
    # scheduler = WarmupConstantSchedule(optimizer, 1)
    # scheduler = WarmupCosineSchedule(optimizer, 1, 1000)
    # scheduler = WarmupLinearSchedule(optimizer, 1, 1000)

    del net, optimizer, scheduler
```

This will run until it (very quickly) uses up all 12GB on my Titan XP GPU. To make sure it was truly the initialization of the scheduler, I also tested

```python
import torch
from pytorch_transformers import WarmupCosineWithHardRestartsSchedule

while True:
    net = torch.nn.Linear(10000, 10000)
    net = net.cuda()

    optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)

    del net, optimizer
```

And did not see the memory accumulation or OOM error.

My question(s) is/are:

- Is this a known problem?
- Am I doing something dumb?
- How might I use a new scheduler for each fold of k-fold cross-validation in a way that doesn't lead to this issue?

Thanks a lot.

Comments:
I am facing  the same issue.When I use  the WarmupLinearSchedule and the 7th epoch  training , I get  a CUDA out of memory issue

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Running `import gc`, then`gc.collect()` and emptying the GPU’s cache should solve the issue temporarily. See #1742
Dose the file /examples/run_lm_finetuning.py provide a demo to pre-train a BERT
Author: bytekongfrombupt
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I'm curious that if the file run_lm_finetuning.py can pre-train a BERT. But if so, why the file named finetuning instead of pre-training? Thanks!

Comments:
Hi, BERT uses the NSP task as well as the MLM task during its pre-training phase. The `run_lm_finetuning.py` script only does MLM so you would need to modify it to pre-train BERT the same way it was done in the paper.
XLM-R Support
Author: josecannete
Pull request: no

## ❓ Questions & Help

Hello!

Is there a way to use XLM-R (https://github.com/pytorch/fairseq/blob/master/examples/xlmr/README.md) with the library of transformers? maybe via RoBERTa? can you provide some guidance on this please?

Thank you in advance

Comments:
The latest news about using XLM-R model with Transformers are discussed in #1769 
Briefly, **at the moment it's not possible to use this model with Transformers directly**.

> ##  Questions & Help
> Hello!
> 
> Is there a way to use XLM-R (https://github.com/pytorch/fairseq/blob/master/examples/xlmr/README.md) with the library of transformers? maybe via RoBERTa? can you provide some guidance on this please?
> 
> Thank you in advance

Thank you,

I'm closing this one and keep an eye in #1769
fix summary_type value of SequenceSummary
Author: tamuhey
Pull request: yes

from #1845


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=h1) Report
> Merging [#1846](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0477b307c7501ea76e01b03cb387a2312db752b3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1846/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1846   +/-   ##
=======================================
  Coverage   84.08%   84.08%           
=======================================
  Files          97       97           
  Lines       14316    14316           
=======================================
  Hits        12037    12037           
  Misses       2279     2279
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1846/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `89.64% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=footer). Last update [0477b30...d08a338](https://codecov.io/gh/huggingface/transformers/pull/1846?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, thanks!
summary_type value  of SequenceSummary is incorrect
Author: tamuhey
Pull request: no

https://github.com/huggingface/transformers/blob/0477b307c7501ea76e01b03cb387a2312db752b3/transformers/modeling_utils.py#L731

I think `if hasattr(config, 'summary_use_proj')` is incorrect, `if hasattr(config, 'summary_type')` is correct.

Comments:
Thanks!
Update modeling_utils.py by adding "DUMMY_INPUTS" after "logger" variable.
Author: RubensZimbres
Pull request: yes

This Pull Request fixes a probable bug found using Transformers in Anaconda, using TFBertForSequenceClassification, Tensorflow 2.0.0b0, according to:

https://github.com/huggingface/transformers/issues/1810#issuecomment-554572471



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=h1) Report
> Merging [#1847](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0477b307c7501ea76e01b03cb387a2312db752b3?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1847/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1847      +/-   ##
==========================================
+ Coverage   84.08%   84.08%   +<.01%     
==========================================
  Files          97       97              
  Lines       14316    14317       +1     
==========================================
+ Hits        12037    12038       +1     
  Misses       2279     2279
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1847/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `89.67% <100%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=footer). Last update [0477b30...07bda77](https://codecov.io/gh/huggingface/transformers/pull/1847?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I think this bug is already fixed on master.

It should be fixed the latest release (2.2.1).
Feel free to reopen if it's not the case.
Why add the arguments 'head_mask' and when to use this arguments
Author: AMANKB
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
        **head_mask**: (`optional`) ``torch.FloatTensor`` of shape ``(num_heads,)`` or ``(num_layers, num_heads)``:
            Mask to nullify selected heads of the self-attention modules.
            Mask values selected in ``[0, 1]``:
            ``1`` indicates the head is **not masked**, ``0`` indicates the head is **masked**.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Examples in Colab
Author: redditTroll
Pull request: no
Labels: name: wontfix

Hi all , does anyone have a Colab sample to share ? 

Comments:
Why not simply run the example scripts in colab yourself?

I'm not exactly sure how to set it up , this is a pretty popular library so I was thinking their might be a blog post out there 

https://huggingface.co/transformers/notebooks.html

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
`distilroberta-base` link missing
Author: felicitywang
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

According to the current master code, link for `distilroberta-base` isn't provided. 
https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/transformers/configuration_distilbert.py#L28-L33

Comments:
It is located under `configuration_roberta.py`, see it [here](https://github.com/huggingface/transformers/blob/1c542df7e554a2014051dd09becf60f157fed524/transformers/configuration_roberta.py#L31) :)

Thanks @stefan-it ! Missed the readme part of calling `distilroberta-base` with `RobertaModel` instead of `DistilBertModel`. Closing.
cannot import name 'get_linear_schedule_with_warmup' from 'transformers.optimization'
Author: FOXaaFOX
Pull request: no

## ❓ Questions & Help
cannot import name 'get_linear_schedule_with_warmup' from 'transformers.optimization'
<!-- A clear and concise description of the question. -->


Comments:
This could be related to this issue here: https://github.com/huggingface/transformers/issues/1837 :)

I copied the&nbsp;get_linear_schedule_with_warmup function code add to my project in the&nbsp;transformers/optimization.py
and then&nbsp;it worked&nbsp;
Thank you for developing such an brilliant library.




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"Stefan Schweter"<notifications@github.com&gt;;
发送时间:&nbsp;2019年12月5日(星期四) 上午7:19
收件人:&nbsp;"huggingface/transformers"<transformers@noreply.github.com&gt;;
抄送:&nbsp;"FOXaaFOX"<zhangyueaijia@qq.com&gt;;"Author"<author@noreply.github.com&gt;;
主题:&nbsp;Re: [huggingface/transformers] cannot import name 'get_linear_schedule_with_warmup' from 'transformers.optimization' (#2056)




This could be related to this issue here: #1837 :)
 
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
save all 12 layers outputs for each token
Author: vr25
Pull request: no

Hi,

I am following [this](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) post to create sentence vectors from the last 4 layers or sum all 12 layers and so on.

Considering a given document has 10,000 tokens and I am planning to use  "bert_base_uncased", I was wondering how to save all 12 layers for each token efficiently. Also, further, how to retrieve these layers in order to build sentence vectors.

Thanks!

Comments:
Hi, to retrieve all layers you can set the `output_hidden_states` flag to `True` in your model configuration:

```py
from transformers import BertConfig, BertModel

config = BertConfig.from_pretrained("bert-base-uncased")
config.output_hidden_states = True
model = BertModel.from_pretrained("bert-base-uncased", config=config)

outputs = model(inputs)
hidden_states = outputs[-1]
```

The model will now output an additional value which will be a tuple of size 1 + n_layer:
`hidden_states[0]`: output of the embedding layer
`hidden_states[1]`: output of the first layer
...
`hidden_states[12]`: output of the last layer
Pre-training a smaller version of BERT on own data
Author: paul-you
Pull request: no

## ❓ Questions & Help
Hello, 
<!-- A clear and concise description of the question. -->
I want to load the first 3-4 layers of BERT, pre-train those on my own data and then fine-tune the model on the target task. Is this possible?

I took a look at the [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) and changed the lines 478-479:

`    config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          cache_dir=args.cache_dir if args.cache_dir else None)`
to
`    config = BertConfig(num_hidden_layers=3)
`
I'm not sure, if this change is sufficient and if the first or last three layers will be taken from the pre-trained BERT?

Is the _next sentence prediction_ pre-training task implemented somewhere ? 

Thanks in Advance 





Comments:
Hi, you might be interested in freezing some layers rather than initializing a model with only 3 layers. Freezing some layers mean that they won't be affected by the backpropagation.

If that's what you're looking for, @BramVanroy's answer may be of help: https://github.com/huggingface/transformers/issues/1431

We do not have an NSP example in our examples.
save_pretrained on CamembertTokenizer
Author: YohannLeFaou
Pull request: no

## 🐛 Bug

This is probably something you already know, but `save_pretrained` for `CamembertTokenizer` seems not working at the moment.

Here is the error message I get:

```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-38-52393f32f8a0> in <module>
      1 results = {}
      2 if args.do_eval and args.local_rank in [-1, 0]:
----> 3     tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
      4     checkpoints = [args.output_dir]
      5     if args.eval_all_checkpoints:

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    281 
    282         """
--> 283         return cls._from_pretrained(*inputs, **kwargs)
    284 
    285 

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
    410 
    411         # Instantiate tokenizer.
--> 412         tokenizer = cls(*init_inputs, **init_kwargs)
    413 
    414         # Save inputs and kwargs for saving and re-loading with ``save_pretrained``

/mnt/azmnt/code/Users/adm/transformers/transformers/tokenization_camembert.py in __init__(self, vocab_file, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, additional_special_tokens, **kwargs)
     55         self.max_len_sentences_pair = self.max_len - 4  # take into account special tokens
     56         self.sp_model = spm.SentencePieceProcessor()
---> 57         self.sp_model.Load(str(vocab_file))
     58         # HACK: These tokens were added by fairseq but don't seem to be actually used when duplicated in the actual
     59         # sentencepiece vocabulary (this is the case for 
 and 


/anaconda/envs/azureml_py36/lib/python3.6/site-packages/sentencepiece.py in Load(self, filename)
    116 
    117     def Load(self, filename):
--> 118         return _sentencepiece.SentencePieceProcessor_Load(self, filename)
    119 
    120     def LoadOrDie(self, filename):

OSError: Not found: "None": No such file or directory Error #2

```

Best

Comments:
This should be fixed in #1860 which should be merged shortly