noamaanMulla-03/transformers-issues huggingface dataset:
Training BERT behind a proxy server
Author: PeliconA
Pull request: no

When I try to run BERT training, I get the following error during the vocabulary download:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /models.huggingface.co/bert/bert-large-uncased-vocab.txt (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbb55377828>: Failed to establish a new connection: [Errno 110] Connection timed out'))

I am running the script behind a proxy server which I suspect is the cause of this error. Is there any way to remedy this?


Comments:
You can download the Tensorflow weights from Google's BERT repo and convert them as detailed in the readme of the present repo.

Hi tomwolf,

I am new to XLNet and have the same issue as above. 

Could you direct me to the readme of the issue? I am not able to find it.

I modified my code to resolve the issue to 

**model_file_address = '/myfolder/XLNetWork/xlnet-base-cased-config.json'**

But I get the below error on the line : 
**model = XLNetForSequenceClassification.from_pretrained(model_file_address,num_labels=len(tag2idx))**

Error:
**UnpicklingError: invalid load key, '{'.**

I am pretty much stucked.

Your help will be appreciated.

Thanks,

Saul


@SaulML Did you solve it? I got the same error, i.e. `UnpicklingError: invalid load key, '{'.`, when I tried to load pretrained bert using `model = BertForSequenceClassification.from_pretrained("/PathToBert/uncased_L-12_H-768_A-12/bert_config.json", num_labels=2)`, thanks in advance!

You can now supply a `proxies` argument to `from_pretrained` when you are using proxies.
Check the doc and docstrings.

Got the same error as @SaulML  and @iamxpy. Has anyone solved it?

Got the same unpickling error. Was it solved?

I think you need to just have the path as the directory rather than the config file.
saving pretrained models that were obtained from another model
Author: dar-tau
Pull request: no

I am trying to save a pretrained model that I created from RobertaForQuestionAnswering by changing some layers.
However, when I load the model with from_pretrained, my new layers disappear.. It makes sense since binary seems to save only the model weights, but I wonder if there's a way to work around this?

Comments:
@dar-tau even though the file stores only weights, but if you added more layers those weights should be saved right? And when you try to load that model into RobertaForQuestionAnswering without your extra layers it should fail. 

Thanks for your reply. 
I'm using AutoModelForQuestionAnswering.from_pretrained(...) and not RobertaForQuestionAnswering (and as a matter of fact I'm actually replacing layers rather than just adding some).

What I am aiming for is a way to make it "forget" it originated from Roberta, and save the entire model.
My desire is that it will be loadable with from_pretrained(..) and semantically equivalent to:
torch.save(model, "file.pt") 
model=torch.load("file.pt")

@dar-tau I don't think this is something you can do with transformers, you need to probably do it using torch directly. You might want to check the config.json file which is saved.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Another way to express masked_index = torch.nonzero(input_ids == self.tokenizer.mask_token_id, as_tuple=False)
Author: moh-yani
Pull request: no

Dear,

We use PyTorch 1.1.0 to run a masked word completion with BERT. However, we found an error
`TypeError: nonzero() got an unexpected keyword argument 'as_tuple'`

The error refers to this:
`masked_index = torch.nonzero(input_ids == self.tokenizer.mask_token_id, as_tuple=False)`

Is there another way to express the syntax above with keep using PyTorch 1.1.0?

Best regards,

Mohammad YANI



Comments:
Hello! Unfortunately, recent transformers versions only work with torch 1.4.0+. The README is incorrect respective to that, and I'll update it in the coming days.

> Hello! Unfortunately, recent transformers versions only work with torch 1.4.0+. The README is incorrect respective to that, and I'll update it in the coming days.

Thank you for the response.

Okay. Does it mean that there is no another way for expressing `masked_index = torch.nonzero(input_ids == self.tokenizer.mask_token_id, as_tuple=False)` in pytorch 1.1.0? Is it still possible to use `torch.where(...)`? if yes, how to express it with `torch.where()`?

This is because the available machine we used is in an old driver version of CUDA.

Sincerely,
MY

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Accept tensorflow-rocm package when checking TF availability
Author: mvsjober
Pull request: yes

When working [on AMD GPU:s the TensorFlow package is called `tensorflow-rocm`](https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learning.html#tensorflow). This trivial PR just adds that to the list of accepted package names when checking for TensorFlow availability in `src/transformers/file_utils.py`.

At least the basic benchmark `run_benchmark_tf.py` seems to work fine on AMD MI50 and MI100 GPUs

Comments:
fix head_mask for albert encoder part(`AlbertTransformer`)
Author: baeseongsu
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #8323

Fix a subtle bug for using a submodule `AlbertTransformer` in `AlbertModel`.


## Who can review?

@LysandreJik 

Comments:
Great, thanks for the fix @baeseongsu ! Could you run the code quality tool to ensure the PR fits the quality requirements?

```
pip install -U -e .[quality]
make fixup
```

@LysandreJik Sure 👍
fix typo in command
Author: vipulraheja
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Is it correct to load weights from task A to train task B
Author: lacls
Pull request: no

> I want to use 2 tasks of modelling including (a) Causal language modelling &  
> (b) Mask language modelling for training my new added tokens 

My pseudo-code is below
```
##add new tokenizer
model_name = "vinai/phobert-large"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
with open("tokenizer_vocab.txt","r") as wr_file:
   new_tokens=wr_file.read().splitlines()
_________________
## Train the (a) task - Casual language modeling
added_tokens = tokenizer.add_tokens([tokens for i in new_tokens])
model.resize_token_embeddings(len(tokenizer))

## train the model following the hugging face released 
https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb#scrollTo=JAscNNUD3l-P

##save_the_model
model.save_pretrained('weights_tokenizer/')
tokenizer.save_pretrained('weights_tokenizer/')
___________________
## Train the (b) task - Masked language modeling
model = AutoModelForMaskedLM.from_pretrained("weights_tokenizer/")
tokenizer= AutoTokenizer.from_pretrained("weights_tokenizer/")

```
_My question here is it correct when I want to train the (b) task with weights from the (a) (because I think I can somehow more enrich the tokenizer)._

Or there is any solution that I can train my tokenizer in both these two tasks. And I then can use the weights (from training on both two tasks above) to train my model with the task (C)

I do appreciate your time and sharing.


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

Thanks for your reply @LysandreJik Ly
Fix typo in docstring
Author: eldarkurtic
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Re-styling in seq2seq attention
Author: sgugger
Pull request: yes

# What does this PR do?

After commenting the same things on multiple new models, I looked around and found everyone was copying this code I don't like from `BartAttention`. So fixing the source to avoid seeing it in the future :-)

Comments:
Tests should be fixed now as well: https://github.com/huggingface/transformers/pull/11615
Fix RNG saves in distributed mode.
Author: sgugger
Pull request: yes

# What does this PR do?

The newly introduced RNG saves in the Trainer checkpoints can yield to an error when one of the process of local_rank != 0 arrives to the end of the function faster then the process local_rank=0 arrived at the beginning: in this case the subfolder "checkpoint-xxx" is not created yet, so trying to save inside fails.

This PR fixes that.

Fixes #11618

Comments:
[fairscale] rng states saving fails in an extended multi-gpu test
Author: stas00
Pull request: no

@sgugger, following up on the RNG states PR
```
RUN_SLOW=1 pytest tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_fully_sharded_ddp_fp16 -sv
```
fails on multi-gpu scheduled CI:

```
File "/__w/transformers/transformers/examples/pytorch/translation/run_translation.py", line 589, in <module>
  main()
File "/__w/transformers/transformers/examples/pytorch/translation/run_translation.py", line 522, in main
  train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1316, in train
  self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1397, in _maybe_log_save_evaluate
  self._save_checkpoint(model, trial, metrics=metrics)
File "/__w/transformers/transformers/src/transformers/trainer.py", line 1534, in _save_checkpoint
stderr: Saving model checkpoint to /tmp/tmpb6tfhvqb/checkpoint-1
  torch.save(rng_states, os.path.join(output_dir, f"rng_state_{local_rank}.pth"))
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 369, in save
  with _open_file_like(f, 'wb') as opened_file:
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
  return _open_file(name_or_buffer, mode)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 211, in __init__
  super(_open_file, self).__init__(open(name, mode))
stderr: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpb6tfhvqb/checkpoint-1/rng_state_1.pth'
```

https://github.com/huggingface/transformers/runs/2514006341?check_suite_focus=true

Oddly enough it works fine for me if I try it locally.



Comments:
Found the root: the "checkpoint-1" folder is only created by process 0 when calling "model.save_pretrained", so if process 1 is faster and arrives at `_save_checkpoint` first, it will not see this folder (there is almost no saving on process 1, so it goes through this method quickly).

Will send a fix shortly.
Add BigBirdPegasus
Author: thevasudevgupta
Pull request: yes

# What does this PR do?
This PR will add Google's BigBird-Pegasus. Extending #10183

Following checkpoints will be added:

- [x] [bigbird-pegasus-large-pubmed](https://huggingface.co/google/bigbird-pegasus-large-pubmed)
- [x] [bigbird-pegasus-large-arxiv](https://huggingface.co/google/bigbird-pegasus-large-arxiv)
- [x] [bigbird-pegasus-large-bigpatent](https://huggingface.co/google/bigbird-pegasus-large-bigpatent)

It is verified that uploaded models work correctly, see:

- BigBird Pegasus Arxiv: https://colab.research.google.com/drive/1ntBBkiDgccbKwKmOECB8VWEFeFmZebLN?usp=sharing
- BigBird Pegasus BigPatent: https://colab.research.google.com/drive/1RKI0BG3JUy4Hn8VdIzNLE5QduwtaiXYZ?usp=sharing
- BigBird Pegasus Pubmed: https://colab.research.google.com/drive/1LebnFVp4unqZWRx5gez1hVyqR9cibIoH?usp=sharing

Here a notebook showing how well BigBirdPegasus works on long-document summarization: https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

Comments:
For running conversion script for `BigBirdPegasus`:

```shell
python3 src/transformers/models/bigbird_pegasus/convert_bigbird_pegasus_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-pegasus-large-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-pegasus-large-arxiv
```

For running conversion script for bigbird-roberta `EncoderDecoderModel`:

```shell
python3 src/transformers/models/bigbird_pegasus/convert_bigbird_roberta_tf_to_pytorch.py --tf_ckpt_path src/tf_ckpt/bigbird-roberta-arxiv/model.ckpt-300000 --save_dir src/google/bigbird-roberta-arxiv
```

@LysandreJik, yes we are planning to add this [notebook](https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) with a few modifications.

@patrickvonplaten, Test failing on CircleCi: `tests/test_modeling_bigbird_pegasus.py::BigBirdPegasusStandaloneDecoderModelTest::test_decoder_model_attn_mask_past` is passing for me locally.

Everything else is fixed!!
Add the ImageClassificationPipeline
Author: LysandreJik
Pull request: yes

This PR adds the `ImageClassificationPipeline`. It is tested on DeiT and ViT and should enable the inference API for these models.

Since I encountered an issue with the `AutoExtractor`, I fixed it as seen with @sgugger (namely switched it from using names such as `"deit"` to using the configuration class as a key, similarly to what we do in tokenizers and the other auto classes.

Please let me know if you would like me to split this PR into multiple PRs for simpler reviews, happy to do so.

@NielsRogge, happy to have your review.

Comments:
LGTM, thank you for adding this.

I should have addressed all comments. 

@Narsil, if you could review once again when you have time, would love your feedback on the updated tests and on the updated `__init__` method.
Unable to save and load RoBERTa model using tensorflow
Author: kiruthiga-A
Pull request: no
Labels: name: wontfix

I have trained my model with Roberta-base and tested, it works.
But when i try to save the model I get following error:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/save.py in list_dependencies(self, obj)
    126                   obj._object_identifier,  # pylint: disable=protected-access
    127                   name,
--> 128                   extra_dependencies.keys()))
    129         yield base.TrackableReference(name, extra_dependencies[name])
    130       else:

ValueError: Error when exporting object <tensorflow.python.keras.layers.core.Activation object at 0x7fef7d287128> of with identifier=_tf_keras_layer. The object has an attribute named regularization_losses, which is reserved. List of all reserved attributes: dict_keys(['regularization_losses', 'variables', 'trainable_variables', 'keras_api'])


Unable to solve it.

Is it some obvious mistake am doing?


# Code:


path = '/path/model_1'
tf.keras.models.save_model(model_roberta_base,path, overwrite=True,include_optimizer=False,save_format='tf')
new_model = tf.keras.models.load_model(path)
result =predct_func(new_model,"Technology driven by Medical imaging and Artificial Intellignence")
plot_result(result)

Comments:
Hey @kiruthiga-A,

Could you post a complete code example so that we can reproduce the error? If the complete code is too long, a google colab would be super useful.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have this same exact issue importing a model from HuggingFace
How to freeze Camembert model for Classification tasks? 
Author: siwarBM
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):


## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Comments:
See #400

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.