noamaanMulla-03/transformers-issues huggingface dataset:
typo
Author: WrRan
Pull request: yes

# What does this PR do?
fix typo
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable 

Fixes # (issue)
-->

## Before submitting
- [Y] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hugging Face model Bio_ClinicalBERT producing 404 error
Author: NicoleJaneway
Pull request: no

I'm building a Named Entity Recognition (NER) model using the Hugging Face implementation of emilyalsentzer/Bio_ClinicalBERT. Up to today, I've had no issues with the model. Today it's not working as expected.


Question 1 - today, trying to train using:

MODEL_NAME = 'emilyalsentzer/Bio_ClinicalBERT'
model = text.sequence_tagger('bilstm-bert', preproc, bert_model=MODEL_NAME)
results in this error: 404 Client Error: Not Found for url: https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT/resolve/main/tf_model.h5

Does Hugging Face offer any kind of health check to ascertain the status of their models?

Question 2 - working with files (model.h5, model.json, and preproc.sav) I'd saved from earlier training iterations, I'm getting the same 404 error shown above. I don't understand wherein these files the call to Hugging Face is occurring. It doesn't seem to be in the .json, and the .h5 and .sav file formats are hard to inspect. Read more about what these files are: https://medium.com/analytics-vidhya/how-to-deploy-your-neural-network-model-using-ktrain-ae255b134c77

Back in February, I'd used these exact model.h5, model.json, and preproc.sav files to run the NER app using Streamlit, no problem. Not sure if this is temporary issue with Bio_ClinicalBERT or if I need to retool my original approach due to potentially permanent problems with this transformer model.

Comments:
Hi @NicoleJaneway ,

I think this issue is similar to the following one in `ktrain` repo:

https://github.com/amaiya/ktrain/issues/367

"Problem" is, that there's no TensorFlow compatible model found on the hub (more precisely the `tf_model.h5` one). One good "workaround" would be if the model owner (pinging @EmilyAlsentzer here) would upload such a model to avoid these message :hugs: 

Thanks, @stefan-it!  Unfortunately, with the 404 error, my app is no longer working.  I posted a new [ktrain issue](https://github.com/amaiya/ktrain/issues/369) about it.  In my experience, the creator has been amazingly responsive, so let's see what comes of the question.

Hello @NicoleJaneway, looking at the repository and its commit history, I don't think there ever was a `.h5` file uploaded.

Could you share the code you're using when using local files so that we can see what's going on? If using local files, `transformers` should look locally before looking on the server, so you shouldn't get a 404 error

Hey @LysandreJik, thanks for trying to help - I don't have this project up on a public github yet.  I'll let you know when I do.
'SequenceClassifierOutput' object has no attribute 'log_softmax'
Author: AngThanos
Pull request: no

Hi there, 

I'm trying to transfer the pre-trained ViT model (model base patch 16, image size 224) on Cassava Leaf Disease Dataset. However, when I started to train the model, I encountered an error: 'SequenceClassifierOutput' object has no attribute 'log_softmax' which is shown in details in the attached image.
Can someone help me with this error?

Many thanks.
![1](https://user-images.githubusercontent.com/41022754/119471445-7ebaeb80-bd73-11eb-9aeb-21f6c984a8b2.PNG)




Comments:
Hello! Could you share the code you have that led to this error? Thanks

This is my source code.  I use google colab. Thank  you for your fast reply.
[https://drive.google.com/file/d/1zRKRolc-IuKAt_J96gTCoO801r6eXu2q/view?usp=sharing](url)


I think the issue is that you have:
```py

class CassvaImgClassifier(nn.Module):
    def __init__(self, model_arch, n_class, pretrained=False):
        super().__init__()
        self.model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
        ... 

    def forward(self, x):
        x = self.model(x)
        return x

```
as a model; if I follow closely enough you're getting the outputs with:

```py
image_preds = model(imgs)
```

These will really be the outputs of the ViT model, which is a [`SequenceClassifierOutput`](https://huggingface.co/transformers/main_classes/output.html#transformers.modeling_outputs.SequenceClassifierOutput) as you can see from the [ViT docs](https://huggingface.co/transformers/model_doc/vit.html#transformers.ViTForImageClassification.forward)

I suppose you're interested in the `logits`, so you would have to do:
```py
image_preds = model(imgs).logits
```
instead.

Hope that helps.

Thank you very much. It works for me.
Enable memory metrics in tests that need it
Author: LysandreJik
Pull request: yes

PR https://github.com/huggingface/transformers/pull/11851 was merged without updating the tests to reflect the change in the argument default.

Explicitly specified the need for memory metrics for these tests.

Merging now to have CI pass, feel free to comment if that's not the right approach @stas00 @sgugger 

Comments:
Thanks a lot for catching and fixing @LysandreJik !
[trainer] multi-node tweaks
Author: stas00
Pull request: no

As I'm using Trainer in a multi-node setup, I will use this issue to post the things that could be improved for that type of env.

1. Repeated logging for non-rank-0 process rank-0 machine:

I gathered all these that get repeated 16 times on a 16 nodes machine:

```
[INFO|trainer.py:1145] 2021-05-20 20:16:39,037 >> ***** Running training *****
[INFO|trainer.py:1146] 2021-05-20 20:16:39,037 >>   Num examples = 1000
[INFO|trainer.py:1147] 2021-05-20 20:16:39,037 >>   Num Epochs = 1
[INFO|trainer.py:1148] 2021-05-20 20:16:39,037 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:1149] 2021-05-20 20:16:39,037 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1150] 2021-05-20 20:16:39,037 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1151] 2021-05-20 20:16:39,037 >>   Total optimization steps = 4

100%|██████████| 4/4 [00:02<00:00,  1.95it/s][INFO|trainer.py:1341] 2021-05-20 20:16:41,214 >> 
{'train_runtime': 2.185, 'train_samples_per_second': 1.831, 'epoch': 1.0}
Training completed. Do not forget to share your model on huggingface.co/models =)

INFO:__main__:*** Evaluate ***
[INFO|trainer.py:2115] 2021-05-20 20:16:41,690 >> ***** Running Evaluation *****
[INFO|trainer.py:2117] 2021-05-20 20:16:41,690 >>   Num examples = 200
[INFO|trainer.py:2120] 2021-05-20 20:16:41,690 >>   Batch size = 4
```

Probably should check not only rank of the process, but also the rank of the machine, right?

@sgugger 

Comments:
Mmm I guess there should be some argument controlling this: when I'm using multi-node I launch the command on two separate machines and have two separate terminals, so having both output the logs is helpful to know where each is at.

Absolutely agree for a few nodes! This becomes an issue on 64+ nodes ;) 

Let's have a flag that by default it logs on each node, and can be turned off if wanted.

This is all new so I'm first just sharing the things that can be improved

One other thing to figure out is pytorch error handling, when the launcher crashes it generated 64 interleaved tracebacks - impossible to understand what went wrong half the time... But that's not trainer-related...
# 🖥 Benchmarking `transformers`
Author: turnertye74
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

__Originally posted by @turnertye74 in https://github.com/huggingface/transformers/issues/11865__

Comments:
fixed a small typo in the CONTRIBUTING doc
Author: stsuchi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

I found a small typo in CONTRIBUTING.md. The fix was at the top of the second sentence around 5th paragraph after 4 bullet points as in "In particular there is a special [Good First
Issue](https://github.com/huggingface/transformers/contribute) listing. *It* will give you ..." 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Multi-node training for casual language modeling example does not work
Author: hamid-ramezani
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0.dev0
- Platform: Linux-4.19.0-14-amd64-x86_64-with-debian-10.8
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes 

### Who can help

@sgugger 

@patrickvonplaten, @LysandreJik 


## Information

Model I am using (Bert, XLNet ...): GPT-2

The problem arises when using:
* [x] my own modified scripts: 
```
nproc_per_node=4

  python -m torch.distributed.launch \
      --nproc_per_node=$nproc_per_node \
      --nnodes=2 \
      --node_rank=0 \
      --master_addr="192.168.1.1" \
      --master_port=1234  run_clm.py \
      --model_name_or_path gpt2 \
      --block_size 256 \
      --dataset_name wikitext \
      --dataset_config_name wikitext-2-raw-v1 \
      --do_train \
      --do_eval \
      --overwrite_output_dir \
      --num_train_epochs 1 \
      --output_dir /tmp/test-clm
```



The tasks I am working on is:
* [x] an official GLUE/SQUaD task: language modeling
* [x] my own task or dataset: wikitext

## To reproduce

Steps to reproduce the behavior:

1. Have two nodes with at least 4 GPUs each. 
2. In the first machine, run the above script.
3. In the second machine, run a script same as above except with the flag `--node_rank=1` instead of `--node_rank=0`.

I have waited for almost 15 mins. Nothing has happened. The training did not get started. 

## Expected behavior

The training gets started.  



Comments:
Are you sure the port is open between the two machines? Not having any output is usually a symptom of that. I've tried on my side and I can run the script on multi-nodes.

@sgugger Thanks for the reply :)

> Are you sure the port is open between the two machines?

yes I made sure by giving different port numbers; none of them worked and I got this message after 15 mins:

`RuntimeError: connect() timed out.` 



No they need to have the same port number, otherwise they can't connect to each other.

Thank you very much. 

> No they need to have the same port number

Here I meant I gave the same port number to both sides, but I tried multiple times with some numbers to make sure the port is open :) 

But no worries, the issue is solved. I tried with the actual IP address of one of the machines and that solved the issue. 

Glad you solved your issue!
What does the output of feature-extraction pipeline represent?
Author: orenpapers
Pull request: no
Labels: name: wontfix

I am using the feature-extraction pipeline:
```
nlp_fe = pipeline('feature-extraction')
nlp_fe('there is a book on the desk')
```
As an output I get a list with one element - that is a list with 9 elements - that is a list of 768 features (floats).
What is the output represent? What is every element of the lists, and what is the meaning of the 768 float values?
Thanks

Comments:
They are embeddings generated from the model. (Bert -Base Model I guess. cause it has a hidden representation of 768 dim). You get 9 elements:- one contextual embedding for each word in your sequence. These values of embeddings represent some hidden features that are not easy to interpret.

So the pipeline will just return the last layer encoding of Bert?
So what is the differance with a code like

```
input_ids = torch.tensor(bert_tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  
outputs = bert_model(input_ids)
hidden_states = outputs[-1][1:]  # The last hidden-state is the first element of the output tuple
layer_hidden_state = hidden_states[n_layer]
return layer_hidden_state
```
Also, does BERT encoding have similar traits as word2vec? e.g. similar word will be closer, France - Paris = England - London , etc?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> So the pipeline will just return the last layer encoding of Bert?
> So what is the differance with a code like
> 
> ```
> input_ids = torch.tensor(bert_tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  
> outputs = bert_model(input_ids)
> hidden_states = outputs[-1][1:]  # The last hidden-state is the first element of the output tuple
> layer_hidden_state = hidden_states[n_layer]
> return layer_hidden_state
> ```
> 
> Also, does BERT encoding have similar traits as word2vec? e.g. similar word will be closer, France - Paris = England - London , etc?

Hi @orko19,
Did you understand the difference from 'hidden_states' vs. 'feature-extraction pipeline'? I'd like to understand it as well
Thanks!

@merleyc I do not! Please share if you do :)

The outputs between "last_hidden_state" and "feature-extraction pipeline" are same, you can try by yourself

"feature-extraction pipeline" just helps us do some jobs from tokenize words to embedding
parallelize and deparallelize method for GPT-Neo series model
Author: Ankit-Dhankhar
Pull request: no

# 🚀 Feature request

Parallelize and deparallelize methods for distribution of attention modules across multiple GPUs.

## Motivation
Finetuning GPT Neo 2.7B model on 12 GB GPU gives out of memory error. Having a parallelize method would allow us to train that model by splitting attention modules across multiple GPUs of smaller VRAM.

## Your contribution
Considering [this line](https://github.com/huggingface/transformers/blob/daf0d6a97bb0225a2571a2612b8285e2c3913992/src/transformers/models/gpt2/modeling_gpt2.py#L522) in GPT2 code and the absence of doc for parallelize method in [GPT2 documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2model), wanted to know if these methods are still supported. If not, what is the recommended method for fine-tuning large transformer models like GPT-Neo?

If they are still supported, I can take up this task and submit PR for both methods as well as documentation fix.





Comments:
This is answered in #11054.

(I'm in a similar situation as you. I'm just going to go with the suggestion and use DeepSpeed instead of model parallelism.)

Thanks, didn't saw that. Parallelism notes are also awesome.
[Wav2Vec2ForCTC] example typo fixed
Author: madprogramer
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue) n/a

Fixed Typo:
In the example code for `transformers.Wav2Vec2ForCTC` loss was being computed on `transcription` instead of the `target_transcription` variable. An acquaintance of mine noticed the error, and that it had been corrected elsewhere, namely in a [code snippet for a fairseq example](https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
How to save and load model from local path in pipeline api ?
Author: yanan1116
Pull request: no

In `from_pretrained` api, the model can be loaded from local path by passing the `cache_dir`. However, I have not found any parameter when using `pipeline` 

for example, `nlp = pipeline("fill-mask" , model = 'distilbert-base-uncased', device=0)`
how to save the downloaded model and load it next time from local path, rather than default cache path ?
Thanks.

Comments:
I don't think it's currently possible, you would have to specify the local path in `model` but it won't ping the custom `cache_dir`.

We would happily welcome a PR that enables that for pipelines, would you be interested in that?

> I don't think it's currently possible, you would have to specify the local path in `model` but it won't ping the custom `cache_dir`.
> 
> We would happily welcome a PR that enables that for pipelines, would you be interested in that?

Thanks for your solution. I prefer to wait for new features in the future.