noamaanMulla-03/transformers-issues huggingface dataset:
Option to change Ray's gridsearch scope
Author: BramVanroy
Pull request: no

# 🚀 Feature request

It would be great if we can get more control over how Ray is selecting the best trial in a hyperparameter search. Currently, the default value for [`get_best_trial`](https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial) is used here:

https://github.com/huggingface/transformers/blob/7c5d79912a21880ce13d77881940458e90d98917/src/transformers/integrations.py#L299

namely `scope="last"`. So for each trial, it will simply take the _last_ checkpoint of that trial, and use its performance to compare with the other trials. This is not always ideal as it may very well be that some trials converge sooner than other and than overfit, leading to poor evaluation scores in their last checkpoints. Fortunately, Ray allows other options, such as `"all"`, which takes the best checkpoint instead of the last, of each trial and compares those.

## Motivation

Currently there is no way to pass this through to Ray from within `transformers` as I can tell, yet it is an important aspect of hyperparameter search.

## Your contribution

I can work on this if requested, although I am still looking for input how to best tackle this. One could simply add an argument to TrainingArguments, e.g. `ray_scope`, and then change this line

https://github.com/huggingface/transformers/blob/7c5d79912a21880ce13d77881940458e90d98917/src/transformers/integrations.py#L299

to

```python
best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3], scope=trainer.args.ray_scope)
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Yes, I'd still like to see this added. I can do a quick PR if needed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Another bump to remind myself that I'll do a PR for this in the coming month.
Broken 'export' using transformers.onnx
Author: RaiAmanRai
Pull request: no

### Feature request

HuggingFace allows an indirect way to export models that aren't supported by the current version of transformers[onnx], mentioned on huggingface website [here](https://huggingface.co/docs/transformers/serialization#implementing-a-custom-onnx-configuration). 

According to the original website, this method also provides a way to change the heads of the models based on users needs. 
Here is the original code mentioned on the website starting from generating the custom onnx configuration, exporting and validating the exported model.

~~~
from typing import Mapping, OrderedDict
from transformers.onnx import OnnxConfig

class DistilBertOnnxConfig(OnnxConfig):
    @property
    def inputs(self) -> Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                ("input_ids", {0: "batch", 1: "sequence"}),
                ("attention_mask", {0: "batch", 1: "sequence"}),
            ]
        )

from transformers import AutoConfig
model_ckpt = "distilbert-base-uncased"
config = AutoConfig.from_pretrained(model_ckpt)
onnx_config_for_seq_clf = DistilBertOnnxConfig(config, task="sequence-classification")

from pathlib import Path
from transformers.onnx import export
from transformers import AutoTokenizer, AutoModel

onnx_path = Path("C:/Users/Hp/zsc/onnx_deberta/model4.onnx")
base_model = AutoModel.from_pretrained(model_ckpt)
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config_for_seq_clf, onnx_config_for_seq_clf.default_onnx_opset, onnx_path)

import onnx
onnx_model = onnx.load("C:/Users/Hp/zsc/onnx_deberta/model4.onnx")
onnx.checker.check_model(onnx_model)

from transformers.onnx import validate_model_outputs

validate_model_outputs(
    onnx_config_for_seq_clf, tokenizer, base_model, onnx_path, onnx_outputs, onnx_config_for_seq_clf.atol_for_validation
)
~~~

But for some reason, the **task** doesn't seem to work. If you opt to only export and directly check the output shape of the model, it is always the same, which is the shape of last hidden layer of the model.
The validation is successful only when the model is exported with the default task, i.e. **task=default**.



### Motivation

This is a related issue raised by me prior to this, mentioned [here](https://github.com/huggingface/transformers/issues/16982), which points out the the wrong output shape generated even after specifying the tasks.
This is possibly a problem where the exported model is unable to change the heads of the exported models based on specified tasks.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Batching does not work with any XGLM model (simple script to reproduce issue included)
Author: ri938
Pull request: no

## Environment info

- `transformers` version: 4.17.0
- Platform: Linux-4.4.0-140-generic-x86_64-with-LinuxMint-18.1-serena
- Python version: 3.7.10
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.3.4 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@patrickvonplaten, @anton-l

Models:

- XGLM

## To reproduce

Here is a simple python script to reproduce the issue.  When using any XGLM model when I start using batching the behaviour changes and I get gibberish results often.  You can also reproduce this issue by padding an input text with a pad token and using attention mask 0 here because it is something to do with handling of attention mask.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
​from transformers import pipeline

# it fails with either of these models which are both xglm
MODEL = 'KoboldAI/fairseq-dense-125M'
MODEL = 'facebook/xglm-564M'
​
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL)
​
text = 'What is the only '
​
p = pipeline('text-generation', model=model, tokenizer=tokenizer)
​
print('expected')
print(p(text, do_sample=False, return_full_text=False, max_new_tokens=64))
​
print('without batching it works as expected and gives same response as above')
print(p([text] * 2 + ['some random text which is longer than the other'], do_sample=False, return_full_text=False, max_new_tokens=64))
​
print('with batching its broken and output is different even though the operation is deterministic')
print(p([text] * 2 + ['some random text which is longer than the other'], do_sample=False, return_full_text=False, max_new_tokens=64, batch_size=3))
```

Comments:
Super keen to resolve this issue and am willing to contribute if given suggestions.

cc @patil-suraj 

Hey @ri938 ! This is because the inputs are padded to right by default, but for batch generation the padding side should be left. If you set `tokenizer.padding_side = "left"` then it should work as expected.

Thanks this indeed did fix the issue with the facebook model but not the kobold model. I guess in this case its an issue with the kobold model and I shoud raise the issue with the provider of this instead.

Can you provide some Q&A  demo for the xglm model,Thanks

@htthYjh XGLM is an auto-regressive model like (GPT-2, 3) so you could try to do QA by providing some QA prompts and then feeding a que and asking it to generate the answer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
could not run distribution cpu training on two CPU sockets using miprun
Author: sywangyi
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.8.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.0 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


![image](https://user-images.githubusercontent.com/36058628/172326302-8b98b2b0-9121-4e6b-be22-5a3798564909.png)
it does not train one model using two CPU sockets, but each instance trains its own model with below output
[INFO|trainer.py:1472] 2022-06-07 00:51:33,164 >>   Instantaneous batch size per device = 12
[INFO|trainer.py:1473] 2022-06-07 00:51:33,164 >>   Total train batch size (w. parallel, distributed
 & accumulation) = 12


### Expected behavior

```shell
should output following log if two cpu sockets are used for distributed training and the training performance should be doubled comparing with training with one CPU socket. 
[INFO|trainer.py:1472] 2022-06-07 00:51:33,164 >>   Instantaneous batch size per device = 12
[INFO|trainer.py:1473] 2022-06-07 00:51:33,164 >>   Total train batch size (w. parallel, distributed
 & accumulation) = 24
```


Comments:
In run_mlm.py the group_texts function incorrectly splits the text into lists of chars
Author: simonhughes22
Pull request: no
Labels: name: bug

### System Info

```shell
Testing the code for the run_mlm.py - https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py to pull it into a separate notebook, I came across an issue with how the group_texts function is being used. Right now it correctly concatenates the input_ids and other lists, but it also operates on the text column (as that is one of the keys in the examples dict()) and incorrectly slices those strings as arrays, based on the token sequence length, which does not apply to the text string length. Further more it produces a list of lists of chars, not a list of strings. I am concerned this may cause issues for the subsequent DataCollatorForLanguageModeling or any downstream classes that rely on the original input text. If this text were tokenized, the function would work but in current code it appears to retain the original text strings.
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

``` python
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
    # customize this part to your needs.
    if total_length >= max_seq_length:
        total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [ t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=N_CPU,
    load_from_cache_file=True,
    desc=f"Grouping texts in chunks of {max_seq_length}",
)
```

### Expected behavior

```shell
For it to group and concatenate the tokenized_data correctly without messing up the text field.
```


Comments:
opened in accident, you are deleting that column
Add FP16 Support for SageMaker Model Parallel
Author: haohanchen-aws
Pull request: yes


(This PR is still pending some changes/fixes)
This PR adds support for SageMaker Model Parallel with FP16.

- To Enable fp16 with SMMP, user needs to add `fp16: True` to `SM_HP_MP_PARAMETERS`, when there's mismatch between `SM_HP_MP_PARAMETERS` and trainer args, a warning log would be printed and `SM_HP_MP_PARAMETERS` will be used as truth.
- Remove amp related stuff for SMMP as SMMP manages it's own half precision.
- Added `clip_master_grads` for grad clipping
- Some minor fixes


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?




Comments:
_The documentation is not available anymore as the PR was closed or merged._

@philschmid Please review or suggest review, thanks!

Can you just run `make style` on your branch to fix the code quality issue? Thanks!
How to generate data using beam search from a custom gpt2 model?
Author: nrjvarshney
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
I have a custom model with classification and an LM head.
`
        

        self.config = AutoConfig.from_pretrained("gpt2", num_labels=3)
        self.base_model = AutoModel.from_pretrained("gpt2", config=self.config)
        self.classifier = nn.Sequential(
            nn.Linear(self.config.hidden_size, self.config.num_labels),
        )

        self.lm_head = nn.Linear(self.base_model.config.n_embd, self.base_model.config.vocab_size, bias=False)`

I want to generate the sentences using this model (given the initial prefix) via beam search.
How can I achieve that?

I know that LM with double head exists but it's not fit for my usecase

Comments:
Here's an example using beam search with GPT-2:

```py
from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

input_ids = tokenizer("The day starts with", return_tensors='pt')['input_ids']
print(tokenizer.decode(model.generate(input_ids, num_beams=3)[0]))
```
Result:
```
The day starts with a long walk to the top of the hill.

The first thing you
```

[Here's the doc for the `generate` method](https://huggingface.co/transformers/main_classes/model.html#transformers.generation_utils.GenerationMixin.generate)

Thanks @LysandreJik for the response but I have a custom model, I want to know how can I generate using my model

@LysandreJik - How can I generate sentences using my custom model?

I would recommend you check the[ source code for the `generate` method](https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L111) and see how the beam search is implemented. It is not trivial, however.

Maybe @sshleifer and @patrickvonplaten have better tips on how to best do this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@nrjvarshney Did you find any suitable way to use `generate` function for a custom model? I am facing a similar issue with a model of mine, and would be really grateful if you could let me know how to solve the issue.
Fix doc builder Dockerfile
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the docker file in `transformers-doc-builder`.

~~(We don't need to run `doc-builder build` in the DockerFile, right? I think it is only for the CI runs.)~~

Currently, `Doc builder (Docker image build)` fails, see [this run](https://github.com/huggingface/transformers/actions/runs/2388105533).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The goal of running it inside the dockerfile is to ensure it actually works before publishing the image, so that it doesn't fail at runtime. Do you know why it failed in the first place?

Thank you, I understand it better now. I will check why it fails inside docker build.

@LysandreJik 

The problem comes from the fact that `$PR_NUMBER` is not defined in the docker image build (`doc-builder`). We can use `main` instead, right?

From `--help`, I saw

```
--version VERSION     Version of the documentation to generate. Will default to the version of the package module (using `main` for a version containing dev).
```

Changed `pr_$PR_NUMBER` to `main`.

Changed to 

```
RUN doc-builder build transformers transformers/docs/source/en --build_dir doc-build-dev --notebook_dir notebooks/transformers_doc --clean --version main
```
works.

Job run page
https://github.com/huggingface/transformers/runs/6775059643?check_suite_focus=true
clm example training script uses larger train/eval data than it should
Author: mayankjobanputra
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.12.3
- Platform: Linux-4.15.0-180-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: (True)
- Using distributed or parallel set-up in script?: (True)
```


### Who can help?

@sgugger @patil-suraj 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Just run `examples/pytorch/language-modeling/run_clm.py` script with `max_train_samples` parameter with value 1, it'll still group first 1024 tokens regardless.

This is also a bug across libraries (i.e., tensorflow, FLAX) and also for `max_eval_samples` parameter as well.

### Expected behavior

```shell
The script should only use the max number of samples specified from the dataset. This happens because the grouping takes place before selecting the number of samples.
```

Grouping: (https://github.com/huggingface/transformers/blob/dcb08b99f44919425f8ba9be9ddcc041af8ec25e/examples/pytorch/language-modeling/run_clm.py#L447)

Dataset selection:(https://github.com/huggingface/transformers/blob/dcb08b99f44919425f8ba9be9ddcc041af8ec25e/examples/pytorch/language-modeling/run_clm.py#L460)



Comments:
If you approve that this is a legitimate bug, then please let me know I will open the PR.

This is completely intended and not a bug. Sample/example is meant as one processed training/evaluation example, which is what is done here.

Yeah maybe for my purposes I needed it to filter the number of samples before grouping and I thought it would be same for others. 

Thanks for the quick response. Closing the issue :)
Rag end2end new
Author: shamanez
Pull request: yes

# What does this PR do?

I revamped the **[rag-end2end-retriever](https://github.com/huggingface/transformers/tree/main/examples/research_projects/rag-end2end-retriever)** to be compatible with the latest versions of PL and RAY. This was requested by @aaronmueller. 

@patrickvonplaten  

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Ok for me! Could we maybe though delete the commented out code?

Thanks a lot for the update. I did remove unwanted comments. Please let me know if I need to do anything more.
Include a comment to reflect Amy's contributions
Author: sayakpaul
Pull request: yes

This PR adds a note to `src/transformers/modeling_tf_pytorch_utils.py` to reflect @amyeroberts's contributions suggested in https://github.com/huggingface/transformers/pull/17571. It is an oversight on my end that I forgot to mention in the first place. 

I hope it's viewed as a mistake and not as a plagiarism attempt. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger forgot tagging you.

Hey @sayakpaul, that's really admirable, thank you for that. I personally don't think it needs to be in a comment, as the code isn't where attribution lies, and would clutter the code with non-technical details. The attribution lives in git, and that's where we should do something if you want to add a mention of Amy for those lines of code.

How about doing something as simple as switching a if/else statement (or any other kind of no-op change) and having Amy as author/co-author?

@LysandreJik see if it's okay now.

Let's see if this works, thanks a lot!
is it possible to split the embedding layers from bert during fine tuning and inference
Author: zhufeijuanjuan
Pull request: no

in my bert model, parameters in embedding layers are huge since the vocabulary size is very big. So I plan to split the embedding layer from the entire model and use it as a independent module to calculate word embeddings. Is it possible to do like this? thanks


Comments:
Hello, 

You can get the embeddings layer with model.embeddings as per 
https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/models/bert/modeling_bert.py#L848

```
print (model.embeddings)
#BertEmbeddings(
  (word_embeddings): Embedding(30522, 768, padding_idx=0)
  (position_embeddings): Embedding(512, 768)
  (token_type_embeddings): Embedding(2, 768)
  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
```

Bonus: Could also just extract the embedding output as follows without splitting, I'm not sure why you are splitting?
https://github.com/huggingface/transformers/issues/1827#issuecomment-554028292



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.