noamaanMulla-03/transformers-issues huggingface dataset:
The `convert_tokens_to_string` method fails when the added tokens include a space
Author: SaulLu
Pull request: no

## Environment info
```
- `transformers` version: 4.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
```

## Information

The bug that was identified [for the decode method regarding added tokens with space ](https://github.com/huggingface/transformers/issues/1133)also occurs with the `convert_tokens_to_string` method.

## To reproduce

```python
from transformers import AutoTokenizer, AddedToken

def print_tokenizer_result(text, tokenizer):
    text = tokenizer.convert_tokens_to_string(tokenizer.tokenize(text))
    print(f"convert_tokens_to_string method: {text}")

model_name = "patrickvonplaten/norwegian-roberta-base"

tokenizer_init = AutoTokenizer.from_pretrained(model_name)
tokenizer_init.save_pretrained("local_tokenizer")

model_name = "local_tokenizer"
tokenizer_s = AutoTokenizer.from_pretrained(model_name, use_fast=False)
tokenizer_f = AutoTokenizer.from_pretrained(model_name, use_fast=True)

new_token = "token with space"
tokenizer_s.add_tokens(AddedToken(new_token, lstrip=True))
tokenizer_f.add_tokens(AddedToken(new_token, lstrip=True))

text = "Example with token with space"
print("Output for the fast:")
print_tokenizer_result(text, tokenizer_f)
print("\nOutput for the slow:")
print_tokenizer_result(text, tokenizer_s)
```

## Expected behavior

No error or the fact that `convert_tokens_to_string` is a private method.


Comments:
One way to solve this problem is to move the logic implemented in the `_decode` method into the `convert_tokens_to_string` method. 
https://github.com/huggingface/transformers/blob/9d88be57785dccfb1ce104a1226552cd216b726e/src/transformers/tokenization_utils.py#L933-L946

The only issue will be that we'll need to add new argument (`skip_special_tokens`, `clean_up_tokenization_spaces`, `spaces_between_special_tokens`) to the `convert_tokens_to_string` method of the slow tokenizers.

Another option is to make the method `convert_tokens_to_string`  private.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
typo IBERT in `__repr__`
Author: scratchmex
Pull request: no

should be `quant_mode: {self.quant_mode}` in here

https://github.com/huggingface/transformers/blob/71e602725b90f63f404109bae9f72cbdf755477b/src/transformers/models/ibert/quant_modules.py#L150-L155

Comments:
Missing torch.no_grad in run_xxx_no_trainer.py
Author: JunnYu
Pull request: no
Labels: name: bug

### System Info

```shell
None
```


### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

missing `with torch.no_grad():` in some `run_xxx_no_trainner.py` file.


### Expected behavior

```shell
add `with torch.no_grad():`.
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix ViTMAEModelTester
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `ViTMAE` test has `(TF)ViTMAEModelTester` not setting `decoder` configs, like `decoder_hidden_size`, `decoder_intermediate_size` etc.

For the model class `(TF)ViTMAEForPretraining`, the default values for these attributes in `ViTMAEConfig` are used, and cause some tests being slow and consuming quite a lot of memory.

This PR fixes it.

(This might be `one` of the reasons where sometimes CI tests get `node down: Not properly terminated` -> `replacing crashed worker gw7` -> hang forever -> timeout)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Spanish docs - Fix nits and wording
Author: omarespejel
Pull request: yes

# What does this PR do?
Fix wording and nits in the merged Spanish documentation.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
@sgugger links to current docs that don't exist (eg `./main_classes/pipelines`) don't work. Do you think it would be worth linking them to the English docs while the Spanish versions become available? Or should we wait?

fyi @osanseviero 

_The documentation is not available anymore as the PR was closed or merged._

I had been told that links to page that don't exist in one language would be automatically resolved to the English version, cc @mishig25 

Currently, the links lead to an error if the doc is not yet translated. For example, this fragment in [`autoclass_tutorial`](https://huggingface.co/docs/transformers/main/es/autoclass_tutorial) leads a to an error for not having [model_doc/auto.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/auto.mdx)
 translated yet.
> Finalmente, las clases AutoModelFor te permiten cargar un modelo preentrenado para una tarea dada (revisa [aquí](https://huggingface.co/docs/transformers/main/es/model_doc/auto) para conocer la lista completa de tareas disponibles).

Please let me know if any help is required. Meanwhile, I will summon more community for the translation.

This PR fixes some wording and format in the Spanish docs. It would be ready to merge IMO.

I opened issue #17461 regarding the problem with the links.

Thanks for fixing!
Disk offload fix
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the disk offloading for pretrained models (requires latest accelerate main branch) and adds a test. The test passes locally for GPT-2, GPT-J, OPT and T5 (the only model where it's activated right now).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix no_trainer examples to properly calculate the number of samples
Author: muellerzr
Pull request: yes
Labels: name: PyTorch

name: Examples

# Fix number of samples for `no_trainer` scripts

## What does this add?

This PR fixes all of the no_trainer scripts to properly use the right number of training steps after the length of the dataloader was changed with `accelerator.prepare`

## Why is it needed?

Currently in a multi-process setup, the progress bar still shows the old number of samples. As a result the old number of steps before breaking is set at the original amount, even though the length of the dataloaders changed. The progress bar reflects this too.

Simplified example:

If the dataloader starts with 128 batches, if 2 GPUs are used then each dataloader has 64 batches. As a result the progress bar should use `64`, and the break condition needs to also know there is only 64. Both currently use 128 still

## What parts of the API does this impact?

### User-facing:

All scripts have a recalculation of the max_train_steps after `accelerate.prepare`

## Basic Usage Example(s):

```python
    # Prepare everything with our `accelerator`.
    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
    )

    # We need to recalculate our total training steps
    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
    args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
```

## When would I use it, and when wouldn't I?

While this is always used, technically it is only needed when the number of nodes > 1. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @muellerzr, @sgugger, in case I specify the argument `max_train_steps` instead of `num_train_epochs` while launching the training script, I need to recalculate the `num_train_epochs` after `accelerate.prepare` instead of `max_train_steps` right? Am I missing something?

@kowndinya-renduchintala we already do this for you 😄 

https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py#L425
Docker image build in parallel
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove `needs` in `.github/workflows/build-docker-images.yml`, as it can run in parallel now.

See this [run page](https://github.com/huggingface/transformers/actions/runs/2389099113) v.s. the [previous run page](https://github.com/huggingface/transformers/actions/runs/2388105533), with 14 mins. v.s. 40 mins.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

From the job run page, I saw

```
#18 exporting to image
#18 pushing layers 35.6s done
#18 pushing manifest for docker.io/huggingface/transformers-all-latest-gpu:latest@sha256:d8523684a112bff61a2899a69e06e05e26c507778df4754454b95c3dcf244012
#18 pushing manifest for docker.io/huggingface/transformers-all-latest-gpu:latest@sha256:d8523684a112bff61a2899a69e06e05e26c507778df4754454b95c3dcf244012 0.3s done
#18 DONE 289.9s
ImageID
  sha256:08ed1b5cc8db313f116b58d86292b2e109f0552737088fba4a5c672012bca3ae
Digest
  sha256:d8523684a112bff61a2899a69e06e05e26c507778df4754454b95c3dcf244012
```
so it looks fine to me. But I will run it again and verify the images on docker hub to make sure!
Fix checkpoint name
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix some `eleutherai/gpt-neox-20b` to `EleutherAI/gpt-neox-20b`.

The uppercase vs. lowercase in the checkpoint name matters, otherwise I got error like

`is not a local folder and is not a valid model identifier listed on`

Comments:
cc @zphang  :-)

_The documentation is not available anymore as the PR was closed or merged._
Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()`
Author: Witiko
Pull request: yes

# What does this PR do?

For tokenizers with added tokens that only consist of a small number of special tokens or any number of special tokens with consecutive token IDs, this pull request reduces the time complexity of creating the trie from quadratic to linear. This can easily reduce the time it takes to load a pretrained tokenizer with many added tokens from tens of minutes to seconds.

Fixes #16936.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@SaulLu @Narsil


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@Narsil Thank you for your review! A better fix would be if we could just accumulate all special and non-special tokens and then call `tokenizer.add_tokens()` once or twice at the end. However, since we need to maintain the order of tokens in the vocab, we need to call `tokenizer.add_tokens()` each time we encounter a token of a different type.

For malicious input, where special and non-special tokens are interleaved, my approach would lead to no speed-up compared to the current code but no slow-down either. For the typical case, where special and non-special tokens form two contiguous regions of identifiers, my approach leads to half the optimum speed and has only small diff against the current code.

I agree with @SaulLu's review - I'll let @Narsil take a final look and merge if he approves :) 

Thank you for the PR, @Witiko!

@LysandreJik: Thank you for your review. In https://github.com/huggingface/transformers/issues/16936#issuecomment-1127621789, @Narsil indicated that they don't think they can do a proper review of this PR due to lack of time and familiarity with the code base.

@Witiko , I think everyone now agrees that this PR is a good step.

I'll let @LysandreJik decide when to merge. 


This pull request is now linked from [the witiko/mathberta model in the Model Hub][1], which is adversely affected by #16936. This will allow users to manually apply the fix before it has been merged and released, so that they can use the model.

 [1]: https://huggingface.co/witiko/mathberta#how-to-use
Name of LayerNorm parameter in RobertaLMHead.
Author: silencio94
Pull request: no

As you know, current training codes using `weight decay`, usually detect target parameters based on their names.

e.g.)
```javascript
no_decay = ["bias", "LayerNorm.weight"]
optimizer_grouped_parameters = [
    {"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
     "weight_decay": args.weight_decay},
    {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
     "weight_decay": 0.0},
    ]
```

In the code below 
,https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/models/roberta/modeling_roberta.py#L696,
Layernorm module defined as "layer_norm". Hence "layer_norm" naming, weight-decay will be applied to RobertaLMHead.layer_norm, different from intent.

```javascript
class RobertaLMHead(nn.Module):
    """Roberta Head for masked language modeling."""

    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

        self.decoder = nn.Linear(config.hidden_size, config.vocab_size)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        self.decoder.bias = self.bias

    def forward(self, features, **kwargs):
        x = self.dense(features)
        x = gelu(x)
        x = self.layer_norm(x)

        # project back to size of vocabulary with bias
        x = self.decoder(x)

        return x

    def _tie_weights(self):
        # To tie those two weights if they get disconnected (on TPU or when the bias is resized)
        self.bias = self.decoder.bias
```

Is there any special reason? 
I don't think it will cause a significant impact in training model, but I wonder what intention it was.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Source link of transformers.pipeline is broken
Author: ftnext
Pull request: no

This is about **documentation link error**.

There is a issue on the page https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.pipeline

<img width="550" alt="image" src="https://user-images.githubusercontent.com/21273221/164890499-54a79a1a-12c7-4776-bcf7-c39bf6f3efc1.png">

**Procedure**
Click the link of `<source>`, then go to https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/pipelines.py#L372 but see 404 Page not found on GitHub.

**Correct link**
I found that the correct link is https://github.com/huggingface/transformers/blob/v4.18.0/src/transformers/pipelines/__init__.py#L372

---

It seems that the content is generated by

https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/docs/source/en/main_classes/pipelines.mdx#L123

Comments:
cc @sgugger @mishig25 

`doc-builder` doesn't like objects defined in `__init__`, will send a fix.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error building docs locally:  No matching distribution found for ray[tune]; extra == "docs"
Author: venkatasg
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.17.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.0
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@sgugger @stevhl

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

On my system, when I try to install the packages necessary to build documentation using `pip install -e "[.docs]"`, I get the following error:

```
ERROR: Could not find a version that satisfies the requirement ray[tune]; extra == "docs" (from transformers[docs]) (from versions: none)
ERROR: No matching distribution found for ray[tune]; extra == "docs"
```

I'm trying to build the documentation locally as I'm a contributor to the [Dash user contributed datasets](https://github.com/Kapeli/Dash-User-Contributions#contribute-a-new-docset).

I'm developing on an Apple silicon Mac, but the conda environment is setup as a `x86_64` environment.


### Expected behavior

Documentation can be built and viewed locally.



Comments:
Not sure why you're opening an issue here as it seems you'd like `ray` to have a package installable on your end :-).

I can't guarantee the doc will build without all integrations, but it might work to build the doc without Ray installed.

I'm going to close the issue now, since I've found that using `python=3.7` fixed the issue with install `ray`. However, I seem to be running into more problems now. Firstly, it looks like the module `black` is not listed as a dependency, I got an error that `doc_builder` requires `black` to work. However, even after installing `black`, I'm running into a new error:

```
fish: 'doc-builder build transformers…' terminated by signal SIGILL (Illegal instruction)
```

I'm not sure what's causing this error -- I might open a new issue if I find that its something related to this repo rather than my setup.

EDIT: The issue seems to be more general. After installing the dependencies and `doc-builder`, even trying to import the `transformers` package exits with the SIGILL error.
Make all configs nicely readable
Author: patrickvonplaten
Pull request: no
Labels: name: Good Second Issue

### Feature request

All configs nicely readable (tokenizers & feature extractor)

### Motivation

https://huggingface.co/facebook/opt-30b/discussions/1

### Your contribution

Happy to do it early next week - happy if someone else wants to take it over though! 

Comments:
in strong favor of this!

PR: https://github.com/huggingface/transformers/pull/17457
Opt in flax and tf
Author: ArthurZucker
Pull request: yes
Labels: name: Core: Modeling

# What does this PR do?

Adds support for OPT in both Flax and TF

## Who can review?

@patrickvonplaten, @LysandreJik @younesbelkada @patil-suraj @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Should we close the other PR? Let me know once it's ready for a review :-)

Superseeds https://github.com/huggingface/transformers/pull/17227 and https://github.com/huggingface/transformers/pull/17226

Cool, very nice job @ArthurZucker ! 

Could you as a final safety guard also add TFOPT and FlaxOPT to the documentation test suite? 

See: https://github.com/huggingface/transformers/tree/main/docs#docstring-testing

Can I merge @LysandreJik @sgugger ? (failing test are not related to OPT) 

@patil-suraj could you quickly check Flax and maybe @gante go over TF OPT?

Thanks all for the reviews 😄 🥳
Setup for Italian translation and add quicktour.mdx translation
Author: mfumanelli
Pull request: yes

I created the folder for files translated into Italian and made the translation of quicktour.mdx and index.mdx files.

The folder (called 'it') is located in transformers/docs/source/.
The translated documents are located in the transformers/docs/source/it folder.

Fixes [#17459](https://github.com/huggingface/transformers/issues/17459)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

@omarespejel @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This looks great! Thanks @mfumanelli for the PR and opening the venue for an Italian 🇮🇹 translation!

Since this is the first IT translation could please add `it` to the languages section in the `.github/workflows` ([here](https://github.com/huggingface/transformers/blob/main/.github/workflows/build_documentation.yml) and [here](https://github.com/huggingface/transformers/blob/main/.github/workflows/build_pr_documentation.yml))?

Thank you @mfumanelli 🇮🇹! LGTM, @sgugger :)