noamaanMulla-03/transformers-issues huggingface dataset:
Improve and fix ImageSegmentationPipeline
Author: alaradirik
Pull request: yes

# What does this PR do?
- Fixes the image segmentation pipeline test failures caused by changes to the postprocessing methods of supported models
- Updates the ImageSegmentationPipeline tests
- Improves docs, adds 'task' argument to optionally perform semantic, instance or panoptic segmentation

Note: `test_small_model_pt` test is skipped due to a random weight initialization error when loading the `hf-internal-testing/tiny-detr-mobilenetsv3-panoptic` model.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ X] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts @sgugger thank you for the review! All comments are addressed, I'll merge the branch once all tests are passing.

> Thanks for making these changes and improving the pipeline ⭐
> 
> I think the PR is good to go as is 👍 Would just like to see a bit more test coverage of the different segmentation tasks. Just one per task in the pipeline, possibly adapting existing ones to avoid making the test suite significantly slower. Have you visualised or counted the number of different pixels between the outputs on this branch and main as validation?

Thank you! Yes, I visualized the segments and they are the same.
Rework pipeline tests
Author: sgugger
Pull request: yes

# What does this PR do?

The test fetcher has been pretty good at identifying which tests to run and even if we're not testing everything on each commit anymore, we've mostly avoided bad surprises. 

Except for pipeline tests.

This is because the pipeline tests are structured in a way that makes it hard for the test fetcher to guess it has to run them, as they don't see to rely on anything else than the pipeline.

Also running pipeline tests is annoying as you have to remember to activate a special env variable. It made sense back in the days where we had all tests in one folder, but now that they are nicely structured, this is completely unnecessary.

Thus this PR proposes two things:
1. remove the special marker for pipeline tests and the corresponding env variable. We know they are all in `tests/pipelines`.
2. run all pipeline tests any time there is some code change warranting at least one test, like we do for the examples. They are taking the same time roughly, and since the pipelines are a good integration test, I think it actually makes more sense to test those all the times than the examples.

Comments:
Note: Flax tests are currently failing because it tries to run the tests inside the pipelines folder, will fix this tomorrow by having all non-pipeline jobs not run any of the tests in the pipelines folder. The `--ignore` flag from pytest does not work for some reason, but the test fetcher can probably fix that somehow ^^.

_The documentation is not available anymore as the PR was closed or merged._

I love this PR !

> Note: Flax tests are currently failing because it tries to run the tests inside the pipelines folder

Can't we make the tests parsable even when having neither PT nor TF ?

Here this:

`from transformers import DetrForSegmentation` seems to be the culprit (in `tests/pipelines/test_pipelines_for_segmentation.py`).

Shouldn't we have dummy models when `torch` is not available ?

> Can't we make the tests parsable even when having neither PT nor TF ?

I can certainly do that too, but the pipeline tests are isolated to not be run by the other jobs, so they shouldn't even be run.

> I can certainly do that too, but the pipeline tests are isolated to not be run by the other jobs, so they shouldn't even be run.

I'm thinking about regular user using JAX (or non libraries actually) doing `pytest -sv tests/` . IMO it'd be nice if the command ran instead of crashing.
Support for converting LayoutLM to ONNX
Author: nishprabhu
Pull request: no

# 🚀 Feature request

Transformers currently provides ready configurations for converting BERT, BART, RoBERTa and several other models to ONNX. Can we extend this to also support LayoutLM?

## Motivation

ONNX is quickly becoming the default runtime environment in many production settings. Ideally, all models supported by the library should have an easy path to conversion.

## Your contribution

I am willing to submit a PR that implements this.


Comments:
Sure that would be great. LayoutLM is literally only adding 4 additional embedding layers to BERT:

https://github.com/huggingface/transformers/blob/a3f96f366a49bbe2cbdeaebd2e32ccdc1260a1d6/src/transformers/models/layoutlm/modeling_layoutlm.py#L66-L69

So I guess it won't be that difficult to support?

cc @mfuntowicz 

The guide written here is very helpful: https://huggingface.co/transformers/serialization.html?highlight=onnx#converting-an-onnx-model-using-the-transformers-onnx-package

Thanks! It was very useful!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

dumb question - how do i format the ORT inputs for LayoutLM onnx? does anyone have an example of LayoutLM ONNX inference?

I'm trying to pass in the output of a collator into the onxx session. Its not liking the bounding box tensor since it its dimensions are different than input_ids, token_type_ids and attention_mask.
Big Bird cannot be converted to ONNX
Author: cigrainger
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.17.5-x86_64-with-glibc2.33
- Python version: 3.9.6
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ydshie

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```bash
python -m transformers.onnx --model=google/bigbird-roberta-base bigbird
```

Returns:
```bash
ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: 6.103515625e-05
```

### Expected behavior

I expect the export to work and return: `All good, model saved at: bigbird/model.onnx`.

Comments:
cc @vumichien, have you encountered this when contributing the BigBird ONNX exporter?

@LysandreJik I didn't encounter this problem when contributing the BigBird ONNX exporter. I do check again by running `RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -k "bigbird"` , and all the tests are still passed. 

Maybe we could put a higher tolerance in that case. WDYT @lewtun ?

Hey @cigrainger I'm not able to reproduce this behaviour using either CPU or GPU. Could you please try running this again with the latest state of the `main` branch in `transformers` and report back if the issue persists?

If yes, we can certainly increase the tolerance as Lysandre suggested :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Create TF port of BigBird
Author: E-Aho
Pull request: no
Labels: name: New model

### Model description

[BigBird](https://arxiv.org/abs/2007.14062) is an open source transformer model architecture for longer sequences, and is implemented in the Transformer library already in PyTorch and Flax, but not yet in TensorFlow. This issue tracks the implementation of a TensorFlow version of the model.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

[Location of current implementations](https://github.com/huggingface/transformers/tree/main/src/transformers/models/big_bird)

Comments:
Currently starting to work on this :)
Input_embeds for Albert
Author: lsyysl9711
Pull request: no

Hello! I am checking the source codes for Albert's inputs_embeds. But I think the open sourced codes do not match with the document:

In the document, the description for inputs_embeds says that the shape should be the of shape (batch_size, sequence_length, hidden_size).

But in the source codes(https://huggingface.co/transformers/v3.4.0/_modules/transformers/modeling_albert.html) we have the followings:

self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.embedding_size)
self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.embedding_size)
position_embeddings = self.position_embeddings(position_ids)
token_type_embeddings = self.token_type_embeddings(token_type_ids)
embeddings = inputs_embeds + position_embeddings + token_type_embeddings


So in order to make the final additions' operations valid, the shape between inputs_embeds and the other two should be the same. But the default embedding size is 128 while the default hidden size is 768("albert-base-v2"). So if we use the method described in document, the compiler will output errors.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding multiprocessing option to transformers.pipelines.automatic_speech_recognition 
Author: mehrzadai
Pull request: no

### Feature request

Hi, 
in `transformers.pipelines.automatic_speech_recognition` in case of `self.type = ctc_wit_lm`, the `postprocess` method uses `self.decoder.decode_beams(items)`. This is too slow and does decoding iterative.
`decoder.decode_beams_batch(pool,items)` is able to make things faster and pharallel.

### Motivation

`transformers.pipelines.automatic_speech_recognition` works really slow in complex `ctc_with_lm` scenarios.

### Your contribution

`None`

Comments:
WDYT @Narsil?

Hi, I don't mind having a parameter for that for sure.

The biggest reason I don't think it should be the defaults is that some users might already be using different processes for different pipelines so doing parallelism twice is usually hurtful.

Also, do you mind providing small benchmarks to see the performance improvement ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
wav2vec2 : No MSELoss implementation
Author: LaurenceYozi
Pull request: no
Labels: name: bug

### System Info

modeling_wav2vec2.py  : 1822 lines only implements CrossEntropyLoss()
  if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.config.num_labels), labels.view(-1))


labels (torch.LongTensor of shape (batch_size,), optional) — Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).
Returns


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model = AutoModelForAudioClassification.from_pretrained(
    "facebook/wav2vec2-base", num_labels=1)  

### Expected behavior

Calculate MSELoss

Comments:
Hey @LaurenceYozi

Indeed! The MSE loss is not implemented. Would you like to open a PR to add this? You can refer to BERT for the loss function/labels logic:
https://github.com/huggingface/transformers/blob/cfd623a859890c6d106610d3c688064eadc7bd61/src/transformers/models/bert/modeling_bert.py#L1578-L1598
You should be able to copy this almost one-for-one from BERT to Wav2Vec2!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `ViTMSNForImageClassification` doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

`>>> torch.manual_seed`  requires `# doctest: +IGNORE_RESULT`, otherwise we get 

```
Expected nothing
Got:
    <torch._C.Generator object at 0x7fe914fdbcd0>
```
I missed this detail when reviewing #19183

Confirmed it works now (on GCP CPU VM)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip `BloomEmbeddingTest.test_embeddings` for PT < 1.10
Author: ydshieh
Pull request: yes

# What does this PR do?

The test `BloomEmbeddingTest.test_embeddings` will load a pretrained-model in `bfloat16`. For PyTorch < 1.10, we have
```bash
RuntimeError: "LayerNormKernelImpl" not implemented for 'BFloat16'
```
so let's skip it to keep Past CI clean 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@LysandreJik For this case, it is specific to test (checkpoint is `bigscience/bigscience-small-testing` which has `bfloat16` in the config file).
Cannot Import BigBirdModel
Author: jaideep11061982
Pull request: no
Labels: name: bug

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 3.0.2
- Platform: Linux-5.10.133+-x86_64-with-debian-9.9
- Python version: 3.6.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
----> 2 from transformers import BigBirdTokenizer,BigBirdModel

ImportError: cannot import name 'BigBirdTokenizer'
```

```
----> 4 from transformers import (AlbertModel, AlbertTokenizer, BartModel, BigBirdModel, BigBirdTokenizer,
      5                           BartTokenizer, BertModel, BertTokenizer,
      6                           CamembertModel, CamembertTokenizer, CTRLModel,

ImportError: cannot import name 'BigBirdModel'
```

### Expected behavior

Model should get imported without the error

Comments:
I don't think transformers version 3.0.2 contains the BigBird model. I think updating your version of the transformers package should solve the issue.
I also see that you are using Python version 3.6.6. The latest version of the transformers package requires Python >=3.7.0, so I guess you would also need to update your installed Python version. 
Hope this helps!

Which version does


On Wed, 7 Sep, 2022, 1:43 pm Manish Sridhar, ***@***.***>
wrote:

> I don't think transformers version 3.0.2 contains the BigBird model. I
> think updating your version of the transformers package should solve the
> issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/18914#issuecomment-1239059464>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AJDFNZADFM4WMFVCK4LEDSLV5BFD5ANCNFSM6AAAAAAQGPXTJY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


@jaideep11061982 I believe the model was added in v4.5.0, but I think either @sgugger or @NielsRogge will be able to better comment on the exact version that you could use.

Yes, it was introduced in v4.5.0.

In general, please do not open issues without updating to some recent version of Transformers, v3.0.2 is more than two years old and bugs are fixed as continuous development of the new versions, so you need to upgrade to the latest releases to see the fixes anyway.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
OWL-ViT
Author: FrancescoSaverioZuppichini
Pull request: no

### Feature request

Dear all,

It looks like [OWL-ViT](https://huggingface.co/docs/transformers/model_doc/owlvit) doesn't support image conditioned detection.

### Motivation

 Image conditioned detection is the most appealing feature of this model

### Your contribution

No, just pointing this out. Thanks

Comments:
Hi Francesco, this feature was already asked and a PR to add this feature can be found here: #18891 

nice, thanks @NielsRogge
[WIP] Fix ImageSegmentationPipeline
Author: alaradirik
Pull request: yes

# What does this PR do?
- Fixes the ImageSegmentationPipeline test errors caused by a `[recent PR](https://github.com/huggingface/transformers/pull/19205)` that adds new post-processing methods to DETR. 

The following PRs will need to be merged first to ensure DETR and MaskFormer have consistent `post_process_panoptic_segmentation` and `post_process_semantic_segmentation outputs `:
- https://github.com/huggingface/transformers/pull/19172
- https://github.com/huggingface/transformers/pull/19262

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
     This issue fixes a pipeline error caused by recent changes to image segmentation post-processing methods of DETR.
     https://github.com/huggingface/transformers/pull/19248
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19264). All of your documentation changes will be reflected on that endpoint.

The issues addressed by this PR are fixed in this [PR](https://github.com/huggingface/transformers/pull/19367).
Add post_process_semantic_segmentation method to SegFormer
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds a post_process_semantic_segmentation method to `SegFormerFeatureExtractor` with optional resizing. This model doesn't support instance or panoptic segmentation. 

I will open separate PRs to make sure the naming and outputs of post_process methods of segmentation models are consistent.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@NielsRogge  FYI, I will also need to open a PR to edit `ImageSegmentationPipeline` after making sure all post-processing methods across segmentation models are consistent in terms of naming and functionality.

Hey @alaradirik, please also ping a core maintainer for review before merging PRs.
Fix DETR docs example, add post_process_object_detection to DETR docs
Author: alaradirik
Pull request: yes

# What does this PR do?
- Fixes DETR docs example
- Adds post_process_object_detection method to DETR docs


## Before submitting
- [ X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
I'm re-running the documentation build to check that the doc is built correctly, will merge afterwards :+1: 

_The documentation is not available anymore as the PR was closed or merged._