noamaanMulla-03/transformers-issues huggingface dataset:
Adding TensorFlow port of LeViT
Author: ariG23498
Pull request: no
Labels: name: WIP

### Feature request

To add the TensorFlow port of the [LeViT](https://arxiv.org/abs/2104.01136) architecture. The architecture is currently present in the Transformers library in [PyTorch](https://github.com/huggingface/transformers/blob/main/src/transformers/models/levit/modeling_levit.py). 

### Motivation

[LeViT](https://arxiv.org/abs/2104.01136) is a family of architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. The TensorFlow port would be an addition to the hybrid architecutre families.

### Your contribution

I would like to make the contribution by building out the TensorFlow port.

Tagging: @amyeroberts who could assign me to the task of adding the TensorFlow port of the model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am still working on this.
(TF) model.generate with beam search to an exportable tf.function
Author: piEsposito
Pull request: no

### Feature request

It would be good if we could export the `tf.function` `model.generate` with beam search, so it is servable with TF Serving. 

### Motivation

When we try to export a `model.generate` with `do_sample=True` to a `tf.function` and save it, we get an error due to Transformers and Tensorflow Internals. It would be good if, such as what is done in #18372, we could do for beam search:

```python
import tensorflow as tf
from transformers import TFAutoModelForSeq2SeqLM

class MyOwnModel(tf.Module):
    def __init__(self, model_path="t5-small"):
        super(MyOwnModel, self).__init__()
        self.model = TFAutoModelForSeq2SeqLM.from_pretrained(model_path)

    @tf.function(input_signature=(tf.TensorSpec((None, 32), tf.int32, name="input_ids"),
                                  tf.TensorSpec((None, 32), tf.int32, name="attention_mask")), jit_compile=True)
    def serving(self, input_ids, attention_mask):
        outputs = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, max_new_tokens=32,
                                   return_dict_in_generate=True, do_sample=True)
        return {"sequences": outputs["sequences"]}

model = MyOwnModel()
export_dir = "./t5-example"

tf.saved_model.save(
    model,
    export_dir,
    signatures={
        "serving_default":
            model.serving
    })
```

That way, we would be able to serve TF text to text models with beam search in TF Serving, which would be a blast. Otherwise we would have to implement our own servers which is not ideal.

### Your contribution

I can try submitting a PR, but would be glad if I could get some guidance because this one seems harsh. It seems like we would have to remove lots of `shape` related operations. 

Comments:
cc @gante 

Hi @piEsposito 👋 Thanks for raising this issue -- there were indeed two related problems, both fixed in #19773 :)

@gante you always bring good news when I'm here to bother you folks from Hugging Face. Thanks!
TF: sample generation compatible with XLA and dynamic batch sizes
Author: gante
Pull request: yes

# What does this PR do?

Fixes #19747 (sample generation failing with XLA and dynamic batch sizes)

There were two distinct problems:
- `_expand_inputs_for_generation` was crashing when the input was a `TensorSpec` with dynamic batch size (used e.g. with tf.Serving). Fix: `tensor.shape` -> `tf.shape`
- Obtaining a new seed with `tf.random.Generator` is broken with `tf.function` for two distinct reasons: 1) the protection we added [here](https://github.com/huggingface/transformers/pull/18044/files) makes unseeded `sample` crash unless we call `sample` with eager execution first (see [here](https://www.tensorflow.org/api_docs/python/tf/random/set_global_generator) why); 2) TF 2.10 + XLA crashes on the line that instantiates it. Fix: replace TF-suggested method to generate seed with `tf.random.Generator` with `tf.experimental.numpy.random.randint`, since we were creating a Generator from a non-deterministic state anyways 🤷 Notes: the new `tf.experimental.numpy.random.randint` strategy seems to be slightly faster (23.5 ms -> 21.6 ms per generate call on a benchmark with XLA, nvidia 3090,  and `t5-small`) and is retrocompatible up to TF 2.4.0.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Install tf2onnx dev version
Author: ydshieh
Pull request: yes

# What does this PR do?

A temporary fix for the error on CI: `ValueError: from_keras requires input_signature`.

Reference: [this commit](https://github.com/onnx/tensorflow-onnx/commit/ddca3a5eb2d912f20fe7e0568dd1a3013aee9fa3)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@LysandreJik I guess ready to merge :-) ?
Fix docker image build
Author: ydshieh
Pull request: yes

# What does this PR do?

The docker image build fails (for deepspeed-releated docker file) since #19532. It seems `nvcr.io/nvidia` is somehow different, and we can't build 2 images in a single GH job.

Current failed run
https://github.com/huggingface/transformers/actions/runs/3278157954

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Not at all an expert on docker (quite the opposite :sweat_smile: ) so will let @LysandreJik comment on this one!
Add zero-shot object detection pipeline
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

### Feature request

We currently have [OWL-ViT](https://huggingface.co/docs/transformers/main/model_doc/owlvit) in the library, which is capable of performing zero-shot object detection.

It would be great to have a pipeline for this task, with the following API:

```
from transformers import pipeline

pipe = pipeline("zero-shot-object-detection")
pipe("cats.png", ["cat", "remote"])
```

This pipeline could default to the https://huggingface.co/google/owlvit-base-patch32 checkpoint. Also check out the [demo notebook](https://github.com/huggingface/notebooks/blob/main/examples/zeroshot_object_detection_with_owlvit.ipynb) that showcases the model.

This can be implemented similar to [other pipelines](https://github.com/huggingface/transformers/tree/main/src/transformers/pipelines) (we already have one for zero-shot image classificaiton with CLIP, so it would be very similar to that one). For an example PR that added a pipeline, see https://github.com/huggingface/transformers/pull/11598.

### Motivation

Pipelines are great for abstracting away all the complexity for quick inference with a model.

### Your contribution

I can assist with this, together with @Narsil.

Comments:
As seen with @alaradirik this morning, this could also be leveraging the custom pipeline feature that was implemented last week, especially if this pipeline works with a very limited number of artchitectures.

cf https://github.com/huggingface/transformers/pull/18079

cc @alaradirik 

Can I take this up and work on it?

Hi @MocktaiLEngineer! Of course, you can also @NielsRogge, @Narsil or me if you need any help or have any questions.

cc @sgugger as we chatted about it as well 

I was going to do a custom pipeline on this today actually, as the dev advocates want more examples of it :-)

Hi @NielsRogge , if no one is working on it, can i take this up?

Hi @sahamrit, I don't think anyone is working on this right now but I'd need to double check with @NielsRogge and @sgugger 

Yes you can take a stab at it. Pinging @OlivierDehaene that might be able to provide guidance too.
Swin2sr
Author: venkat-natchi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #19568


## Who can review?
@NielsRogge 



Comments:
WIP. 

Found the equivalents for PatchEmb, PatchMerging, SwinV2Stage

Hi @venkat-natchi,

I actually played around with the Swin2SR model this weekend and got a [working implementation](https://github.com/NielsRogge/transformers/tree/add_swin2sr/src/transformers/models/swin2sr) already.

I see you're still at the start of the process, so would you be interested in working on another model [from this list](https://github.com/huggingface/transformers/issues?q=is%3Aissue+is%3Aopen+label%3A%22New+model%22)?



Sure, make sense. I was actually going through the papers in the weekend to get the high level understanding of it. 

I thought of picking [this](https://github.com/huggingface/transformers/issues/19631) one up now, but seems to be closed. Do you know why?

If possible could you point me to any one model, which was not taken up so far. Thanks. 




Do you have an email address? I'll set up a Slack channel so we can discuss :)

sure. Mine is venkatachalam.natchiappan@gmail.com

Hey, @venkat-natchi, would you like to collaborate on Adding EDSR to HuggingFace? I closed that issue because I felt it was irrelevant.

> Hey, @venkat-natchi, would you like to collaborate on Adding EDSR to HuggingFace? I closed that issue because I felt it was irrelevant.

Sure, I am interested. 

That's great! then i will reopen that issue and tag u there

I'll close this PR as you'll work on a new model.
PT <-> TF for composite models
Author: ydshieh
Pull request: yes

# What does this PR do?

Make `from_pretrained` (cross loading PT/TF) available for composite models. Ensure `enc_to_dec_proj` layer is correctly loaded.

With this PR, we can get the PT/TF same output for the test in #19719

I will apply the same change to Text and Speech composite models if the current version is approved.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
add doc for hyperparameter search
Author: sywangyi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
add doc for hpo

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @sgugger



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@yao-matrix
fix HPO DDP GPU problem
Author: sywangyi
Pull request: yes

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
https://github.com/huggingface/transformers/issues/18609

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @sgugger

Comments:
@sgugger @@spigo900 please try with PR, it works for me to do HPO DDP with GPU

@yao-matrix

_The documentation is not available anymore as the PR was closed or merged._
oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with …
Author: sywangyi
Pull request: yes

…PyTorch 1.12.1

raise error in this condition and update the doc

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

Fixes # (issue)
torch 1.12.1 does not work with intel ccl 1.12.0

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger


Comments:
@yao-matrix @sgugger please notice the issue and help review

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19132). All of your documentation changes will be reflected on that endpoint.

@sgugger I agree with your point, so I upload another PR (https://github.com/huggingface/transformers/pull/19151) to only update the doc.

we will release a new oneCCL 1.12.1 to work with torch 1.12.1
HPO: keep the original logic if there's only one process, pass the tr…
Author: sywangyi
Pull request: yes

…ial to trainer

     need to find out solution for following cases
     *if we need to use trial in model_init, how to do it for non-main rank, sync the model with rank0 in app?
     *how to use optuna prune feature for DDP, if we do it in rank0, how does other rank know it.

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @sgugger


Comments:
@yao-matrix @sgugger please review the patch.

_The documentation is not available anymore as the PR was closed or merged._

 need to find out solution for following cases
 *if we need to use trial in model_init, how to do it for non-main rank, sync the model with rank0 in app?
 *how to use optuna prune feature for DDP, if we do it in rank0, how does other rank know it.
add DDP HPO support for sigopt. only main_process will have HPO, and …
Author: sywangyi
Pull request: yes

…pass argument to other process

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
HPO do not support DDP now. add support in sigopt backend

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @sgugger


Comments:
@sgugger @yao-matrix. please help to review

_The documentation is not available anymore as the PR was closed or merged._

@sgugger thanks for the review. the torch.distributed.broadcast_object_list only support list(each element is pickable) input, does not support dict class
add opt "flush_denormal" in training_args.
Author: sywangyi
Pull request: yes

To solve the low performance issue caused by denormal numbers, user could enable this opt

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?
[Denormal number](https://en.wikipedia.org/wiki/Denormal_number) is used to store extremely small numbers which are close to 0. Computations with denormal numbers are remarkably slower than normalized number. To solve the low performance issue caused by denormal numbers, users can use the following opt "flush_denormal"

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger, please help review it


Comments:
@yao-matrix @sgugger please help review

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18574). All of your documentation changes will be reflected on that endpoint.

Thanks for your PR but I don't completely understand why this should leave in the Trainer? Users can just add `torch.set_flush_denormal(True)` to their script.

Hi, I think not all the data scientists are familiar with the option, if we add it to the training arg. at least they could see it from --help  and get the point. @sgugger

> Thanks for your PR but I don't completely understand why this should leave in the Trainer? Users can just add `torch.set_flush_denormal(True)` to their script.



Maybe it could be documented somewhere in this case, but I doubt adding yet another training argument is going to make it any more visible. There are currently 90 of them.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.