Could swin-tiny-patch4-window7-224 be traced by using torch.jit.trace? 
Author: heylamourding
Pull request: no

### System Info

torch version: 1.12.0+cu102
transformers version: 4.18.0 
model: microsoft/swin-tiny-patch4-window7-224

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


Hi team, I want to traced the swin model. 

My first try is directly tracing pretrained model. 
```
from transformers import SwinModel, SwinConfig
import types
model = SwinModel.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
model.eval()

x = torch.randn(1,3,224,224)
if not hasattr(model, 'forward_'): model.forward_ = model.forward
# change the forward to make it traceable
model.forward = types.MethodType(lambda self,x: self.forward_(x).last_hidden_state, model)
traced = torch.jit.trace(model, x)
```
Error shows 

> 
> ---------------------------------------------------------------------------
> TracingCheckError                         Traceback (most recent call last)
> <ipython-input-55-622b7b3243c5> in <module>
>       8 # change the forward to make it traceable
>       9 model.forward = types.MethodType(lambda self,x: self.forward_(x).last_hidden_state, model)
> ---> 10 traced = torch.jit.trace(model, x)
>      11 # try:
>      12 #     traced = torch.jit.trace(model, x)
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
>     748 
>     749     if isinstance(func, torch.nn.Module):
> --> 750         return trace_module(
>     751             func,
>     752             {"forward": example_inputs},
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
>     990                     )
>     991                 else:
> --> 992                     _check_trace(
>     993                         [inputs],
>     994                         func,
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
>      25         def decorate_context(*args, **kwargs):
> ...
> 		- %pooler : __torch__.torch.nn.modules.pooling.___torch_mangle_6083.AdaptiveAvgPool1d = prim::GetAttr[name="pooler"](%self.1)
> 		?                                                               ^ -
> 		+ %pooler : __torch__.torch.nn.modules.pooling.___torch_mangle_6338.AdaptiveAvgPool1d = prim::GetAttr[name="pooler"](%self.1)
> 		?                                                               ^^

Then, I tried to disable pooling layer by directly declare raw swin structure. 

```
configuration = SwinConfig()
configuration.patch_norm = False
model = SwinModel(configuration, add_pooling_layer = False, use_mask_token = False)   
model.eval()

x = torch.randn(1,3,224,224)
if not hasattr(model, 'forward_'): model.forward_ = model.forward
# change the forward to make it traceable
model.forward = types.MethodType(lambda self,x: self.forward_(x).last_hidden_state, model)
```

Error shows 

> 
> ---------------------------------------------------------------------------
> TracingCheckError                         Traceback (most recent call last)
> <ipython-input-56-c1e3f68ee2a2> in <module>
>      13 # torch.jit.save(traced, f'model/inferentia/trial3_traced.pt')
>      14 
> ---> 15 traced = torch.jit.trace(model, x)
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
>     748 
>     749     if isinstance(func, torch.nn.Module):
> --> 750         return trace_module(
>     751             func,
>     752             {"forward": example_inputs},
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
>     990                     )
>     991                 else:
> --> 992                     _check_trace(
>     993                         [inputs],
>     994                         func,
> 
> ~/miniconda3/envs/fsic_easy/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
>      25         def decorate_context(*args, **kwargs):
>      26             with self.clone():
> ---> 27                 return func(*args, **kwargs)
> ...
> 		- %layernorm : __torch__.torch.nn.modules.normalization.___torch_mangle_6592.LayerNorm = prim::GetAttr[name="layernorm"](%self.1)
> 		?                                                                        ^^^
> 		+ %layernorm : __torch__.torch.nn.modules.normalization.___torch_mangle_6846.LayerNorm = prim::GetAttr[name="layernorm"](%self.1)
> 		?                                                                        ^^^
> 

May I know is it possible to trace the model? 

### Expected behavior

Expect swin model can be traced. The traced model can be tried for aws-neuron. ..

Comments:
cc @NielsRogge 

Normally it should work, see https://github.com/huggingface/transformers/issues/17476 for details

Hi @NielsRogge , thanks. I tried the approach mentioned in https://github.com/huggingface/transformers/issues/17476. It works. 

```
from transformers import SwinModel, SwinConfig
import types
model = SwinModel.from_pretrained("microsoft/swin-tiny-patch4-window7-224")
model.eval()
if not hasattr(model, 'forward_'): model.forward_ = model.forward
model.forward = types.MethodType(lambda self,x: self.forward_(x).last_hidden_state, model)
x = torch.randn(1,3,224,224)
traced = torch.jit.trace(model, x, check_trace = False) 
```

However, I tried to convert traced model into neuron format and deploied to inferentia. 
 I followed the [tutorial](https://huggingface.co/docs/transformers/main/en/torchscript) and ran code below: 
```
torch.neuron.trace(model, x, strict = False)
```
It showed error below. **_May I know is SwinT convertible in neuron format?_** 
```
INFO:Neuron:There are 23 ops of 3 different types in the TorchScript that are not compiled by neuron-cc: aten::adaptive_avg_pool1d, aten::index, aten::roll, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 1813, fused = 1135, percent fused = 62.6%
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$2347; falling back to native python function call
ERROR:Neuron:torch.jit.trace error. The PyTorch-Neuron trace Python API uses the
torch.jit.trace function in PyTorch to generate ScriptFunction models for execution
on Inferentia. Due to this, your exisiting PyTorch model must be torch jit traceable.
```
 

Hello @heylamourding,

it seems like that Inferentia1 (`neuron-sdk`) is missing support for some operators for the `SWIN` model, that's not a transformers issue more a neuron-sdk issue. 
i saw you already opened an issue in the `neuron-sdk` repository: https://github.com/aws-neuron/aws-neuron-sdk/issues/626



Hi @philschmid, thanks for your reply! Sorry for creating inappropriate issue here. I will close the issue.