Faketensor issue when using torch inductor as backend with Trainer API
Author: YuchengT
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: Linux-5.15.0-1030-aws-x86_64-with-glibc2.10
- Python version: 3.8.0
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 2.0.0a0+git45d775c (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the error:

```
pip3 install numpy --pre torch --force-reinstall --index-url https://download.pytorch.org/whl/nightly/cu117
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .

cd examples/pytorch/image-classification
pip install -r requirements.txt
python run_image_classification.py \
    --dataset_name food101   --output_dir ./food101_outputs/ \
    --remove_unused_columns False     --do_train  --learning_rate 2e-5  \
    --num_train_epochs 1  --report_to none   --per_device_train_batch_size 1 \
    --logging_strategy steps     --logging_steps 10   --save_strategy epoch \
    --overwrite_output_dir  --torch_compile_backend inductor
```

### Expected behavior

In the forward pass we saw 

```
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 82, in __call__
    return self.dynamo_ctx(self._orig_mod.__call__)(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 215, in _fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 343, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object
    transformations(instructions, code_options)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1862, in run
    super().run()
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 619, in run
    and self.step()
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 583, in step
    getattr(self, inst.opname)(inst)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in STORE_ATTR
    self.output.compile_subgraph(
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 579, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 626, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 713, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(1, 3, 224, 224)), cuda:0), tensor([[[[ 1.5585e-02,  5.1153e-02,  5.5507e-02,  ...,  8.7095e-02,
            1.0724e-01,  1.1972e-01],
          [ 1.2528e-02,  1.3805e-02,  1.9514e-02,  ...,  8.3470e-02,
            3.8538e-02,  8.5673e-02],
          [ 1.6835e-02,  1.1395e-02, -4.2227e-03,  ...,  7.6267e-02,
            3.2913e-02,  3.5811e-02],
          ...,
          [-1.3910e-02, -5.2311e-03, -5.4728e-02,  ..., -5.9774e-02,
           -6.8372e-02,  1.9745e-02],
          [ 3.9740e-02, -4.4819e-02, -2.2511e-02,  ...,  1.0378e-02,
           -4.0710e-02,  8.1610e-02],
          [ 6.9672e-02,  8.1102e-02,  3.7697e-02,  ...,  9.5118e-02,
            1.1510e-01,  1.5316e-01]],

         [[-6.2993e-02, -3.8614e-02, -4.2429e-02,  ..., -5.9603e-02,
           -2.5599e-02, -2.1458e-02],
          [-8.3695e-03, -1.0892e-02, -8.3388e-03,  ..., -3.2603e-02,
           -4.5211e-02, -3.2238e-04],
          [ 4.6061e-02,  3.2819e-02,  1.3283e-02,  ..., -3.5023e-02,
           -3.9311e-02, -3.1661e-02],
          ...,
          [ 3.6763e-02,  2.0482e-02, -5.9279e-02,  ..., -2.3277e-02,
           -3.5168e-02,  4.1823e-02],
          [ 5.9616e-02, -4.6340e-02, -5.6928e-02,  ...,  2.1146e-02,
           -2.8068e-02,  8.1620e-02],
          [ 3.2009e-02,  1.2772e-02, -5.2345e-02,  ...,  4.5677e-02,
            6.9799e-02,  9.9707e-02]],

         [[-1.1234e-01, -3.7040e-02,  2.3741e-03,  ..., -2.5491e-02,
           -2.3041e-02, -4.9780e-02],
          [-8.1879e-02, -1.8625e-02,  2.3212e-02,  ...,  2.8488e-02,
           -8.9298e-03, -8.8660e-03],
          [-5.3666e-02, -8.0642e-03,  3.1249e-04,  ...,  2.4810e-02,
            1.9670e-03, -2.3820e-02],
          ...,
          [-2.9322e-02,  2.7237e-02,  7.6372e-03,  ..., -5.7884e-03,
           -4.6314e-02, -2.3849e-02],
          [-1.5772e-02, -6.9045e-02, -3.1001e-02,  ..., -1.5394e-02,
           -9.2838e-02, -1.6544e-02],
          [-6.0532e-02, -5.9463e-02, -1.1458e-01,  ..., -8.0818e-02,
           -5.7851e-02, -3.6628e-02]]],


        [[[-2.9805e-02,  2.8396e-02,  2.7912e-02,  ...,  4.0566e-03,
           -3.9771e-02, -3.0932e-02],
          [-3.8447e-02,  7.3079e-02,  8.8570e-02,  ...,  5.9645e-02,
            7.1729e-02, -2.3781e-02],
          [-6.7763e-02, -9.5788e-03,  2.2610e-02,  ...,  6.9609e-02,
            6.6766e-02,  2.1204e-02],
          ...,
          [-9.7874e-02,  3.7861e-02,  1.4769e-01,  ...,  4.2415e-02,
            2.4433e-02, -1.9694e-02],
          [-1.0486e-01,  9.1625e-02,  6.1902e-02,  ...,  5.4587e-02,
            6.1803e-02, -7.3962e-02],
          [-8.3595e-02, -3.1282e-02,  4.5430e-03,  ...,  1.0579e-01,
            9.3442e-03, -3.1741e-02]],

         [[-1.3991e-02,  3.7737e-02,  2.1331e-02,  ..., -5.3170e-04,
           -4.6416e-02, -3.0628e-02],
          [-2.4599e-02,  7.4020e-02,  8.8880e-02,  ...,  6.2086e-03,
            2.5795e-02, -5.7271e-02],
          [-3.8102e-02,  1.2179e-02,  4.9777e-02,  ..., -1.9636e-03,
           -7.1444e-03, -4.5585e-02],
          ...,
          [-1.0808e-01,  1.5686e-02,  1.1100e-01,  ...,  5.2311e-02,
            4.0869e-02,  2.2863e-02],
          [-9.2463e-02,  1.0301e-01,  7.3106e-02,  ...,  5.4267e-02,
            7.1780e-02, -5.0014e-02],
          [-5.0767e-02,  1.1884e-02,  5.5425e-02,  ...,  1.0279e-01,
            1.1630e-02, -2.9744e-02]],

         [[-1.2795e-02,  2.0818e-02, -1.5076e-02,  ...,  5.3417e-02,
            3.3919e-02,  8.3990e-02],
          [-3.6925e-02,  1.5391e-02, -2.9710e-03,  ...,  1.6262e-02,
            4.1795e-02,  2.0504e-02],
          [-6.3379e-02, -4.8387e-02, -3.3549e-02,  ...,  1.4104e-02,
            4.7213e-03,  1.5434e-02],
          ...,
          [-5.8642e-02,  1.2647e-02,  7.3773e-02,  ..., -3.3242e-02,
            2.6407e-03,  4.4312e-02],
          [-5.1496e-02,  1.1767e-01,  5.7953e-02,  ..., -2.6282e-02,
            3.3894e-02, -3.7081e-02],
          [-3.9964e-02,  3.5065e-02,  7.8275e-02,  ...,  2.7511e-02,
           -3.0828e-02, -4.1590e-02]]],


        [[[-1.5722e-02, -4.7355e-04, -1.0641e-02,  ...,  1.7975e-03,
            8.2088e-03,  2.2358e-03],
          [ 1.3006e-02,  2.2377e-02,  4.6318e-03,  ..., -8.6258e-03,
           -9.6003e-03, -1.8025e-02],
          [ 1.1088e-02,  2.8006e-02,  1.0182e-02,  ..., -1.2203e-02,
           -1.4415e-02, -2.4993e-02],
          ...,
          [ 1.8237e-02,  1.0154e-02,  4.7651e-03,  ..., -3.4567e-03,
            2.9223e-03,  1.2099e-02],
          [ 2.3695e-02,  5.8175e-03,  6.3596e-03,  ..., -5.2218e-03,
           -2.5360e-04,  2.2794e-02],
          [ 6.2578e-03, -4.3371e-03, -1.8502e-02,  ..., -1.2459e-02,
            5.3634e-03,  2.5850e-02]],

         [[-1.6234e-02,  8.7551e-03,  6.5956e-03,  ...,  3.4186e-02,
            4.4762e-02,  4.3195e-02],
          [-8.9584e-04,  2.0318e-02,  8.5710e-03,  ...,  1.4135e-02,
            1.1658e-02,  8.4339e-03],
          [-9.6351e-03,  2.2278e-02,  8.8168e-03,  ...,  3.5157e-03,
            3.5562e-03, -5.8602e-03],
          ...,
          [ 2.7858e-03,  2.6561e-03,  1.9480e-03,  ..., -2.0423e-03,
            1.7712e-03,  9.1982e-03],
          [-3.3212e-03, -5.4636e-03, -2.3775e-03,  ..., -4.5449e-03,
           -1.1336e-03,  1.2337e-02],
          [-2.9789e-02, -2.6257e-02, -2.9101e-02,  ..., -1.4381e-02,
           -8.4865e-04,  9.4145e-03]],

         [[-3.9820e-02, -2.2362e-03, -7.1783e-03,  ..., -6.4696e-03,
           -9.6364e-04, -8.4148e-03],
          [-1.7378e-02,  1.6933e-02,  7.9727e-03,  ..., -1.1679e-03,
           -6.7231e-03, -1.6826e-02],
          [-2.1956e-02,  1.7214e-02,  1.0109e-02,  ...,  5.0059e-03,
           -2.2617e-03, -1.9757e-02],
          ...,
          [ 1.0005e-02,  1.8804e-02,  1.7285e-02,  ...,  1.2554e-02,
            8.4794e-03, -1.2349e-03],
          [ 3.7699e-03,  1.2804e-02,  2.3074e-02,  ...,  8.5308e-03,
            4.7621e-03,  3.3861e-03],
          [-1.5654e-02,  1.4163e-03,  1.6649e-03,  ...,  1.0598e-03,
            5.9754e-03,  1.9494e-03]]],


        ...,


        [[[-4.0359e-02,  1.7736e-02,  6.0642e-02,  ..., -3.0918e-02,
           -4.0575e-02,  8.9583e-03],
          [-4.5315e-02,  2.3188e-02,  7.3064e-02,  ..., -4.0640e-02,
           -3.9918e-02, -1.0623e-02],
          [-2.8732e-02,  3.8204e-03,  6.9796e-02,  ..., -3.4637e-02,
           -4.3759e-02, -2.8479e-02],
          ...,
          [ 3.1028e-03, -1.0449e-02,  1.7222e-02,  ...,  1.2165e-01,
            4.6210e-02, -4.2192e-02],
          [ 7.6139e-03,  2.3000e-02,  9.2082e-03,  ...,  4.5240e-02,
            2.7699e-02, -5.6189e-02],
          [ 2.3990e-02, -1.4030e-02, -1.0267e-02,  ..., -5.2204e-02,
           -6.0534e-02, -7.1400e-02]],

         [[-1.1183e-01, -4.2477e-02,  6.3755e-03,  ..., -1.0254e-01,
           -1.1569e-01, -6.7280e-02],
          [-9.4214e-02, -8.5278e-03,  5.2520e-02,  ..., -6.8358e-02,
           -7.9058e-02, -5.9130e-02],
          [-7.1334e-02, -1.6083e-02,  7.3228e-02,  ..., -2.8043e-02,
           -5.2757e-02, -4.9763e-02],
          ...,
          [ 2.0692e-02,  7.6735e-03,  3.0739e-02,  ...,  1.7656e-01,
            8.8030e-02, -1.7911e-02],
          [ 2.6301e-02,  4.7867e-02,  2.6775e-02,  ...,  8.4734e-02,
            6.1141e-02, -3.5701e-02],
          [ 2.9517e-02,  3.1440e-04,  2.3788e-03,  ..., -2.9185e-02,
           -4.0404e-02, -5.2867e-02]],

         [[-6.7530e-02, -2.4295e-02, -5.6700e-03,  ..., -6.6664e-02,
           -5.8257e-02, -1.5316e-02],
          [-4.9842e-02, -1.2484e-03,  2.4207e-02,  ..., -5.5344e-02,
           -4.6049e-02, -1.7707e-02],
          [-3.7976e-02, -1.2565e-02,  3.6258e-02,  ..., -3.3223e-02,
           -4.0654e-02, -1.9860e-02],
          ...,
          [ 2.3920e-04, -1.2323e-02, -2.5662e-03,  ...,  6.3977e-02,
            2.5765e-02, -2.7954e-02],
          [ 5.6197e-03,  2.9101e-02, -2.5730e-03,  ...,  1.9924e-02,
            2.6986e-02, -2.9991e-02],
          [ 9.6282e-03, -3.3303e-03,  4.9659e-04,  ..., -2.7544e-02,
           -3.6161e-02, -4.3468e-02]]],


        [[[ 6.2803e-02,  7.9755e-02,  8.1322e-02,  ..., -3.7440e-03,
            4.9733e-03, -3.9191e-02],
          [-4.2880e-02, -4.4536e-02, -4.7230e-02,  ..., -7.2959e-02,
           -7.5629e-02, -6.4150e-02],
          [-6.8180e-02, -5.6262e-02, -6.1507e-02,  ..., -4.4115e-02,
           -3.9107e-02, -6.6999e-02],
          ...,
          [ 1.7710e-02,  8.7420e-02,  3.3490e-02,  ...,  1.2845e-02,
            4.8843e-02,  2.8497e-02],
          [ 3.1498e-02,  2.7061e-02,  7.8619e-03,  ...,  6.8658e-02,
            5.4993e-02,  6.3310e-02],
          [-9.4789e-02, -6.8985e-02, -1.4324e-01,  ..., -6.7484e-03,
            4.5337e-02,  2.0077e-02]],

         [[ 8.4717e-02,  5.7770e-02,  3.7002e-02,  ..., -1.3588e-02,
           -7.0228e-03, -6.0760e-02],
          [-1.5979e-02, -6.4630e-02, -9.1545e-02,  ..., -9.8412e-02,
           -9.4532e-02, -9.2499e-02],
          [-2.8392e-02, -6.5403e-02, -1.0047e-01,  ..., -8.3756e-02,
           -6.3615e-02, -9.5850e-02],
          ...,
          [ 5.9997e-02,  1.0705e-01,  4.9321e-02,  ...,  3.5627e-02,
            5.4496e-02, -1.2566e-02],
          [ 6.3830e-02,  2.6649e-02,  7.0561e-03,  ...,  9.2897e-02,
            5.9108e-02,  2.4363e-02],
          [-4.8049e-02, -5.9966e-02, -1.3561e-01,  ...,  2.1068e-02,
            5.2889e-02,  1.5655e-03]],

         [[ 6.2001e-02,  2.0778e-02,  1.1975e-02,  ..., -2.0197e-03,
            6.9852e-03, -4.6863e-02],
          [ 1.3178e-02, -3.5991e-02, -5.1943e-02,  ..., -2.9544e-02,
           -2.5600e-02, -3.3762e-02],
          [ 2.7143e-02, -2.0521e-02, -6.2057e-02,  ..., -1.6369e-02,
            1.0089e-02, -2.7409e-02],
          ...,
          [ 5.1592e-02,  8.0635e-02,  4.0372e-02,  ..., -1.1472e-02,
           -1.3918e-02, -9.3905e-02],
          [ 8.2424e-02,  2.7950e-02,  3.8630e-02,  ...,  2.2609e-02,
           -1.9679e-02, -5.3972e-02],
          [ 5.0708e-02,  1.6264e-02, -4.1324e-02,  ..., -2.9146e-02,
           -2.8712e-03, -4.6670e-02]]],


        [[[-1.3285e-02, -1.1488e-03,  3.0550e-03,  ..., -9.4483e-03,
           -8.8926e-03, -9.0441e-04],
          [ 2.3043e-03, -4.1523e-03, -5.2203e-03,  ...,  9.1216e-05,
           -9.0951e-03,  2.5220e-03],
          [-6.0988e-03, -1.1074e-02, -5.9025e-03,  ...,  1.6161e-03,
           -8.8638e-03, -1.3972e-03],
          ...,
          [-5.4505e-03, -4.5738e-03, -1.0316e-03,  ...,  3.4947e-04,
           -6.1689e-03, -4.7928e-03],
          [ 2.6408e-03, -2.8769e-03, -5.0605e-03,  ...,  1.3172e-04,
           -1.3570e-03, -2.5045e-03],
          [-5.3083e-04, -9.0542e-03, -7.9351e-03,  ...,  4.1531e-03,
           -2.9358e-03, -7.6401e-03]],

         [[ 7.8418e-03,  1.8411e-02,  2.0441e-02,  ...,  5.7535e-03,
            8.6325e-03,  1.8935e-02],
          [ 1.7873e-02,  8.4761e-03,  7.8354e-03,  ...,  8.8990e-03,
            5.7851e-03,  1.6686e-02],
          [ 6.2208e-03, -3.4983e-03,  1.7564e-03,  ...,  4.4534e-03,
           -3.9801e-03,  4.1031e-03],
          ...,
          [ 2.9908e-03,  3.0588e-03,  1.2445e-03,  ...,  4.4925e-04,
            1.0658e-03,  7.1672e-03],
          [ 1.6074e-02,  8.3498e-03,  5.7661e-03,  ...,  9.4217e-03,
            1.0780e-02,  1.6278e-02],
          [ 4.4408e-03,  1.5656e-03, -2.5045e-04,  ...,  9.6282e-03,
            6.9736e-03,  6.3294e-03]],

         [[-7.2360e-03, -1.6965e-03,  3.0952e-03,  ...,  1.5861e-03,
           -6.5436e-03,  4.5501e-03],
          [ 2.8352e-03, -9.9491e-03, -6.2758e-03,  ..., -2.5671e-03,
           -1.2278e-02, -1.3398e-03],
          [ 1.9139e-04, -8.5651e-03, -1.9316e-03,  ...,  2.4267e-04,
           -1.0509e-02, -5.0672e-03],
          ...,
          [-5.4899e-03, -6.1185e-03,  1.0800e-03,  ...,  3.4343e-03,
           -4.8832e-03,  7.7482e-04],
          [-6.7859e-03, -1.3454e-02, -8.1208e-03,  ...,  5.4349e-04,
           -7.4328e-03,  3.7456e-04],
          [-1.5332e-02, -1.7777e-02, -1.3543e-02,  ..., -1.2068e-03,
           -1.5574e-02, -8.6474e-03]]]], device='cuda:0',
       grad_fn=<BroadcastBackward>), tensor([-1.6090e-02,  1.2174e-02,  2.2797e-01,  4.6908e-02, -1.5499e-01,
         8.7547e-02,  1.3502e-01,  1.2800e-02,  8.2932e-02, -4.2087e-01,
         3.8552e-03, -2.6860e-02,  1.4235e-02, -5.7877e-03, -3.6805e-02,
        -3.5540e-02,  5.5290e-03,  9.2909e-02,  1.2771e-02, -3.0965e-02,
        -6.3441e-02, -8.4934e-04, -5.3447e-03, -2.5515e-02, -4.1445e-03,
         2.5515e-02,  1.8479e-02, -2.5615e-02,  4.0568e-02, -2.0309e-02,
        -2.8299e-03, -5.2244e-03, -2.2531e-02, -1.0226e-03, -1.7576e-02,
         1.1028e-03, -2.5013e-02, -3.2375e-02,  1.4297e-02,  1.2362e-02,
        -3.5994e-02,  4.0352e-02, -3.7467e-02, -5.9556e-03, -4.2830e-01,
         3.5867e-01,  1.1702e-02, -1.1946e-02,  1.0084e-01, -1.4606e-02,
        -1.0271e-02,  4.5181e-01,  5.3849e-03, -1.2856e-02,  1.4235e-03,
        -1.7222e-03, -2.2668e-02, -2.2627e-03,  3.5256e-02, -1.5487e-01,
        -3.0596e-02,  1.7638e-02, -1.6498e-02, -2.2896e-03,  1.1674e-01,
         1.4380e-02,  7.1714e-02,  6.4757e-03,  1.4729e-02,  1.0424e-01,
         2.1318e-02,  7.3172e-01, -2.7957e-03,  1.0743e-02, -5.3283e-01,
         3.7074e-03, -8.4154e-03, -1.1694e-02, -1.3869e-02, -1.8652e-02,
         2.4801e-02,  3.8952e-03,  3.9827e-02,  1.4166e-02, -7.6014e-01,
         4.1432e-01, -1.9246e-01,  1.1102e-01, -1.4642e-02,  5.7801e-03,
         2.8745e-03, -1.7657e-02, -7.7463e-02, -6.4326e-02,  1.2057e-02,
        -1.1613e-02, -4.0119e-02, -1.8979e-02, -2.7809e-02, -2.3785e-02,
         2.2651e-02, -2.4268e-02,  5.2309e-03,  1.9109e-02, -3.7954e-03,
        -1.9735e-02, -2.8117e-02, -5.8240e-02, -1.0201e-01,  6.7421e-04,
         4.4703e-02, -3.0308e-04, -3.6796e-02,  2.3805e-04,  3.2304e-01,
        -3.5212e-01, -4.4832e-02,  5.7270e-03,  5.8176e-03, -1.7689e-02,
         5.5221e-03, -6.2864e-03,  2.8066e-02,  3.8595e-02, -1.3975e-02,
        -2.1706e-02,  1.7213e-02, -1.1424e-02,  1.0731e-02, -5.9880e-04,
         2.5505e-02, -7.9028e-03, -3.2949e-02,  3.8501e-03, -8.8245e-03,
        -3.4819e-03, -1.4605e-02, -9.3169e-03, -6.8412e-02,  9.3665e-03,
        -3.4788e-03,  6.8371e-03,  6.4590e-03, -6.6017e-02,  8.7025e-01,
        -4.8855e-02,  2.0873e-02,  3.9021e-04,  9.8548e-03,  5.1253e-03,
         1.0060e-02, -5.7132e-02, -2.5164e-03,  7.5240e-01, -6.7990e-03,
        -4.7859e-01, -6.1399e-03, -1.1962e-03,  1.1866e-03,  3.5237e-03,
         4.2073e-02,  5.0811e-03,  3.5187e-02,  7.6161e-03, -1.8199e-02,
        -2.5168e-02,  1.0928e-02, -1.8055e-02, -8.7963e-03,  1.2136e-02,
         2.9134e-02, -4.7957e-02,  1.9883e-03, -2.1588e-02, -1.2183e-03,
        -5.0964e-02, -1.0120e-02, -5.1278e-03,  9.2718e-03, -1.0521e-02,
        -9.1500e-01, -6.9778e-03, -1.1852e-05, -2.7351e-02, -6.3665e-03,
         2.6623e-02, -1.0697e-02,  1.3013e+00, -4.2087e-03,  3.5010e-03,
        -9.0631e-03, -1.2495e-02, -1.1143e-02, -6.4557e-02, -7.4548e-03,
         2.5077e-02,  6.0618e-03,  3.8729e-02,  1.9512e-03, -1.5695e-02,
         4.1066e-02,  7.1876e-02,  1.1941e-02,  1.1479e-02, -2.1025e-02,
        -4.4586e-02, -1.2142e-01,  3.6297e-01, -1.1479e-02, -7.0550e-03,
        -3.0732e-03, -1.0475e-01, -3.1307e-02,  2.6413e-02,  1.8010e-04,
        -2.9713e-03,  6.5173e-03, -9.2936e-03, -2.7098e-02,  3.4719e-02,
        -5.4723e-02, -4.5547e-02, -2.4742e-02, -3.0855e-03,  8.1147e-03,
         1.3071e-02,  6.4248e-02,  8.1228e-04,  2.4757e-02,  1.4534e-02,
         2.5024e-02,  8.5607e-02,  6.8466e-03,  5.1933e-03,  8.6611e-02,
        -4.6842e-01, -5.6616e-02,  1.6048e-01,  6.1547e-03,  2.4924e-02,
        -3.3897e-02,  7.5299e-01, -2.8823e-02,  5.1894e-03,  1.5493e-02,
        -2.8037e-02, -3.5234e-02, -6.3110e-01,  2.6496e-02, -2.0996e-03,
         8.1190e-01,  5.2454e-01, -9.2737e-03,  1.5295e-02,  7.9412e-03,
        -4.2548e-02, -9.7852e-03, -2.3154e-02, -9.9202e-03,  8.1742e-03,
         2.4342e-05, -8.2740e-03, -5.5718e-03,  6.1778e-01, -2.5267e-02,
         1.5646e-02, -4.1497e-02, -1.9631e-02,  8.7076e-02,  8.0120e-03,
        -2.8781e-03, -5.4196e-02, -5.0515e-01, -3.8567e-03, -5.6713e-03,
        -8.2977e-03,  5.7573e-03, -3.0785e-02, -8.8448e-03,  5.0190e-01,
         2.7427e-02, -2.2315e-02,  3.7897e-03,  5.3910e-03, -1.7446e-02,
        -6.7904e-02, -6.4136e-02,  3.0637e-02,  1.7055e-02, -4.0486e-03,
         4.0048e-04,  8.4181e-03, -2.9910e-03, -1.7335e-02,  2.8932e-02,
         3.4725e-02, -3.8263e-02, -6.6158e-03,  2.3055e-02,  3.3611e-03,
        -4.1740e-02,  5.7093e-01,  1.0084e-02, -1.0451e-02,  5.0783e-02,
        -1.5320e-02, -1.3032e-02,  3.7760e-02,  1.7139e-02,  5.1414e-02,
        -1.8133e-02,  4.9087e-03, -1.3765e-01,  1.8895e-02,  1.6932e-02,
         2.7277e-03,  1.8141e-02, -6.8402e-03, -1.1778e-01,  1.0169e-02,
         1.6853e-01, -1.3144e-04,  4.2427e-02, -2.6230e-02, -2.1131e-02,
         4.9175e-02, -1.0456e-02, -2.5849e-02,  4.4606e-02, -1.7229e-05,
        -1.1238e-02,  1.5116e-03,  1.2077e-02,  2.6054e-03,  6.9948e-02,
         1.5055e-02, -1.9745e-02,  1.9165e-03, -4.6319e-02, -3.2234e-02,
        -1.7079e-01, -3.5179e-02, -7.9092e-02,  1.5927e-02, -7.5581e-03,
         3.9424e-01,  4.5398e-02,  1.0123e-03, -3.0889e-02,  8.9127e-03,
         1.3117e-04, -1.2365e-02,  1.4814e-01, -2.2632e-02, -1.2881e-02,
        -3.4497e-02,  8.9060e-03, -3.0747e-01,  1.1530e-02, -5.1192e-03,
         2.6323e-02,  3.6447e-03, -9.0179e-02, -1.0916e-03,  1.1193e-02,
        -1.7883e-02, -1.8083e-02,  4.2735e-03, -8.7878e-03,  7.8851e-01,
        -8.4941e-03, -3.0741e-02,  2.3667e-02,  1.3072e-02,  4.4558e-02,
        -5.2301e-03,  8.5091e-05, -2.0978e-02,  5.6533e-03, -4.4536e-02,
        -2.2528e-02,  2.7267e-03,  2.7645e-01, -3.3284e-02, -1.3758e-01,
        -3.8739e-02, -2.1205e-02,  1.4522e-02,  3.3279e-02, -4.9861e-01,
        -2.6866e-03,  4.7418e-02, -1.0412e-02, -5.0259e-03,  5.9364e-02,
         1.5291e-02,  1.2832e-02,  9.9840e-04, -4.6879e-01,  6.3109e-03,
        -6.6550e-03, -4.7005e-01,  1.1081e-02,  6.3798e-03, -2.3701e-02,
        -6.1321e-04,  9.4518e-03, -1.8441e-02,  1.9444e-02, -3.2495e-02,
        -3.9736e-02,  1.2732e-02, -1.5738e-03,  1.7463e-02, -1.2696e-01,
         3.1286e-01, -5.8906e-02,  1.7822e-02, -5.9794e-02, -1.9892e-02,
         6.0129e-03, -2.2110e-02,  8.3146e-02,  1.1826e-03, -1.5764e-01,
        -6.6654e-03, -2.4652e-02,  2.9888e-02,  1.3662e-02,  2.2539e-02,
        -4.5877e-02,  1.7355e-02,  3.1344e-03,  3.8609e-03,  4.8026e-03,
         1.1118e-02,  1.0949e-01, -1.5221e-02,  1.9277e-02, -6.6302e-03,
        -7.1109e-03,  7.4188e-04, -6.3697e-03, -3.1777e-02,  1.2641e-03,
         5.6136e-03, -1.3858e-02,  5.9672e-01, -2.9905e-02, -7.8330e-01,
         8.2061e-04,  1.9875e-03,  4.8229e-02, -4.0310e-03, -7.4141e-01,
        -8.2589e-02,  4.6838e-03, -1.1728e-03, -6.8659e-02, -2.1787e-02,
        -2.3413e-02,  3.8247e-03, -3.5208e-02,  8.0469e-03, -6.0692e-03,
        -1.8382e-02, -6.8114e-04,  8.2483e-01,  4.2232e-02,  2.6186e-02,
        -3.5071e-02,  2.9950e-01, -3.9087e-02,  2.8266e-02, -3.4465e-02,
         4.0329e-04, -2.1029e-02,  1.3218e-02,  5.7879e-01, -1.1618e-02,
        -1.7881e-02, -5.4950e-02, -1.3292e-03,  2.9873e-02, -3.3105e-03,
        -1.5792e-01, -1.4564e-02,  3.8663e-02, -8.6169e-02, -1.5880e-01,
        -4.9658e-02, -1.6112e-03,  1.9248e-02, -4.6184e-03, -4.1773e-02,
        -2.6493e-03,  2.0980e-02,  1.2356e-02,  1.5418e-02,  8.0534e-01,
         1.7136e-02, -1.9514e-02,  3.1425e-03,  4.3635e-03,  9.3242e-03,
         5.9385e-02, -1.6063e-02,  9.7715e-01, -1.0699e-01,  3.2006e-02,
        -1.7259e-02, -2.2930e-02, -8.4327e-01, -1.0589e-02,  4.8237e-02,
         9.1879e-03,  8.5088e-01, -3.9914e-03, -6.3607e-02,  2.3347e-02,
        -2.6078e-02, -1.2018e-02, -3.9092e-01,  1.0435e-01, -4.1202e-02,
         1.4983e-02,  1.4928e-02,  1.6260e-02,  2.1786e-02,  6.1203e-01,
        -4.3203e-03,  2.1811e-02,  2.8324e-02, -1.6943e-02,  1.0289e-03,
        -4.2235e-03, -3.9832e-02, -3.0189e-02, -3.2481e-03,  1.6847e-02,
        -4.3612e-02,  5.2142e-01, -2.0684e-02,  3.4631e-02,  1.9373e-03,
         1.2069e-01, -1.9904e-02,  9.1863e-03,  4.1725e-03, -8.4121e-02,
        -2.2211e-03,  5.6895e-03, -1.3401e-02, -1.5374e-02, -2.5099e-02,
         1.9727e-02, -1.7755e-01,  2.9352e-03, -2.2476e-01,  2.6460e-03,
         1.8885e-02,  1.3120e-02, -3.8352e-02, -5.8401e-03, -1.1220e-02,
        -1.9525e-02,  1.6225e-02,  1.8265e-02, -1.9113e-02,  4.5447e-01,
         8.7089e-03, -6.5228e-03, -9.9888e-03,  6.5381e-02, -1.6652e-02,
        -3.9072e-03, -1.2806e-01, -6.4722e-03,  6.2611e-05, -4.1967e-02,
        -2.6738e-02,  8.5272e-02, -6.5992e-03,  2.0409e-02, -1.6775e-02,
        -8.2542e-02, -9.1505e-03,  2.3265e-01, -2.0393e-02, -2.5408e-02,
         3.7184e-01,  1.1421e-02, -1.4207e-04, -1.0234e-01,  1.9930e-03,
         6.7073e-02, -1.1965e-02,  6.2451e-04, -4.2905e-02,  1.9731e-03,
        -2.7423e-02, -1.0493e-02, -5.5997e-02,  3.5540e-03,  5.0565e-03,
         1.5316e-04,  1.9976e-02, -6.1887e-02, -1.9239e-02, -2.8151e-02,
         1.4671e-02,  3.8599e-03, -1.0113e-01,  3.7388e-02,  1.2063e-02,
         9.1354e-03,  1.3958e-02,  1.5136e-03, -1.1466e-02, -1.6999e-02,
        -7.6677e-02, -4.8566e-03, -2.4719e-01,  1.0974e-02,  8.4056e-03,
         5.8064e-03, -2.4864e-01, -3.7818e-01,  5.0208e-02, -3.0094e-02,
         2.8905e-04,  4.9162e-03,  1.2000e-02, -1.7560e-02, -1.5956e-01,
        -9.3537e-03, -3.1023e-02,  7.7834e-01, -8.3942e-01, -9.3088e-03,
        -1.3086e-02,  5.0306e-01, -1.4037e-03, -2.2345e-01,  1.4390e-03,
         3.5904e-01, -1.2583e-02,  2.1788e-02,  3.1457e-03, -1.3125e-02,
        -3.9513e-02, -1.5309e-02, -7.3315e-04, -2.7821e-02,  6.5916e-04,
         9.3677e-03, -1.9807e-02, -6.9219e-03,  2.0447e-01, -1.7606e-02,
        -2.0071e-02, -1.5081e-02,  3.1485e-01,  8.7187e-03, -2.3410e-02,
        -3.4283e-03, -5.5052e-03, -5.2397e-02,  4.0907e-02,  1.0465e-03,
        -4.0619e-01,  2.4019e-02,  5.6711e-01, -8.7051e-02, -5.8143e-03,
        -6.2758e-03,  6.5046e-03,  5.2147e-01,  2.7435e-01, -1.7838e-03,
         1.8702e-02,  9.8643e-03,  2.2815e-02, -1.3442e-02, -1.9221e-03,
        -2.5178e-02,  2.8828e-01, -9.8848e-03, -8.6106e-03, -1.6368e-01,
         8.5245e-03,  2.3074e-02,  1.0927e-02,  2.3362e-03, -5.0312e-03,
        -1.4515e-03,  6.0329e-03,  1.7541e-03,  2.2712e-02, -6.2920e-02,
         1.9621e-02, -5.9920e-03,  9.3095e-03, -1.2057e-02, -3.4766e-02,
         7.0962e-02,  3.8835e-01, -6.9094e-03,  1.4411e-02,  2.9177e-02,
         5.3793e-03, -2.1615e-02,  2.6376e-02, -1.1585e-03,  4.0472e-02,
         1.1851e-02,  1.5705e-02,  3.8531e-02,  6.8253e-03,  1.0445e-02,
        -9.5043e-03, -1.3340e-02,  6.6893e-03,  7.8108e-03,  8.2606e-03,
         7.3462e-02,  4.5651e-01, -1.9310e-02, -5.4200e-02, -3.6836e-03,
         8.5394e-03,  5.6307e-03, -2.4418e-02, -7.0981e-03, -2.5901e-01,
        -2.2678e-02,  6.2244e-02,  2.3416e-02,  1.8405e-03, -1.6070e-02,
        -2.2669e-03, -3.3542e-02,  5.1731e-01, -2.7881e-02, -7.9796e-02,
         4.5613e-03, -4.0021e-03,  4.8400e-03, -3.7174e-03,  2.0248e-02,
         2.0703e-02,  6.0141e-03,  3.3654e-02, -1.0999e-02, -4.6471e-01,
        -2.7094e-01, -3.0672e-02,  6.7943e-03,  4.8118e-02, -1.1301e+00,
         2.7343e-03, -3.5806e-02, -1.0150e-01, -5.5286e-03, -5.1283e-03,
        -1.7647e-03, -1.4113e-02, -4.1544e-01], device='cuda:0',
       grad_fn=<BroadcastBackward>), [16, 16], [0, 0], [1, 1], False, [0, 0], 1), **{}) 

While executing %self_vit_embeddings_patch_embeddings_projection : [#users=1] = call_module[target=self_vit_embeddings_patch_embeddings_projection](args = (%pixel_values,), kwargs = {})
Original traceback:
  File "/home/ubuntu/transformers/src/transformers/models/vit/modeling_vit.py", line 175, in forward
    embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
 |   File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
 |   File "/home/ubuntu/transformers/src/transformers/models/vit/modeling_vit.py", line 117, in forward
    embeddings = self.patch_embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
 |   File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
 |   File "/home/ubuntu/transformers/src/transformers/models/vit/modeling_vit.py", line 573, in forward
    embedding_output = self.embeddings(
 |   File "/home/ubuntu/anaconda3/envs/py38_nightly/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
 |   File "/home/ubuntu/transformers/src/transformers/models/vit/modeling_vit.py", line 787, in forward
    outputs = self.vit(
```

Comments:
Diving into this issue, the forward pass of a ViT model (or also ResNet) segfaults on my side (I don't have your error message, but you are using multiple GPUs with DataParallel if I read the traceback correctly, which is probably not supported). I'll reach out to the PyTorch team.

Ok, the segmentation fault actually came from a mix of torchvision stable and torch nightlies. With 
```
pip3 install --pre torch torchvision --force-reinstall --index-url https://download.pytorch.org/whl/nightly/cu117
```
I don't get the segfaults when running the forward passes and I can run the example on one GPU with torch inductor.

Thanks for checking in. I just tried to restrict the use to a single GPU, the original Faketensor issue is gone and training goes normal.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.