RuntimeError: p.block != nullptr && p.block->ptr != nullptr INTERNAL ASSERT FAILED at "../c10/cuda/CUDACachingAllocator.cpp":1275, please report a bug to PyTorch.
Labels: triaged, module: assert failure, module: CUDACachingAllocator

### 🐛 Describe the bug

self.model = timm.create_model('swin_large_patch4_window12_384', num_classes=4, pretrained=False).to(device)
pre = torch.load(
              'model_swin_large_patch4_window12_384.pth', map_location=device)
new_state = OrderedDict()
for k, v in pre.items():  # 去除关键字”model"
     name = k[6:]
     new_state[name] = v
self.model.load_state_dict(new_state)
self.model.eval()


If the code is written as follows——
        self.model = timm.create_model('swin_large_patch4_window12_384', num_classes=10, pretrained=False).to(device)
        pre = torch.load(
              'model_swin_large_patch4_window12_384.pth', map_location=device)
        new_state = OrderedDict()
        for k, v in pre.items():  # 去除关键字”model"
            name = k[6:]
            new_state[name] = v
        self.model.load_state_dict(new_state)
        for param in self.model.parameters():
              param.requires_grad = False
        self.model.eval()
this error will not appear！！

### Error logs

  outputs = model_D5(img, imfo)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/zqf/embryo_prediction_rate/ExtractModel.py", line 25, in forward
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/timm/models/swin_transformer.py", line 568, in forward
    x = self.forward_features(x)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/timm/models/swin_transformer.py", line 558, in forward_features
    x = self.layers(x)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/timm/models/swin_transformer.py", line 420, in forward
    x = self.blocks(x)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/timm/models/swin_transformer.py", line 325, in forward
    x = x + self.drop_path(self.mlp(self.norm2(x)))
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/timm/models/layers/mlp.py", line 27, in forward
    x = self.fc1(x)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/pyt/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: p.block != nullptr && p.block->ptr != nullptr INTERNAL ASSERT FAILED at "../c10/cuda/CUDACachingAllocator.cpp":1275, please report a bug to PyTorch.


### Minified repro

_No response_

### Versions

pytorch 1.12.0+cu113


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh