DISABLED test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
Labels: oncall: distributed, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/distributed%2Ffsdp%2Ftest_fsdp_tp_integration.py%3A%3ATestTPFSDPIntegration%3A%3Atest_fsdp_tp_checkpoint_integration)).

This test starts to fail on periodic multigpu job recently https://github.com/pytorch/pytorch/actions/runs/4942837753/jobs/8837502304

```
=================================== FAILURES ===================================
__________ TestTPFSDPIntegration.test_fsdp_tp_checkpoint_integration ___________
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 541, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 760, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 805, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 657, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 543, in wrapper
    fn()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_tp_integration.py", line 331, in test_fsdp_tp_checkpoint_integration
    tp_fsdp_model.load_state_dict(state_dict)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2056, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FullyShardedDataParallel:
	size mismatch for _fsdp_wrapped_module.net1.weight: copying a param with shape torch.Size([4, 5]) from checkpoint, the shape in current model is torch.Size([8, 5]).
	size mismatch for _fsdp_wrapped_module.net1.bias: copying a param with shape torch.Size([4]) from checkpoint, the shape in current model is torch.Size([8]).
	size mismatch for _fsdp_wrapped_module.net2.weight: copying a param with shape torch.Size([4, 4]) from checkpoint, the shape in current model is torch.Size([4, 8]).
	While copying the parameter named "_fsdp_wrapped_module.net2.bias", whose dimensions in the model are torch.Size([4]) and whose dimensions in the checkpoint are torch.Size([4]), an exception occurred : ('aten.copy_.default: got mixed distributed and non-distributed tensors.',).


----------------------------- Captured stdout call -----------------------------
Process 2 terminated with exit code 10, terminating remaining processes.
------------------------------ Captured log call -------------------------------
INFO     numba.cuda.cudadrv.driver:driver.py:245 init
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 0 with pid 52714
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 1 with pid 52715
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 2 with pid 52716
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 3 with pid 52717
----------------------------- Captured stdout call -----------------------------
Process 0 terminated with exit code 10, terminating remaining processes.
------------------------------ Captured log call -------------------------------
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 0 with pid 52791
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 1 with pid 52792
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 2 with pid 52793
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 3 with pid 52794
----------------------------- Captured stdout call -----------------------------
Process 1 terminated with exit code 10, terminating remaining processes.
------------------------------ Captured log call -------------------------------
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 0 with pid 52868
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 1 with pid 52869
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 2 with pid 52870
INFO     torch.testing._internal.common_distributed:common_distributed.py:589 Started process 3 with pid 52871
- generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-620ff0cafbb720dd.xml -
=========================== short test summary info ============================
FAILED [5.7867s] distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_checkpoint_integration - RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 657, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 543, in wrapper
    fn()
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper
    return func(*args, **kwargs)
  File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_tp_integration.py", line 331, in test_fsdp_tp_checkpoint_integration
    tp_fsdp_model.load_state_dict(state_dict)
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2056, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for FullyShardedDataParallel:
	size mismatch for _fsdp_wrapped_module.net1.weight: copying a param with shape torch.Size([4, 5]) from checkpoint, the shape in current model is torch.Size([8, 5]).
	size mismatch for _fsdp_wrapped_module.net1.bias: copying a param with shape torch.Size([4]) from checkpoint, the shape in current model is torch.Size([8]).
	size mismatch for _fsdp_wrapped_module.net2.weight: copying a param with shape torch.Size([4, 4]) from checkpoint, the shape in current model is torch.Size([4, 8]).
	While copying the parameter named "_fsdp_wrapped_module.net2.bias", whose dimensions in the model are torch.Size([4]) and whose dimensions in the checkpoint are torch.Size([4]), an exception occurred : ('aten.copy_.default: got mixed distributed and non-distributed tensors.',).
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu