TestTagsCPU.test_tags__refs_constant_pad_nd_cpu_float32 flaky with dynamo & pytest
Labels: module: ci, triaged

### 🐛 Describe the bug

Recently, the dynamo config has failed 3 times seemingly flakily with the following:
```
[gw3] [ 96%] RERUN test_ops.py::TestTagsCPU::test_tags__refs_constant_pad_nd_cpu_float32 
2022-07-19T20:27:06.4357892Z test_ops.py::TestTagsCPU::test_tags__refs_constant_pad_nd_cpu_float32 
2022-07-19T20:27:06.4368462Z [gw3] [ 96%] RERUN test_ops.py::TestTagsCPU::test_tags__refs_constant_pad_nd_cpu_float32 
2022-07-19T20:27:06.4404288Z test_ops.py::TestTagsCPU::test_tags__refs_constant_pad_nd_cpu_float32 
2022-07-19T20:27:14.1210808Z [gw3] [ 96%] FAILED test_ops.py::TestTagsCPU::test_tags__refs_constant_pad_nd_cpu_float32 
2022-07-19T20:27:14.1211730Z 
2022-07-19T20:27:14.1214386Z =================================== FAILURES ===================================
2022-07-19T20:27:14.1215474Z ___________ TestTagsCPU.test_tags__refs_constant_pad_nd_cpu_float32 ____________
2022-07-19T20:27:14.1216133Z [gw3] linux -- Python 3.7.13 /opt/conda/bin/python
2022-07-19T20:27:14.1216393Z Traceback (most recent call last):
2022-07-19T20:27:14.1216653Z   File "/var/lib/jenkins/workspace/test/test_ops.py", line 1480, in test_tags
2022-07-19T20:27:14.1216880Z     @onlyCPU
2022-07-19T20:27:14.1217101Z   File "/var/lib/jenkins/workspace/test/test_ops.py", line 1491, in test_tags
2022-07-19T20:27:14.1217370Z     rs = op(input, *sample.args, **sample.kwargs)
2022-07-19T20:27:14.1217800Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_methods_invocations.py", line 953, in __call__
2022-07-19T20:27:14.1218094Z     return self.op(*args, **kwargs)
2022-07-19T20:27:14.1218586Z   File "/opt/conda/lib/python3.7/site-packages/torch/_refs/__init__.py", line 2168, in constant_pad_nd
2022-07-19T20:27:14.1218865Z     prims.copy_to(c_output, c_input)
2022-07-19T20:27:14.1219216Z   File "/opt/conda/lib/python3.7/site-packages/torch/_ops.py", line 49, in __call__
2022-07-19T20:27:14.1219469Z     return self._op(*args, **kwargs or {})
2022-07-19T20:27:14.1219829Z   File "/opt/conda/lib/python3.7/site-packages/torch/_prims/__init__.py", line 374, in _autograd_impl
2022-07-19T20:27:14.1220153Z     return BackwardsNotSupported.apply(args_spec, *flat_args)
2022-07-19T20:27:14.1220535Z   File "/opt/conda/lib/python3.7/site-packages/torch/_prims/__init__.py", line 364, in forward
2022-07-19T20:27:14.1220781Z     return _prim(*args, **kwargs)
2022-07-19T20:27:14.1221103Z   File "/opt/conda/lib/python3.7/site-packages/torch/_ops.py", line 49, in __call__
2022-07-19T20:27:14.1221368Z     return self._op(*args, **kwargs or {})
2022-07-19T20:27:14.1221718Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/_python_dispatch.py", line 74, in wrapped
2022-07-19T20:27:14.1221993Z     return f(self, *args, **kwargs)
2022-07-19T20:27:14.1222256Z   File "/var/lib/jenkins/workspace/test/test_ops.py", line 1468, in __torch_dispatch__
2022-07-19T20:27:14.1222538Z     def __torch_dispatch__(self, func, types, args=(), kwargs=None):
2022-07-19T20:27:14.1222831Z   File "/var/lib/jenkins/workspace/test/test_ops.py", line 1462, in check_inplace_view
2022-07-19T20:27:14.1223105Z     assert torch.Tag.inplace_view in func.tags
2022-07-19T20:27:14.1223351Z AssertionError: assert <Tag.inplace_view: 3> in []
2022-07-19T20:27:14.1223658Z  +  where <Tag.inplace_view: 3> = <class 'torch.Tag'>.inplace_view
2022-07-19T20:27:14.1223941Z  +    where <class 'torch.Tag'> = torch.Tag
2022-07-19T20:27:14.1224252Z  +  and   [] = <OpOverload(op='prims.copy_to', overload='default')>.tags
```

Most recent logs: https://github.com/pytorch/pytorch/runs/7430470789?check_suite_focus=true 

It looks like the test has turned flaky ever since turning on pytest in #79898, but this is an observation and not necessarily the reason why. Ideally, it'd be great to fix this on the dynamo side if possible but, if not, we can exclude dynamo from using pytest.

ccing @ezyang @clee2000 

### Versions

In CI

cc @ezyang @gchanan @zou3519 @seemethere @malfet @pytorch/pytorch-dev-infra