DDPOptimizer+inductor OOMs with hf_GPT2_large and timm_vision_transformer_large
Labels: triaged, oncall: pt2

Repro:

`gpui --oversubscribe python run_benchmark.py distributed --ngpus 1 --model torchbenchmark.models.hf_GPT2_large.Model --trainer torchbenchmark.util.distributed.core_model.trainer.Trainer --job_dir /fsx/users/dberard/scratch-local/bench-fast/benchmark/logs --cluster local --torchdynamo inductor --optimize_dynamo_ddp`

Error:
```
Traceback (most recent call last):
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/submission.py", line 54, in process_job
    result = delayed.result()
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/submitit/core/utils.py", line 133, in result
    self._result = self.function(*self.args, **self.kwargs)
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/distributed/submit.py", line 123, in __call__
    return trainer_class(self.args, model_class, model_args=self.model_args).measure()
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/distributed/core_model/trainer.py", line 90, in measure
    self.benchmark.invoke()
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/model.py", line 243, in invoke
    self.train()
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 157, in _fn
    return fn(*args, **kwargs)
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 123, in train
    def train(self):
  File "/fsx/users/dberard/scratch-local/bench-fast/benchmark/torchbenchmark/util/framework/huggingface/model_factory.py", line 124, in <graph break in train>
    outputs = self.model(**self.example_inputs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/autograd/function.py", line 270, in apply
    return user_fn(self, *args)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/functorch/_src/aot_autograd.py", line 464, in backward
    CompiledFunction.compiled_bw = aot_config.bw_compiler(
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/optimizations/backends.py", line 555, in _wrapped_bw_compiler
    return disable(bw_compiler(*args, **kwargs))
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 85, in time_wrapper
    r = func(*args, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 353, in bw_compiler
    return compile_fx_inner(
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 450, in debug_wrapper
    compiled_fn = compiler_fn(gm, example_inputs, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_inductor/debug.py", line 178, in inner
    return fn(*args, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 120, in compile_fx_inner
    compiled_fn = cudagraphify(
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 85, in time_wrapper
    r = func(*args, **kwargs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 183, in cudagraphify
    return cudagraphify_impl(model, inputs, static_input_idxs)
  File "/data/home/dberard/miniconda/envs/bench-fast/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 249, in cudagraphify_impl
    model(list(static_inputs))
  File "/tmp/torchinductor_dberard/wm/cwmx6z66cmrszqbfpvgyixq7nrq23aagw7qu7or5ijuzlvwfucn4.py", line 470, in call
    buf9 = empty_strided((1280, 1280), (1280, 1), device='cuda', dtype=torch.float32)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 39.41 GiB total capacity; 31.47 GiB already allocated; 7.50 MiB free; 38.03 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try sett
ing max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh