[Inductor] [CPU] performance regression with TORCHINDUCTOR_FREEZING=1
Labels: triaged, oncall: pt2, module: cpu inductor

### 🐛 Describe the bug

There are 6 performance regression from https://github.com/pytorch/pytorch/issues/93531#issuecomment-1630223790

|   |  2023-07-09 nightly  |   |     |   |  2023-07-06 nightly  |   |     |   |  Result Comp  |     |   |
|  ---- |  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
|  model | batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|Background_Matting	|1	|0.782193|	0.350721461|	0.274331872|	1	|1.00737	|0.279395653	|0.281454799	|0.78	|1.03	|0.8|
|doctr_det_predictor	|1	|1.090279|	0.148405348|	0.161803234|	1	|1.713053	 |0.095406578	|0.163436525	|0.64	|1.01	|0.64|
|functorch_dp_cifar10	|64	|0.622732|	0.009190167|	0.005723011|	64	|1.008348	 |0.005596095	|0.005642811	|0.62	|0.99|	0.61|
|gmlp_s16_224	|128	|1.068468|	0.658434753|	0.703516464|	128	|1.227975	 |0.587295424	|0.721184098	|0.87|	1.03	|0.89|
|resmlp_12_224	|128	|0.749039|	0.415152565|	0.310965462|	128	|1.237528	 |0.259625741	|0.321294124	|0.61|	1.03	|0.63|
|tnt_s_patch16_224|	1	|1.173545	|0.094649226	|0.111075126	|1	|1.367958	|0.081854891	|0.111974053	|0.86	|1.01	|0.86

SW information:

SW	| Nightly commit	| Main commit
-- | -- | --
Pytorch|[9b5a84f](https://github.com/pytorch/pytorch/commit/9b5a84f)|[dd6c38c](https://github.com/pytorch/pytorch/commit/dd6c38c)
Torchbench|/|[8526eabb](https://github.com/pytorch/benchmark/commit/8526eabb)
torchaudio|[a233cc1](https://github.com/pytorch/audio/commit/a233cc1)|[1e117f5](https://github.com/pytorch/audio/commit/1e117f5)
torchtext|[90ea46c](https://github.com/pytorch/text/commit/90ea46c)| [8546bbb](https://github.com/pytorch/text/commit/8546bbb)
torchvision|[2ab2f74](https://github.com/pytorch/vision/commit/2ab2f74)|[657027f](https://github.com/pytorch/vision/commit/657027f)
torchdata|[9ed0325](https://github.com/pytorch/data/commit/9ed0325)|[901b483](https://github.com/pytorch/data/commit/901b483)
dynamo_benchmarks|[6226b7d](https://github.com/pytorch/pytorch/commit/6226b7d)|/


### Versions

```bash
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so:${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libjemalloc.so
export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:-1,muzzy_decay_ms:-1"

export TORCHINDUCTOR_FREEZING=1
CORES=$(lscpu | grep Core | awk '{print $4}')
export OMP_NUM_THREADS=$CORES

python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/torchbench.py --inference --inference --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only Background_Matting  --cold_start_latency

python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/torchbench.py --inference --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only doctr_det_predictor  --cold_start_latency

python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/torchbench.py --inference --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only functorch_dp_cifar10 --cold_start_latency

python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/torchbench.py --inference --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only gmlp_s16_224  --cold_start_latency

python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/timm_models.py --inference --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only resmlp_12_224  --cold_start_latency

python -m torch.backends.xeon.run_cpu --core_list 0 --ncores_per_instance 1 benchmarks/dynamo/timm_models.py --inference --performance --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only tnt_s_patch16_224  --cold_start_latency


```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305