nn.AdaptiveMaxPool2d returns identical results within a batch
Labels: high priority, module: nn, module: cuda, triaged, module: correctness (silent), bug

### 🐛 Describe the bug

When using nn.AdaptiveMaxPool2d(output_size=(1, 1)) as global max pooling after LayerNorm, it returns identical results within a batch. The code is as follows.

```
import torch
from torch import nn
import torch.nn.functional as F

class LayerNorm2d(nn.LayerNorm):
    # normalization along channels
    def forward(self, x):
        # batch_size, channel, hight, width = x.size()
        x = x.permute(0, 2, 3, 1)
        x = F.layer_norm(x, self.normalized_shape, self.weight, self.bias, self.eps)
        x = x.permute(0, 3, 1, 2)
        return x

class TestConv(nn.Module):
    def __init__(self, in_channel, out_channel):
        super().__init__()
        self.norm = LayerNorm2d(in_channel)
        self.conv = nn.Conv2d(in_channel, out_channel, kernel_size=3, stride=1, padding=1, bias=False)
    def forward(self, x):
        x = self.norm(x)
        x = self.conv(x)
        return x
 
print(torch.__version__)
model = TestConv(2, 1).cuda().eval()
pooling = nn.AdaptiveMaxPool2d(output_size=(1, 1))

data_in = torch.rand((2,2,3,3)).cuda()
data_out1 = model(data_in)
data_out2 = pooling(data_out1)
print(data_out1)
print(data_out2)
```
In approximately 90% of cases, the elements of data_out2 are identical, which, I think, is incorrect.
For example, the above code may act as:
```
1.13.1+cu117
tensor([[[[-0.6695, -0.5291, -0.1605],
          [ 0.9633,  0.1471, -0.0945],
          [-0.0090,  0.7883,  0.7220]]],
        [[[ 0.2343, -0.4815, -0.1492],
          [-0.6429,  0.2614, -0.0867],
          [ 0.8783,  0.7894,  0.7118]]]], device='cuda:0', grad_fn=<ConvolutionBackward0>)
tensor([[[[0.9633]]],
        [[[0.9633]]]], device='cuda:0', grad_fn=<AdaptiveMaxPool2DBackward0>)
```

### Versions

PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-78-generic-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.3.109
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA RTX A4000
GPU 1: NVIDIA RTX A4000
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA RTX A4000
GPU 4: NVIDIA RTX A4000

Nvidia driver version: 525.125.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  16
Socket(s):           1
NUMA node(s):        4
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7302 16-Core Processor
Stepping:            0
CPU MHz:             1500.000
CPU max MHz:         3000.0000
CPU min MHz:         1500.0000
BogoMIPS:            5989.01
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-3,16-19
NUMA node1 CPU(s):   4-7,20-23
NUMA node2 CPU(s):   8-11,24-27
NUMA node3 CPU(s):   12-15,28-31
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.13.1
[pip3] torchtext==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39h6c91a56_3  
[conda] numpy-base                1.21.5           py39ha15fc14_3  
[conda] numpydoc                  1.4.0            py39h06a4308_0  
[conda] torch                     1.13.1                   pypi_0    pypi
[conda] torchaudio                0.13.1                   pypi_0    pypi
[conda] torchvision               0.14.1                   pypi_0    pypi

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ptrblck