Incorrect and inconsistent outputs from CrossEntropyLoss(reduction="none") with torch.float16 dtype
Labels: high priority, triage review, module: nn, module: cuda, triaged, module: correctness (silent)

### 🐛 Describe the bug

I'm trying to evaluate the loss on a fairly large tensor, and I get different results between the first eval and any subsequent evals, and all evals are incorrect for a chunk of the output. @albanD you've been great help before, let me know what you think.

Minimal repro (requires min 40GB GPU RAM):

```
import torch

batch_size = 70
seq_len = 2048
vocab_size = 50000

shift_labels = torch.zeros(batch_size, seq_len-1, dtype=torch.long).to("cuda")
logits = torch.ones(batch_size, seq_len-1, vocab_size, dtype=torch.float16).to("cuda")

loss_fct = torch.nn.CrossEntropyLoss(reduction="none")

# Evaluate loss first time
nll = loss_fct(logits.permute(0, 2, 1), shift_labels).float()
print(nll)
# This gives:
# tensor([[10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#       [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#       [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        ...,
#       [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000],
#       [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000],
#       [-0.0000, -0.0000, -0.0000,  ..., -0.0000, -0.0000, -0.0000]])

# Evaluate loss second time
nll = loss_fct(logits.permute(0, 2, 1), shift_labels).float()
print(nll)
# This gives
# tensor([[10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        ...,
#        [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
#        [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000],
#        [-1.0000, -1.0000, -1.0000,  ..., -1.0000, -1.0000, -1.0000]])
```

The expected outputs for both calls should be:
```
# tensor([[10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        ...,
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203],
#        [10.8203, 10.8203, 10.8203,  ..., 10.8203, 10.8203, 10.8203]])
```

### Versions

Collecting environment information...
PyTorch version: 2.1.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.27.7
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1049-azure-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.128
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version: 470.199.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             24
On-line CPU(s) list:                0-23
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7V13 64-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 24
Socket(s):                          1
Stepping:                           1
BogoMIPS:                           4890.87
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          768 KiB (24 instances)
L1i cache:                          768 KiB (24 instances)
L2 cache:                           12 MiB (24 instances)
L3 cache:                           96 MiB (3 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-23
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.1
[pip3] torch==2.1.0
[pip3] triton==2.1.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @kadeng @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ptrblck