Reproducibility issue between Intel and AMD CPUs

chiragjn: <p>I am using <code>torch==1.8.1+cpu</code> and <code>transformers==4.7.0</code> in a python 3.6 docker container installed using pip.<br>
Here is the build info</p>
<pre><code class="lang-auto">PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, 
</code></pre>
<p>With this, I am getting different results for the same model forward on an Intel CPU and an AMD CPU. I have read about non deterministic ops but none of them seem to be matching my situation</p>
<pre><code class="lang-auto">base_model = 'google/bert_uncased_L-12_H-256_A-4'

import os
os.environ['CUDA_VISIBLE_DEVICES'] = ''
import torch
import numpy as np
import random
from transformers import BertConfig, BertTokenizerFast, BertForSequenceClassification

torch.set_printoptions(precision=11)
print(torch.cuda.is_available())
def set_random_seed(seed_value: int, use_cuda: bool = False):
    np.random.seed(seed_value)
    torch.manual_seed(seed_value)
    random.seed(seed_value)
    torch.use_deterministic_algorithms(True)
    if use_cuda:
        torch.cuda.manual_seed(seed_value)
        torch.cuda.manual_seed_all(seed_value)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = True


set_random_seed(2019, use_cuda=False)
tokenizer = BertTokenizerFast.from_pretrained(base_model, do_lower_case=True, max_len=256)
bert_config = BertConfig.from_pretrained(base_model, num_labels=2)
model = BertForSequenceClassification.from_pretrained(base_model, config=bert_config)
batch_text_or_text_pairs = [['hello', 'world']]
all_label_ids = [1]


inputs = tokenizer.batch_encode_plus(
    batch_text_or_text_pairs=batch_text_or_text_pairs,
    add_special_tokens=True,
    padding='max_length',
    truncation='longest_first',
    max_length=256,
    stride=0,
    is_split_into_words=False,
    pad_to_multiple_of=None,
    return_tensors='pt',
    return_token_type_ids=True,
    return_attention_mask=True,
    return_overflowing_tokens=False,
    return_special_tokens_mask=False,
    return_offsets_mapping=False,
    return_length=False,
    verbose=False,
)

inputs['labels'] = torch.tensor(all_label_ids, dtype=torch.long)
model.zero_grad()
model = model.train()
model(**inputs)
</code></pre>
<p>On two Intel CPUs</p>
<pre><code class="lang-auto">Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
</code></pre>
<p>this consistently gives</p>
<pre><code class="lang-auto">SequenceClassifierOutput(loss=tensor(0.53043878078, grad_fn=&lt;NllLossBackward&gt;), logits=tensor([[-0.31894901395,  0.03818603605]], grad_fn=&lt;AddmmBackward&gt;), hidden_states=None, attentions=None)
</code></pre>
<p>on AMD CPUs</p>
<pre><code class="lang-auto">AMD EPYC 7V12 64-Core Processor
</code></pre>
<p>this consistently gives</p>
<pre><code class="lang-auto">SequenceClassifierOutput(loss=tensor(0.56962001324, grad_fn=&lt;NllLossBackward&gt;), logits=tensor([[-0.26781406999, -0.00332138361]], grad_fn=&lt;AddmmBackward&gt;), hidden_states=None, attentions=None)
</code></pre>
<p>Should I compile torch from source using Eigen or OpenBLAS for reproducibility? Or is something else going on that I can’t control?<br>
My libtorch*.so does not link against libmkl</p>
<pre><code class="lang-auto">./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so:
	linux-vdso.so.1 (0x00007ffc05543000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9a89df6000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9a89c73000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f9a89a49000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9a89a2f000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9a89a0e000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a8984d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9a8a21d000)
./usr/local/lib/python3.6/site-packages/torch/lib/libcaffe2_detectron_ops.so:
	linux-vdso.so.1 (0x00007fff351c0000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007f5448580000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f5448356000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f544832b000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007f5436c5e000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007f54369c7000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5436844000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f543683d000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f54366b9000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f543669f000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f54364de000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f544a508000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007f54364d4000)
./usr/local/lib/python3.6/site-packages/torch/lib/libcaffe2_module_test_dynamic.so:
	linux-vdso.so.1 (0x00007fff5618a000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007f18e8fa5000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007f18d78d8000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007f18d7641000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f18d74b3000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f18d7499000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f18d72d8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f18e93c5000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f18d70ac000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f18d708b000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007f18d7081000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f18d707c000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f18d6ef9000)
./usr/local/lib/python3.6/site-packages/torch/lib/libcaffe2_observers.so:
	linux-vdso.so.1 (0x00007ffc705cf000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007fe0e9fc0000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007fe0d88f3000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fe0d865c000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe0d8631000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe0d84ad000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe0d8493000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe0d82d0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe0ea437000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fe0d80a6000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe0d809c000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe0d8097000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe0d7f14000)
./usr/local/lib/python3.6/site-packages/torch/lib/libjitbackend_test.so:
	linux-vdso.so.1 (0x00007fff49fad000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007fde84056000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007fde72989000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fde726f2000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fde72564000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fde7254a000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fde72389000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fde8448a000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fde7215d000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fde7213c000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fde72132000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fde7212d000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fde71faa000)
./usr/local/lib/python3.6/site-packages/torch/lib/libshm.so:
	linux-vdso.so.1 (0x00007ffc4e5fa000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007fa1936a9000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa193695000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fa1933fe000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007fa181d31000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa181d10000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa181b8c000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa181b70000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa1819af000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa18182c000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fa181602000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa193ac6000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa1815fd000)
./usr/local/lib/python3.6/site-packages/torch/lib/libtorchbind_test.so:
	linux-vdso.so.1 (0x00007ffef2bef000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007f5ec4810000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007f5eb3143000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007f5eb2eac000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5eb2d1e000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5eb2d04000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5eb2b43000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5ec4c7a000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f5eb2917000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5eb28f6000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5eb28ec000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5eb28e7000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5eb2764000)
./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so:
	linux-vdso.so.1 (0x00007ffd9bdf3000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fc91be38000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc91be0d000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc91be03000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc91bde9000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc91bde4000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc91bc61000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fc91b9c8000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc91b844000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc91b683000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc92d731000)
./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_global_deps.so:
	linux-vdso.so.1 (0x00007ffc720d9000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8f8f1f3000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8f8f070000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8f8f06b000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007f8f8ee41000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8f8ec80000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8f8f423000)
./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_python.so:
	linux-vdso.so.1 (0x00007ffc78bb5000)
	libshm.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libshm.so (0x00007fcd33fbb000)
	libtorch.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so (0x00007fcd33da7000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fcd33d93000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fcd33afc000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007fcd2242f000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcd2240e000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcd22407000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcd22283000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcd22269000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcd220a8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcd35274000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcd21f25000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fcd21cfb000)
./usr/local/lib/python3.6/site-packages/torch/lib/libtorch.so:
	linux-vdso.so.1 (0x00007ffcb1d87000)
	libtorch_cpu.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so (0x00007fcbee57e000)
	libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcbee55a000)
	libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcbee399000)
	libgomp-a34b3233.so.1 =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libgomp-a34b3233.so.1 (0x00007fcbee16f000)
	libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcbee14e000)
	librt.so.1 =&gt; /lib/x86_64-linux-gnu/librt.so.1 (0x00007fcbee144000)
	libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcbee13d000)
	libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcbedfba000)
	libc10.so =&gt; /./usr/local/lib/python3.6/site-packages/torch/lib/libc10.so (0x00007fcbedd23000)
	libstdc++.so.6 =&gt; /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcbedb9f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcbffe61000)
</code></pre>

ptrblck: <p>I don’t think there is a guarantee to get bitwise identical results using different hardware.<br>
Enabling deterministic results makes sure to get the same values on the currently used system/setup.</p>

chiragjn: <p>Ah, that’s disappointing <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I tried three more things</p>
<ol>
<li>Predicting with an already trained model - the confidence scores are not exact but the precision is still very high (absolute difference &lt;= 1e-6)</li>
<li>Just changing <code>model.train()</code> to <code>model.eval()</code> - Same as 1, not exact but still very high precision</li>
<li>Checking pseudo random generation by using <code>torch.rand()</code> after setting seed. The results here were exact</li>
</ol>
<p>I wonder what operations in this training/model architecture scenario make the difference so large. Since putting the model in <code>train</code> mode causes this I am going to check the outputs layer by layer once - maybe something is up with dropout, batch norm, layer norm layers</p>

ptrblck: <aside class="quote no-group" data-username="chiragjn" data-post="3" data-topic="144779">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chiragjn/48/46933_2.png" class="avatar"> chiragjn:</div>
<blockquote>
<p>Since putting the model in <code>train</code> mode causes this I am going to check the outputs layer by layer once - maybe something is up with dropout, batch norm, layer norm layers</p>
</blockquote>
</aside>
<p>That’s a good idea as I wouldn’t entirely exclude the possibility of a numerical issue in one of your setups (e.g. caused in the framework directly, by a 3rd party library etc.).</p>

chiragjn: <p>So I am back after some debugging. Here are my findings:</p>
<p>Dropout kernels produce different masks across different CPUs (Intel vs AMD) and GPUs (V100 vs T4). They only produce the same mask when run on the exact same device.</p>
<p>LayerNorm CPU and CUDA kernels don’t produce the exact same output but at least the output is close enough - CPU outputs are exactly matching when comparing across different cpus, GPU outputs are exactly matching when comparing across different GPUs.</p>
<p>My minimal reproduction code along with runs from three different machines is here: <a href="https://github.com/chiragjn/torch-1.8.1-reproducibility-bug" class="inline-onebox" rel="noopener nofollow ugc">GitHub - chiragjn/torch-1.8.1-reproducibility-bug</a></p>
<p>In summary</p>
<h2>
<a name="torchnnlayernorm-1" class="anchor" href="#torchnnlayernorm-1"></a><code>torch.nn.LayerNorm</code>
</h2>
<div class="md-table">
<table>
<thead>
<tr>
<th>LayerNorm (train mode)</th>
<th>Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz</th>
<th>AMD EPYC 7V12 64-Core Processor</th>
<th>Intel(R) Core™ i7-8565U CPU @ 1.80GHz</th>
<th>Tesla V100-PCIE-16GB, 460.27.04</th>
<th>Tesla T4, 460.27.04</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz</strong></td>
<td>Exact</td>
<td>-</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>AMD EPYC 7V12 64-Core Processor</strong></td>
<td>Exact</td>
<td>Exact</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>Intel(R) Core™ i7-8565U CPU @ 1.80GHz</strong></td>
<td>Exact</td>
<td>Exact</td>
<td>Exact</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>Tesla V100-PCIE-16GB, 460.27.04</strong></td>
<td><strong>Close enough</strong></td>
<td><strong>Close enough</strong></td>
<td><strong>Close enough</strong></td>
<td>Exact</td>
<td>-</td>
</tr>
<tr>
<td><strong>Tesla T4, 460.27.04</strong></td>
<td><strong>Close enough</strong></td>
<td><strong>Close enough</strong></td>
<td><strong>Close enough</strong></td>
<td>Exact</td>
<td>Exact</td>
</tr>
</tbody>
</table>
</div><h2>
<a name="torchnndropout-2" class="anchor" href="#torchnndropout-2"></a><code>torch.nn.Dropout</code>
</h2>
<div class="md-table">
<table>
<thead>
<tr>
<th>Dropout (train mode)</th>
<th>Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz</th>
<th>AMD EPYC 7V12 64-Core Processor</th>
<th>Intel(R) Core™ i7-8565U CPU @ 1.80GHz</th>
<th>Tesla V100-PCIE-16GB, 460.27.04</th>
<th>Tesla T4, 460.27.04</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz</strong></td>
<td>Exact</td>
<td>-</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>AMD EPYC 7V12 64-Core Processor</strong></td>
<td><strong>Different</strong></td>
<td>Exact</td>
<td>-</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>Intel(R) Core™ i7-8565U CPU @ 1.80GHz</strong></td>
<td>Exact</td>
<td><strong>Different</strong></td>
<td>Exact</td>
<td>-</td>
<td>-</td>
</tr>
<tr>
<td><strong>Tesla V100-PCIE-16GB, 460.27.04</strong></td>
<td><strong>Different</strong></td>
<td><strong>Different</strong></td>
<td><strong>Different</strong></td>
<td>Exact</td>
<td>-</td>
</tr>
<tr>
<td><strong>Tesla T4, 460.27.04</strong></td>
<td><strong>Different</strong></td>
<td><strong>Different</strong></td>
<td><strong>Different</strong></td>
<td><strong>Different</strong></td>
<td>Exact</td>
</tr>
</tbody>
</table>
</div><p>We can live with LayerNorm, but dropout masks being different might result in significantly different models after quite some number of batches especially when dropout is used almost after every layer <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Thanks for the update. The pseudorandom number generators would use different implementations which is why random operations such as dropout are not producing the same “random” numbers.<br>
Yes, I think you are right that “different” models would be learned, but as long as the random sampling process is not broken, I would expect to see a similar model performance.<br>
If the success of the model training itself depends on a specific random seed, it would be a bad sign for the stability of the overall training routine.</p>

chiragjn: <p>While I agree, I can’t write any automated tests to check training routines and have them pass on any virtual machine.</p>
<p>But apart from that, I am wondering how come <code>torch.rand</code> and a few other randomization ops can be made deterministic across all kinds of devices but dropout can’t be? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>I will try to dig into the kernel code myself when I get the time later but would love an answer if someone is already familiar with the code</p>

ptrblck: <p>I don’t think that’s the case. At least I see the expected differences:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
print(torch.randn(10))
# &gt; tensor([-2.0748,  0.8152, -1.1281,  0.8386, -0.4471, -0.5538, -0.8776, -0.5635,
#           0.5434, -0.8192])

torch.manual_seed(2809)
print(torch.randn(10, device='cuda'))
# &gt; tensor([ 0.5603,  1.8841, -0.1417, -0.6460, -2.2665, -1.3027,  0.3552, -0.5376,
#           0.1375,  1.6309], device='cuda:0')
</code></pre>

chiragjn: <p>Ah, I see, in my experiments, I have always been creating my input on CPU and then moving them to my test device. Fair enough I won’t expect the same output when comparing CPU vs GPU</p>
<p>But still, <code>torch.rand</code> works (or at least so far seems to) exactly the same when I move from Intel to AMD and/or V100 to T4 but <code>torch.nn.Dropout</code> definitely doesn’t <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<p>Might have to read hardware manuals now to check RNG implementations</p>

chiragjn: <p>Just leaving this here, the Bernoulli kernels are indeed customized for Intel:<br>
on Intel CPUs, vectorized RNG from mkl would be used and on AMD it falls back to default one</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L484-L544">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L484-L544" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L484-L544" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L484-L544</a></h4>



    <pre class="onebox">      <code class="lang-cpp">
        <ol class="start lines" start="484" style="counter-reset: li-counter 483 ;">
            <li>#if !AT_MKL_ENABLED()</li>
            <li>void bernoulli_scalar_kernel(Tensor&amp; self, double p, c10::optional&lt;Generator&gt; gen) {</li>
            <li>  bernoulli_scalar_kernel_default(self, p, gen);</li>
            <li>}</li>
            <li>#else</li>
            <li>void bernoulli_scalar_kernel(Tensor &amp;self, double p, c10::optional&lt;Generator&gt; gen) {</li>
            <li>  if (cpuinfo_initialize() &amp;&amp; cpuinfo_vendor_intel == cpuinfo_get_processor(0)-&gt;core-&gt;vendor) {</li>
            <li>    CPUGeneratorImpl* generator = get_generator_or_default&lt;CPUGeneratorImpl&gt;(gen, detail::getDefaultCPUGenerator());</li>
            <li>    int64_t seed;</li>
            <li>    {</li>
            <li>      // See Note [Acquire lock when using random generators]</li>
            <li>      std::lock_guard&lt;std::mutex&gt; lock(generator-&gt;mutex_);</li>
            <li>      seed = generator-&gt;random();</li>
            <li>    }</li>
            <li>    int64_t n = self.numel();</li>
            <li>    bool contig = self.is_contiguous();</li>
            <li>
            </li>
<li>    AT_DISPATCH_ALL_TYPES_AND(at::ScalarType::Bool, self.scalar_type(), "bernoulli_scalar_cpu_", [&amp;] {</li>
            <li>      at::Tensor tmp_int_tensor;</li>
            <li>      if (std::is_same&lt;scalar_t, int&gt;::value &amp;&amp; contig) {</li>
        </ol>
      </code>
    </pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cpu/UnaryOpsKernel.cpp#L484-L544" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>For CUDA I am not sure what can be done but the RNG for some reason does not behave the same across GPUs</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cuda/DistributionTemplates.h#L625-L643">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cuda/DistributionTemplates.h#L625-L643" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cuda/DistributionTemplates.h#L625-L643" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/56b43f4fec1f76953f15a627694d4bba34588969/aten/src/ATen/native/cuda/DistributionTemplates.h#L625-L643</a></h4>



    <pre class="onebox">      <code class="lang-h">
        <ol class="start lines" start="625" style="counter-reset: li-counter 624 ;">
            <li>template&lt;typename RNG&gt;</li>
            <li>void bernoulli_kernel(Tensor&amp; self, const Tensor&amp; p_, RNG gen) {</li>
            <li>  PhiloxCudaState rng_engine_inputs;</li>
            <li>  {</li>
            <li>    // See Note [Acquire lock when using random generators]</li>
            <li>    std::lock_guard&lt;std::mutex&gt; lock(gen-&gt;mutex_);</li>
            <li>    rng_engine_inputs = gen-&gt;philox_cuda_state(10);</li>
            <li>  }</li>
            <li>  auto p = std::get&lt;0&gt;(expand_inplace(self, p_.to(kCUDA)));</li>
            <li>  AT_DISPATCH_ALL_TYPES_AND3(</li>
            <li>    at::ScalarType::Half, at::ScalarType::BFloat16, at::ScalarType::Bool, self.scalar_type(), "bernoulli_tensor_cuda_self_", [&amp;] {</li>
            <li>      using self_t = scalar_t;</li>
            <li>      AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, p.scalar_type(), "bernoulli_tensor_cuda_p_", [&amp;] {</li>
            <li>        using p_t = scalar_t;</li>
            <li>        return bernoulli_tensor_cuda_kernel&lt;self_t, p_t&gt;(self, p, rng_engine_inputs);</li>
            <li>      });</li>
            <li>   });</li>
            <li>}</li>
            <li>
        </li>
</ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-python">def check(seed, device, size):
    import torch
    torch.set_printoptions(precision=11)
    torch.manual_seed(seed)
    torch.use_deterministic_algorithms(True)
    if 'cuda' in device:
        torch.cuda.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = True
    device = torch.device(device)
    x = torch.rand(*size)
    x = x.to(device)
    print(x)
    a = x.bernoulli_(p=0.5)
    print(a.sum(axis=0))
    print(a.sum(axis=1))

check(42, 'cuda', (256, 256))
</code></pre>
<p>For some reason, the difference doesn’t occur until a particular size is hit (more than ~60K elements in my observations). Now I don’t know if this is torch incorrectly using the curand APIs or a bug within CUDA itself <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>