Issue 1:
🐛 Bug

When use gpu to train a two-convolution-layer model, the random behavior is different in the running of two times.

To Reproduce

write this code in some file, on my computer the source code is named try.py

import torch
import numpy as np
import random

torch.manual_seed(123)
random.seed(123)
np.random.seed(123)
torch.cuda.manual_seed(123)
torch.backends.cudnn.enabled=False
torch.backends.cudnn.deterministic=True

conv1 = torch.nn.Conv2d(3, 32, 3, 1, 1).cuda()
conv2 = torch.nn.Conv2d(32, 32, 3, stride=1, padding=1, dilation=1).cuda()
# fc = torch.nn.Linear(32, 10)
criteria = torch.nn.CrossEntropyLoss()

torch.nn.init.kaiming_normal_(conv1.weight, 1)
torch.nn.init.kaiming_normal_(conv2.weight, 1)
params = list(conv1.parameters()) + list(conv2.parameters())
optim = torch.optim.SGD(params, lr=1e-4, momentum=0.9, weight_decay=1e-4)

for i in range(10):
inten = torch.randn(16, 3, 64, 64).cuda()
lb = torch.randint(0, 10, (16, 64, 64)).cuda()

optim.zero_grad()
feat = conv1(inten)
feat = conv2(feat)
loss = criteria(feat, lb)
loss.backward()
optim.step()
print(loss.item())

run the above code for twice, and see the loss printed to the screen:

CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101936340332
3.944667100906372
3.9629740715026855
3.9574058055877686
3.9502787590026855
3.956238031387329
3.9543356895446777
3.950388193130493
3.946221351623535

CUDA_VISIBLE_DEVICES=0 python try.py
3.9505960941314697
3.962101697921753
3.944666862487793
3.9629745483398438
3.9574055671691895
3.9502787590026855
3.956238031387329
3.9543352127075195
3.950387954711914
3.9462215900421143

Since my practical model has a big variance, this random behavior can bring a final score variance of as much as 0.5%, which makes it too difficult for me to tune the hyper-parameters. What is the cause of this randomness, and how could I reduce it please?

Environment

Collecting environment information...

PyTorch version: 1.1.0a0+95ce796

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: CentOS Linux 7 (Core)

GCC version: (GCC) 5.4.0

CMake version: version 3.14.0

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 9.0.176

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

GPU 2: GeForce GTX 1080 Ti

GPU 3: GeForce GTX 1080 Ti

Nvidia driver version: 390.67

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.16.3

[pip] torch==1.1.0

[pip] torchvision==0.2.2.post3

[pip] torchvision-nightly==0.2.3

[conda] Could not collect

Issue 2:
Hi,

I came across this problem while using pytorch:

The dot product between 2 vectors with floating precision can become arbitrarily small/large. This problem does not occur when using the sum and product operation separately:

v1, v2 = torch.FloatTensor([1., 1.]), torch.FloatTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))
(-3.6893488147419103e+19, -1.9999998807907104)

Dot product with tensors with double precision seems ok:

v1, v2 = torch.DoubleTensor([1., 1.]), torch.DoubleTensor([-1., -0.9999999])
float(v1 @ v2), float(torch.sum(v1 * v2))
(-1.9999999000000002, -1.9999999000000002)

System Info

PyTorch version: 0.4.0

Is debug build: No

CUDA used to build PyTorch: None

OS: Mac OSX 10.13.4

GCC version: Could not collect

CMake version: Could not collect

Python version: 3.6

Is CUDA available: No

CUDA runtime version: No CUDA

GPU models and configuration: No CUDA

Nvidia driver version: No CUDA

cuDNN version: No CUDA

Versions of relevant libraries:

[pip] numpy (1.14.0)

[pip] numpydoc (0.7.0)

[pip] torch (0.4.0)

[pip] torchvision (0.2.1)

[conda] pytorch 0.4.0 py36_cuda0.0_cudnn0.0_1 pytorch

[conda] torchvision 0.2.1 py36_1 pytorch