Issue 1:
🐛 Bug

To Reproduce

Steps to reproduce the behavior:

import numpy as np
import torch

a = np.array([[[ -29.8316, 506.7845, 5400.1382],
[ -155.6416, 73.0718, 5448.8071],
[ -52.9619, -309.7045, 5251.0830],
[ -300.4998, -332.3928, 5156.6812],
[ -258.2405, 99.6090, 5244.6816],
[ -209.4844, 548.8338, 5290.7637],
[ -176.7308, -321.0486, 5203.8818],
[ -109.1576, -529.7281, 5123.8906],
[ -140.1912, -780.1214, 5074.6050],
[ -118.9348, -970.2283, 5058.5991],
[ 13.8925, -279.8529, 5421.0684],
[ 35.7916, -470.1449, 5257.7383],
[ -19.7603, -716.9181, 5140.2725],
[ -259.0900, -690.1336, 5050.5923],
[ -370.6709, -448.5993, 5134.1772],
[ -462.2866, -290.8295, 5307.6274],
[ -153.1819, -886.9761, 5130.1655]],

[[ -29.6097, 507.0160, 5400.0415],
[ -156.1013, 73.2300, 5446.2266],
[ -52.8395, -309.0523, 5247.8511],
[ -299.8065, -331.7188, 5151.9609],
[ -258.0193, 99.6480, 5243.2402],
[ -209.7421, 548.7822, 5290.6899],
[ -176.3229, -320.3856, 5199.9062],
[ -108.0304, -530.1070, 5123.3218],
[ -140.6482, -779.8441, 5071.7817],
[ -118.2912, -970.1854, 5056.8403],
[ 10.7354, -281.1750, 5414.3501],
[ 40.3466, -470.5802, 5251.2056],
[ -19.6162, -718.2531, 5137.8745],
[ -258.3210, -688.7090, 5046.1133],
[ -368.4557, -443.3939, 5120.0488],
[ -460.3479, -281.6299, 5289.6309],
[ -153.4338, -886.6000, 5127.5791]],

[[ -29.7913, 507.1184, 5400.0073],
[ -156.4464, 73.1930, 5444.3994],
[ -52.3020, -308.4679, 5245.2900],
[ -299.1556, -330.9422, 5149.0630],
[ -258.1010, 100.1828, 5241.8062],
[ -210.1292, 549.2088, 5290.5708],
[ -175.7287, -319.7050, 5197.1768],
[ -107.6006, -531.0508, 5125.0386],
[ -142.8517, -779.9221, 5071.1001],
[ -118.7543, -970.0274, 5056.5713],
[ 25.9413, -271.5096, 5396.5942],
[ 45.9570, -469.2388, 5242.0957],
[ -21.8357, -718.7902, 5137.6470],
[ -258.9276, -687.5615, 5042.6875],
[ -361.8696, -434.6036, 5099.1748],
[ -458.3741, -250.1555, 5240.7217],
[ -154.4211, -886.6248, 5127.2637]]])

a = torch.from_numpy(a)
a -= a[:,7:8,:]
print(a)

output

tensor([[[ 79.3260, 1036.5126, 276.2476],
[ -46.4840, 602.7999, 324.9165],
[ 56.1957, 220.0236, 127.1924],
[ -191.3422, 197.3353, 32.7906],
[ -149.0829, 629.3371, 120.7910],
[ -100.3268, 1078.5619, 166.8731],
[ -67.5732, 208.6795, 79.9912],
[ 0.0000, 0.0000, 0.0000],
[ -140.1912, -780.1214, 5074.6050], #<---here
[ -118.9348, -970.2283, 5058.5991],
[ 13.8925, -279.8529, 5421.0684],
[ 35.7916, -470.1449, 5257.7383],
[ -19.7603, -716.9181, 5140.2725],
[ -259.0900, -690.1336, 5050.5923],
[ -370.6709, -448.5993, 5134.1772],
[ -462.2866, -290.8295, 5307.6274],
[ -153.1819, -886.9761, 5130.1655]],

[[ 78.4207, 1037.1230, 276.7197],
[ -48.0709, 603.3370, 322.9048],
[ 55.1909, 221.0547, 124.5293],
[ -191.7761, 198.3882, 28.6391],
[ -149.9889, 629.7550, 119.9184],
[ -101.7117, 1078.8892, 167.3681],
[ -68.2925, 209.7214, 76.5844],
[ 0.0000, 0.0000, 0.0000],
[ -140.6482, -779.8441, 5071.7817],
[ -118.2912, -970.1854, 5056.8403],
[ 10.7354, -281.1750, 5414.3501],
[ 40.3466, -470.5802, 5251.2056],
[ -19.6162, -718.2531, 5137.8745],
[ -258.3210, -688.7090, 5046.1133],
[ -368.4557, -443.3939, 5120.0488],
[ -460.3479, -281.6299, 5289.6309],
[ -153.4338, -886.6000, 5127.5791]],

[[ 77.8093, 1038.1692, 274.9687],
[ -48.8458, 604.2438, 319.3608],
[ 55.2986, 222.5829, 120.2514],
[ -191.5550, 200.1086, 24.0244],
[ -150.5004, 631.2336, 116.7676],
[ -102.5286, 1080.2596, 165.5322],
[ -68.1281, 211.3458, 72.1382],
[ 0.0000, 0.0000, 0.0000],
[ -142.8517, -779.9221, 5071.1001],
[ -118.7543, -970.0274, 5056.5713],
[ 25.9413, -271.5096, 5396.5942],
[ 45.9570, -469.2388, 5242.0957],
[ -21.8357, -718.7902, 5137.6470],
[ -258.9276, -687.5615, 5042.6875],
[ -361.8696, -434.6036, 5099.1748],
[ -458.3741, -250.1555, 5240.7217],
[ -154.4211, -886.6248, 5127.2637]]], dtype=torch.float64)

Expected behavior

tensor([[[ 79.3260, 1036.5126, 276.2476],
[ -46.4840, 602.7999, 324.9165],
[ 56.1957, 220.0236, 127.1924],
[ -191.3422, 197.3353, 32.7906],
[ -149.0829, 629.3371, 120.7910],
[ -100.3268, 1078.5619, 166.8731],
[ -67.5732, 208.6795, 79.9912],
[ 0.0000, 0.0000, 0.0000],
[ -140.1912, -780.1214, 5074.6050], #<---here
[ -118.9348, -970.2283, 5058.5991],
[ 13.8925, -279.8529, 5421.0684],
[ 35.7916, -470.1449, 5257.7383],
[ -19.7603, -716.9181, 5140.2725],
[ -259.0900, -690.1336, 5050.5923],
[ -370.6709, -448.5993, 5134.1772],
[ -462.2866, -290.8295, 5307.6274],
[ -153.1819, -886.9761, 5130.1655]],
...

Environment

PyTorch version: 0.4.0
Is debug build: No
CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: TITAN Xp
GPU 2: TITAN Xp
GPU 3: TITAN Xp
GPU 4: TITAN Xp
GPU 5: TITAN Xp
GPU 6: TITAN Xp
GPU 7: TITAN Xp

Nvidia driver version: 396.54
cuDNN version: Probably one of the following:
/usr/local/cuda-9.0/lib64/libcudnn.so.7.3.0
/usr/local/cuda-9.0/lib64/libcudnn_static.a
/usr/local/cuda-9.2/lib64/libcudnn.so.7.2.1
/usr/local/cuda-9.2/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] numpy (1.15.2)
[pip] torch (0.4.0)
[pip] torchfile (0.1.0)
[pip] torchsample (0.1.3, /home/nonews/src/torchsample)
[pip] torchvision (0.2.1)
[conda] torch 0.4.0 <pip>
[conda] torchfile 0.1.0 <pip>
[conda] torchsample 0.1.3 <pip>
[conda] torchvision 0.2.1 <pip>

Additional context

Ptrblck was able to reproduce this issue in 1.0.0a0+dfad8b6 and 1.0.0.dev20181007 on Nov 11.

The output is caused by changing of tensor a before the op finish.

Issue 2:
Issue description

I have a 80x256x256x10 FC network I'm using for policy gradient, but when I do a forward pass on it it takes 70-100ms (!!) to execute. Puzzling-ly, after some condition that I can't identify, the forward pass suddenly speeds up permanently to < 1-3ms (I couldn't get a reproducible example of that though). Exiting the code during execution seems to imply that it's spending a lot of time in torch.addmm(bias, input, weight.t()). Adding pytorch.set_num_threads(1) seems to fix it, but I don't know why. Sorry if this is a duplicate (I suspect it may be) but I looked and couldn't find anything in the issues

Code example

Here's a (sort of) minimal example that reproduces the slowness:

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import time

class GaussianMLP(nn.Module):
def __init__(self, num_inputs, action_dim, init_std=1, learn_std=True, nonlinearity="tanh"):
super(GaussianMLP, self).__init__()

actor_dims = (256, 256)
critic_dims = (256, 256)

# create actor network
self.actor_layers = nn.ModuleList()
self.actor_layers += [nn.Linear(num_inputs, actor_dims[0])]
for l in range(len(actor_dims) - 1):
in_dim = actor_dims[l]
out_dim = actor_dims[l + 1]
self.actor_layers += [nn.Linear(in_dim, out_dim)]

self.mean = nn.Linear(actor_dims[-1], action_dim)

# create critic network
self.critic_layers = nn.ModuleList()
self.critic_layers += [nn.Linear(num_inputs, critic_dims[0])]
for l in range(len(critic_dims) - 1):
in_dim = critic_dims[l]
out_dim = critic_dims[l + 1]
self.critic_layers += [nn.Linear(in_dim, out_dim)]

self.vf = nn.Linear(critic_dims[-1], 1)

if nonlinearity == "relu":
self.nonlinearity = F.relu
else:
self.nonlinearity = torch.tanh

self.train()

def forward(self, inputs):
start = time.time()
x = inputs
for l in self.critic_layers:
x = self.nonlinearity(l(x))
value = self.vf(x)

x = inputs
for l in self.actor_layers:
x = self.nonlinearity(l(x))
x = self.mean(x)

x = torch.tanh(x)
print(time.time() - start)

return value, x

policy = GaussianMLP(80, 10, nonlinearity="relu", init_std=np.exp(-2), learn_std=False)

for _ in range(100):
s = torch.rand(1, 80)
policy(s)

And here's an iPython stacktrace from randomly exiting that code while it's running:

61 for _ in range(100):
62 s = torch.rand(1, 80)
---> 63 policy(s)

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
475 result = self._slow_forward(*input, **kwargs)
476 else:
--> 477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
479 hook_result = hook(self, input, result)

~/p-morais/cassie-sim-to-real/gaussian_mlp.py in forward(self, inputs)
49 x = inputs
50 for l in self.actor_layers:
---> 51 x = self.nonlinearity(l(x))
52 x = self.mean(x)
53

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
475 result = self._slow_forward(*input, **kwargs)
476 else:
--> 477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
479 hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
53
54 def forward(self, input):
---> 55 return F.linear(input, self.weight, self.bias)
56
57 def extra_repr(self):

~/.local/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
1022 if input.dim() == 2 and bias is not None:
1023 # fused op is marginally faster
-> 1024 return torch.addmm(bias, input, weight.t())
1025
1026 output = input.matmul(weight.t()))

System Info

PyTorch version: 0.4.1

Is debug build: No

CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.1 LTS

GCC version: (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0

CMake version: version 3.10.2

Python version: 3.6

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration: GPU 0: TITAN X (Pascal)

Nvidia driver version: 390.77

cuDNN version: Could not collect

Versions of relevant libraries:

[pip3] numpy (1.15.3)

[pip3] torch (1.0.0a0+c029c83)

[pip3] torchvision (0.2.1)

[conda] pytorch 0.4.0 py36_cuda8.0.61_cudnn7.1.2_1 pytorch

[conda] pytorch-cpu 0.4.1 py36_cpu_1 pytorch

[conda] torchfile 0.1.0

[conda] torchvision-cpu 0.2.1 py36_1 pytorch