Implementation of arbitrary differentiable functions

Veril: Is it possible to just use arbitrary differentiable/supported functions to create other functions without having to implement their backward as described in examples? One thing I really liked about TF is how you can just create an arbitrary compute graph of differentiable pieces. It’s not obvious how to do that here unless I’m missing something?

jekbradbury: 

Veril: I see, so what then is the reason behind “def backward(self, grad_output):” when implementing your own modules? Why isn’t that redundant?

jekbradbury: 

Veril: It seems that you can’t use numpy constants in such a construct, it leads it being stuck on the CPU : <SCODE>def gelu(x):
	return 0.5 * x * (1 + F.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x*x*x)))
<ECODE> However if you replace the square root with 0.79788456080 everything works fine. Intentional?

apaszke: What’s the error you’re getting on the GPU? You’re only doing scalar ops from numpy.

Veril: Not getting any error. It just seems to never get around to training, GPU isn’t busy and a single CPU core is at 100%.

apaszke: And what’s the stack trace once you interrupt the script?

Veril: <SCODE>^CProcess Process-1:
Traceback (most recent call last):
  File "/home/rrr/anaconda/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "main.py", line 121, in <module>
    self.run()
  File "/home/rrr/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 26, in _worker_loop
    output = net(input)
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    r = index_queue.get()
  File "/home/rrr/anaconda/lib/python2.7/multiprocessing/queues.py", line 378, in get
    result = self.forward(*input, **kwargs)
  File "main.py", line 79, in forward
    x = gelu(self.fc1(x))
  File "main.py", line 61, in gelu
    return 0.5 * x * (1 + F.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x*x*x)))
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/autograd/variable.py", line 818, in __iter__
    return recv()
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 21, in recv
    buf = self.recv_bytes()
KeyboardInterrupt
    return iter(map(lambda i: self[i], range(self.size(0))))
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/autograd/variable.py", line 818, in <lambda>
    return iter(map(lambda i: self[i], range(self.size(0))))
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/autograd/variable.py", line 68, in __getitem__
    return Index(key)(self)
  File "/home/rrr/anaconda/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 16, in forward
    result = i.index(self.index)
KeyboardInterrupt<ECODE>

apaszke: 

Veril: Well I’m not sure what the problem is but changing np.sqrt(2 / np.pi) to its value immediately fixes the problem. So I can’t say it’s unrelated.

apaszke: I think the only fix we can do is to add scalar types to torch. We’ve been talking about that for some time now, and it’s probably going to happen, but rather in some farther future.