Repository: Theano/Theano

Issue:
numerical inaccuracies with new backend
Using the new backend results in significant numerical discrepancies when computing gradients over certain graphs. In the past I had observed slower training with the new backend, but had not been able to come up with a simple way to reproduce the problem until now. Most of the examples from "DeepLearningTutorials" work fine with either backend. 


To reproduce: run the following code with each different backend and look at the output.  



```
import os
device = 'gpu1'
os.environ['THEANO_FLAGS'] = "floatX=float32, device={}".format(device)


import theano
import theano.tensor as T
import numpy as np
import pickle

k = T.iscalar("k")
A = T.matrix("A")

np.random.seed(123)
u1 = np.linalg.svd(np.random.randn(500,500))[0].astype(T.config.floatX)
u2 = theano.shared(np.linalg.svd(np.random.randn(500,500))[0].astype(T.config.floatX))


result, updates = theano.scan(fn=lambda prior_result, A: T.dot(u2 +theano.map(lambda x:T.nnet.softmax(x)[0], prior_result)[0],T.tanh(T.dot(A,prior_result))),
                              outputs_info=A,
                              non_sequences=A,
                              n_steps=k)


final_result = result[-1]

power = theano.function(inputs=[A,k], outputs=T.grad(T.nnet.softmax(final_result)[0][0],u2), updates=updates)


z = power(u1,200)

f = open('test' + device + '.pickle', 'wb')
pickle.dump(np.array(z), f)
f.close()

print(np.array(z[0][:10]))


```

Also note that the results are close between device=cpu and device=gpu, but device=cuda returns different values. 

Update: Edited to remove code comments. The code is modified from the theano scan tutorial. 




Discussion:
I can only confirm. The results of the `cpu` and `gpu` differ in the 3rd or 4th digit. But with the `cuda` backend it differs already in the 1st or second digit. The `config.nvcc.fastmath`  has no effect on this. The precision (`float32` vs `float64`) has very little effect on this.
I can see there is a difference, but I cannot say if it is outside what can be expected due to different summation orders for instance.
The computation graphs look really similar, disabling cuDNN did not make any difference, nor did `alloc_empty_to_zeros`.
However, `optimizer=fast_compile` seemed to give results close to the CPU and old back-end.
`optimizer_excluding=inplace` seems to give the old results back as well.
I'll try to run with DebugMode, but since the differences are really small, I doubt it can detect anything.
If you want to push further, feel free to enable or disable inplace optimizations one by one. It seems it could be any of:
- scan
- add
- tanh
- setsubtensor
- incsubtensor
> I can see there is a difference, but I cannot say if it is outside what can be expected due to different summation orders for instance.

In this case wouldn't we expect better results when floatX=float64? 


Setting optimizer_excluding=local_inplace_setsubtensor gives the correct results. 

And I call them "correct" because they're much closer to the results obtained by device=cpu floatX=float64, which I assume is the gold standard. Given that using device=cuda produces a difference in results that's several orders of magnitude greater than device=gpu, I think it's a serious bug. 
More notes:

The problem still exists when T.tanh is removed.

There is no substantial difference in results when computing the functions output 
```
T.nnet.softmax(final_result)[0][0]
```
instead of its gradient. So the function output is correct, but the gradient is not. 


What is your cuda driver version?
Are you using the current master of Theano?

In case those are ok, there may be a problem with the code for inplace setsubtensor.
> What is your cuda driver version?

Cuda 8.0

> Are you using the current master of Theano?

 Yes
The issue appears somehow linked to the nested scan.
If I replace `theano.map(lambda x:T.nnet.softmax(x)[0], prior_result)[0]` by `T.nnet.softmax(prior_result)`, I manage to get the "good" results, even without disabling inplace optimizations.
I don't know yet if it is something related to scan that makes it unsafe to operate inplace in the IncSubtensor, or if the IncSubtensor itself is doing something wrong.
Also, disabling C code for GpuIncSubtensor and falling back on perform seems to work.
Here's another example that produces inaccuracies on the order of 10e-4 with only 20 iterations. One of the 10 printed values even has the wrong sign. 


```
import os
device = 'cuda1'
os.environ['THEANO_FLAGS'] = "floatX=float32, device={},lib.cnmem=0.0".format(device)


import theano
import theano.tensor as T
import numpy as np
import pickle

k = T.iscalar("k")
A = T.matrix("A")

np.random.seed(123)
u1 = theano.shared(np.linalg.svd(np.random.randn(100,100))[0].astype(T.config.floatX), borrow=False)
u2 = theano.shared(np.linalg.svd(np.random.randn(100,100))[0].astype(T.config.floatX), borrow=False)


result, updates = theano.scan(fn=lambda prior_result: T.dot(u2 + theano.map(lambda x:T.nnet.softmax(x)[0], prior_result)[0],T.tanh(T.dot(u1,prior_result))),
                              outputs_info=u1,
                              n_steps=k)

final_result = result[-1]

power = theano.function(inputs=[k], outputs=T.grad(T.nnet.softmax(final_result)[0][0],u1),updates=updates)

z = power(20)

f = open('test' + device + '.pickle', 'wb')
pickle.dump(np.array(z), f)
f.close()

print(np.array(z)[0][:10])

``` 
I think I got it, I'm preparing a fix.
Awesome! Glad to hear it. 

Resolving patch:
diff --git a/theano/gpuarray/subtensor.py b/theano/gpuarray/subtensor.py
--- a/theano/gpuarray/subtensor.py
+++ b/theano/gpuarray/subtensor.py
@@ -300,8 +300,9 @@ def make_view_array(self, x, view_ndim):
         size_t dims[%(view_ndim)s];
         for(int i=0; i<%(view_ndim)s; i++)
             dims[i] = xview_dims[i];
+
         zview = pygpu_fromgpudata(%(x)s->ga.data,
-                                  xview_offset,
+                                  %(x)s->ga.offset + xview_offset,
                                   %(x)s->ga.typecode,
                                   %(view_ndim)s,
                                   dims,
@@ -396,7 +397,7 @@ def c_code_cache_version(self):
         parent_version = super(GpuIncSubtensor, self).c_code_cache_version()
         if not parent_version:
             return
-        return parent_version + (8,)
+        return parent_version + (9,)
 
 
 class GpuAdvancedSubtensor1(HideC, tensor.AdvancedSubtensor1):


Test patch:
diff --git a/theano/gpuarray/tests/test_subtensor.py b/theano/gpuarray/tests/test_subtensor.py
--- a/theano/gpuarray/tests/test_subtensor.py
+++ b/theano/gpuarray/tests/test_subtensor.py
@@ -6,6 +6,7 @@
 from theano import tensor
 from theano.compile import DeepCopyOp
 from theano.tensor.tests import test_subtensor
+from theano.tests import unittest_tools as utt
 
 from ..basic_ops import HostFromGpu, GpuFromHost, GpuContiguous
 from ..elemwise import GpuDimShuffle
@@ -136,6 +137,22 @@ def test_incsub_f16():
     assert np.allclose(rval, rep)
 
 
+def test_incsub_offset():
+    # Test for https://github.com/Theano/Theano/issues/5670
+
+    # Build a GPU variable which value will have an offset (x1)
+    x = gpuarray_shared_constructor(np.zeros(5, dtype=theano.config.floatX))
+    x1 = x[1:]
+    # Use inc_subtensor on it
+    y = tensor.vector()
+    z = tensor.inc_subtensor(x1[2:], y)
+    # Use updates so that inc_subtensor can happen inplace
+    f = theano.function([y], z, updates={x: z}, mode=mode_with_gpu)
+    utt.assert_allclose(
+        f([1, 2]),
+        np.array([0, 0, 1, 2], dtype=theano.config.floatX))
+
+
 class G_advancedsubtensor(test_subtensor.TestAdvancedSubtensor):
     def shortDescription(self):
         return None