Repository: Theano/Theano

Issue:
inc_subtensor of column vector
Trying to set a column vector using inc_subtensor gives an error.  E.g.:

The following code attempts to set the first two columns in a matrix and works fine:

```
import numpy, theano 
import theano.tensor as T
mat = theano.shared(numpy.zeros((10,10)))
temp = numpy.asarray(range(20))
temp.shape = (10,2)
mat_inc = theano.shared(temp)
mat = T.inc_subtensor(mat[:,[0,1]], mat_inc)
print mat.eval()
```

The following code attempts to set the first one columns in a matrix, but returns an error:

```
import numpy, theano
import theano.tensor as T
mat = theano.shared(numpy.zeros((10,10)))
temp = numpy.asarray(range(20))
temp.shape = (10,1)
mat_inc = theano.shared(temp)
mat = T.inc_subtensor(mat[:,[0]], mat_inc)
print mat.eval()
```

The following code works as expected, but isn't as general.  I'd like to use the above approach, but if I don't know which columns I want to set beforehand, it causes a problem.  

```
import numpy, theano
import theano.tensor as T
mat = theano.shared(numpy.zeros((10,10)))
temp = numpy.asarray(range(10))
mat_inc = theano.shared(temp)
mat = T.inc_subtensor(mat[:,0], mat_inc)
print mat.eval()
```



Discussion:
There is a mistake in your second code snippet. I think you wanted to give this code:

```
import numpy, theano
import theano.tensor as T
mat = theano.shared(numpy.zeros((10,10)))
#temp = numpy.asarray(range(20))
temp = numpy.asarray(range(10))
temp.shape = (10,1)
mat_inc = theano.shared(temp)
mat = T.inc_subtensor(mat[:,[0]], mat_inc)
print mat.eval()

```

This will fail with the error you probably had. I tried quickly to fix it, but today I need to work on other stuff.
```

Thanks, that is what I meant. 

On a side note, I had previously been trying to update a subset of the weight matrix, but it seems that subtensor calls are expensive computationally.  I'm still trying to wrap my head around theano and make sure I do things properly, but what are some common applications for the subtensor function?  

This will probably help you make the partial update efficient:

http://deeplearning.net/software/theano/tutorial/faq_tutorial.html?highlight=inc_subtensor#how-to-update-a-subset-of-weights

On Thu, Oct 29, 2015 at 3:19 PM, traversc notifications@github.com wrote:

> Thanks, that is what I meant.
> 
> On a side note, I had previously been trying to update a subset of the
> weight matrix, but it seems that subtensor calls are expensive
> computationally. I'm still trying to wrap my head around theano and make
> sure I do things properly, but what are some common applications for the
> subtensor function?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3575#issuecomment-152292569.

The fix would be to not do reshape/dimshuffle when we build the graph to use AdvancedSubtensor1 instead of AdvancedSubtensor. Now, on the CPU we support the full AdvancedSubtensor. We need to profile it we need to make a graph optimization to do the conversion.

If we do this change before  we implement AdvancedSubtensor on the GPU, we will need an optimization, otherwise, we need to time which version is faster.

I don't think we have time for the 0.9 release, so I bump that to 0.10 release.

This advanced, in the new back-end, we have GpuAdvancedSubtensor, we started working on GpuAdvancedIncSubtensor.
I just tried that code, and the issue now seems different now, it has to do with unexpected broadcasting patterns in DimShuffle.

Resolving patch:
diff --git a/theano/tensor/subtensor.py b/theano/tensor/subtensor.py
--- a/theano/tensor/subtensor.py
+++ b/theano/tensor/subtensor.py
@@ -1157,7 +1157,11 @@ def inc_subtensor(x, y, inplace=False, set_instead_of_inc=False,
             inplace=inplace,
             set_instead_of_inc=set_instead_of_inc,
             tolerate_inplace_aliasing=tolerate_inplace_aliasing)
-        return x.owner.op(inner_incsubtensor, *x.owner.inputs[1:])
+        # The broadcastable pattern of inner_x may not be the same as
+        # the one of x, so we have to build a new dimshuffle here,
+        # instead of reusing x.owner.op().
+        return inner_incsubtensor.dimshuffle(x.owner.op.new_order)
+
     elif isinstance(x.owner.op, theano.tensor.Reshape):
         # This case happens when the indices are not arranged as a vector, but
         # as a higher-dimensional array. This is handled by the subtensor


Test patch:
diff --git a/theano/tensor/tests/test_subtensor.py b/theano/tensor/tests/test_subtensor.py
--- a/theano/tensor/tests/test_subtensor.py
+++ b/theano/tensor/tests/test_subtensor.py
@@ -1325,6 +1325,20 @@ def test_matrix_idx(self):
         utt.assert_allclose(a2val[2], mval[2] * 3)
         utt.assert_allclose(a2val[3], mval[3] * 2)
 
+    def test_inc_bcastableidx(self):
+        idx = tensor.constant([0])
+        c_inc = tensor.col()
+        m_inc = tensor.matrix()
+        out1 = inc_subtensor(self.m[:, idx], c_inc)
+        out2 = inc_subtensor(self.m[:, idx], m_inc)
+
+        f = theano.function([self.m, c_inc, m_inc], [out1, out2])
+        mval = self.rng.random_sample((10, 5))
+        incval = self.rng.random_sample((10, 1)).astype(config.floatX)
+
+        out1val, out2val = f(mval, incval, incval)
+        utt.assert_allclose(out1val, out2val)
+
 
 inplace_increment_missing = SkipTest(
     "inc_subtensor with advanced indexing not enabled. "