Repository: Theano/Theano

Issue:
Non-unit value on shape on a broadcastable dimension.
I am developing a very simple program which uses `theano.scan` function to loop through an array of vectors (in the end, I intend to develop a LSTM layer for my program). The problem is that I always get the error **Non-unit on shape of a broadcastable dimension** when compiling the function. I believe the `updates` param is the cause because as long as I put it there to compile a function, the error will occur. Here is the code:

```
import theano
import theano.tensor as T
from utils import *
import numpy as np

class LSTM:
    def __init__(self, X, in_size, out_size):
        self.X = X
        self.in_size = in_size
        self.out_size = out_size
        self.W_x = init_weights((in_size, out_size), "W_x")

        def _active(x, pre_h):
            x = T.reshape(x, (1, in_size))
            pre_h = T.dot(x, self.W_x)
            return pre_h

        h, updates = theano.scan(_active, sequences=X,
            outputs_info = [T.alloc(floatX(0.), 1, out_size)])

        self.activation = h

if __name__ == "__main__":
    X = T.matrix('X')
    in_size = 2
    out_size = 4
    lstm = LSTM(X, in_size, out_size)
    value = lstm.activation
    cost = T.mean(value)
    params = [lstm.W_x]

    updates = []
    for p in params:
        gp = T.grad(cost, p)
        updates.append((p, p - 0.1*gp))

    f = theano.function([X], outputs = cost, updates=updates)

    test = f(np.random.rand(10, in_size))
    print test
```

In the code, I use some functions loaded from utils.py, which is like this:

```
#pylint: skip-file
import numpy as np
import theano
import theano.tensor as T

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def init_weights(shape, name):
    return theano.shared(floatX(np.random.randn(*shape) * 0.1), name)

def init_gradws(shape, name):
    return theano.shared(floatX(np.zeros(shape)), name)

def init_bias(size, name):
    return theano.shared(floatX(np.zeros((size,))), name)
```

I have been searching for a while but could not find any solution for this problem. In addition, I can not see any problem with my code. If I do not use theano.scan, the code will be running just fine.

Can you see any problem in my code? Do you have any advice to solve this problem?

Thank you in advance



Discussion:
The mismatch seems to come from the default initial storage used to try and make scan work inplace.
That storage is initialized to a zero-shaped array, here an array of shape (0, 0) for a 2D variable. Then, as soon as a value is provided, then the right shape will be used.
Here, since the variable is a row (broadcastable pattern (True, False)), a shape of (0, 0) is not valid.
The solution would be to make it (1, 0) instead, I believe.
As a temporary workaround, you can try to make x and pre_h vectors instead of rows, or full matrices (replacing the 1 in reshapes with a shared variable with a value of 1).

Resolving patch:
diff --git a/theano/scan_module/scan_op.py b/theano/scan_module/scan_op.py
--- a/theano/scan_module/scan_op.py
+++ b/theano/scan_module/scan_op.py
@@ -773,8 +773,11 @@ def make_thunk(self, node, storage_map, compute_map, no_recycling,
                         # function exectution. Also, since an update is
                         # defined, a default value must also be (this is
                         # verified by DebugMode). Use an array of size 0 but
-                        # the right ndim and dtype.
-                        default_val = numpy.zeros([0] * inp.ndim,
+                        # the right ndim and dtype (use a shape of 1 on
+                        # broadcastable dimensions, 0 on the others).
+                        default_shape = [1 if _b else 0
+                                         for _b in inp.broadcastable]
+                        default_val = numpy.zeros(default_shape,
                                                   dtype=inp.dtype)
                         wrapped_inp = In(variable=inp, value=default_val,
                                          update=self.outputs[output_idx])


Test patch:
diff --git a/theano/scan_module/tests/test_scan.py b/theano/scan_module/tests/test_scan.py
--- a/theano/scan_module/tests/test_scan.py
+++ b/theano/scan_module/tests/test_scan.py
@@ -5466,3 +5466,29 @@ def test_outputs_taps_check():
     outputs_info = {'initial': y, 'taps': [-1, -1]}
     assert_raises(ValueError, theano.scan, f, x, outputs_info)
     print('done')
+
+
+def test_default_value_broadcasted():
+    def floatx(X):
+        return numpy.asarray(X, dtype=theano.config.floatX)
+
+    def init_weights(shape, name):
+        return theano.shared(floatx(numpy.random.randn(*shape) * 0.1), name)
+
+    X = theano.tensor.matrix('X')
+    in_size = 2
+    out_size = 4
+    W_x = init_weights((in_size, out_size), "W_x")
+
+    def _active(x, pre_h):
+        x = theano.tensor.reshape(x, (1, in_size))
+        pre_h = theano.tensor.dot(x, W_x)
+        return pre_h
+
+    value, scan_updates = theano.scan(_active, sequences=X,
+                                      outputs_info=[theano.tensor.alloc(floatx(0.), 1, out_size)])
+    cost = theano.tensor.mean(value)
+    gW_x = theano.tensor.grad(cost, W_x)
+    updates = [(W_x, W_x - 0.1 * gW_x)]
+    f = theano.function([X], outputs=cost, updates=updates)
+    f(numpy.random.rand(10, in_size).astype(X.dtype))