Repository: Theano/Theano

Issue:
ValueError during graph optimization - "Cannot construct a ufunc with more than 32 operands (requested number were: inputs = 34 and outputs = 1)"
I'm trying to track down the cause of an error during graph optimization that I'm seeing with Theano 0.7.0. Here's some code that will reproduce it:

``` python
import numpy as np
import theano
from theano import tensor as te
from theano.gradient import jacobian, hessian


# hidden parameters to learn
# ---------------------------------------------------------------------
P = te.dvector('P')     # packed vector of model parameters
A = P[0]
B = P[1]
C = P[2:4]
sigma_min = P[4]
sigma_ratio = P[5]
theta = P[6]
sigma_maj = sigma_ratio * sigma_min

# known parameters
# ---------------------------------------------------------------------
X = te.dmatrix('X')     # [2, m], (x, y) positions of stimulus
Z = te.dvector('Z')     # [m,], response magnitudes

# bivariate Gaussian model
# ---------------------------------------------------------------------
sin2_theta = te.sin(theta) ** 2
cos2_theta = te.cos(theta) ** 2
sin_2theta = te.sin(2 * theta)
sigma2_maj = sigma_maj ** 2
sigma2_min = sigma_min ** 2
a = (cos2_theta / 2. / sigma2_maj) + (sin2_theta / 2. / sigma2_min)
b = (-sin_2theta / 4. / sigma2_maj) + (sin_2theta / 4. / sigma2_min)
c = (sin2_theta / 2. / sigma2_maj) + (cos2_theta / 2. / sigma2_min)
M = te.stack([a, b, b, c]).reshape((2, 2))
D = (X.T - C).T
Q = (D * te.dot(M, D)).sum(0)
Z_hat = A * te.exp(-Q) + B

_fit_err = Z - Z_hat
_loss = 0.5 * te.dot(_fit_err, _fit_err)   # fast sum of squares
_jac = jacobian(_loss, P)
_hess = hessian(_loss, P)

# Theano functions
# ---------------------------------------------------------------------
forward = theano.function([P, X], Z_hat)
loss = theano.function([P, X, Z], _loss)
jacobian = theano.function([P, X, Z], _jac)
hessian = theano.function([P, X, Z], _hess) #<-error disappears if this line is commented

# sample input/output
# ---------------------------------------------------------------------
P = np.r_[1., 0., 1., 2., 1., 3., np.pi / 4.]
P0 = np.random.randn(7)
X = np.mgrid[-5:5:1000j,-5:5:1000j].reshape(2, -1)
Y = forward(P, X)
L = loss(P0, X, Y)
J = jacobian(P0, X, Y)
H = hessian(P0, X, Y)
```

If I try to run this script with the line `hessian = theano.function([P, X, Z], _hess)` uncommented, I see the following optimization error:

``` python
ERROR (theano.gof.opt): SeqOptimizer apply <theano.tensor.opt.FusionOptimizer object at 0x7f9e22400d90>
ERROR (theano.gof.opt): Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/gof/opt.py", line 195, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/gof/opt.py", line 81, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5498, in apply
    new_outputs = self.optimizer(node)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5433, in local_fuse
    n = OP(C)(*inputs).owner
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/elemwise.py", line 496, in __init__
    scalar_op.nout)
ValueError: Cannot construct a ufunc with more than 32 operands (requested number were: inputs = 35 and outputs = 1)

```

Thinking it might be a version conflict, I've tried various different versions of Theano (0.6.0 and 0.7.0) and numpy (1.8.1, 1.9.1 and 1.10.0.dev0+3c06833), and hit the same error every time. Since the error seems to be specific to computing the Hessian, I've also tried implementing this myself [as described here](http://deeplearning.net/software/theano/tutorial/gradients.html#computing-the-hessian), rather than using the `theano.gradient.hessian` convenience function:

``` python
...
_jac = jacobian(_loss, P)
# _hess = hessian(_loss, P)
_hess, updates = theano.scan(lambda i, _jac, P: te.grad(_jac[i], P),
                             sequences=te.arange(_jac.shape[0]),
                             non_sequences=[_jac, P])
hessian = theano.function([P, X, Z], _hess, updates=updates)
...
```

Again, this gives me a very similar optimization error:

``` python
ERROR (theano.gof.opt): SeqOptimizer apply <theano.tensor.opt.FusionOptimizer object at 0x7f7b0c48fe50>
ERROR (theano.gof.opt): Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/gof/opt.py", line 195, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/gof/opt.py", line 81, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5498, in apply
    new_outputs = self.optimizer(node)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5445, in local_fuse
    ret = local_fuse(n)
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/opt.py", line 5433, in local_fuse
    n = OP(C)(*inputs).owner
  File "/home/alistair/.venvs/resonant_pipeline/lib/python2.7/site-packages/theano/tensor/elemwise.py", line 496, in __init__
    scalar_op.nout)
ValueError: Cannot construct a ufunc with more than 32 operands (requested number were: inputs = 35 and outputs = 1)
```

As far as I can tell this error doesn't seem to be fatal, in the sense that my `hessian()` function can still be called, and seems to give reasonable output (although I haven't checked it rigorously yet).

Any ideas about what might be causing this?

Alistair



Discussion:
We normally fuse elementwise operations together and we can support an almost unlimited (1024) number of parameters with our C code implementation.  However numpy only supports 32 inputs.

This means that you don't have a C compiler or theano didn't find it.  We can limit the fusion amount to 32 inputs for all graphs, but this might lead to a loss of performance in some cases.

I still have a PR that changes the limit to 32 so that it works with no C code.


Resolving patch:
diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -5453,7 +5453,7 @@ def check_input(inputs):
 # ###############
 # # Loop fusion #
 # ###############
-def local_elemwise_fusion_op(OP, max_input_fct=lambda node: 1024,
+def local_elemwise_fusion_op(OP, max_input_fct=lambda node: 32,
                              maker=None):
     """
     We parametrize it to make it work for Elemwise and GpuElemwise op.
@@ -5468,10 +5468,8 @@ def local_elemwise_fusion_op(OP, max_input_fct=lambda node: 1024,
                           limit how many ops we fuse together to avoid busting
                           that 256 limit.
 
-                          On the CPU we limit to 1024 input variable
-                          to the resulting fused op. This is big
-                          enough that if we hit it, I'm not sure it
-                          will affect performance.
+                          On the CPU we limit to 32 input variables
+                          since that is the maximum numpy support.
     """
     if maker is None:
         def maker(node, scalar_op):


Test patch: