Repository: Theano/Theano

Issue:
Quick python op with gradient is difficult
For pymc3 it is important to easily specify gradients for custom ops. Currently this is a little complex. 

`as_op` is very nice but can't specify gradient. Inheriting from Op is good but requires specifying `make_node` which novice users  won't understand.

Would be lovely to be able to specify `itypes` and `otypes` when inheriting from Op. 

More here:
https://groups.google.com/forum/#!searchin/theano-users/simplest$20op$20with$20gradient$20function/theano-users/ByysGhDmep8/NOFR0uE6UL4J



Discussion:
Just to tell that I think this is relatively easy to do. It do not need any
deep knoledge of Theano, but knowing python.

No time to work shorlty on this, so if someone is interrested, don,t
hesitate to see the url of the description, it tell how to implement it.

On Tue, Jun 30, 2015 at 5:11 PM, John Salvatier notifications@github.com
wrote:

> For pymc3 it is important to easily specify gradients for custom ops.
> Currently this is a little complex.
> 
> as_op is very nice but can't specify gradient. Inheriting from Op is good
> but requires specifying make_node which novice users won't understand.
> 
> Would be lovely to be able to specify itypes and otypes when inheriting
> from Op.
> 
> More here:
> 
> https://groups.google.com/forum/#!searchin/theano-users/simplest$20op$20with$20gradient$20function/theano-users/ByysGhDmep8/NOFR0uE6UL4J
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3094.

@nouiz , is someone working on this issue ? if not, can I take this ?

Nobody is working on this. You can take it. But be warned, it is more advanced then your previous issue you helped on.

Mostly, the as_op method use a class FromFunctionOp. In that case, there is a make_node implementation that use the self.itypes and self.otypes. You should move that make_node implementation in the Op class. If the class do not have itypes or otypes defined, the make_node should raise NotImplementedError().

This would need test inside theano/got/tests/test_op.py. You can base your tests on the tests for as_op().

If you didn't read that tutorial page, read it. It should give you the back-ground information on what is needed:

http://www.deeplearning.net/software/theano/tutorial/extending_theano.html

Mostly, we want to make it easier to make some type of op as as_op is doing, but with the full capacity of an Op.

thanks 

@nouiz , Will go through them and get back if i have any question. Thanks! :+1: 

@nouiz , sorry for the delayed reply. I just went through the tutorials and the files. From what i can understand, the make_node method is a mandatory method while using theano.Op, which checks if the input variables are compatible with the current op. But from what i can see, the make_node implementation here(https://github.com/Theano/Theano/blob/master/theano/gof/op.py#L503 ) just raises an error if it's not overridden. Where should I look for as_op implementation that uses self.itypes and self.otypes ?

@nouiz Ignore my previous comment. I should be checking this method (https://github.com/Theano/Theano/blob/master/theano/compile/ops.py ) , and raising the error if itypes or otypes is not defined, right ?

Yes. Move from the ops.py file the make node fct to the Op class. And raise
the the attributes aren't defined.
Le 4 déc. 2015 17:07, "Ramana Subramanyam" notifications@github.com a
écrit :

> @nouiz https://github.com/nouiz Ignore my previous comment. I should be
> checking this method (
> https://github.com/Theano/Theano/blob/master/theano/compile/ops.py ) ,
> and raising the error if itypes or otypes is not defined, right ?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3094#issuecomment-162094873.


Resolving patch:
diff --git a/theano/compile/ops.py b/theano/compile/ops.py
--- a/theano/compile/ops.py
+++ b/theano/compile/ops.py
@@ -518,16 +518,6 @@ def __hash__(self):
     def __str__(self):
         return 'FromFunctionOp{%s}' % self.__fn.__name__
 
-    def make_node(self, *inputs):
-        if len(inputs) != len(self.itypes):
-            raise ValueError("We expected %d inputs but got %d." %
-                             (len(self.itypes), len(inputs)))
-        if not all(inp.type == it for inp, it in zip(inputs, self.itypes)):
-            raise TypeError(
-                "We expected inputs of types '%s' but got types '%s' " %
-                (str([inp.type for inp in inputs]), str(self.itypes)))
-        return theano.Apply(self, inputs, [o() for o in self.otypes])
-
     def perform(self, node, inputs, outputs):
         outs = self.__fn(*inputs)
         if not isinstance(outs, (list, tuple)):
diff --git a/theano/gof/op.py b/theano/gof/op.py
--- a/theano/gof/op.py
+++ b/theano/gof/op.py
@@ -38,6 +38,7 @@ class CLinkerObject(object):
     Standard elements of an Op or Type used with the CLinker.
 
     """
+
     def c_headers(self):
         """
         Optional: Return a list of header files required by code returned by
@@ -59,7 +60,8 @@ def c_headers(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_headers", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_headers", type(self), self.__class__.__name__)
 
     def c_header_dirs(self):
         """
@@ -82,7 +84,10 @@ def c_header_dirs(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_header_dirs", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_header_dirs",
+            type(self),
+            self.__class__.__name__)
 
     def c_libraries(self):
         """
@@ -105,7 +110,8 @@ def c_libraries(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_libraries", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_libraries", type(self), self.__class__.__name__)
 
     def c_lib_dirs(self):
         """
@@ -128,7 +134,8 @@ def c_lib_dirs(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_lib_dirs", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_lib_dirs", type(self), self.__class__.__name__)
 
     def c_support_code(self):
         """
@@ -144,7 +151,10 @@ def c_support_code(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_support_code", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_support_code",
+            type(self),
+            self.__class__.__name__)
 
     def c_code_cache_version(self):
         """
@@ -182,7 +192,10 @@ def c_compile_args(self):
             Subclass does not implement this method.
 
         """
-        raise utils.MethodNotDefined("c_compile_args", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_compile_args",
+            type(self),
+            self.__class__.__name__)
 
     def c_no_compile_args(self):
         """
@@ -203,7 +216,10 @@ def c_no_compile_args(self):
             The subclass does not override this method.
 
         """
-        raise utils.MethodNotDefined("c_no_compile_args", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "c_no_compile_args",
+            type(self),
+            self.__class__.__name__)
 
     def c_init_code(self):
         """
@@ -505,14 +521,9 @@ def make_node(self, *inputs):
         Required: return an Apply instance representing the
         application of this Op to the provided inputs.
 
-        All subclasses should over-ride this function.
-
-        Raises
-        ------
-        MethodNotDefined : the subclass does not override this method.
-
         """
-        raise utils.MethodNotDefined("make_node", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "make_node", type(self), self.__class__.__name__)
 
     @classmethod
     def _get_test_value(cls, v):
@@ -542,7 +553,7 @@ def _get_test_value(cls, v):
                     "For compute_test_value, one input test value does not"
                     " have the requested type.\n")
                 tr = getattr(v.tag, 'trace', [])
-                if type(tr) is list and len(tr) > 0:
+                if isinstance(tr, list) and len(tr) > 0:
                     detailed_err_msg += (
                         " \nBacktrace when that variable is created:\n")
                     # Print separate message for each element in the list
@@ -612,10 +623,14 @@ def __call__(self, *inputs, **kwargs):
                 except AttributeError:
                     # no test-value was specified, act accordingly
                     if config.compute_test_value == 'warn':
-                        warnings.warn('Warning, Cannot compute test value: input %i (%s) of Op %s missing default value' % (i, ins, node), stacklevel=2)
+                        warnings.warn(
+                            'Warning, Cannot compute test value: input %i (%s) of Op %s missing default value' %
+                            (i, ins, node), stacklevel=2)
                         run_perform = False
                     elif config.compute_test_value == 'raise':
-                        raise ValueError('Cannot compute test value: input %i (%s) of Op %s missing default value' % (i, ins, node))
+                        raise ValueError(
+                            'Cannot compute test value: input %i (%s) of Op %s missing default value' %
+                            (i, ins, node))
                     elif config.compute_test_value == 'ignore':
                         # silently skip test
                         run_perform = False
@@ -623,7 +638,9 @@ def __call__(self, *inputs, **kwargs):
                         import pdb
                         pdb.post_mortem(sys.exc_info()[2])
                     else:
-                        raise ValueError('%s is invalid for option config.compute_Test_value' % config.compute_test_value)
+                        raise ValueError(
+                            '%s is invalid for option config.compute_Test_value' %
+                            config.compute_test_value)
 
             # if all inputs have test-values, run the actual op
             if run_perform:
@@ -653,7 +670,8 @@ def __call__(self, *inputs, **kwargs):
 
                 for output in node.outputs:
                     # Check that the output has been computed
-                    assert compute_map[output][0], (output, storage_map[output][0])
+                    assert compute_map[output][
+                        0], (output, storage_map[output][0])
 
                     # add 'test_value' to output tag, so that downstream ops can use these
                     # numerical values as inputs to their perform method.
@@ -714,7 +732,7 @@ def R_op(self, inputs, eval_points):
             "own op, implement the R_op method." %
             (self, self.__class__.__name__))
 
-    def perform(self, node, inputs, output_storage):
+    def perform(self, node, inputs, output_storage, params=None):
         """
         Required: Calculate the function on the inputs and put the variables in
         the output storage. Return None.
@@ -746,7 +764,10 @@ def perform(self, node, inputs, output_storage):
             The subclass does not override this method.
 
         """
-        raise utils.MethodNotDefined("perform", type(self), self.__class__.__name__)
+        raise utils.MethodNotDefined(
+            "perform", type(self), self.__class__.__name__,
+            "Did you used Theano flags mode=FAST_COMPILE?"
+            " You can use optimizer=fast_compile instead.")
 
     def do_constant_folding(self, node):
         """
@@ -810,7 +831,8 @@ def __str__(self):
 
     def __eq__(self, other):
         if hasattr(self, '__props__'):
-            return (type(self) == type(other) and self._props() == other._props())
+            return (type(self) == type(other) and self._props() ==
+                    other._props())
         else:
             return NotImplemented
 
@@ -933,6 +955,25 @@ def make_thunk(self, node, storage_map, compute_map, no_recycling):
         # condition: either there was no c_code, or it failed
         return self.make_py_thunk(node, storage_map, compute_map, no_recycling)
 
+    def make_node(self, *inputs):
+
+        if not hasattr(self, 'itypes'):
+            raise NotImplementedError("You can either define itypes and otypes,\
+             or implement make_node")
+
+        if not hasattr(self, 'otypes'):
+            raise NotImplementedError("You can either define itypes and otypes,\
+             or implement make_node")
+
+        if len(inputs) != len(self.itypes):
+            raise ValueError("We expected %d inputs but got %d." %
+                             (len(self.itypes), len(inputs)))
+        if not all(inp.type == it for inp, it in zip(inputs, self.itypes)):
+            raise TypeError(
+                "We expected inputs of types '%s' but got types '%s' " %
+                (str([inp.type for inp in inputs]), str(self.itypes)))
+        return theano.Apply(self, inputs, [o() for o in self.otypes])
+
 
 def get_test_value(v):
     """
@@ -1203,7 +1244,9 @@ class COp(Op):
     """
 
     section_re = re.compile(r'^#section ([a-zA-Z0-9_]+)$', re.MULTILINE)
-    backward_re = re.compile(r'^THEANO_(APPLY|SUPPORT)_CODE_SECTION$', re.MULTILINE)
+    backward_re = re.compile(
+        r'^THEANO_(APPLY|SUPPORT)_CODE_SECTION$',
+        re.MULTILINE)
     # This is the set of allowed markers
     SECTIONS = set([
         'init_code', 'init_code_apply', 'init_code_struct',
@@ -1303,8 +1346,9 @@ def load_c_code(self):
                 n = 1
                 while n < len(split):
                     if split[n] not in self.SECTIONS:
-                        raise ValueError("Unknown section type (in file %s): %s" %
-                                         (self.func_files[i], split[n]))
+                        raise ValueError(
+                            "Unknown section type (in file %s): %s" %
+                            (self.func_files[i], split[n]))
                     if split[n] not in self.code_sections:
                         self.code_sections[split[n]] = ""
                     self.code_sections[split[n]] += split[n + 1]
@@ -1372,7 +1416,9 @@ def get_c_macros(self, node, name, check_input=None):
                 macro_name = "DTYPE_" + vname
                 macro_value = "npy_" + v.dtype
 
-                define_macros.append(define_template % (macro_name, macro_value))
+                define_macros.append(
+                    define_template %
+                    (macro_name, macro_value))
                 undef_macros.append(undef_template % macro_name)
 
                 d = numpy.dtype(v.dtype)
@@ -1380,13 +1426,17 @@ def get_c_macros(self, node, name, check_input=None):
                 macro_name = "TYPENUM_" + vname
                 macro_value = d.num
 
-                define_macros.append(define_template % (macro_name, macro_value))
+                define_macros.append(
+                    define_template %
+                    (macro_name, macro_value))
                 undef_macros.append(undef_template % macro_name)
 
                 macro_name = "ITEMSIZE_" + vname
                 macro_value = d.itemsize
 
-                define_macros.append(define_template % (macro_name, macro_value))
+                define_macros.append(
+                    define_template %
+                    (macro_name, macro_value))
                 undef_macros.append(undef_template % macro_name)
 
         # Generate a macro to mark code as being apply-specific


Test patch:
diff --git a/theano/compile/tests/test_ops.py b/theano/compile/tests/test_ops.py
--- a/theano/compile/tests/test_ops.py
+++ b/theano/compile/tests/test_ops.py
@@ -16,7 +16,7 @@
 # reachable from pickle (as in it cannot be defined inline)
 @as_op([dmatrix, dmatrix], dmatrix)
 def mul(a, b):
-    return a*b
+    return a * b
 
 
 class OpDecoratorTests(utt.InferShapeTester):
diff --git a/theano/gof/tests/test_op.py b/theano/gof/tests/test_op.py
--- a/theano/gof/tests/test_op.py
+++ b/theano/gof/tests/test_op.py
@@ -156,6 +156,7 @@ def test_op_struct(self):
 
 
 class TestMakeThunk(unittest.TestCase):
+
     def test_no_c_code(self):
         class IncOnePython(Op):
             """An Op with only a Python (perform) implementation"""
@@ -233,6 +234,26 @@ def c_code(self, node, name, inputs, outputs, sub):
             self.assertRaises((NotImplementedError, utils.MethodNotDefined),
                               thunk)
 
+    def test_no_make_node(self):
+        class DoubleOp(Op):
+            """An Op without make_node"""
+
+            __props__ = ()
+
+            itypes = [T.dmatrix]
+            otypes = [T.dmatrix]
+
+            def perform(self, node, inputs, outputs):
+                inp = inputs[0]
+                output = outputs[0]
+                output[0] = inp * 2
+
+        x_input = T.dmatrix('x_input')
+        f = theano.function([x_input], DoubleOp()(x_input))
+        inp = numpy.random.rand(5, 4)
+        out = f(inp)
+        assert numpy.allclose(inp * 2, out)
+
 
 def test_test_value_python_objects():
     for x in ([0, 1, 2], 0, 0.5, 1):