Repository: Theano/Theano

Issue:
Opt: IncSubtensor(zeros, x, ...) -> SetSubtensor(zeros, x,  ...)
See https://github.com/Theano/Theano/pull/5086/commits/1649d3eac9742347254d6de512954f3767233e56

I suppose this can stay in the final graph if there is no add.



Discussion:
I'm not convinced there is anything to gain from doing that.
Apart maybe in the case where we have both inc_subtensor and set_subtensor with the same indices, then they could be merged.

I think you didn't understand what I want by this optimization.
The goal is to transforme  IncSubtensor(zeros, x , slices/ints) to
SetSubtensor(zeros, x slices/ints)

There is no merge needed. This is a local opt on only 1 node with the
condition that the source is zeros.

This will save execution time as we won't need to read the zeros, we will
only overwrite them. I didn't saw this as a bottleneck, but I saw this in
some graph. I don't remember which graph, it was maybe tests, not real
model. This is low priority.

On Mon, Oct 17, 2016 at 11:49 PM, Pascal Lamblin notifications@github.com
wrote:

> I'm not convinced there is anything to gain from doing that.
> Apart maybe in the case where we have both inc_subtensor and set_subtensor
> with the same indices, then they could be merged.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/5094#issuecomment-254399589, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AALC-3slBOQxv4coHMoXOcIvLCZz09gJks5q1EHVgaJpZM4KWhNZ
> .

OK, let me clarify.
I don't think that reading the zeros will make a measurable difference in practice. In the inplace case, we will have to write to that memory anyway, in the no-inplace case the rest of the array will need to be copied.
However, if for some reason there is both `set_subtensor(zeros[idx], x)` and `inc_subtensor(zeros[idx], x)` in the same graph, then having a standard form may help merge them (and potentially further computation as well). I have no idea if it happens in real life, though.


Resolving patch:
diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -1,6 +1,5 @@
 from __future__ import absolute_import, print_function, division
-"""
-Tensor optimizations addressing the ops in basic.py.
+""" Tensor optimizations addressing the ops in basic.py.
 """
 # TODO: intelligent merge for mul/add
 # TODO: 0*x -> 0
@@ -3411,6 +3410,24 @@ def local_incsubtensor_of_zeros(node):
             return
 
 
+@register_canonicalize
+@register_specialize
+@gof.local_optimizer([IncSubtensor])
+def local_incsubtensor_of_zeros_to_setsubtensor(node):
+    """
+    IncSubtensor(zeros, x, ...) -> SetSubtensor(zeros, x, ...)
+    """
+    if (isinstance(node.op, (IncSubtensor)) and not node.op.set_instead_of_inc):
+        x = node.inputs[0]
+
+        if isinstance(x, T.Constant) and not np.any(x.data):
+            return [IncSubtensor(node.op.idx_list,
+                                 node.op.inplace,
+                                 set_instead_of_inc=True,
+                                 destroyhandler_tolerate_aliased=node.op.destroyhandler_tolerate_aliased,
+                                 )(*node.inputs)]
+
+
 @register_canonicalize('local_setsubtensor_of_allocs')
 @register_stabilize('local_setsubtensor_of_allocs')
 @gof.local_optimizer([IncSubtensor])
diff --git a/theano/tensor/subtensor.py b/theano/tensor/subtensor.py
--- a/theano/tensor/subtensor.py
+++ b/theano/tensor/subtensor.py
@@ -573,8 +573,9 @@ def grad(self, inputs, grads):
         else:
             # For best optimization, we let this as an inc.
             # This allow the opt local_IncSubtensor_serialize to apply first.
-            # We need to implement an optimization that will convert this to a
-            # set subtensor.
+            # We have an optimization that will convert this to a
+            # set subtensor here at:
+            # theano/tensor/opt.py:local_incsubtensor_of_zeros_to_setsubtensor()
             first = IncSubtensor(self.idx_list)(x.zeros_like(),
                                                 gz, *rest)
         return ([first] + [DisconnectedType()()] * len(rest))


Test patch:
diff --git a/theano/tensor/tests/test_opt.py b/theano/tensor/tests/test_opt.py
--- a/theano/tensor/tests/test_opt.py
+++ b/theano/tensor/tests/test_opt.py
@@ -3001,6 +3001,37 @@ def test_incsubtensor_allocs1(self):
         assert np.all([not isinstance(n.op, tensor.IncSubtensor)
                        for n in f.maker.fgraph.toposort()])
 
+    def test_incsubtensor_x_zeros(self):
+        x = tensor.constant(np.asarray(np.zeros((4, 4)),
+                                       dtype=config.floatX))
+        y = tensor.matrix()
+        z = tensor.inc_subtensor(x[:4], y)
+        f = theano.function([y], z)
+        inc_nodes = [n for n in f.maker.fgraph.toposort()
+                     if isinstance(n.op, tensor.IncSubtensor)]
+
+        assert(len(inc_nodes) == 1)
+        node_is_set_instead_of_inc = inc_nodes[0].op.set_instead_of_inc
+        mode = theano.config.mode
+        assert((mode != "FAST_COMPILE" and node_is_set_instead_of_inc) or
+               (mode == "FAST_COMPILE" and not node_is_set_instead_of_inc))
+        test_X = np.random.random((4, 4)).astype(config.floatX)
+        utt.assert_allclose(f(test_X), test_X)
+
+        # also check the flag doesn't get set if first input is not zeros:
+        not_all_zeros = np.zeros((4, 4))
+        not_all_zeros[1, 0] = 0.001
+        x = tensor.constant(np.asarray(not_all_zeros, dtype=config.floatX))
+        y = tensor.matrix()
+        z = tensor.inc_subtensor(x[:4], y)
+        f = theano.function([y], z)
+        inc_nodes = [n for n in f.maker.fgraph.toposort()
+                     if isinstance(n.op, tensor.IncSubtensor)]
+        assert(len(inc_nodes) == 1)
+        assert(inc_nodes[0].op.set_instead_of_inc is False)
+        test_X = np.random.random((4, 4)).astype(config.floatX)
+        utt.assert_allclose(f(test_X), test_X + not_all_zeros)
+
     def test_advancedincsubtensor1_allocs0(self):
         x = tensor.matrix()
         y = tensor.matrix()