Repository: Theano/Theano

Issue:
local_useless_split considered harmful
Hi,

I noticed a slowdown in my code on the GPU between 0.7 and 0.8.  When I look at the compilation graph it turns out that some GpuFromHost nodes that should have been merged are no longer being merged.  To make a long story short (long story available on request) the reason for this is that the input to these nodes is an Assert node with a rather unhelpful message:

```
 | |   |   | | | | | |GpuFromHost [id IL] ''   254
 | |   |   | | | | | | |Assert{msg='Theano Assert failed!'} [id IM] ''   238
(a bunch of stuff...) 221
 | |   |   | | | | | |   |Elemwise{eq,no_inplace} [id IS] ''   232
 | |   |   | | | | | |     |Shape_i{1} [id IT] ''   227
 | |   |   | | | | | |     | | (my stuff)   221
 | |   |   | | | | | |     |Shape_i{1} [id IU] ''   14
 | |   |   | | | | | |       |W_uc0 [id BN]
```

As noted elsewhere, Assert nodes have the unpleasant property of blocking optimizations - and though there seems to be a local optimization that hoists them above HostFromGpu nodes, obviously we can't do that for GpuFromHost nodes.

It turs out that these Assert nodes were introduced in revision 83d458b5 from #4440 - at least, they should have a meaningful error message, but also, leaving in the "useless" Spilt nodes (which are actually the derivatives of equally "useless" Join nodes) doesn't cause a performance hit, while these Assert nodes make my code up to 40% slower!

In order to avoid creating "useless" Split nodes I have to put extra code that checks before calling theano.tensor.concatenate() that its input isn't a list of a single item.  If we still think that local_useless_split is a useful optimization, maybe there is some way to make concatenate() avoid creating useless Join/Split nodes in the first place?



Discussion:
I did a quick PR to try to fix this:
https://github.com/Theano/Theano/pull/4898

can you test it?

The commit you tell introduced the problem, I don't think it could have
caused that as it don't change how optimization work (just keep more
information for user error message). But anyway, we can just implement the
missing lift operation.

On Wed, Aug 24, 2016 at 11:36 AM, David Huggins Daines <
notifications@github.com> wrote:

> Hi,
> 
> I noticed a slowdown in my code on the GPU between 0.7 and 0.8. When I
> look at the compilation graph it turns out that some GpuFromHost nodes that
> should have been merged are no longer being merged. To make a long story
> short (long story available on request) the reason for this is that the
> input to these nodes is an Assert node with a rather unhelpful message:
> 
>  | |   |   | | | | | |GpuFromHost [id IL] ''   254
>  | |   |   | | | | | | |Assert{msg='Theano Assert failed!'} [id IM] ''   238
> (a bunch of stuff...) 221
>  | |   |   | | | | | |   |Elemwise{eq,no_inplace} [id IS] ''   232
>  | |   |   | | | | | |     |Shape_i{1} [id IT] ''   227
>  | |   |   | | | | | |     | | (my stuff)   221
>  | |   |   | | | | | |     |Shape_i{1} [id IU] ''   14
>  | |   |   | | | | | |       |W_uc0 [id BN]
> 
> As noted elsewhere, Assert nodes have the unpleasant property of blocking
> optimizations - and though there seems to be a local optimization that
> hoists them above HostFromGpu nodes, obviously we can't do that for
> GpuFromHost nodes.
> 
> It turs out that these Assert nodes were introduced in revision 83d458b
> https://github.com/Theano/Theano/commit/83d458b5d01128c5870a2bcde4a9cd7286e9a965
> from #4440 https://github.com/Theano/Theano/pull/4440 - at least, they
> should have a meaningful error message, but also, leaving in the "useless"
> Spilt nodes (which are actually the derivatives of equally "useless" Join
> nodes) doesn't cause a performance hit, while these Assert nodes make my
> code up to 40% slower!
> 
> In order to avoid creating "useless" Split nodes I have to put extra code
> that checks before calling theano.tensor.concatenate() that its input isn't
> a list of a single item. If we still think that local_useless_split is a
> useful optimization, maybe there is some way to make concatenate() avoid
> creating useless Join/Split nodes in the first place?
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/4897, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AALC-3hx7TC2T8cC8Tx4GeXLDE3B6UtJks5qjGT4gaJpZM4JsJVc
> .

Hi,

This helps a bit, but not as much as removing the Assert entirely would,
because it is still blocking the merging of some other operations higher up
(a bunch of GpuIncSubtensor, GpuAlloc, and GpuDot22).  It doesn't make
sense to continue hoisting it because those operations (GpuDot22 in
particulary) would actually get broken if the thing the Assert is testing
turns out to be false.  So my code is still 20% slower even with the fix.

Is this Assert really necessary?  In particular this first assert seems
guaranteed to be true (even though it won't look that way to
remove_useless_assert):

```
    if node.op.len_splits == 1:
        x, axis, splits = node.inputs
        out = assert_op(x, T.eq(splits.shape[0], 1))
```

Because shouldn't splits.shape[0] always be 1 if node.op.len_splits is 1?
There is a compile-time assert for this in Split()
(theano.tensor.basic:3628).

I realize that defensive programming is a good idea but I wonder whether
these checks are already handled (or could be handled) at compile time so
as to avoid mysterious optimization problems (and again, the Asserts are
not very useful since they don't even print a meaningful error messages).

On Wed, Aug 24, 2016 at 12:01 PM, Frédéric Bastien <notifications@github.com

> wrote:
> 
> I did a quick PR to try to fix this:
> https://github.com/Theano/Theano/pull/4898
> 
> can you test it?
> 
> The commit you tell introduced the problem, I don't think it could have
> caused that as it don't change how optimization work (just keep more
> information for user error message). But anyway, we can just implement the
> missing lift operation.
> 
> On Wed, Aug 24, 2016 at 11:36 AM, David Huggins Daines <
> notifications@github.com> wrote:
> 
> > Hi,
> > 
> > I noticed a slowdown in my code on the GPU between 0.7 and 0.8. When I
> > look at the compilation graph it turns out that some GpuFromHost nodes
> > that
> > should have been merged are no longer being merged. To make a long story
> > short (long story available on request) the reason for this is that the
> > input to these nodes is an Assert node with a rather unhelpful message:
> > 
> > | | | | | | | | |GpuFromHost [id IL] '' 254
> > | | | | | | | | | |Assert{msg='Theano Assert failed!'} [id IM] '' 238
> > (a bunch of stuff...) 221
> > | | | | | | | | | |Elemwise{eq,no_inplace} [id IS] '' 232
> > | | | | | | | | | |Shape_i{1} [id IT] '' 227
> > | | | | | | | | | | | (my stuff) 221
> > | | | | | | | | | |Shape_i{1} [id IU] '' 14
> > | | | | | | | | | |W_uc0 [id BN]
> > 
> > As noted elsewhere, Assert nodes have the unpleasant property of blocking
> > optimizations - and though there seems to be a local optimization that
> > hoists them above HostFromGpu nodes, obviously we can't do that for
> > GpuFromHost nodes.
> > 
> > It turs out that these Assert nodes were introduced in revision 83d458b
> > <https://github.com/Theano/Theano/commit/83d458b5d01128c5870a2bcde4a9cd
> > 7286e9a965>
> > from #4440 https://github.com/Theano/Theano/pull/4440 - at least, they
> > should have a meaningful error message, but also, leaving in the
> > "useless"
> > Spilt nodes (which are actually the derivatives of equally "useless" Join
> > nodes) doesn't cause a performance hit, while these Assert nodes make my
> > code up to 40% slower!
> > 
> > In order to avoid creating "useless" Split nodes I have to put extra code
> > that checks before calling theano.tensor.concatenate() that its input
> > isn't
> > a list of a single item. If we still think that local_useless_split is a
> > useful optimization, maybe there is some way to make concatenate() avoid
> > creating useless Join/Split nodes in the first place?
> > 
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > https://github.com/Theano/Theano/issues/4897, or mute the thread
> > <https://github.com/notifications/unsubscribe-auth/AALC-
> > 3hx7TC2T8cC8Tx4GeXLDE3B6UtJks5qjGT4gaJpZM4JsJVc>
> > .
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/4897#issuecomment-242118813, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ADK8UMlJcg2Ot4TzX1-zCu1OcqOvZd1bks5qjGrTgaJpZM4JsJVc
> .

Sorry, I just realized that the error-checking code that is redundant with these Asserts (there are two of them) is actually run-time code in Split.perform().  So I suppose they are not actually useless.

The problem is that the slowdown that is caused by adding these Assert nodes is much greater than the slowdown that would be caused by keeping the vacuous Split nodes.

Ideally, we would fix the other root problem of the slow down. Can you do
again what you did and find the part that is still on CPU? I can probably
fix it.

The assert don't cause problem to you know, but there is two cases. We
didn't removed it when it was possible, or it would trigger with some other
shapes inputs.

On Wed, Aug 24, 2016 at 2:32 PM, David Huggins Daines <
notifications@github.com> wrote:

> Sorry, I just realized that the error-checking code that is redundant with
> these Asserts (there are two of them) is actually run-time code in
> Split.perform(). So I suppose they are not actually useless.
> 
> The problem is that the slowdown that is caused by adding these Assert
> nodes is much greater than the slowdown that would be caused by keeping the
> vacuous Split nodes.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/4897#issuecomment-242164936, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AALC-9rgP73_0R5sDAdzlEvRDAZ1-oBoks5qjI45gaJpZM4JsJVc
> .

I reopen, as the PR didn't fully fix it.

Hi, sorry for not following up on this.

I think I mentioned before that I have actually worked around the problem
in my own code, like this:

if len(inputs) == 1:
    x_in = inputs[0]
else:
    x_in = T.concatenate(inputs, axis=1)

This avoids creating a useless Join node, and thus a useless Split node in
the gradient, thus there is no problem with optimizations getting blocked.

The part that is on the CPU is a custom Op that I've implemented internally
(and will probably reimplement for th GPU eventually) - but the problem
isn't that things are staying on the CPU when they could be on the GPU,
it's that the same operation is getting applied repeatedly because the
optimizer can't combine it due to the presence of an Assert.

I'm actually unsure of the way to proceed because I am not sure what the
best solution is, maybe it is reasonable to close it, I don't know.

On Wed, Aug 24, 2016 at 4:48 PM, Frédéric Bastien notifications@github.com
wrote:

> Ideally, we would fix the other root problem of the slow down. Can you do
> again what you did and find the part that is still on CPU? I can probably
> fix it.
> 
> The assert don't cause problem to you know, but there is two cases. We
> didn't removed it when it was possible, or it would trigger with some other
> shapes inputs.
> 
> On Wed, Aug 24, 2016 at 2:32 PM, David Huggins Daines <
> notifications@github.com> wrote:
> 
> > Sorry, I just realized that the error-checking code that is redundant
> > with
> > these Asserts (there are two of them) is actually run-time code in
> > Split.perform(). So I suppose they are not actually useless.
> > 
> > The problem is that the slowdown that is caused by adding these Assert
> > nodes is much greater than the slowdown that would be caused by keeping
> > the
> > vacuous Split nodes.
> > 
> > —
> > You are receiving this because you commented.
> > Reply to this email directly, view it on GitHub
> > https://github.com/Theano/Theano/issues/4897#issuecomment-242164936,
> > or mute
> > the thread
> > <https://github.com/notifications/unsubscribe-auth/AALC-9rgP73_
> > 0R5sDAdzlEvRDAZ1-oBoks5qjI45gaJpZM4JsJVc>
> > 
> > .
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/4897#issuecomment-242202958, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ADK8UH_xYjomV5L7qD85u85Z55vNwHpUks5qjK4FgaJpZM4JsJVc
> .

We started to continue to work on this. We have been able to generate one test case where we can remove the assert. But I'm not sure this would cover your cases.

The easiest way to verify that, is to send us the full debugprint of the compiled Theano function without your work around. We need to see all the code around the assert (above and bellow it). Can you give us that?

thanks


Resolving patch:
diff --git a/theano/gpuarray/opt.py b/theano/gpuarray/opt.py
--- a/theano/gpuarray/opt.py
+++ b/theano/gpuarray/opt.py
@@ -591,7 +591,7 @@ def local_gpua_alloc2(node):
         return
     if (isinstance(node.op, tensor.Alloc) and
         all(c != 'output' and
-            c.op == tensor.join and
+            isinstance(c.op, tensor.Join) and
             all(i.owner and
                 i.owner.op in [host_from_gpu, tensor.alloc]
                 for i in c.inputs[1:])
diff --git a/theano/sandbox/cuda/opt.py b/theano/sandbox/cuda/opt.py
--- a/theano/sandbox/cuda/opt.py
+++ b/theano/sandbox/cuda/opt.py
@@ -2207,7 +2207,7 @@ def local_gpualloc(node):
             # if all clients are on gpu
             replace = True
         elif all([c != 'output' and
-                  c.op == tensor.join and
+                  isinstance(c.op, tensor.Join) and
                   all(i.owner and
                       i.owner.op in [host_from_gpu, tensor.alloc]
                       for i in c.inputs[1:])
diff --git a/theano/tensor/basic.py b/theano/tensor/basic.py
--- a/theano/tensor/basic.py
+++ b/theano/tensor/basic.py
@@ -4162,9 +4162,18 @@ def infer_shape(self, node, ishapes):
         return [tuple(out_shapes)]
 
 
-"""
+join_ = Join()
+pprint.assign(Join, printing.FunctionPrinter('join'))
+
+
+def join(axis, *tensors_list):
+    """
     Convenience function to concatenate `TensorType`s along the given axis.
 
+    This function will not add the op in the graph when it is not useful.
+    For example, in the case that the list of tensors to be concatenated
+    is one, it will just return the tensor.
+
     Parameters
     ----------
     tensors : list of tensors (or list-like)
@@ -4181,12 +4190,11 @@ def infer_shape(self, node, ishapes):
         former case, the axis is fixed at construction, while in the
         latter it may vary over time depending on the value of the
         `axis` variable.
-
-"""
-
-join = Join()
-
-pprint.assign(Join, printing.FunctionPrinter('join'))
+    """
+    if len(tensors_list) == 1:
+        return tensors_list[0]
+    else:
+        return join_(axis, *tensors_list)
 
 
 def roll(x, shift, axis=None):


Test patch:
diff --git a/theano/gpuarray/tests/test_basic_ops.py b/theano/gpuarray/tests/test_basic_ops.py
--- a/theano/gpuarray/tests/test_basic_ops.py
+++ b/theano/gpuarray/tests/test_basic_ops.py
@@ -378,7 +378,7 @@ def test_gpujoin_gpualloc():
                                             T.ones_like(b)) + 4,
                              mode=mode_with_gpu)
     assert sum([node.op == T.alloc for node in f.maker.fgraph.toposort()]) == 2
-    assert sum([node.op == T.join for node in f.maker.fgraph.toposort()]) == 1
+    assert sum([node.op == T.join_ for node in f.maker.fgraph.toposort()]) == 1
     assert sum([isinstance(node.op, GpuAlloc)
                 for node in f_gpu.maker.fgraph.toposort()]) == 2
     assert sum([node.op == gpu_join
diff --git a/theano/sandbox/cuda/tests/test_basic_ops.py b/theano/sandbox/cuda/tests/test_basic_ops.py
--- a/theano/sandbox/cuda/tests/test_basic_ops.py
+++ b/theano/sandbox/cuda/tests/test_basic_ops.py
@@ -945,7 +945,7 @@ def test_gpujoin_gpualloc():
         mode=mode_with_gpu)
 
     assert sum([node.op == T.alloc for node in f.maker.fgraph.toposort()]) == 2
-    assert sum([node.op == T.join for node in f.maker.fgraph.toposort()]) == 1
+    assert sum([isinstance(node.op, T.Join) for node in f.maker.fgraph.toposort()]) == 1
     assert sum([isinstance(node.op, B.GpuAlloc)
                 for node in f_gpu.maker.fgraph.toposort()]) == 2
     assert sum([node.op == B.gpu_join
diff --git a/theano/tensor/tests/test_basic.py b/theano/tensor/tests/test_basic.py
--- a/theano/tensor/tests/test_basic.py
+++ b/theano/tensor/tests/test_basic.py
@@ -4372,6 +4372,25 @@ def test_join_inplace():
     assert numpy.allclose(f(data, 0), [3, 4, 5])
 
 
+def test_join_oneInput():
+    """Test join when only 1 input is given.
+
+    This functions tests the case when concatenate is called
+    on an array of tensors but the array has only one element.
+    In this case, we would like to avoid the computational
+    overhead of concatenation of one element.
+    """
+    x_0 = theano.tensor.fmatrix()
+    x_1 = theano.tensor.fmatrix()
+    x_2 = theano.tensor.fvector()
+    join_0 = theano.tensor.concatenate([x_0], axis=1)
+    join_1 = theano.tensor.concatenate([x_0, x_1, theano.tensor.shape_padright(x_2)],
+                                       axis=1)
+
+    assert join_0 is x_0
+    assert join_1 is not x_0
+
+
 class test_comparison(unittest.TestCase):
     """Test <, >, <=, >=, == and !=