Repository: numpy/numpy

Issue:
PR #10559  failed to fix einsum (optimize=True) broadcasting bug
#10559 introduced the [following code](https://github.com/numpy/numpy/blob/a9e9343ecf5a6564eb09f1d7078c407db2ff7d9e/numpy/core/einsumfunc.py#L1107) to prevent dispatching `numpy.tensordot` in a case where einsum was broadcasting over a singleton dimension.
```python
        # Handle broadcasting vs BLAS cases
        if blas:
            # Checks have already been handled
            input_str, results_index = einsum_str.split('->')
            input_left, input_right = input_str.split(',')
            if 1 in tmp_operands[0] or 1 in tmp_operands[1]:
                left_dims = {dim: size for dim, size in
                             zip(input_left, tmp_operands[0].shape)}
                right_dims = {dim: size for dim, size in
                              zip(input_right, tmp_operands[1].shape)}
                # If dims do not match we are broadcasting, BLAS off
                if any(left_dims[ind] != right_dims[ind] for ind in idx_rm):
                    blas = False
```
However, this checks to see if `1` occurs within the operand array itself rather than the shape of the operand. Incidentally, this likely produced a nasty performance regression.

Thus the line

`if 1 in tmp_operands[0] or 1 in tmp_operands[1]` 

should be 

`if 1 in tmp_operands[0].shape or 1 in tmp_operands[1].shape`

This wasn't caught by the [unit test](https://github.com/numpy/numpy/blob/a9e9343ecf5a6564eb09f1d7078c407db2ff7d9e/numpy/core/tests/test_einsum.py#L486) because arrays of ones were used 🌌

This leads to the following behavior:

```python
>>> x = np.array([0., 1., 0.])  # contains 1, no blas
>>> y = np.array([0.0])
>>> np.einsum("i,i", x, y, optimize=True)
0.
```

```python
>>> x = np.array([0., -1., 0.])  # doesn't contain 1, yes blas
>>> y = np.array([0.0])
>>> np.einsum("i,i", x, y, optimize=True)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-184-b0dcea8eedea> in <module>()
      1 x = np.array([0., -1., 0.])
      2 y = np.array([0.0])
----> 3 np.einsum("i,i", x, y, optimize=True)

c:\anaconda\envs\py36\lib\site-packages\numpy\core\einsumfunc.py in einsum(*operands, **kwargs)
   1132 
   1133             # Contract!
-> 1134             new_view = tensordot(*tmp_operands, axes=(tuple(left_pos), tuple(right_pos)))
   1135 
   1136             # Build a new view if needed

c:\anaconda\envs\py36\lib\site-packages\numpy\core\numeric.py in tensordot(a, b, axes)
   1281                 axes_b[k] += ndb
   1282     if not equal:
-> 1283         raise ValueError("shape-mismatch for sum")
   1284 
   1285     # Move the axes to sum over to the end of "a"

ValueError: shape-mismatch for sum
```




Discussion:
Ouch, that is definitely a bug- can you patch this?

In addition I noticed that the fix doesn't allow GEMM's for non-broadcasting cases which is a legitimate use. I patched this in `opt_einsum` itself in a very different [manner](https://github.com/dgasmith/opt_einsum/commit/4dbe46eede0fbc5d4e6d4ce11b732cef9cb1d100). I should likely port this over when I get a chance.
Excuse me, why numpy arrows `np.einsum("i,i", x, y, optimize=True)` when dimension of x and y is different?

In [document,](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.einsum.html) `np.einsum('i,i', a, b) is equivalent to np.inner(a,b)`.
But np.inner disallows np.inner(x, y) when dimension of x and y.

I thought that same labels means same label dimension.

For example,
```
>>> numpy.einsum('ii', numpy.ones((3, 1)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ryo/workspace/chainer-env/lib/python3.5/site-packages/numpy/core/einsumfunc.py", line 1069, in einsum
    return c_einsum(*operands, **kwargs)
ValueError: dimensions in operand 0 for collapsing index 'i' don't match (3 != 1)
```
This error message apparently seems to indicate that the number of dimensions should be the same.
Yes, for inner products the dimensions of both arrays should be the same. It is unclear what this operation should be for different sized arrays. Both `einsum` and `inner` bounce inputs like this.

The shown error message discusses the size of individual dimensions not the number of dimensions? Sorry not entirely sure what the second example is getting at there.
> Both einsum and inner bounce inputs like this.

But np.inner raises error, einsum does not raise error?

```
>>> np.einsum("i, i", np.ones(1), np.ones(3))
3.0
>>> np.inner(np.ones(1), np.ones(3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: shapes (1,) and (3,) not aligned: 1 (dim 0) != 3 (dim 0)
```
IIUC, einsum broadcasts inputs, try ones(2) instead of ones(1).

@dgasmith So this still needs a fix?
The original issue still needs a fix. There are two ways to approach this 1) fix the typo as shown in the script 2) use a more comprehensive solution that is more flexible in what is permissible via BLAS (implemented in `opt_einsum`). I would like to do 2), but it is quite apparent that the many uses of `einsum` are quite extreme and not always documented so I am bit hesitant to do 2) without some extreme testing.
@dgasmith I'm sorry for dropping off the map on this. I haven't had time to circle back. It sounds like the best mode of operation is to do the typo-fix, since that is pretty trivial and does resolve the issue, and then eventually bring in your additional logic to enable BLAS (and hopefully add to the documentation).

I can try to get around to the typo fix this evening.

Regarding unit tests, I originally caught this issue via my unit tests for my [autograd library](https://github.com/rsokl/MyGrad/blob/master/tests/linalg/test_einsum.py). These test both broadcasting and non-broadcasting operations. Currently, they exclude the optimization flag (because of the present issue), but it can be added easily.

Please feel free to leverage these if they would be any help to you.
@rsokl Great, thanks for fixing the current issue. If you have a moment can you add a tests that fails under the current code? It would be great to continue to cover previous issues in the tests so that we do not regress.

It is probably worth fixing the current issue and rerunning your code with the optimize flag on to see if thats clean. If not please do make a new issue with a code snippet for that as well!

Resolving patch:
diff --git a/numpy/core/einsumfunc.py b/numpy/core/einsumfunc.py
--- a/numpy/core/einsumfunc.py
+++ b/numpy/core/einsumfunc.py
@@ -1109,7 +1109,7 @@ def einsum(*operands, **kwargs):
             # Checks have already been handled
             input_str, results_index = einsum_str.split('->')
             input_left, input_right = input_str.split(',')
-            if 1 in tmp_operands[0] or 1 in tmp_operands[1]:
+            if 1 in tmp_operands[0].shape or 1 in tmp_operands[1].shape:
                 left_dims = {dim: size for dim, size in
                              zip(input_left, tmp_operands[0].shape)}
                 right_dims = {dim: size for dim, size in


Test patch:
diff --git a/numpy/core/tests/test_einsum.py b/numpy/core/tests/test_einsum.py
--- a/numpy/core/tests/test_einsum.py
+++ b/numpy/core/tests/test_einsum.py
@@ -489,8 +489,16 @@ def check_einsum_sums(self, dtype, do_opt=False):
         assert_array_equal(np.einsum('ij,ij->j', p, q, optimize=True),
                            [10.] * 2)
 
-        p = np.ones((1, 5))
-        q = np.ones((5, 5))
+        # a blas-compatible contraction broadcasting case which was failing
+        # for optimize=True (ticket #10930)
+        x = np.array([2., 3.])
+        y = np.array([4.])
+        assert_array_equal(np.einsum("i, i", x, y, optimize=False), 20.)
+        assert_array_equal(np.einsum("i, i", x, y, optimize=True), 20.)
+
+        # all-ones array was bypassing bug (ticket #10930)
+        p = np.ones((1, 5)) / 2
+        q = np.ones((5, 5)) / 2
         for optimize in (True, False):
             assert_array_equal(np.einsum("...ij,...jk->...ik", p, p,
                                          optimize=optimize),
@@ -498,7 +506,7 @@ def check_einsum_sums(self, dtype, do_opt=False):
                                          optimize=optimize))
             assert_array_equal(np.einsum("...ij,...jk->...ik", p, q,
                                          optimize=optimize),
-                               np.full((1, 5), 5))
+                               np.full((1, 5), 1.25))
 
     def test_einsum_sums_int8(self):
         self.check_einsum_sums('i1')