Repository: google/TensorNetwork

Issue:
ncon throws an obscure error for a valid contraction order
Hey, I just noticed that the following code throws an obscure error:
```python 
d = 2
D1 = 4
D2= 5
a = tf.random_uniform(shape = [D1,D2,d,d])
b = tf.random_uniform(shape = [D1,d,D1])
c = tf.random_uniform(shape = [D2,d,D2])
f = tf.random_uniform(shape = [d,d,d,d])

tn.ncon([a, b, c, f], [[1,3,5,4], [1,2,-1], [3,4,-2], [5,-4,2,-3]])
```
The order above is not chosen smartly, but `ncon` should either throw an interpretable error, or do the contraction


Discussion:
What error does it throw?
The error is thrown by `network_components.py`.

ValueError                                Traceback (most recent call last)
<ipython-input-36-ece68fe98643> in <module>
      7 f = tf.random_uniform(shape = [d,d,d,d])
      8 
----> 9 tn.ncon([a, b, c, f], [[1,3,5,4], [1,2,-1], [3,4,-2], [5,-4,2,-3]])

~/workspace/git_repos/TensorNetwork/tensornetwork/ncon_interface.py in ncon(tensors, network_structure, con_order, out_order)
     80   while con_edges:
     81     e = con_edges.pop(0)  # pop so that older nodes can be deallocated
---> 82     nodes = e.get_nodes()
     83 
     84     nodes_set = set(nodes)

~/workspace/git_repos/TensorNetwork/tensornetwork/network_components.py in get_nodes(self)
    428   def get_nodes(self) -> List[Optional[Node]]:
    429     """Get the nodes of the edge."""
--> 430     return [self.node1, self.node2]
    431 
    432   def update_axis(self, old_axis: int, old_node: Node, new_axis: int,

~/workspace/git_repos/TensorNetwork/tensornetwork/network_components.py in node1(self)
    459     val = self._node1()
    460     if val is None:
--> 461       raise ValueError("node1 for edge '{}' no longer exists.".format(self))
    462     return val
    463 

ValueError: node1 for edge '5' no longer exists.
That's strange... This seems less like a bad error message and more like a real bug. I'll investigate some more today.
I think the problem is related to the bad contraction order. `ncon` calls `contract_between` (if I'm not mistaken). In the specified order above, contraction 2 and 5 could be done in one go, but I specified 4 and 3 to be done between 2 and 5. I think `ncon` should throw an error in this case telling the user that he chose a stupid contraction order (@amilsted did you implemented something along these lines?).

Ok I see what you mean now.

Well, I think it's important to maintain `contract_between` since it gives huge speed up. So we have 2 options of what we can do.

1. Check if the edge is still in the network before trying to contract it. If it's not in the network, just skip it.

2. Throw an error saying the contraction order is wrong.

I'm leaning more towards 1 since it'll be easier to write. I'm not sure there is any advantage in forcing the user to account for our use of `contract_between`
I totally agree with maintaining `contract_between`. I also agree to have 1 in any case, since no matter how the user specifies the order, it should at least in principle be contracted (unless the order is ill defined). As for 2, it would be nice to at least throw a warning so the user is aware of his bad choice of contraction order. @amilsted mentioned at some point he had this implemented already because it is very easy to do it on the `ncon` level.  I think we can quickly fix the issue on the `ncon` level as well
There is code to provide a reasonable error message for these kinds of ordering issues, but this looks like a case I missed. Possibly this is because the tensor network objects now raise exceptions where they didn't previously. Anyway, it's easy enough to handle this in the ncon code. Will submit a PR.
Yeah that's what I think happened too. We added additional error messages to deal with our usage of weakref. Seems to have broken some assumptions in ncon. 

Resolving patch:
diff --git a/tensornetwork/ncon_interface.py b/tensornetwork/ncon_interface.py
--- a/tensornetwork/ncon_interface.py
+++ b/tensornetwork/ncon_interface.py
@@ -16,6 +16,7 @@
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
+import warnings
 from typing import Any, Sequence, List, Optional, Union, Text, Tuple, Dict, Any
 from tensornetwork import network
 from tensornetwork import network_components
@@ -73,31 +74,46 @@ def ncon(tensors: Sequence[Tensor],
   tn, con_edges, out_edges = ncon_network(
       tensors, network_structure, con_order=con_order, out_order=out_order)
 
-  # Contract assuming all edges connecting a given pair of nodes are adjacent
-  # in con_order. If this is not the case, the contraction is sub-optimal
-  # so we throw an exception.
-  prev_nodes = []
+  # Reverse the list so we can pop from the end: O(1).
+  con_edges = con_edges[::-1]
   while con_edges:
-    e = con_edges.pop(0)  # pop so that older nodes can be deallocated
-    nodes = e.get_nodes()
-
-    nodes_set = set(nodes)
-    if nodes_set != set(prev_nodes):
-      if not nodes_set.issubset(tn.nodes_set):
-        # the node pair was already contracted
-        raise ValueError("Edge '{}' is not adjacent to other edges connecting "
-                         "'{}' and '{}' in the contraction order.".format(
-                             e, nodes[0], nodes[1]))
-      if not con_edges and len(tn.nodes_set) == 2:
-        # If this already produces the final output, order the edges 
-        # here to avoid transposes in some cases.
-        tn.contract_between(
-            *nodes,
-            name="con({},{})".format(*nodes),
-            output_edge_order=out_edges)
+    nodes_to_contract = con_edges[-1].get_nodes()
+    edges_to_contract = tn.get_shared_edges(*nodes_to_contract)
+
+    # Eat up all parallel edges that are adjacent in the ordering.
+    adjacent_parallel_edges = set()
+    for edge in reversed(con_edges):
+      if edge in edges_to_contract:
+        adjacent_parallel_edges.add(edge)
       else:
-        tn.contract_between(*nodes, name="con({},{})".format(*nodes))
-      prev_nodes = nodes
+        break
+    con_edges = con_edges[:-len(adjacent_parallel_edges)]
+
+    # In an optimal ordering, all edges connecting a given pair of nodes are
+    # adjacent in con_order. If this is not the case, warn the user.
+    leftovers = edges_to_contract - adjacent_parallel_edges
+    if leftovers:
+      warnings.warn(
+        "Suboptimal ordering detected. Edges {} are not adjacent in the "
+        "contraction order to edges {}, connecting nodes {}. Deviating from "
+        "the specified ordering!".format(
+          list(map(str, leftovers)),
+          list(map(str, adjacent_parallel_edges)),
+          list(map(str, nodes_to_contract)))
+        )
+      con_edges = [e for e in con_edges if e not in edges_to_contract]
+
+    if set(nodes_to_contract) == tn.nodes_set:
+      # If this already produces the final output, order the edges
+      # here to avoid transposes in some cases.
+      tn.contract_between(
+        *nodes_to_contract,
+        name="con({},{})".format(*nodes_to_contract),
+        output_edge_order=out_edges)
+    else:
+      tn.contract_between(
+        *nodes_to_contract,
+        name="con({},{})".format(*nodes_to_contract))
 
   # TODO: More efficient ordering of products based on out_edges
   res_node = tn.outer_product_final_nodes(out_edges)


Test patch:
diff --git a/tensornetwork/ncon_interface_test.py b/tensornetwork/ncon_interface_test.py
--- a/tensornetwork/ncon_interface_test.py
+++ b/tensornetwork/ncon_interface_test.py
@@ -15,113 +15,118 @@
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
+import pytest
 import numpy as np
 import tensorflow as tf
 tf.compat.v1.enable_v2_behavior()
 from tensornetwork import ncon_interface
 
 
-class NconTest(tf.test.TestCase):
-
-  def test_sanity_check(self):
-    result = ncon_interface.ncon([tf.ones(
-        (2, 2)), tf.ones((2, 2))], [(-1, 1), (1, -2)])
-    self.assertAllClose(result, tf.ones((2, 2)) * 2)
-
-  def test_order_spec(self):
-    a = tf.ones((2, 2))
-
-    result = ncon_interface.ncon([a, a], [(-1, 1), (1, -2)], out_order=[-1, -2])
-    self.assertAllClose(result, tf.ones((2, 2)) * 2)
-
-    result = ncon_interface.ncon([a, a], [(-1, 1), (1, -2)], con_order=[1])
-    self.assertAllClose(result, tf.ones((2, 2)) * 2)
-
-    result = ncon_interface.ncon(
-        [a, a], [(-1, 1), (1, -2)], con_order=[1], out_order=[-1, -2])
-    self.assertAllClose(result, tf.ones((2, 2)) * 2)
-
-  def test_order_spec_noninteger(self):
-    a = tf.ones((2, 2))
-    result = ncon_interface.ncon(
-        [a, a], [('o1', 'i'), ('i', 'o2')],
-        con_order=['i'],
-        out_order=['o1', 'o2'])
-    self.assertAllClose(result, tf.ones((2, 2)) * 2)
-
-  def test_invalid_network(self):
-    a = tf.ones((2, 2))
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 1), (1, 2)])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 2)])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (3, 1)])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 0.1)])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 't')])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(0, 1), (1, 0)])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1,), (1, 2)])
-
-  def test_invalid_order(self):
-    a = tf.ones((2, 2))
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 1)], con_order=[2, 3])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a], [(1, 2), (2, 1)], out_order=[-1])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon(
-          [a, a], [('i1', 'i2'), ('i1', 'i2')], con_order=['i1'], out_order=[])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon(
-          [a, a], [('i1', 'i2'), ('i1', 'i2')],
-          con_order=['i1', 'i2'],
-          out_order=['i1'])
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon(
-          [a, a], [('i1', 'i2'), ('i1', 'i2')],
-          con_order=['i1', 'i1', 'i2'],
-          out_order=[])
-
-  def test_out_of_order_contraction(self):
-    a = tf.ones((2, 2, 2))
-    with self.assertRaises(ValueError):
-      ncon_interface.ncon([a, a, a], [(-1, 1, 3), (1, 3, 2), (2, -2, -3)])
-
-  def test_output_order(self):
-    a = np.random.randn(2, 2)
-    res = ncon_interface.ncon([a], [(-2, -1)])
-    self.assertAllClose(res, a.transpose())
-
-  def test_outer_product(self):
-    a = np.array([1, 2, 3])
-    b = np.array([1, 2])
-    res = ncon_interface.ncon([a, b], [(-1,), (-2,)])
-    self.assertAllClose(res, np.kron(a, b).reshape((3, 2)))
-    res = ncon_interface.ncon([a, a, a, a], [(1,), (1,), (2,), (2,)])
-    self.assertEqual(res.numpy(), 196)
-
-  def test_trace(self):
-    a = tf.ones((2, 2))
-    res = ncon_interface.ncon([a], [(1, 1)])
-    self.assertEqual(res.numpy(), 2)
-
-  def test_small_matmul(self):
-    a = np.random.randn(2, 2)
-    b = np.random.randn(2, 2)
-    res = ncon_interface.ncon([a, b], [(1, -1), (1, -2)])
-    self.assertAllClose(res, a.transpose() @ b)
-
-  def test_contraction(self):
-    a = np.random.randn(2, 2, 2)
-    res = ncon_interface.ncon([a, a, a], [(-1, 1, 2), (1, 2, 3), (3, -2, -3)])
-    res_np = a.reshape((2, 4)) @ a.reshape((4, 2)) @ a.reshape((2, 4))
-    res_np = res_np.reshape((2, 2, 2))
-    self.assertAllClose(res, res_np)
-
-
-if __name__ == '__main__':
-  tf.test.main()
+def test_sanity_check():
+  result = ncon_interface.ncon([tf.ones(
+      (2, 2)), tf.ones((2, 2))], [(-1, 1), (1, -2)])
+  np.testing.assert_allclose(result, tf.ones((2, 2)) * 2)
+
+
+def test_order_spec():
+  a = tf.ones((2, 2))
+
+  result = ncon_interface.ncon([a, a], [(-1, 1), (1, -2)], out_order=[-1, -2])
+  np.testing.assert_allclose(result, tf.ones((2, 2)) * 2)
+
+  result = ncon_interface.ncon([a, a], [(-1, 1), (1, -2)], con_order=[1])
+  np.testing.assert_allclose(result, tf.ones((2, 2)) * 2)
+
+  result = ncon_interface.ncon(
+      [a, a], [(-1, 1), (1, -2)], con_order=[1], out_order=[-1, -2])
+  np.testing.assert_allclose(result, tf.ones((2, 2)) * 2)
+
+
+def test_order_spec_noninteger():
+  a = tf.ones((2, 2))
+  result = ncon_interface.ncon(
+      [a, a], [('o1', 'i'), ('i', 'o2')],
+      con_order=['i'],
+      out_order=['o1', 'o2'])
+  np.testing.assert_allclose(result, tf.ones((2, 2)) * 2)
+
+
+def test_invalid_network():
+  a = tf.ones((2, 2))
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 1), (1, 2)])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 2)])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (3, 1)])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 0.1)])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 't')])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(0, 1), (1, 0)])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1,), (1, 2)])
+
+
+def test_invalid_order():
+  a = tf.ones((2, 2))
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 1)], con_order=[2, 3])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon([a, a], [(1, 2), (2, 1)], out_order=[-1])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon(
+        [a, a], [('i1', 'i2'), ('i1', 'i2')], con_order=['i1'], out_order=[])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon(
+        [a, a], [('i1', 'i2'), ('i1', 'i2')],
+        con_order=['i1', 'i2'],
+        out_order=['i1'])
+  with pytest.raises(ValueError):
+    ncon_interface.ncon(
+        [a, a], [('i1', 'i2'), ('i1', 'i2')],
+        con_order=['i1', 'i1', 'i2'],
+        out_order=[])
+
+
+def test_out_of_order_contraction():
+  a = tf.ones((2, 2, 2))
+  with pytest.warns(UserWarning, match='Suboptimal ordering'):
+    ncon_interface.ncon([a, a, a], [(-1, 1, 3), (1, 3, 2), (2, -2, -3)])
+
+
+def test_output_order():
+  a = np.random.randn(2, 2)
+  res = ncon_interface.ncon([a], [(-2, -1)])
+  np.testing.assert_allclose(res, a.transpose())
+
+
+def test_outer_product():
+  a = np.array([1, 2, 3])
+  b = np.array([1, 2])
+  res = ncon_interface.ncon([a, b], [(-1,), (-2,)])
+  np.testing.assert_allclose(res, np.kron(a, b).reshape((3, 2)))
+  res = ncon_interface.ncon([a, a, a, a], [(1,), (1,), (2,), (2,)])
+  assert res.numpy() == 196
+
+
+def test_trace():
+  a = tf.ones((2, 2))
+  res = ncon_interface.ncon([a], [(1, 1)])
+  assert res.numpy() == 2
+
+
+def test_small_matmul():
+  a = np.random.randn(2, 2)
+  b = np.random.randn(2, 2)
+  res = ncon_interface.ncon([a, b], [(1, -1), (1, -2)])
+  np.testing.assert_allclose(res, a.transpose() @ b)
+
+
+def test_contraction():
+  a = np.random.randn(2, 2, 2)
+  res = ncon_interface.ncon([a, a, a], [(-1, 1, 2), (1, 2, 3), (3, -2, -3)])
+  res_np = a.reshape((2, 4)) @ a.reshape((4, 2)) @ a.reshape((2, 4))
+  res_np = res_np.reshape((2, 2, 2))
+  np.testing.assert_allclose(res, res_np)