Repository: rwth-i6/returnn

Issue:
Dim tag derived_from_tag/same_as cycle due to conv+padding
I have the following exemplary config reproducing the problem:
```
#!returnn.py
# kate: syntax python;

use_tensorflow = True

task = "initialize_model"
train = {"class": "Task12AXDataset", "num_seqs": 1000}

network = {
        "Conv1": {'class': 'conv', 'from': 'data', 'activation': None, 'with_bias': True, 'n_out': 9, 'filter_size': (4,), 'padding': 'same', 'in_spatial_dims': ['T'], 'strides': (2,)},
        "ConstantPad": {'class': 'pad', 'mode': 'constant', 'axes': ['T'], 'padding': [(1, 1)], 'from': 'Conv1', 'value': 0},
        "Conv2": {'class': 'conv', 'from': 'ConstantPad', 'activation': None, 'with_bias': True, 'n_out': 9, 'filter_size': (3,), 'padding': 'valid', 'in_spatial_dims': ['T'], 'groups': 9},
        "add": {'class': 'combine', 'kind': 'add', 'from': ['Conv1', 'Conv2']},
        "output": {"class": "softmax", "loss": "ce", "from": ["add"]},
}

log_verbosity = 5
```

This config gives the error:
```
  File "/u/dierkes/returnn/returnn/tf/util/data.py", line 870, in Dim.undefined
    line: assert id(base) not in visited  # should not have cycles. normally this should never be triggered
    locals:
      id = <builtin> <built-in function id>
      base = <local> Dim{'Conv2:conv:s0'[B]}
      visited = <local> {140193235905552: Dim{'Conv2:conv:s0'[B]}, 140193235855968: Dim{'Conv2:conv:s0'[B]}, 140193235561152: Dim{'Conv2:conv:s0'[B]}, 140193235534224: Dim{'1+Conv1:conv:s0+1'[B]}, 140193235535712: Dim{'1+Conv1:conv:s0+1'[?]}}
AssertionError
```

Full stack-trace is posted below. 

Apparently the problem is that there is a circle in the dimension ancestry of the time output dimension of `Conv2`. This is problematic in `Dim.undefined` where the problem is detected. The circle is the following: `Conv2` time dimension links to `ConstantPad` time dimension which links back to `Conv2` time dimension. I think the problem is created in when for the layer `Conv2` the function `get_out_data_from_opts` is called:
In `Conv2.get_out_data_from_opts` the function `set_output_dim_tag` is called https://github.com/rwth-i6/returnn/blob/ae42d2473e44f8bce4fe04f88b021b4d0bffea7b/returnn/tf/layers/basic.py#L5300
In the function `set_output_dim_tag` we then calculate the out tag in https://github.com/rwth-i6/returnn/blob/ae42d2473e44f8bce4fe04f88b021b4d0bffea7b/returnn/tf/layers/basic.py#L5043
and finally set the calculated output tag to be `same_as` the `out_tag` in https://github.com/rwth-i6/returnn/blob/ae42d2473e44f8bce4fe04f88b021b4d0bffea7b/returnn/tf/layers/basic.py#L5048
Apparently because of the structure of our network the calculated output tag is the time dimension of `Conv1` and then appears to be overwritten to the out tag of `Conv2`. Thus the time dimension ancestor of `ConstantPad` is linking now to `Conv2` time dimension instead of `Conv1` creating the circle.

This problem is also relevant to the issue in the pytorch-to-returnn converter https://github.com/rwth-i6/pytorch-to-returnn/issues/124

A proposed solution to the problem you can find in the PR draft #1055


Discussion:

Resolving patch:
diff --git a/returnn/tf/util/data.py b/returnn/tf/util/data.py
--- a/returnn/tf/util/data.py
+++ b/returnn/tf/util/data.py
@@ -895,7 +895,7 @@ def declare_same_as(self, other):
     assert self.can_be_used_as_dim() and other.can_be_used_as_dim()  # declare_same_as does not make sense otherwise
     self._maybe_update()
     self._validate_in_current_graph()
-    if self is other:
+    if self == other:
       return
     other_same_base = other.get_same_base()
     if self is other_same_base or self.same_as is other_same_base:
@@ -907,6 +907,12 @@ def declare_same_as(self, other):
       # We actually want it to be the other way around.
       other_same_base.declare_same_as(self_same_as)
       return
+    other_derived_bases = set(other.get_derived_bases_list())
+    self_derived_bases = set(self.get_derived_bases_list())
+    assert other_derived_bases != self_derived_bases
+    if self_derived_bases.issubset(other_derived_bases):
+      # Avoid cycles on derived_from_tag. https://github.com/rwth-i6/returnn/issues/1054
+      return other.declare_same_as(self)
     if self_same_as is not self:
       assert not self_same_as.same_as
       if self_same_as is other_same_base:


Test patch: