Repository: rwth-i6/returnn

Issue:
Invalid argument: ConcatOp : Dimensions of inputs should match
```
TensorFlow exception: 2 root error(s) found.
  (0) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [19,20,35,1024] vs. shape[2] = [19,8,35,1024]
 [[node output/rec/concat_am_att_lm_masked/concat_sources/concat (defined at /setups/combined/2021-05-31/tools/returnn/returnn/tf/layers/basic.py:122) ]]
 [[objective/loss/loss/Sum/_599]]
  (1) Invalid argument: ConcatOp : Dimensions of inputs should match: shape[0] = [19,20,35,1024] vs. shape[2] = [19,8,35,1024]
 [[node output/rec/concat_am_att_lm_masked/concat_sources/concat (defined at /setups/combined/2021-05-31/tools/returnn/returnn/tf/layers/basic.py:122) ]]
```
[Full log and partial config is here.](https://gist.github.com/albertz/db6847a3a5032addfca30801015a8bda)
I don't really know what the problem is yet.

In the feed dict, the `Neg_1` looks suspicious:
```
Feed dict:
  <tf.Tensor 'Neg_1:0' shape=(?,) dtype=int32>: shape (35,), dtype int32, min/max 1/7, ([1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 2 1 1 1 7 1 1 1 1 1 1])
  <tf.Tensor 'extern_data/placeholders/batch_dim:0' shape=() dtype=int32>: int(35)
  <tf.Tensor 'extern_data/placeholders/data/data:0' shape=(?, ?, 40) dtype=float32>: shape (35, 112, 40), dtype float32, min/max -4.506656/4.565903, mean/stddev -5.71718e-10/0.9186281, Data{'data', [B,T|'time'[B],F|F'audio'(40)]}
  <tf.Tensor 'extern_data/placeholders/data/data_dim0_size:0' shape=(?,) dtype=int32>: shape (35,), dtype int32, min/max 53/112, ([ 61  67  72  81  83  54  90  91  76  93  95  96  97  98  99 100  93 101
 101 102 103 104 104 105 105  53 107 104 108 109 110 111 111 112 112])
  <tf.Tensor 'extern_data/placeholders/orth_classes/orth_classes:0' shape=(?, ?) dtype=int32>: shape (35, 7), dtype int32, min/max 0/444, Data{'orth_classes', [B,T|'out-spatial'[B]], dtype='int32', sparse_dim=Dim{F'vocab'(1030)}, available_for_inference=False}
  <tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>: bool(True)
```
This could be some dim tag mess up.

In another run, I also saw this:
```
NativeOp assertion failed: Ndarray_DIMS(X)[0] == T, 8 == 20
NativeOp assertion failed: Ndarray_DIMS(Y)[0] == T, 8 == 20
NativeOp assertion failed: Ndarray_DIMS(C)[0] == T, 8 == 20
NativeOp assertion failed: Ndarray_DIMS(H)[0] == T, 8 == 20
```
And:
```
...
layer /output(rec-subnet-output)/'label_emit_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'la
bel_log_prob:feature-dense'(1030)] float32
layer /output(rec-subnet-output)/'output_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'(emit_
prob0:feature-dense)+(label_log_prob:feature-dense)'(1031)] float32
layer /output(rec-subnet-output)/'rna_alignment': [B,T|'lstm0_pool:conv:s0'[B]] int32 sparse_dim=Dim{F'rna_align
ment:sparse-dim'(1031)}
layer /output(rec-subnet-output)/'output_prob': [B] float32
Warning: assuming dim tags are same with different size placeholders: <tf.Tensor 'lstm0_pool/Neg_1:0' shape=(?,)
 dtype=int32> vs <tf.Tensor 'extern_data/placeholders/orth_classes/orth_classes_dim0_size:0' shape=(?,) dtype=in
t32>
Warning: assuming dim tags are same with different size placeholders: <tf.Tensor 'lstm0_pool/Neg_1:0' shape=(?,)
 dtype=int32> vs <tf.Tensor 'extern_data/placeholders/orth_classes/orth_classes_dim0_size:0' shape=(?,) dtype=in
t32>
```
Net construction log:
```
DEPRECATION WARNING: Missing "from" in layer definition: /source
This will be disallowed with behavior_version 1.
layer /'data': [B,T|'time'[B],F|F'audio'(40)] float32
layer /'source': [B,T|'time'[B],F|F'audio'(40)] float32
WARNING:tensorflow:From /u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/network.py:2153: calling Ze
ros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future vers
ion.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
layer /'source0': [B,T|'time'[B],F'audio'(40),F|F'source0_split_dims1'(1)] float32]         32 segments ~  2min
DEPRECATION WARNING: Explicitly specify in_spatial_dims when there is more than one spatial dim in the input.
This will be disallowed with behavior_version 8.
layer /'conv0': [B,T|'time'[B],F'audio'(40),F|F'conv0:channel'(32)] float32
layer /'conv0p': [B,T|'time'[B],'conv0p:conv:s1'(20),F|F'conv0:channel'(32)] float32
layer /'conv1': [B,T|'time'[B],'conv0p:conv:s1'(20),F|F'conv1:channel'(32)] float32
layer /'conv1p': [B,T|'time'[B],'conv1p:conv:s1'(10),F|F'conv1:channel'(32)] float32
layer /'conv_merged': [B,T|'time'[B],F|F'conv1p:conv:s1*conv1:channel'(320)] float32
DEPRECATION WARNING: MergeDimsLayer, only keep_order=True is allowed
This will be disallowed with behavior_version 6.
layer /'lstm0_fw': [T|'time'[B],B,F|F'lstm0_fw:feature'(512)] float32
WARNING:tensorflow:From /u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/util/basic.py:1416: calling
 VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in 
a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
layer /'lstm0_bw': [T|'time'[B],B,F|F'lstm0_bw:feature'(512)] float32             ]         76 segments ~ 12min
layer /'lstm0_pool': [B,T|'lstm0_pool:conv:s0'[B],F|F'lstm0_fw:feature+lstm0_bw:feature'(1024)] float32
layer /'lstm1_fw': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature'(512)] float32
layer /'lstm1_bw': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_bw:feature'(512)] float32
layer /'encoder0': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)] float32
layer /'encoder': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)] float32
layer /'enc_ctx0': [T|'lstm0_pool:conv:s0'[B],B,F|F'enc_ctx0:feature-dense'(200)] float32
layer /'enc_ctx_win': [B,T|'lstm0_pool:conv:s0'[B],'enc_ctx_win:window'(5),F|F'enc_ctx0:feature-dense'(200)] flo
at32
DEPRECATION WARNING: Do not specify axis or axes in a way that depends on the order of the axes.
This will be disallowed with behavior_version 7.
layer /'enc_val': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)] float32
layer /'enc_val_win': [B,T|'lstm0_pool:conv:s0'[B],'enc_val_win:window'(5),F|F'lstm1_fw:feature+lstm1_bw:feature
'(1024)] float32
layer /'data:orth_classes': [B,T|'out-spatial'[B]] int32 sparse_dim=Dim{F'vocab'(1030)}
layer /'lm_input0': [B,T|'out-spatial'[B]] int32 sparse_dim=Dim{F'vocab'(1030)}
layer /'lm_input1': [B,T|'1+(out-spatial)'[?]] int32 sparse_dim=Dim{F'vocab'(1030)}
layer /'lm_input': [B,T|'1+(out-spatial)'[B]] int32 sparse_dim=Dim{F'vocab'(1030)}
layer /'output': [T|'lstm0_pool:conv:s0'[B],B] int32 sparse_dim=Dim{F'vocab'(1030)}
Rec layer 'output' (search False, train 'globals/train_flag:0') sub net:
  Input layers moved out of loop: (#: 0)
    None
  Output layers moved out of loop: (#: 24)
    output_prob
    output_emit
    output_is_not_blank
    output
    rna_alignment
    output_log_prob
    blank_log_prob
    label_emit_log_prob
    label_log_prob
    emit_log_prob
    emit_prob0
    readout
    readout_in
    att
    att_weights
    att_weights1
    att_weights0
    att_energy
    enc_ctx_win
    att_query
    am
    enc_val_win
    :i
    lm_masked
  Layers in loop: (#: 0)
    None
  Unused layers: (#: 5)
    const0
    const1
    lm
    out_str
    prev_out_non_blank
layer /output(rec-subnet-output)/'data:source': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:featu
re'(1024)] float32
layer /output(rec-subnet-output)/'am': [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)
] float32
layer /output(rec-subnet-output)/':i': [T|'lstm0_pool:conv:s0'[B]] int32
layer /output(rec-subnet-output)/'enc_ctx_win': [B,T|'lstm0_pool:conv:s0'[B],'enc_ctx_win:window'(5),F|F'enc_ctx
0:feature-dense'(200)] float32
layer /output(rec-subnet-output)/'att_query': [T|'lstm0_pool:conv:s0'[B],B,F|F'att_query:feature-dense'(200)] fl
oat32
layer /output(rec-subnet-output)/'att_energy': [B,T|'lstm0_pool:conv:s0'[B],'enc_ctx_win:window'(5),F|'att_energ
y:dot:dummy-var2'(1)] float32
layer /output(rec-subnet-output)/'att_weights0': [B,T|'lstm0_pool:conv:s0'[B],F|'att_energy:dot:dummy-var2'(1),'
enc_ctx_win:window'(5)] float32
layer /output(rec-subnet-output)/'att_weights1': [B,T|'lstm0_pool:conv:s0'[B],F|'att_energy:dot:dummy-var2'(1),'
enc_ctx_win:window'(5)] float32
layer /output(rec-subnet-output)/'att_weights': [B,T|'lstm0_pool:conv:s0'[B],F|'(att_energy:dot:dummy-var2)*enc_
ctx_win:window'(5)] float32
layer /output(rec-subnet-output)/'enc_val_win': [B,T|'lstm0_pool:conv:s0'[B],'enc_val_win:window'(5),F|F'lstm1_f
w:feature+lstm1_bw:feature'(1024)] float32
layer /output(rec-subnet-output)/'att': [B,T|'lstm0_pool:conv:s0'[B],F|F'lstm1_fw:feature+lstm1_bw:feature'(1024
)] float32
layer /output(rec-subnet-output)/'lm_masked': [T|'1+(out-spatial)'[B],B,F|F'lstm0:feature'(1024)] float32
layer /output(rec-subnet-output)(extra._internal.masked(lm_masked))/lm_masked/'input_embed': [B,T|'1+(out-spatia
l)'[B],F|F'input_embed:feature-dense'(256)] float32
layer /output(rec-subnet-output)(extra._internal.masked(lm_masked))/lm_masked/'lstm0': [T|'1+(out-spatial)'[B],B
,F|F'lstm0:feature'(1024)] float32
layer /output(rec-subnet-output)(extra._internal.masked(lm_masked))/lm_masked/'output': [T|'1+(out-spatial)'[B],
B,F|F'lstm0:feature'(1024)] float32
layer /output(rec-subnet-output)/'readout_in': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'feature:rea
dout_in_output'(1000)] float32
DEPRECATION WARNING: All inputs
 - Data{'am_output', [T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)]}
 - Data{'att_output', [B,T|'lstm0_pool:conv:s0'[B],F|F'lstm1_fw:feature+lstm1_bw:feature'(1024)]}
 - Data{'lm_masked_output', [T|'1+(out-spatial)'[B],B,F|F'lstm0:feature'(1024)]}
require broadcasting to 
  Data{'am_output', [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'lstm1_fw:feature+lstm1_bw:feature'(102
4)]}.
This must be explicitly allowed, e.g. by specifying out_shape.
This will be disallowed with behavior_version 4.
layer /output(rec-subnet-output)/'readout': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'feature:readou
t_in_output//2'(500)] float32
layer /output(rec-subnet-output)/'emit_prob0': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'emit_prob0:
feature-dense'(1)] float32
layer /output(rec-subnet-output)/'data:orth_classes': [B,T|'out-spatial'[B]] int32 sparse_dim=Dim{F'vocab'(1030)
}
layer /output(rec-subnet-output)/'output': [B,T|'out-spatial'[B]] int32 sparse_dim=Dim{F'vocab'(1030)}
layer /output(rec-subnet-output)/'output_is_not_blank': [B,T|'out-spatial'[B]] bool
layer /output(rec-subnet-output)/'output_emit': [B,T|'out-spatial'[B]] bool
layer /output(rec-subnet-output)/'blank_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'emit_pr
ob0:feature-dense'(1)] float32
layer /output(rec-subnet-output)/'label_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'label_l
og_prob:feature-dense'(1030)] float32
layer /output(rec-subnet-output)/'emit_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'emit_pro
b0:feature-dense'(1)] float32
layer /output(rec-subnet-output)/'label_emit_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'la
bel_log_prob:feature-dense'(1030)] float32
layer /output(rec-subnet-output)/'output_log_prob': [T|'lstm0_pool:conv:s0'[B],'1+(out-spatial)'[B],B,F|F'(emit_
prob0:feature-dense)+(label_log_prob:feature-dense)'(1031)] float32
layer /output(rec-subnet-output)/'rna_alignment': [B,T|'lstm0_pool:conv:s0'[B]] int32 sparse_dim=Dim{F'rna_align
ment:sparse-dim'(1031)}
layer /output(rec-subnet-output)/'output_prob': [B] float32
Exception creating layer /'output' of class RecLayer with opts:
{'_name': 'output',
 '_network': <TFNetwork '' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'axis': Dim{'lstm0_pool:conv:s0'[B]},
 'back_prop': True,
 'include_eos': True,
 'n_out': <class 'returnn.util.basic.NotSpecified'>,
 'name': 'output',
 'network': <TFNetwork '' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'output': Data{'output_output', [T|'lstm0_pool:conv:s0'[B],B], dtype='int32', sparse_dim=Dim{F'vocab'(1030)}},
 'sources': [<CopyLayer 'encoder' out_type=Data{[T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature+lstm1_bw:featu
re'(1024)]}>],
 'unit': <_SubnetworkRecCell '/output(rec-subnet)'>}
```



Discussion:
This last warning looks very suspicious:
```
Warning: assuming dim tags are same with different size placeholders: <tf.Tensor 'lstm0_pool/Neg_1:0' shape=(?,) dtype=int32> vs <tf.Tensor 'extern_data/placeholders/orth_classes/orth_classes_dim0_size:0' shape=(?,) dtype=int32>
```
If it mixes up the dims here, marks them incorrectly as same, this could explain such an error.
I temporarily made that dim tag warning an exception (see also #1141), and this is the stacktrace:
```
...
  File "/u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/network.py", line 1068, in TFNetwork._creat
e_layer
    line: layer = layer_class(**layer_desc)
    locals:
      layer = <not found>
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': True, '_network': <TFNetwork '' train=<tf.Tensor '
globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified
'>, 'sources': [<CopyLayer 'encoder' out_type=Data{[T|'lstm0_pool:conv:s0'[B],B,F|F'lstm1_fw:feature..., len = 1
1
  File "/u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/layers/rec.py", line 259, in RecLayer.__ini
t__
    line: y = self._get_output_subnet_unit(self.cell)
    locals:
      y = <not found>
      self = <local> <RecLayer 'output' out_type=Data{[T|'lstm0_pool:conv:s0'[B],B], dtype='int32', sparse_dim=D
im{F'vocab'(1030)}}>
      self._get_output_subnet_unit = <local> <bound method RecLayer._get_output_subnet_unit of <RecLayer 'output
' out_type=Data{[T|'lstm0_pool:conv:s0'[B],B], dtype='int32', sparse_dim=Dim{F'vocab'(1030)}}>>
      self.cell = <local> <_SubnetworkRecCell '/output(rec-subnet)'>
  File "/u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/layers/rec.py", line 1073, in RecLayer._get
_output_subnet_unit
    line: output = cell.get_output()
    locals:
      output = <not found>
      cell = <local> <_SubnetworkRecCell '/output(rec-subnet)'>
      cell.get_output = <local> <bound method _SubnetworkRecCell.get_output of <_SubnetworkRecCell '/output(rec-
subnet)'>>
  File "/u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/layers/rec.py", line 3079, in _SubnetworkRe
cCell.get_output
    line: self.time_dim_tag.declare_same_as(output_data.get_time_dim_tag())
    locals:
      self = <local> <_SubnetworkRecCell '/output(rec-subnet)'>
      self.time_dim_tag = <local> Dim{'lstm0_pool:conv:s0'[B]}
      self.time_dim_tag.declare_same_as = <local> <bound method Dim.declare_same_as of Dim{'lstm0_pool:conv:s0'[
B]}>
      output_data = <local> Data{'orth_classes', [T|'out-spatial'[B],B], dtype='int32', sparse_dim=Dim{F'vocab'(
1030)}}
      output_data.get_time_dim_tag = <local> <bound method Data.get_time_dim_tag of Data{'orth_classes', [T|'out
-spatial'[B],B], dtype='int32', sparse_dim=Dim{F'vocab'(1030)}}>
  File "/u/zeyer/setups/combined/2021-05-31/tools/returnn/returnn/tf/util/data.py", line 979, in Dim.declare_sam
e_as
    line: raise Exception(
            "Warning: assuming dim tags are same with different size placeholders: %r vs %r" % (
              self.dyn_size, other_same_base.dyn_size))
...
Exception: Warning: assuming dim tags are same with different size placeholders: <tf.Tensor 'lstm0_pool/Neg_1:0'
 shape=(?,) dtype=int32> vs <tf.Tensor 'extern_data/placeholders/orth_classes/orth_classes_dim0_size:0' shape=(?
,) dtype=int32>
```

You see from that log the problem. Specifically this:
```
    line: self.time_dim_tag.declare_same_as(output_data.get_time_dim_tag())
    locals:
      self = <local> <_SubnetworkRecCell '/output(rec-subnet)'>
      self.time_dim_tag = <local> Dim{'lstm0_pool:conv:s0'[B]}
      output_data = <local> Data{'orth_classes', [T|'out-spatial'[B],B], dtype='int32', sparse_dim=Dim{F'vocab'(
1030)}}
```
`self.time_dim_tag` of the rec layer is correct. We loop over the encoder frames.
Not sure on `output_data`, this looks wrong, or the `declare_same_as`.

Hm this is tricky. We have a case where the original config clearly was incorrect, but due to bugs (or rather missing extra checks) it worked, because in the end all the broken parts were not used (but still the layers were constructed).

So, what should we do? Either be strict about it and make sure that existing configs keep working, or in this case break an existing. Or rather, it's already broken, so the question is more, fix RETURNN to make it work again, or not.

Specifically, the `output` sublayer in the rec layer is ([via](https://github.com/rwth-i6/returnn-experiments/blob/264d13aef3321d48f685cc9750fd277fb70cc74e/2020-rnn-transducer/configs/rna-tf2.blank0.enc6l-grow2l.scratch-lm.rdrop02.lm1-1024.attwb5-drop02.l2_1e_4.mlr50.config#L778)):
```
            # During training   : targetb = "target"  (RNA-loss)
            # During recognition: targetb = "targetb"
            'output': {
                'class': 'choice', 'target': targetb, 'beam_size': beam_size,
                'from': "output_log_prob", "input_type": "log_prob",
...
```
So in training, all layers are moved out, and this is the target seq. But the loop goes over the enc seq, so different seq len. So the `output` time dim does not match the time dim of the rec layer output, but actually it must match.

Resolving patch:
diff --git a/returnn/tf/layers/rec.py b/returnn/tf/layers/rec.py
--- a/returnn/tf/layers/rec.py
+++ b/returnn/tf/layers/rec.py
@@ -2325,6 +2325,7 @@ def get_output(self):
     :rtype: tf.Tensor
     """
     from returnn.tf.util.basic import check_input_dim, tensor_array_stack, Dim, get_valid_scope_name_from_str
+    from returnn.util.basic import BehaviorVersion
     assert self.parent_rec_layer
     rec_layer = self.parent_rec_layer
 
@@ -3067,6 +3068,14 @@ def cond(i, net_vars, acc_tas, seq_len_info=None, allow_inf_max_len=False):
       layer = self.net.layers[name]
       assert layer.search_choices
 
+    for key in (
+          self.net.used_data_keys |
+          (self.input_layers_net.used_data_keys if self.input_layers_net else set()) |
+          (self.output_layers_net.used_data_keys if self.output_layers_net else set())):
+      if key == "source":
+        continue
+      self.parent_net.used_data_keys.add(key)
+
     with tf.name_scope("output"):
       output_layer = None
       if self.input_layers_net and "output" in self.input_layers_net.layers:
@@ -3076,7 +3085,19 @@ def cond(i, net_vars, acc_tas, seq_len_info=None, allow_inf_max_len=False):
       if output_layer:
         assert isinstance(output_layer, LayerBase)
         output_data = output_layer.output.copy_as_time_major()
-        self.time_dim_tag.declare_same_as(output_data.get_time_dim_tag())
+        if not self.time_dim_tag.is_dim_known():
+          self.time_dim_tag.declare_same_as(output_data.get_time_dim_tag())
+        elif self.time_dim_tag not in output_data.dim_tags:
+          # We allow this for older behavior version to not break some older setups.
+          # https://github.com/rwth-i6/returnn/issues/1140
+          BehaviorVersion.require(
+            False,
+            "%s: time-dim-tag mismatch: self %r vs sub-output-layer %r time-dim-tag %r" % (
+              rec_layer, self.time_dim_tag, output_data, output_data.get_time_dim_tag()), version=13)
+          # No further checks, it would fail anyway.
+          # Replace the actual rec layer output and return.
+          rec_layer.output = output_data
+          return output_data.placeholder
         assert len(rec_layer.output.dim_tags) == len(output_data.dim_tags)
         for tag1, tag2 in zip(rec_layer.output.dim_tags, output_data.dim_tags):
           try:
@@ -3090,24 +3111,14 @@ def cond(i, net_vars, acc_tas, seq_len_info=None, allow_inf_max_len=False):
           # and then created once for the template layer, and again for the real layer.
           # Make sure they are really the same such that we get all information like dyn sizes.
           tag1.declare_same_as(tag2)
-        output = output_data.placeholder
+        return output_data.placeholder
       else:
         assert seq_len is not None
         rec_layer.output.size_placeholder[0] = seq_len
         assert not self.net.layers["output"].get_search_choices()
-        output = tensor_array_stack(
+        return tensor_array_stack(
           self.final_acc_tas_dict["output_output"], stop=max_seq_len, name="output_stack")  # e.g. (time, batch, dim)
 
-    for key in (
-          self.net.used_data_keys |
-          (self.input_layers_net.used_data_keys if self.input_layers_net else set()) |
-          (self.output_layers_net.used_data_keys if self.output_layers_net else set())):
-      if key == "source":
-        continue
-      self.parent_net.used_data_keys.add(key)
-
-    return output
-
   def _get_search_choice_seq(self, search_choices):
     """
     :param SearchChoices search_choices:
diff --git a/returnn/util/basic.py b/returnn/util/basic.py
--- a/returnn/util/basic.py
+++ b/returnn/util/basic.py
@@ -238,7 +238,7 @@ class BehaviorVersion:
   The version will be set after the config is defined at __main__.init_config() or Engine.__init__()
   """
 
-  _latest_behavior_version = 12
+  _latest_behavior_version = 13
   _behavior_version = None  # type: typing.Optional[int]
 
   @classmethod


Test patch:
diff --git a/tests/test_TFNetworkLayer.py b/tests/test_TFNetworkLayer.py
--- a/tests/test_TFNetworkLayer.py
+++ b/tests/test_TFNetworkLayer.py
@@ -5871,14 +5871,16 @@ def test_SliceNdLayer_dyn_size():
 
 def test_SliceNdLayer_multidimensional_start():
   with make_scope() as session:
-    n_out = 5
+    from returnn.tf.util.data import batch_dim, SpatialDim, FeatureDim
+    out_dim = FeatureDim("feat", 3)
+    time_dim = SpatialDim("time")
     n_batch = 3
     max_seq_len = 10
     config = Config({
       "debug_print_layer_output_template": True,
       "extern_data": {
-        "data": {"dim": n_out},
-        "classes": {"dim": n_out, "sparse": True}
+        "data": {"dim_tags": [batch_dim, time_dim, out_dim]},
+        "classes": {"dim_tags": [batch_dim, time_dim], "sparse_dim": out_dim}
       }})
     net = TFNetwork(config=config, train_flag=True)
     net.construct_from_dict({
@@ -5902,7 +5904,7 @@ def test_SliceNdLayer_multidimensional_start():
     input_data = feed[net.extern_data.data["data"].placeholder]
     max_size = numpy.amax(seq_lens[:, None] - starts)
     max_size = max(max_size, 0)
-    assert segments.shape == (n_batch, max_seq_len, max_size, n_out)
+    assert segments.shape == (n_batch, max_seq_len, max_size, out_dim.dimension)
     for b in range(n_batch):
       for t in range(max_seq_len):
         s = starts[b, t]
@@ -5911,7 +5913,7 @@ def test_SliceNdLayer_multidimensional_start():
           orig_seq = numpy.pad(orig_seq, [(0, max_size - len(orig_seq)), (0, 0)], "constant")
         elif len(orig_seq) > max_size:
           orig_seq = orig_seq[:max_size]
-        assert orig_seq.shape == (max_size, n_out)
+        assert orig_seq.shape == (max_size, out_dim.dimension)
         orig_seq = numpy.where((numpy.arange(s, s + max_size) >= seq_lens[b])[:, None], 0.0, orig_seq)
         for t2 in range(max_size):
           numpy.testing.assert_equal(orig_seq[t2], segments[b, t, t2])
@@ -5919,14 +5921,16 @@ def test_SliceNdLayer_multidimensional_start():
 
 def test_SliceNdLayer_multidimensional_size():
   with make_scope() as session:
-    n_out = 5
+    from returnn.tf.util.data import batch_dim, SpatialDim, FeatureDim
+    out_dim = FeatureDim("feat", 3)
+    time_dim = SpatialDim("time")
     n_batch = 3
     max_seq_len = 10
     config = Config({
       "debug_print_layer_output_template": True,
       "extern_data": {
-        "data": {"dim": n_out},
-        "classes": {"dim": n_out, "sparse": True}
+        "data": {"dim_tags": [batch_dim, time_dim, out_dim]},
+        "classes": {"dim_tags": [batch_dim, time_dim], "sparse_dim": out_dim}
       }})
     net = TFNetwork(config=config, train_flag=True)
     net.construct_from_dict({
@@ -5954,7 +5958,7 @@ def test_SliceNdLayer_multidimensional_size():
     input_data = feed[net.extern_data.data["data"].placeholder]
     max_size = numpy.amax(sizes)
     max_size = max(max_size, 0)
-    assert segments.shape == (n_batch, max_seq_len, max_size, n_out)
+    assert segments.shape == (n_batch, max_seq_len, max_size, out_dim.dimension)
     for b in range(n_batch):
       for t in range(max_seq_len):
         s = starts[b, t]
@@ -5965,7 +5969,7 @@ def test_SliceNdLayer_multidimensional_size():
           orig_seq = numpy.pad(orig_seq, [(0, max_size - len(orig_seq)), (0, 0)], "constant")
         elif len(orig_seq) > max_size:
           orig_seq = orig_seq[:max_size]
-        assert orig_seq.shape == (max_size, n_out)
+        assert orig_seq.shape == (max_size, out_dim.dimension)
         orig_seq = numpy.where((numpy.arange(s, s + max_size) >= seq_lens[b])[:, None], 0.0, orig_seq)
         for t2 in range(max_size):
           numpy.testing.assert_equal(orig_seq[t2], segments[b, t, t2])
diff --git a/tests/test_TFNetworkRecLayer.py b/tests/test_TFNetworkRecLayer.py
--- a/tests/test_TFNetworkRecLayer.py
+++ b/tests/test_TFNetworkRecLayer.py
@@ -600,11 +600,17 @@ def _enc_func(source, **_):
 
 
 def test_rec_subnet_with_choice():
+  from returnn.tf.util.data import batch_dim, SpatialDim, FeatureDim
+  in_dim = FeatureDim("feat", 3)
+  out_dim = FeatureDim("classes", 4)
+  time_dim = SpatialDim("time")
   with tf_compat.v1.Session():
     config = Config()
     config.update({
-      "num_outputs": 3,
-      "num_inputs": 4,
+      "extern_data": {
+        "data": {"dim_tags": [batch_dim, time_dim, in_dim]},
+        "classes": {"dim_tags": [batch_dim, time_dim], "sparse_dim": out_dim}
+      },
       "network": {
         "output": {"class": "rec", "from": "data:data", "target": "classes", "unit": {
           "prob": {"class": "softmax", "from": ["prev:output"], "loss": "ce", "target": "classes"},
@@ -1220,12 +1226,16 @@ def test_rec_RecStepInfoLayer_broadcast_moved_out():
       },
     }
   }
+  from returnn.tf.util.data import batch_dim, SpatialDim, FeatureDim
+  in_dim = FeatureDim("feat", 3)
+  out_dim = FeatureDim("classes", 5)
+  time_dim = SpatialDim("time")
   config = Config({
     "debug_print_layer_output_template": True,
     "extern_data": {
-      "data": {"dim": 3},
-      "classes": {"sparse": True, "dim": 5},
-    }
+      "data": {"dim_tags": [batch_dim, time_dim, in_dim]},
+      "classes": {"dim_tags": [batch_dim, time_dim], "sparse_dim": out_dim}
+    },
   })
   from test_TFNetworkLayer import make_feed_dict
   with make_scope() as session:
@@ -6158,6 +6168,93 @@ def test_reclayer_shape_from_initial():
     session.run(out.placeholder, feed_dict=make_feed_dict(net.extern_data))
 
 
+def test_reclayer_time_sync_target_diff():
+  # https://github.com/rwth-i6/returnn/issues/1140
+  from returnn.util.basic import BehaviorVersion
+  from returnn.tf.util.data import batch_dim, SpatialDim, FeatureDim
+  from returnn.tf.layers.rec import _SubnetworkRecCell
+  src_dim = FeatureDim("src-feat", 5)
+  tgt_dim = FeatureDim("tgt-classes", 7)
+  tgt_with_blank_dim = tgt_dim + 1
+  src_time_dim = SpatialDim("src-time")
+  tgt_time_dim = SpatialDim("out-spatial")
+
+  config = Config({
+    "extern_data": {
+      "data": {"dim_tags": [batch_dim, src_time_dim, src_dim]},
+      "classes": {"dim_tags": [batch_dim, tgt_time_dim], "sparse_dim": tgt_dim, "available_for_inference": False},
+      "align_classes": {
+        "dim_tags": [batch_dim, src_time_dim], "sparse_dim": tgt_with_blank_dim, "available_for_inference": False},
+    },
+    "network": {
+      "encoder": {"class": "linear", "activation": "tanh", "n_out": 5, "from": "data:data"},
+
+      "output": {"class": "rec", "from": "encoder", "unit": {
+        "output_prob": {"class": "softmax", "from": "data:source", "out_dim": tgt_with_blank_dim},
+
+        # Note: This is actually not correct to have 'classes' here.
+        # In practice, in search, it would use output_prob and then have actually one more label.
+        # classes also has the wrong spatial dim, which actually causes the error.
+        # However, then this output is actually never used.
+        # We had such training configs for transducer, and we want to make sure that they still work.
+        # In that case, in search, the config switched to a different target, so that is why it worked.
+        'output': {'class': 'choice', 'target': 'classes', 'beam_size': 12, 'from': "output_prob",
+                   "initial_output": 0},
+
+        # Would also look different for recognition.
+        "classes_embed": {"class": "linear", "activation": "tanh", "n_out": 5, "from": "base:data:classes"},
+        "joint": {
+          "class": "combine", "from": ["output_prob", "classes_embed"], "kind": "mul",
+          "allow_broadcast_all_sources": True},
+
+        # Dummy loss. In transducer, this would be the full-sum after joint network.
+        # Here we just need sth to trigger the dependencies.
+        "loss": {
+          "class": "eval", "from": "joint",
+          "eval": "tf.reduce_mean(source(0,auto_convert=False))",
+          "out_type": {"shape": (), "dtype": "float32", "batch_dim_axis": None, "time_dim_axis": None},
+          "loss": "as_is"
+        },
+
+      }, "target": "classes"},
+    }})
+
+  print("Constructing train network (old behavior).")
+  with make_scope() as session:
+    net = TFNetwork(train_flag=True, config=config)
+    orig_behavior_version = BehaviorVersion._behavior_version
+    try:
+      BehaviorVersion._behavior_version = 0
+      # The net dict requires an older behavior version. This is important for the test.
+      # We want to make sure such old config still works.
+      net.construct_from_dict(config.typed_value("network"))
+    finally:
+      BehaviorVersion._behavior_version = orig_behavior_version
+    # Check whether we triggered the dim tag bug.
+    assert src_time_dim != tgt_time_dim
+    net.initialize_params(session)
+    rec_layer = net.get_layer("output")
+    assert isinstance(rec_layer, RecLayer)
+    cell = rec_layer.cell
+    assert isinstance(cell, _SubnetworkRecCell)
+    assert_equal(cell.layers_in_loop, [])
+    loss = net.get_total_loss()
+    from test_TFNetworkLayer import make_feed_dict
+    loss_v = session.run(loss, feed_dict=make_feed_dict(net.extern_data))
+    print("Loss:", loss_v)
+
+  print("Constructing train network (new behavior).")
+  with make_scope():
+    net = TFNetwork(train_flag=True, config=config)
+    try:
+      net.construct_from_dict(config.typed_value("network"))
+    except BehaviorVersion.RequirementNotSatisfied as exc:
+      assert "time-dim-tag mismatch" in str(exc)
+      print("Got expected exception:", exc)
+    else:
+      raise Exception("did not get expected exception")
+
+
 def test_convert_lstm_params_save_load():
   """
   Test conversions from different units to different units.