Repository: rwth-i6/returnn

Issue:
RecStepInfoLayer (`:i`) problem in unbroadcasting
It seems like the commit 7e3b939237e69a396a017e88bf836972f0d63e79 broke part of my config.
The config looks like this:

```
network = {
  "output": {
    "class": "rec", "from": "data",  # time-sync
    "include_eos": True, "back_prop": (task == "train") and train, "unit": {
      "segment_starts": { 
        "class": "switch", "condition": "prev:output_is_not_blank", "true_from": ":i",
        "false_from": "prev:segment_starts", "initial_output": 0
      },
      "segment_lens0": {"class": "combine", "kind": "sub", "from": [":i", "segment_starts"], "is_output_layer": True},
      "output_prob": {
        "class": "softmax", "from": "data:source", "target": "classes", "loss": "ce"
      },
      'output': {
        'class': 'choice', 'target': "classes", 'beam_size': 3, 'from': "output_prob", "input_type": "prob",
        "initial_output": 0,
      }, "output_is_not_blank": {
        "class": "compare", "from": "output", "value": 0, "kind": "not_equal", "initial_output": True
      },
    },
  }
}
```

When I try to run the config after commit 7e3b939237e69a396a017e88bf836972f0d63e79, I get the following error:

[error.txt](https://github.com/rwth-i6/returnn/files/7135243/error.txt). Before that commit, it works. The error seems to be caused by `segment_lens0`. I don't know if the way I am doing it is wrong or if the commit broke something which should work. @albertz 



Discussion:
Error:
```
/home/robin/miniconda3/envs/i6_env/bin/python /mnt/data/src/returnn/rnn.py /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py
2021-09-09 11:07:07.831320: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-09 11:07:07.831335: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Hello, experiment: /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function
RETURNN starting up, version 1.0.0+unknown, date/time 2021-09-09-11-07-08 (UTC+0200), pid 29267, cwd /mnt/data/src/returnn, Python /home/robin/miniconda3/envs/i6_env/bin/python
RETURNN command line options: ['/mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py']
Hostname: robin-HP-ENVY-Laptop-13-ba0xxx
TensorFlow: 2.5.0 (v2.5.0-rc3-213-ga4dfb8d1a71) (<site-package> in /home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow)
Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}.
CUDA_VISIBLE_DEVICES is not set.
Collecting TensorFlow device list...
Local devices available to TensorFlow:
  1/1: name: "/device:CPU:0"
       device_type: "CPU"
       memory_limit: 268435456
       locality {
       }
       incarnation: 14028160239026936064
Train data:
  input: 10 x 1
  output: {'classes': (3, 1), 'data': (10, 2)}
  DummyDataset, sequences: 10, frames: unknown
2021-09-09 11:07:08.644655: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-09 11:07:08.644995: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-09-09 11:07:08.645005: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-09-09 11:07:08.645017: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (robin-HP-ENVY-Laptop-13-ba0xxx): /proc/driver/nvidia/version does not exist
Device not set explicitly, and no GPU found.
Learning-rate-control: no file specified, not saving history (no proper restart possible)
Setup TF session with options {'log_device_placement': False, 'device_count': {'GPU': 0}} ...
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/network.py:455: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:1285: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
layer root/'data' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
layer root/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])
Rec layer 'output' (search False, train 'globals/train_flag:0') sub net:
  Input layers moved out of loop: (#: 2)
    output
    output_is_not_blank
  Output layers moved out of loop: (#: 7)
    print1
    output_prob
    print3
    segment_lens
    const1
    segment_lens0
    print2
  Layers in loop: (#: 2)
    :i
    segment_starts
  Unused layers: (#: 1)
    segments
layer root/output(rec-subnet-input)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-input)/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-input)/'output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet)/'prev:output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B])
layer root/output(rec-subnet)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[B])
layer root/output(rec-subnet-output)/'print1' output: Data(name='print1_output', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-output)/'data:source' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
layer root/output(rec-subnet-output)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-output)/'output_prob' output: Data(name='output_prob_output', batch_shape_meta=[B,T|'time'[B],F|F'feature:output_prob_output'(3)])
layer root/output(rec-subnet-output)/':i' output: Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])
layer root/output(rec-subnet-output)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
layer root/output(rec-subnet-output)/'segment_lens0' output: Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
Exception creating layer root/output(rec-subnet-output)/'segment_lens0' of class CombineLayer with opts:
{'_name': 'segment_lens0',
 '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:6354: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.
    
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:5393: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
 'kind': 'sub',
 'name': 'segment_lens0',
 'network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'output': Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B]),
 'sources': [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>,
             <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]}
Exception occurred during output-net construction of layer 'segment_lens0'.
We had previous exceptions at template construction, which got resolved, but maybe sth is wrong.
Template network (check out types / shapes):
{':i': <_TemplateLayer(RecStepInfoLayer)(:template::i) output/':i' out_type=Data(dtype='int32', batch_shape_meta=[]) (construction stack 'segment_starts')>,
 'const1': <_TemplateLayer(ConstantLayer)(:template:constant) output/'const1' out_type=Data(dtype='int32', batch_shape_meta=[]) (construction stack None)>,
 'data:classes': <_TemplateLayer(SourceLayer)(:template:source) output/'data:classes' out_type=Data(dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B]) (construction stack 'output')>,
 'data:source': <_TemplateLayer(SourceLayer)(:template:source) output/'data:source' out_type=Data(batch_shape_meta=[B,F|F'feature:data'(10)]) (construction stack 'output_prob')>,
 'output': <_TemplateLayer(ChoiceLayer)(:template:choice) output/'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[B]) (construction stack None)>,
 'output_is_not_blank': <_TemplateLayer(CompareLayer)(:template:compare) output/'output_is_not_blank' out_type=Data(dtype='bool', sparse=True, dim=2, batch_shape_meta=[B]) (construction stack 'segment_starts')>,
 'output_prob': <_TemplateLayer(SoftmaxLayer)(:template:softmax) output/'output_prob' out_type=Data(batch_shape_meta=[B,F|F'feature:output_prob_output'(3)]) (construction stack None)>,
 'print1': <_TemplateLayer(PrintLayer)(:template:print) output/'print1' out_type=Data(dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]]) (construction stack None)>,
 'print2': <_TemplateLayer(PrintLayer)(:template:print) output/'print2' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack None)>,
 'print3': <_TemplateLayer(PrintLayer)(:template:print) output/'print3' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack None)>,
 'segment_lens': <_TemplateLayer(CombineLayer)(:template:combine) output/'segment_lens' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'print3')>,
 'segment_lens0': <_TemplateLayer(CombineLayer)(:template:combine) output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'segment_lens')>,
 'segment_starts': <_TemplateLayer(SwitchLayer)(:template:switch) output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'print2')>}
Collected (unique) exceptions during template construction:
(Note that many of these can be ignored, or are expected.)
EXCEPTION while constructing layer 'segment_starts'
NetworkConstructionDependencyLoopException: <TFNetwork 'root/output(rec-subnet)' parent_net=<TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>: Error: There is a dependency loop on layer 'segment_starts'.
Construction stack (most recent first):
  segment_starts
  print2

Exception occurred during output-net construction of layer 'segment_lens'.
Exception occurred during output-net construction of layer 'print3'.
Exception creating layer root/'output' of class RecLayer with opts:
{'_name': 'output',
 '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 '_time_dim_tag': DimensionTag{'time'[B]},
 'back_prop': {'class': 'DummyDataset',
               'input_dim': 10,
               'num_seqs': 10,
               'output_dim': 3,
               'seq_len': 10},
 'include_eos': True,
 'n_out': <class 'returnn.util.basic.NotSpecified'>,
 'name': 'output',
 'network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'output': Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B]),
 'sources': [<SourceLayer 'data' out_type=Data(batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])>],
 'unit': <_SubnetworkRecCell 'root/output(rec-subnet)'>}
Unhandled exception <class 'AssertionError'> in thread <_MainThread(MainThread, started 139834570043776)>, proc 29267.

Thread current, main, <_MainThread(MainThread, started 139834570043776)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
  File "/mnt/data/src/returnn/rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x7f2d80677f70>
  File "/mnt/data/src/returnn/returnn/__main__.py", line 655, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x7f2d80677e50>
  File "/mnt/data/src/returnn/returnn/__main__.py", line 453, in execute_main_task
    line: engine.init_train_from_config(config, train_data, dev_data, eval_data)
    locals:
      engine = <global> <returnn.tf.engine.Engine object at 0x7f2d48940c40>
      engine.init_train_from_config = <global> <bound method Engine.init_train_from_config of <returnn.tf.engine.Engine object at 0x7f2d48940c40>>
      config = <global> <returnn.config.Config object at 0x7f2dc559eee0>
      train_data = <global> <DummyDataset 'train' epoch=None>
      dev_data = <global> None
      eval_data = <global> None
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1031, in Engine.init_train_from_config
    line: self.init_network_from_config(config)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7f2d48940c40>
      self.init_network_from_config = <local> <bound method Engine.init_network_from_config of <returnn.tf.engine.Engine object at 0x7f2d48940c40>>
      config = <local> <returnn.config.Config object at 0x7f2dc559eee0>
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1096, in Engine.init_network_from_config
    line: self._init_network(net_desc=net_dict, epoch=self.epoch)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7f2d48940c40>
      self._init_network = <local> <bound method Engine._init_network of <returnn.tf.engine.Engine object at 0x7f2d48940c40>>
      net_desc = <not found>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition...
      epoch = <local> None
      self.epoch = <local> 1
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1275, in Engine._init_network
    line: self.network, self.updater = self.create_network(
            config=self.config,
            extern_data=extern_data,
            rnd_seed=net_random_seed,
            train_flag=train_flag, eval_flag=self.use_eval_flag, search_flag=self.use_search_flag,
            initial_learning_rate=getattr(self, "initial_learning_rate", None),
            net_dict=net_desc)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7f2d48940c40>
      self.network = <local> None
      self.updater = <local> None
      self.create_network = <local> <bound method Engine.create_network of <class 'returnn.tf.engine.Engine'>>
      config = <not found>
      self.config = <local> <returnn.config.Config object at 0x7f2dc559eee0>
      extern_data = <local> <ExternData data={'data': Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)]), 'classes': Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])}>
      rnd_seed = <not found>
      net_random_seed = <local> 1
      train_flag = <local> <tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>
      eval_flag = <not found>
      self.use_eval_flag = <local> True
      search_flag = <not found>
      self.use_search_flag = <local> False
      initial_learning_rate = <not found>
      getattr = <builtin> <built-in function getattr>
      net_dict = <not found>
      net_desc = <local> {'output': {'class': 'rec', 'from': 'data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition...
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1316, in Engine.create_network
    line: network.construct_from_dict(net_dict)
    locals:
      network = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      network.construct_from_dict = <local> <bound method TFNetwork.construct_from_dict of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 584, in TFNetwork.construct_from_dict
    line: self.construct_layer(net_dict, name, get_layer=get_layer)
    locals:
      self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition...
      name = <local> 'output', len = 6
      get_layer = <local> None
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 910, in TFNetwork.construct_layer
    line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
    locals:
      add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'output', len = 6
      name_with_prefix = <local> 'output', len = 6
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 1055, in TFNetwork.add_layer
    line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
    locals:
      layer = <not found>
      self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'output', len = 6
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 976, in TFNetwork._create_layer
    line: layer = layer_class(**layer_desc)
    locals:
      layer = <not found>
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 11
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 238, in RecLayer.__init__
    line: y = self._get_output_subnet_unit(self.cell)
    locals:
      y = <not found>
      self = <local> <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>
      self._get_output_subnet_unit = <local> <bound method RecLayer._get_output_subnet_unit of <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>>
      self.cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 921, in RecLayer._get_output_subnet_unit
    line: output = cell.get_output()
    locals:
      output = <not found>
      cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      cell.get_output = <local> <bound method _SubnetworkRecCell.get_output of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 2528, in _SubnetworkRecCell.get_output
    line: self._construct_output_layers_moved_out(
            loop_accumulated=self.final_acc_tas_dict, seq_len=seq_len,
            extra_output_layers=extra_output_layers, final_net_vars=final_net_vars)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self._construct_output_layers_moved_out = <local> <bound method _SubnetworkRecCell._construct_output_layers_moved_out of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
      loop_accumulated = <not found>
      self.final_acc_tas_dict = <local> {'output_:i': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output___i'>, 'output_segment_starts': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output_segment_starts'>}
      seq_len = <local> <tf.Tensor 'output/rec/subnet_base/check_seq_len_batch_size/check_input_dim/identity_with_dim_check:0' shape=(?,) dtype=int32>
      extra_output_layers = <local> {'output', 'print1', 'const1', 'print3', 'print2'}, len = 5
      final_net_vars = <local> ([<tf.Tensor 'output/rec/while/Exit_1:0' shape=(?,) dtype=int32>], [])
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3315, in _SubnetworkRecCell._construct_output_layers_moved_out
    line: get_layer(layer_name)
    locals:
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
      layer_name = <local> 'print3', len = 6
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3303, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
    line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      self.net_dict = <local> {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', '..., len = 11
      name = <local> 'print3', len = 6
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 903, in TFNetwork.construct_layer
    line: layer_class.transform_config_dict(layer_desc, network=net, get_layer=get_layer)
    locals:
      layer_class = <local> <class 'returnn.tf.layers.basic.PrintLayer'>
      layer_class.transform_config_dict = <local> <bound method LayerBase.transform_config_dict of <class 'returnn.tf.layers.basic.PrintLayer'>>
      layer_desc = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print3'}
      network = <not found>
      net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 495, in LayerBase.transform_config_dict
    line: d["sources"] = [
            get_layer(src_name)
            for src_name in src_names
            if not src_name == "none"]
    locals:
      d = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print3'}
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
      src_name = <not found>
      src_names = <local> ['segment_lens'], _[0]: {len = 12}
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 496, in <listcomp>
    line: get_layer(src_name)
    locals:
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
      src_name = <local> 'segment_lens', len = 12
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3303, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
    line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      self.net_dict = <local> {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', '..., len = 11
      name = <local> 'segment_lens', len = 12
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 903, in TFNetwork.construct_layer
    line: layer_class.transform_config_dict(layer_desc, network=net, get_layer=get_layer)
    locals:
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_class.transform_config_dict = <local> <bound method LayerBase.transform_config_dict of <class 'returnn.tf.layers.basic.CombineLayer'>>
      layer_desc = <local> {'kind': 'add', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens'}
      network = <not found>
      net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 495, in LayerBase.transform_config_dict
    line: d["sources"] = [
            get_layer(src_name)
            for src_name in src_names
            if not src_name == "none"]
    locals:
      d = <local> {'kind': 'add', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens'}
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
      src_name = <not found>
      src_names = <local> ['segment_lens0', 'const1'], _[0]: {len = 13}
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 496, in <listcomp>
    line: get_layer(src_name)
    locals:
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
      src_name = <local> 'segment_lens0', len = 13
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3303, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
    line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      self.net_dict = <local> {'const1': {'class': 'constant', 'value': 1, 'is_output_layer': True}, 'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', '..., len = 11
      name = <local> 'segment_lens0', len = 13
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7f2dc1455dc0>
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 910, in TFNetwork.construct_layer
    line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
    locals:
      add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'segment_lens0', len = 13
      name_with_prefix = <local> 'segment_lens0', len = 13
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 1055, in TFNetwork.add_layer
    line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
    locals:
      layer = <not found>
      self = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'segment_lens0', len = 13
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 976, in TFNetwork._create_layer
    line: layer = layer_class(**layer_desc)
    locals:
      layer = <not found>
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna..., len = 7
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5882, in CombineLayer.__init__
    line: x = op(sources)
    locals:
      x = <not found>
      op = <local> <bound method CombineLayer._op_kind_sub of <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5972, in CombineLayer._op_kind_sub
    line: return self._op_dense_fn(sources, tf.subtract)
    locals:
      self = <local> <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>
      self._op_dense_fn = <local> <function CombineLayer._op_dense_fn at 0x7f2dc1606ee0>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
      tf = <global> <module 'tensorflow' from '/home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow/__init__.py'>
      tf.subtract = <global> <function subtract at 0x7f2d505d3e50>
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5954, in CombineLayer._op_dense_fn
    line: x = sources[0].output.copy_compatible_to(common_data).placeholder
    locals:
      x = <not found>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
      output = <not found>
      output.copy_compatible_to = <not found>
      common_data = <local> Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
      placeholder = <not found>
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 2043, in Data.copy_compatible_to
    line: v = v.copy_add_dim_by_tag(data.get_dim_tag(target_axis), axis=new_v_axis, unbroadcast=unbroadcast_axis)
    locals:
      v = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])
      v.copy_add_dim_by_tag = <local> <bound method Data.copy_add_dim_by_tag of Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])>
      data = <local> Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
      data.get_dim_tag = <local> <bound method Data.get_dim_tag of Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>
      target_axis = <local> 1
      axis = <not found>
      new_v_axis = <local> 1
      unbroadcast = <local> False
      unbroadcast_axis = <local> False
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1909, in Data.copy_add_dim_by_tag
    line: return self.copy_add_batch_dim(
            batch_dim_axis=axis, batch=batch_info, dim_tag=dim_tag if dim_tag.dimension == 1 else None)
    locals:
      self = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])
      self.copy_add_batch_dim = <local> <bound method Data.copy_add_batch_dim of Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])>
      batch_dim_axis = <not found>
      axis = <local> 1
      batch = <not found>
      batch_info = <local> BatchInfo{}
      dim_tag = <local> DimensionTag{B}
      dim_tag.dimension = <local> None
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1824, in Data.copy_add_batch_dim
    line: return Data(placeholder=placeholder, **data_opts)
    locals:
      Data = <global> <class 'returnn.tf.util.data.Data'>
      placeholder = <local> <tf.Tensor 'output/rec/segment_lens0/i_add_batch_dim:0' shape=(?, 1) dtype=int32>
      data_opts = <local> {'name': 'i', 'dim_tags': [DimensionTag{'time'[B]}, DimensionTag{B}], 'dtype': 'int32', 'batch': BatchInfo{}, 'beam': None, 'time_dim_axis': 0}, len = 6
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1315, in Data.__init__
    line: self._adapt_batch_consistent_dim_tags()
    locals:
      self = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B],Bx])
      self._adapt_batch_consistent_dim_tags = <local> <bound method Data._adapt_batch_consistent_dim_tags of Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B],Bx])>
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1586, in Data._adapt_batch_consistent_dim_tags
    line: self._dim_tags = tuple(tag.get_for_batch(self.batch) for tag in self._dim_tags)
    locals:
      self = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B],Bx])
      self._dim_tags = <local> (DimensionTag{'time'[B]}, DimensionTag{B})
      tuple = <builtin> <class 'tuple'>
      tag = <not found>
      tag.get_for_batch = <not found>
      self.batch = <local> BatchInfo{}
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1586, in <genexpr>
    line: self._dim_tags = tuple(tag.get_for_batch(self.batch) for tag in self._dim_tags)
    locals:
      self = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B],Bx])
      self._dim_tags = <local> (DimensionTag{'time'[B]}, DimensionTag{B})
      tuple = <builtin> <class 'tuple'>
      tag = <local> DimensionTag{'time'[B]}
      tag.get_for_batch = <local> <bound method DimensionTag.get_for_batch of DimensionTag{'time'[B]}>
      self.batch = <local> BatchInfo{}
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 151, in DimensionTag.get_for_batch
    line: if batch.copy_remove_beam() == batch.get_global_base() and batch.beam and same_base.dyn_size_ext:
    locals:
      batch = <local> BatchInfo{}
      batch.copy_remove_beam = <local> <bound method BatchInfo.copy_remove_beam of BatchInfo{}>
      batch.get_global_base = <local> <bound method BatchInfo.get_global_base of BatchInfo{}>
      batch.beam = <local> None
      same_base = <local> DimensionTag{'time'[B]}
      same_base.dyn_size_ext = <local> Data(name='time:var:extern_data:data:dyn_size', dtype='int32', batch_shape_meta=[B])
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 951, in BatchInfo.copy_remove_beam
    line: assert self.virtual_dims
    locals:
      self = <local> BatchInfo{}
      self.virtual_dims = <local> []
AssertionError

Process finished with exit code 1
```

So, error in `CombineLayer` `segment_lens0`, which got moved out, and inputs `:i` and `segment_starts`

```
 'sources': [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>,
             <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]}
```
It fails in `copy_compatible_to`, where it got the common data:
```
common_data = <local> Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
```

But then the main error is in `BatchInfo.copy_remove_beam`.
I pushed some small change to `BatchInfo.copy_remove_beam`. Please try again. Reopen if you still have some other problem. Then please post the new error.
So I just tried again and I now get a different error:

```
/home/robin/miniconda3/envs/i6_env/bin/python /mnt/data/src/returnn/rnn.py /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py
2021-09-10 14:00:56.356147: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-10 14:00:56.356161: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Hello, experiment: /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function
RETURNN starting up, version 1.0.0+unknown, date/time 2021-09-10-14-00-57 (UTC+0200), pid 2740, cwd /mnt/data/src/returnn, Python /home/robin/miniconda3/envs/i6_env/bin/python
RETURNN command line options: ['/mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py']
Hostname: robin-HP-ENVY-Laptop-13-ba0xxx
TensorFlow: 2.5.0 (v2.5.0-rc3-213-ga4dfb8d1a71) (<site-package> in /home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow)
Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}.
CUDA_VISIBLE_DEVICES is not set.
Collecting TensorFlow device list...
Local devices available to TensorFlow:
  1/1: name: "/device:CPU:0"
       device_type: "CPU"
       memory_limit: 268435456
       locality {
       }
       incarnation: 514580541836988322
Train data:
  input: 10 x 1
  output: {'classes': (3, 1), 'data': (10, 2)}
  DummyDataset, sequences: 10, frames: unknown
2021-09-10 14:00:57.200436: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-10 14:00:57.200786: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-09-10 14:00:57.200800: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-09-10 14:00:57.200813: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (robin-HP-ENVY-Laptop-13-ba0xxx): /proc/driver/nvidia/version does not exist
Device not set explicitly, and no GPU found.
Learning-rate-control: no file specified, not saving history (no proper restart possible)
Setup TF session with options {'log_device_placement': False, 'device_count': {'GPU': 0}} ...
Warning: Using extern_data and will ignore num_inputs/num_outputs in config.
Warning: assuming dim tags are same with different size placeholders: DimensionTag{'time'[B?]} vs DimensionTag{'time'[B?]}
layer root/'data:data' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/network.py:459: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
layer root/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:1350: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
Rec layer 'output' (search False, train 'globals/train_flag:0') sub net:
  Input layers moved out of loop: (#: 2)
    output
    output_is_not_blank
  Output layers moved out of loop: (#: 3)
    output_prob
    print2
    segment_lens0
  Layers in loop: (#: 2)
    :i
    segment_starts
  Unused layers: (#: 0)
    None
layer root/output(rec-subnet-input)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-input)/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-input)/'output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet)/'prev:output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B])
layer root/output(rec-subnet)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[B])
layer root/output(rec-subnet-output)/'data:source' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
layer root/output(rec-subnet-output)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
layer root/output(rec-subnet-output)/'output_prob' output: Data(name='output_prob_output', batch_shape_meta=[B,T|'time'[B],F|F'output_prob:feature-dense'(3)])
WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:5452: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
layer root/output(rec-subnet-output)/':i' output: Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])
layer root/output(rec-subnet-output)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
layer root/output(rec-subnet-output)/'segment_lens0' output: Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
Exception creating layer root/output(rec-subnet-output)/'segment_lens0' of class CombineLayer with opts:
{'_name': 'segment_lens0',
 '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'kind': 'sub',
 'name': 'segment_lens0',
 'network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'output': Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B]),
 'sources': [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>,
             <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]}
Exception occurred during output-net construction of layer 'segment_lens0'.
We had previous exceptions at template construction, which got resolved, but maybe sth is wrong.
Template network (check out types / shapes):
{':i': <_TemplateLayer(RecStepInfoLayer)(:template::i) output/':i' out_type=Data(dtype='int32', batch_shape_meta=[]) (construction stack 'segment_lens0')>,
 'data:classes': <_TemplateLayer(SourceLayer)(:template:source) output/'data:classes' out_type=Data(dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B]) (construction stack 'output')>,
 'data:source': <_TemplateLayer(SourceLayer)(:template:source) output/'data:source' out_type=Data(batch_shape_meta=[B,F|F'feature:data'(10)]) (construction stack 'output_prob')>,
 'output': <_TemplateLayer(ChoiceLayer)(:template:choice) output/'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[B]) (construction stack None)>,
 'output_is_not_blank': <_TemplateLayer(CompareLayer)(:template:compare) output/'output_is_not_blank' out_type=Data(dtype='bool', sparse=True, dim=2, batch_shape_meta=[B]) (construction stack 'segment_starts')>,
 'output_prob': <_TemplateLayer(SoftmaxLayer)(:template:softmax) output/'output_prob' out_type=Data(batch_shape_meta=[B,F|F'output_prob:feature-dense'(3)]) (construction stack None)>,
 'print2': <_TemplateLayer(PrintLayer)(:template:print) output/'print2' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack None)>,
 'segment_lens0': <_TemplateLayer(CombineLayer)(:template:combine) output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'print2')>,
 'segment_starts': <_TemplateLayer(SwitchLayer)(:template:switch) output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'segment_lens0')>}
Collected (unique) exceptions during template construction:
(Note that many of these can be ignored, or are expected.)
EXCEPTION while constructing layer 'segment_starts'
NetworkConstructionDependencyLoopException: <TFNetwork 'root/output(rec-subnet)' parent_net=<TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>: Error: There is a dependency loop on layer 'segment_starts'.
Construction stack (most recent first):
  segment_starts
  segment_lens0
  print2

Exception occurred during output-net construction of layer 'print2'.
Exception creating layer root/'output' of class RecLayer with opts:
{'_name': 'output',
 '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 '_time_dim_tag': DimensionTag{'time'[B]},
 'back_prop': {'class': 'DummyDataset',
               'input_dim': 10,
               'num_seqs': 10,
               'output_dim': 3,
               'seq_len': 10},
 'include_eos': True,
 'n_out': <class 'returnn.util.basic.NotSpecified'>,
 'name': 'output',
 'network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
 'output': Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B]),
 'sources': [<SourceLayer 'data:data' out_type=Data(batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])>],
 'unit': <_SubnetworkRecCell 'root/output(rec-subnet)'>}
Unhandled exception <class 'AssertionError'> in thread <_MainThread(MainThread, started 140481628434816)>, proc 2740.

Thread current, main, <_MainThread(MainThread, started 140481628434816)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
  File "/mnt/data/src/returnn/rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x7fc42816ff70>
  File "/mnt/data/src/returnn/returnn/__main__.py", line 655, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x7fc42816fe50>
  File "/mnt/data/src/returnn/returnn/__main__.py", line 453, in execute_main_task
    line: engine.init_train_from_config(config, train_data, dev_data, eval_data)
    locals:
      engine = <global> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
      engine.init_train_from_config = <global> <bound method Engine.init_train_from_config of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
      config = <global> <returnn.config.Config object at 0x7fc46d096ee0>
      train_data = <global> <DummyDataset 'train' epoch=None>
      dev_data = <global> None
      eval_data = <global> None
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1031, in Engine.init_train_from_config
    line: self.init_network_from_config(config)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
      self.init_network_from_config = <local> <bound method Engine.init_network_from_config of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
      config = <local> <returnn.config.Config object at 0x7fc46d096ee0>
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1096, in Engine.init_network_from_config
    line: self._init_network(net_desc=net_dict, epoch=self.epoch)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
      self._init_network = <local> <bound method Engine._init_network of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
      net_desc = <not found>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
      epoch = <local> None
      self.epoch = <local> 1
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1275, in Engine._init_network
    line: self.network, self.updater = self.create_network(
            config=self.config,
            extern_data=extern_data,
            rnd_seed=net_random_seed,
            train_flag=train_flag, eval_flag=self.use_eval_flag, search_flag=self.use_search_flag,
            initial_learning_rate=getattr(self, "initial_learning_rate", None),
            net_dict=net_desc)
    locals:
      self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
      self.network = <local> None
      self.updater = <local> None
      self.create_network = <local> <bound method Engine.create_network of <class 'returnn.tf.engine.Engine'>>
      config = <not found>
      self.config = <local> <returnn.config.Config object at 0x7fc46d096ee0>
      extern_data = <local> <ExternData data={'data': Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)]), 'classes': Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])}>
      rnd_seed = <not found>
      net_random_seed = <local> 1
      train_flag = <local> <tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>
      eval_flag = <not found>
      self.use_eval_flag = <local> True
      search_flag = <not found>
      self.use_search_flag = <local> False
      initial_learning_rate = <not found>
      getattr = <builtin> <built-in function getattr>
      net_dict = <not found>
      net_desc = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
  File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1316, in Engine.create_network
    line: network.construct_from_dict(net_dict)
    locals:
      network = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      network.construct_from_dict = <local> <bound method TFNetwork.construct_from_dict of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 588, in TFNetwork.construct_from_dict
    line: self.construct_layer(net_dict, name, get_layer=get_layer)
    locals:
      self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
      name = <local> 'output', len = 6
      get_layer = <local> None
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 914, in TFNetwork.construct_layer
    line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
    locals:
      add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'output', len = 6
      name_with_prefix = <local> 'output', len = 6
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 1059, in TFNetwork.add_layer
    line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
    locals:
      layer = <not found>
      self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'output', len = 6
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 980, in TFNetwork._create_layer
    line: layer = layer_class(**layer_desc)
    locals:
      layer = <not found>
      layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
      layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 11
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 238, in RecLayer.__init__
    line: y = self._get_output_subnet_unit(self.cell)
    locals:
      y = <not found>
      self = <local> <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>
      self._get_output_subnet_unit = <local> <bound method RecLayer._get_output_subnet_unit of <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>>
      self.cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 923, in RecLayer._get_output_subnet_unit
    line: output = cell.get_output()
    locals:
      output = <not found>
      cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      cell.get_output = <local> <bound method _SubnetworkRecCell.get_output of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 2542, in _SubnetworkRecCell.get_output
    line: self._construct_output_layers_moved_out(
            loop_accumulated=self.final_acc_tas_dict, seq_len=seq_len,
            extra_output_layers=extra_output_layers, final_net_vars=final_net_vars)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self._construct_output_layers_moved_out = <local> <bound method _SubnetworkRecCell._construct_output_layers_moved_out of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
      loop_accumulated = <not found>
      self.final_acc_tas_dict = <local> {'output_:i': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output___i'>, 'output_segment_starts': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output_segment_starts'>}
      seq_len = <local> <tf.Tensor 'output/rec/subnet_base/check_seq_len_batch_size/check_input_dim/identity_with_dim_check:0' shape=(?,) dtype=int32>
      extra_output_layers = <local> {'output', 'print2'}, len = 2
      final_net_vars = <local> ([<tf.Tensor 'output/rec/while/Exit_1:0' shape=(?,) dtype=int32>], [])
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3298, in _SubnetworkRecCell._construct_output_layers_moved_out
    line: get_layer(layer_name)
    locals:
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
      layer_name = <local> 'print2', len = 6
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3286, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
    line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      self.net_dict = <local> {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', 'segment_starts']}, 'output_prob': {'class': 'softmax', 'from': 'data:s..., len = 6
      name = <local> 'print2', len = 6
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 907, in TFNetwork.construct_layer
    line: layer_class.transform_config_dict(layer_desc, network=net, get_layer=get_layer)
    locals:
      layer_class = <local> <class 'returnn.tf.layers.basic.PrintLayer'>
      layer_class.transform_config_dict = <local> <bound method LayerBase.transform_config_dict of <class 'returnn.tf.layers.basic.PrintLayer'>>
      layer_desc = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print2'}
      network = <not found>
      net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 487, in LayerBase.transform_config_dict
    line: d["sources"] = [
            get_layer(src_name)
            for src_name in src_names
            if not src_name == "none"]
    locals:
      d = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print2'}
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
      src_name = <not found>
      src_names = <local> ['segment_lens0'], _[0]: {len = 13}
  File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 488, in <listcomp>
    line: get_layer(src_name)
    locals:
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
      src_name = <local> 'segment_lens0', len = 13
  File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3286, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
    line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
    locals:
      self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
      self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      self.net_dict = <local> {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', 'segment_starts']}, 'output_prob': {'class': 'softmax', 'from': 'data:s..., len = 6
      name = <local> 'segment_lens0', len = 13
      get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 914, in TFNetwork.construct_layer
    line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
    locals:
      add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'segment_lens0', len = 13
      name_with_prefix = <local> 'segment_lens0', len = 13
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 1059, in TFNetwork.add_layer
    line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
    locals:
      layer = <not found>
      self = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
      name = <local> 'segment_lens0', len = 13
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
  File "/mnt/data/src/returnn/returnn/tf/network.py", line 980, in TFNetwork._create_layer
    line: layer = layer_class(**layer_desc)
    locals:
      layer = <not found>
      layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
      layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna..., len = 7
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5923, in CombineLayer.__init__
    line: x = op(sources)
    locals:
      x = <not found>
      op = <local> <bound method CombineLayer._op_kind_sub of <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 6013, in CombineLayer._op_kind_sub
    line: return self._op_dense_fn(sources, tf.subtract)
    locals:
      self = <local> <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>
      self._op_dense_fn = <local> <function CombineLayer._op_dense_fn at 0x7fc4691c6940>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
      tf = <global> <module 'tensorflow' from '/home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow/__init__.py'>
      tf.subtract = <global> <function subtract at 0x7fc3f80cae50>
  File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5995, in CombineLayer._op_dense_fn
    line: x = sources[0].output.copy_compatible_to(common_data).placeholder
    locals:
      x = <not found>
      sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
      output = <not found>
      output.copy_compatible_to = <not found>
      common_data = <local> Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
      placeholder = <not found>
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 2153, in Data.copy_compatible_to
    line: v.sanity_check()
    locals:
      v = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[?],Bx])
      v.sanity_check = <local> <bound method Data.sanity_check of Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[?],Bx])>
  File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1472, in Data.sanity_check
    line: assert tag.dyn_size is not None
    locals:
      tag = <local> DimensionTag{'time'[?]}
      tag.dyn_size = <local> None
AssertionError

Process finished with exit code 1

```
I see. I can later fix that. Can you maybe prepare a minimal config where
that happens? :i is involved here.


Robin Schmitt ***@***.***> schrieb am Fr., 10. Sept. 2021,
14:04:

> So I just tried again and I now get a different error:
>
> /home/robin/miniconda3/envs/i6_env/bin/python /mnt/data/src/returnn/rnn.py /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py
> 2021-09-10 14:00:56.356147: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
> 2021-09-10 14:00:56.356161: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
> Hello, experiment: /mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function
> RETURNN starting up, version 1.0.0+unknown, date/time 2021-09-10-14-00-57 (UTC+0200), pid 2740, cwd /mnt/data/src/returnn, Python /home/robin/miniconda3/envs/i6_env/bin/python
> RETURNN command line options: ['/mnt/data/projects/i6/experiments/transducer/config/configs/segments-tests/segmentation_wo_function.py']
> Hostname: robin-HP-ENVY-Laptop-13-ba0xxx
> TensorFlow: 2.5.0 (v2.5.0-rc3-213-ga4dfb8d1a71) (<site-package> in /home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow)
> Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}.
> CUDA_VISIBLE_DEVICES is not set.
> Collecting TensorFlow device list...
> Local devices available to TensorFlow:
>   1/1: name: "/device:CPU:0"
>        device_type: "CPU"
>        memory_limit: 268435456
>        locality {
>        }
>        incarnation: 514580541836988322
> Train data:
>   input: 10 x 1
>   output: {'classes': (3, 1), 'data': (10, 2)}
>   DummyDataset, sequences: 10, frames: unknown
> 2021-09-10 14:00:57.200436: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
> To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
> 2021-09-10 14:00:57.200786: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
> 2021-09-10 14:00:57.200800: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
> 2021-09-10 14:00:57.200813: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (robin-HP-ENVY-Laptop-13-ba0xxx): /proc/driver/nvidia/version does not exist
> Device not set explicitly, and no GPU found.
> Learning-rate-control: no file specified, not saving history (no proper restart possible)
> Setup TF session with options {'log_device_placement': False, 'device_count': {'GPU': 0}} ...
> Warning: Using extern_data and will ignore num_inputs/num_outputs in config.
> Warning: assuming dim tags are same with different size placeholders: DimensionTag{'time'[B?]} vs DimensionTag{'time'[B?]}
> layer root/'data:data' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
> WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/network.py:459: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
> Instructions for updating:
> Call initializer instance with the dtype argument instead of passing it to the constructor
> layer root/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])
> WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:1350: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
> Instructions for updating:
> Call initializer instance with the dtype argument instead of passing it to the constructor
> Rec layer 'output' (search False, train 'globals/train_flag:0') sub net:
>   Input layers moved out of loop: (#: 2)
>     output
>     output_is_not_blank
>   Output layers moved out of loop: (#: 3)
>     output_prob
>     print2
>     segment_lens0
>   Layers in loop: (#: 2)
>     :i
>     segment_starts
>   Unused layers: (#: 0)
>     None
> layer root/output(rec-subnet-input)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
> layer root/output(rec-subnet-input)/'output' output: Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[B,T|'time'[B]])
> layer root/output(rec-subnet-input)/'output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B,T|'time'[B]])
> layer root/output(rec-subnet)/'prev:output_is_not_blank' output: Data(name='output_is_not_blank_output', dtype='bool', sparse=True, dim=2, batch_shape_meta=[B])
> layer root/output(rec-subnet)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[B])
> layer root/output(rec-subnet-output)/'data:source' output: Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])
> layer root/output(rec-subnet-output)/'data:classes' output: Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])
> layer root/output(rec-subnet-output)/'output_prob' output: Data(name='output_prob_output', batch_shape_meta=[B,T|'time'[B],F|F'output_prob:feature-dense'(3)])
> WARNING:tensorflow:From /mnt/data/src/returnn/returnn/tf/util/basic.py:5452: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
> Instructions for updating:
> Colocations handled automatically by placer.
> layer root/output(rec-subnet-output)/':i' output: Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[B]])
> layer root/output(rec-subnet-output)/'segment_starts' output: Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
> layer root/output(rec-subnet-output)/'segment_lens0' output: Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
> Exception creating layer root/output(rec-subnet-output)/'segment_lens0' of class CombineLayer with opts:
> {'_name': 'segment_lens0',
>  '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
>  'kind': 'sub',
>  'name': 'segment_lens0',
>  'network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
>  'output': Data(name='segment_lens0_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B]),
>  'sources': [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>,
>              <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]}
> Exception occurred during output-net construction of layer 'segment_lens0'.
> We had previous exceptions at template construction, which got resolved, but maybe sth is wrong.
> Template network (check out types / shapes):
> {':i': <_TemplateLayer(RecStepInfoLayer)(:template::i) output/':i' out_type=Data(dtype='int32', batch_shape_meta=[]) (construction stack 'segment_lens0')>,
>  'data:classes': <_TemplateLayer(SourceLayer)(:template:source) output/'data:classes' out_type=Data(dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B]) (construction stack 'output')>,
>  'data:source': <_TemplateLayer(SourceLayer)(:template:source) output/'data:source' out_type=Data(batch_shape_meta=[B,F|F'feature:data'(10)]) (construction stack 'output_prob')>,
>  'output': <_TemplateLayer(ChoiceLayer)(:template:choice) output/'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[B]) (construction stack None)>,
>  'output_is_not_blank': <_TemplateLayer(CompareLayer)(:template:compare) output/'output_is_not_blank' out_type=Data(dtype='bool', sparse=True, dim=2, batch_shape_meta=[B]) (construction stack 'segment_starts')>,
>  'output_prob': <_TemplateLayer(SoftmaxLayer)(:template:softmax) output/'output_prob' out_type=Data(batch_shape_meta=[B,F|F'output_prob:feature-dense'(3)]) (construction stack None)>,
>  'print2': <_TemplateLayer(PrintLayer)(:template:print) output/'print2' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack None)>,
>  'segment_lens0': <_TemplateLayer(CombineLayer)(:template:combine) output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'print2')>,
>  'segment_starts': <_TemplateLayer(SwitchLayer)(:template:switch) output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[B]) (construction stack 'segment_lens0')>}
> Collected (unique) exceptions during template construction:
> (Note that many of these can be ignored, or are expected.)
> EXCEPTION while constructing layer 'segment_starts'
> NetworkConstructionDependencyLoopException: <TFNetwork 'root/output(rec-subnet)' parent_net=<TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>: Error: There is a dependency loop on layer 'segment_starts'.
> Construction stack (most recent first):
>   segment_starts
>   segment_lens0
>   print2
>
> Exception occurred during output-net construction of layer 'print2'.
> Exception creating layer root/'output' of class RecLayer with opts:
> {'_name': 'output',
>  '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
>  '_time_dim_tag': DimensionTag{'time'[B]},
>  'back_prop': {'class': 'DummyDataset',
>                'input_dim': 10,
>                'num_seqs': 10,
>                'output_dim': 3,
>                'seq_len': 10},
>  'include_eos': True,
>  'n_out': <class 'returnn.util.basic.NotSpecified'>,
>  'name': 'output',
>  'network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>,
>  'output': Data(name='output_output', dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B]),
>  'sources': [<SourceLayer 'data:data' out_type=Data(batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)])>],
>  'unit': <_SubnetworkRecCell 'root/output(rec-subnet)'>}
> Unhandled exception <class 'AssertionError'> in thread <_MainThread(MainThread, started 140481628434816)>, proc 2740.
>
> Thread current, main, <_MainThread(MainThread, started 140481628434816)>:
> (Excluded thread.)
>
> That were all threads.
> EXCEPTION
> Traceback (most recent call last):
>   File "/mnt/data/src/returnn/rnn.py", line 11, in <module>
>     line: main()
>     locals:
>       main = <local> <function main at 0x7fc42816ff70>
>   File "/mnt/data/src/returnn/returnn/__main__.py", line 655, in main
>     line: execute_main_task()
>     locals:
>       execute_main_task = <global> <function execute_main_task at 0x7fc42816fe50>
>   File "/mnt/data/src/returnn/returnn/__main__.py", line 453, in execute_main_task
>     line: engine.init_train_from_config(config, train_data, dev_data, eval_data)
>     locals:
>       engine = <global> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
>       engine.init_train_from_config = <global> <bound method Engine.init_train_from_config of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
>       config = <global> <returnn.config.Config object at 0x7fc46d096ee0>
>       train_data = <global> <DummyDataset 'train' epoch=None>
>       dev_data = <global> None
>       eval_data = <global> None
>   File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1031, in Engine.init_train_from_config
>     line: self.init_network_from_config(config)
>     locals:
>       self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
>       self.init_network_from_config = <local> <bound method Engine.init_network_from_config of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
>       config = <local> <returnn.config.Config object at 0x7fc46d096ee0>
>   File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1096, in Engine.init_network_from_config
>     line: self._init_network(net_desc=net_dict, epoch=self.epoch)
>     locals:
>       self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
>       self._init_network = <local> <bound method Engine._init_network of <returnn.tf.engine.Engine object at 0x7fc3efc97490>>
>       net_desc = <not found>
>       net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
>       epoch = <local> None
>       self.epoch = <local> 1
>   File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1275, in Engine._init_network
>     line: self.network, self.updater = self.create_network(
>             config=self.config,
>             extern_data=extern_data,
>             rnd_seed=net_random_seed,
>             train_flag=train_flag, eval_flag=self.use_eval_flag, search_flag=self.use_search_flag,
>             initial_learning_rate=getattr(self, "initial_learning_rate", None),
>             net_dict=net_desc)
>     locals:
>       self = <local> <returnn.tf.engine.Engine object at 0x7fc3efc97490>
>       self.network = <local> None
>       self.updater = <local> None
>       self.create_network = <local> <bound method Engine.create_network of <class 'returnn.tf.engine.Engine'>>
>       config = <not found>
>       self.config = <local> <returnn.config.Config object at 0x7fc46d096ee0>
>       extern_data = <local> <ExternData data={'data': Data(name='data', batch_shape_meta=[B,T|'time'[B],F|F'feature:data'(10)]), 'classes': Data(name='classes', dtype='int32', sparse=True, dim=3, available_for_inference=False, batch_shape_meta=[B,T|'time'[B]])}>
>       rnd_seed = <not found>
>       net_random_seed = <local> 1
>       train_flag = <local> <tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>
>       eval_flag = <not found>
>       self.use_eval_flag = <local> True
>       search_flag = <not found>
>       self.use_search_flag = <local> False
>       initial_learning_rate = <not found>
>       getattr = <builtin> <built-in function getattr>
>       net_dict = <not found>
>       net_desc = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
>   File "/mnt/data/src/returnn/returnn/tf/engine.py", line 1316, in Engine.create_network
>     line: network.construct_from_dict(net_dict)
>     locals:
>       network = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       network.construct_from_dict = <local> <bound method TFNetwork.construct_from_dict of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 588, in TFNetwork.construct_from_dict
>     line: self.construct_layer(net_dict, name, get_layer=get_layer)
>     locals:
>       self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       self.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       net_dict = <local> {'output': {'class': 'rec', 'from': 'data:data', 'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, 'unit': {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': '...
>       name = <local> 'output', len = 6
>       get_layer = <local> None
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 914, in TFNetwork.construct_layer
>     line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
>     locals:
>       add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       name = <local> 'output', len = 6
>       name_with_prefix = <local> 'output', len = 6
>       layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
>       layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 1059, in TFNetwork.add_layer
>     line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
>     locals:
>       layer = <not found>
>       self = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       name = <local> 'output', len = 6
>       layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
>       layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 8
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 980, in TFNetwork._create_layer
>     line: layer = layer_class(**layer_desc)
>     locals:
>       layer = <not found>
>       layer_class = <local> <class 'returnn.tf.layers.rec.RecLayer'>
>       layer_desc = <local> {'include_eos': True, 'back_prop': {'class': 'DummyDataset', 'input_dim': 10, 'output_dim': 3, 'num_seqs': 10, 'seq_len': 10}, '_network': <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'output', 'n_out': <class 'returnn.util.basic.NotSpecified'>, 'sourc..., len = 11
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 238, in RecLayer.__init__
>     line: y = self._get_output_subnet_unit(self.cell)
>     locals:
>       y = <not found>
>       self = <local> <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>
>       self._get_output_subnet_unit = <local> <bound method RecLayer._get_output_subnet_unit of <RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])>>
>       self.cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 923, in RecLayer._get_output_subnet_unit
>     line: output = cell.get_output()
>     locals:
>       output = <not found>
>       cell = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
>       cell.get_output = <local> <bound method _SubnetworkRecCell.get_output of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 2542, in _SubnetworkRecCell.get_output
>     line: self._construct_output_layers_moved_out(
>             loop_accumulated=self.final_acc_tas_dict, seq_len=seq_len,
>             extra_output_layers=extra_output_layers, final_net_vars=final_net_vars)
>     locals:
>       self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
>       self._construct_output_layers_moved_out = <local> <bound method _SubnetworkRecCell._construct_output_layers_moved_out of <_SubnetworkRecCell 'root/output(rec-subnet)'>>
>       loop_accumulated = <not found>
>       self.final_acc_tas_dict = <local> {'output_:i': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output___i'>, 'output_segment_starts': <tf.TensorArray 'output/rec/subnet_base/acc_ta_output_segment_starts'>}
>       seq_len = <local> <tf.Tensor 'output/rec/subnet_base/check_seq_len_batch_size/check_input_dim/identity_with_dim_check:0' shape=(?,) dtype=int32>
>       extra_output_layers = <local> {'output', 'print2'}, len = 2
>       final_net_vars = <local> ([<tf.Tensor 'output/rec/while/Exit_1:0' shape=(?,) dtype=int32>], [])
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3298, in _SubnetworkRecCell._construct_output_layers_moved_out
>     line: get_layer(layer_name)
>     locals:
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>       layer_name = <local> 'print2', len = 6
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3286, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
>     line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
>     locals:
>       self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
>       self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       self.net_dict = <local> {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', 'segment_starts']}, 'output_prob': {'class': 'softmax', 'from': 'data:s..., len = 6
>       name = <local> 'print2', len = 6
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 907, in TFNetwork.construct_layer
>     line: layer_class.transform_config_dict(layer_desc, network=net, get_layer=get_layer)
>     locals:
>       layer_class = <local> <class 'returnn.tf.layers.basic.PrintLayer'>
>       layer_class.transform_config_dict = <local> <bound method LayerBase.transform_config_dict of <class 'returnn.tf.layers.basic.PrintLayer'>>
>       layer_desc = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print2'}
>       network = <not found>
>       net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>   File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 487, in LayerBase.transform_config_dict
>     line: d["sources"] = [
>             get_layer(src_name)
>             for src_name in src_names
>             if not src_name == "none"]
>     locals:
>       d = <local> {'is_output_layer': True, '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'print2'}
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>       src_name = <not found>
>       src_names = <local> ['segment_lens0'], _[0]: {len = 13}
>   File "/mnt/data/src/returnn/returnn/tf/layers/base.py", line 488, in <listcomp>
>     line: get_layer(src_name)
>     locals:
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>       src_name = <local> 'segment_lens0', len = 13
>   File "/mnt/data/src/returnn/returnn/tf/layers/rec.py", line 3286, in _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer
>     line: return self.output_layers_net.construct_layer(self.net_dict, name=name, get_layer=get_layer)
>     locals:
>       self = <local> <_SubnetworkRecCell 'root/output(rec-subnet)'>
>       self.output_layers_net = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       self.output_layers_net.construct_layer = <local> <bound method TFNetwork.construct_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       self.net_dict = <local> {'segment_starts': {'class': 'switch', 'condition': 'prev:output_is_not_blank', 'true_from': ':i', 'false_from': 'prev:segment_starts', 'initial_output': 0}, 'segment_lens0': {'class': 'combine', 'kind': 'sub', 'from': [':i', 'segment_starts']}, 'output_prob': {'class': 'softmax', 'from': 'data:s..., len = 6
>       name = <local> 'segment_lens0', len = 13
>       get_layer = <local> <function _SubnetworkRecCell._construct_output_layers_moved_out.<locals>.get_layer at 0x7fc468fe6160>
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 914, in TFNetwork.construct_layer
>     line: return add_layer(name=name_with_prefix, layer_class=layer_class, **layer_desc)
>     locals:
>       add_layer = <local> <bound method TFNetwork.add_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       name = <local> 'segment_lens0', len = 13
>       name_with_prefix = <local> 'segment_lens0', len = 13
>       layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
>       layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 1059, in TFNetwork.add_layer
>     line: layer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)
>     locals:
>       layer = <not found>
>       self = <local> <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
>       self._create_layer = <local> <bound method TFNetwork._create_layer of <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
>       name = <local> 'segment_lens0', len = 13
>       layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
>       layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna...
>   File "/mnt/data/src/returnn/returnn/tf/network.py", line 980, in TFNetwork._create_layer
>     line: layer = layer_class(**layer_desc)
>     locals:
>       layer = <not found>
>       layer_class = <local> <class 'returnn.tf.layers.basic.CombineLayer'>
>       layer_desc = <local> {'kind': 'sub', '_network': <TFNetwork 'root/output(rec-subnet-output)' parent_layer=<RecLayer 'output' out_type=Data(dtype='int32', sparse=True, dim=3, batch_shape_meta=[T|'time'[B],B])> train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>, '_name': 'segment_lens0', 'sources': [<Interna..., len = 7
>   File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5923, in CombineLayer.__init__
>     line: x = op(sources)
>     locals:
>       x = <not found>
>       op = <local> <bound method CombineLayer._op_kind_sub of <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>>
>       sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
>   File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 6013, in CombineLayer._op_kind_sub
>     line: return self._op_dense_fn(sources, tf.subtract)
>     locals:
>       self = <local> <CombineLayer output/'segment_lens0' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>
>       self._op_dense_fn = <local> <function CombineLayer._op_dense_fn at 0x7fc4691c6940>
>       sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
>       tf = <global> <module 'tensorflow' from '/home/robin/miniconda3/envs/i6_env/lib/python3.9/site-packages/tensorflow/__init__.py'>
>       tf.subtract = <global> <function subtract at 0x7fc3f80cae50>
>   File "/mnt/data/src/returnn/returnn/tf/layers/basic.py", line 5995, in CombineLayer._op_dense_fn
>     line: x = sources[0].output.copy_compatible_to(common_data).placeholder
>     locals:
>       x = <not found>
>       sources = <local> [<InternalLayer output/':i' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B]])>, <InternalLayer output/'segment_starts' out_type=Data(dtype='int32', batch_shape_meta=[T|F|'time'[B],B])>]
>       output = <not found>
>       output.copy_compatible_to = <not found>
>       common_data = <local> Data(name='segment_starts_output', dtype='int32', batch_shape_meta=[T|F|'time'[B],B])
>       placeholder = <not found>
>   File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 2153, in Data.copy_compatible_to
>     line: v.sanity_check()
>     locals:
>       v = <local> Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[?],Bx])
>       v.sanity_check = <local> <bound method Data.sanity_check of Data(name='i', dtype='int32', batch_shape_meta=[T|F|'time'[?],Bx])>
>   File "/mnt/data/src/returnn/returnn/tf/util/data.py", line 1472, in Data.sanity_check
>     line: assert tag.dyn_size is not None
>     locals:
>       tag = <local> DimensionTag{'time'[?]}
>       tag.dyn_size = <local> None
> AssertionError
>
> Process finished with exit code 1
>
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/rwth-i6/returnn/issues/637#issuecomment-916851489>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAON7CKXTKSBFOOSU4UWE3UBHX35ANCNFSM5DWVUSCA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

```
#!returnn/rnn.py
# kate: syntax python;
from returnn.tf.util.basic import DimensionTag

# task
use_tensorflow = True
task = "train"

# data
train = {"class": "DummyDataset", "input_dim": 10, "output_dim": 3, "num_seqs": 10, "seq_len": 10}

time_tag = DimensionTag(kind=DimensionTag.Types.Spatial, description="time")

extern_data = {
  "data": {"dim": 10, "same_dim_tags_as": {"t": time_tag}},  # Gammatone 40-dim
  "classes": {"dim": 3, "sparse": True, "same_dim_tags_as": {"t": time_tag}},
}

network = {
  "output": {
    "class": "rec", "from": "data:data",  # time-sync
    "include_eos": True, "back_prop": (task == "train") and train, "unit": {
      "out": {"class": "reinterpret_data", "from": "prev:output", "set_sparse": False},
      "segment_lens0": {"class": "combine", "kind": "sub", "from": [":i", "out"], "is_output_layer": True},

      "output_prob": {
        "class": "softmax", "from": "data:source", "target": "classes", "loss": "ce"
      },

      'output': {
        'class': 'choice', 'target': "classes", 'beam_size': 3, 'from': "output_prob", "input_type": "prob",
        "initial_output": 0,
      },
    },
  }
}

# trainer
batching = "random"
batch_size = 40
max_seqs = 40
adam = True
learning_rate = 0.01
num_epochs = 1
log_verbosity = 5
```

This one raises the same error but with `i_unrolled` instead of `i`. If you want the error with `i`, just replace the network with the (updated) one from my first comment.

Resolving patch:
diff --git a/returnn/tf/util/data.py b/returnn/tf/util/data.py
--- a/returnn/tf/util/data.py
+++ b/returnn/tf/util/data.py
@@ -135,6 +135,8 @@ def get_for_batch(self, batch):
       # If static dim, no effect.
       assert not self.batch
       return self
+    if batch.is_broadcast():
+      return self  # just leave as-is. should not matter.
     same_base = self.get_same_base()
     # Might be uninitialized in some cases. Assume batch is global.
     if not same_base.batch:
@@ -935,6 +937,12 @@ def is_global_batch(self):
     global_beam_dims = [dim for dim in self.virtual_dims if isinstance(dim, BatchInfo.GlobalBatchDim)]
     return len(global_beam_dims) == 1 and len(self.virtual_dims) == 1
 
+  def is_broadcast(self):
+    """
+    :rtype: bool
+    """
+    return len(self.virtual_dims) == 0
+
   def _make_beam_dim(self, beam):
     """
     :param SearchBeam beam:
@@ -1443,7 +1451,7 @@ def sanity_check(self, ignore_placeholder=False, assume_complete=True):
       assert axis != self.batch_dim_axis, "%s: invalid %s" % (self, tag)
       # Note: tag.kind (feature or spatial) is independent from self.feature_dim_axis.
       if tag.batch and self.batch:
-        assert tag.batch == self.batch
+        assert tag.batch == self.batch or self.batch.is_broadcast()
       if tag.dyn_size_ext:
         assert tag.dyn_size_ext.dtype in {"int32", "int64"}
         assert tag.batch == tag.dyn_size_ext.batch


Test patch:
diff --git a/tests/test_TFNetworkRecLayer.py b/tests/test_TFNetworkRecLayer.py
--- a/tests/test_TFNetworkRecLayer.py
+++ b/tests/test_TFNetworkRecLayer.py
@@ -1109,6 +1109,46 @@ def test_rec_RecStepInfoLayer():
     assert_equal(out_v.tolist(), [0, 1, 2])
 
 
+def test_rec_RecStepInfoLayer_broadcast_moved_out():
+  # https://github.com/rwth-i6/returnn/issues/637
+  net_dict = {
+    "output": {
+      "class": "rec", "from": "data",
+      "unit": {
+        "segment_starts": {
+          "class": "switch", "condition": "prev:output_is_not_blank", "true_from": ":i",
+          "false_from": "prev:segment_starts", "initial_output": 0
+        },
+        "segment_lens0": {"class": "combine", "kind": "sub", "from": [":i", "segment_starts"], "is_output_layer": True},
+        "output_prob": {
+          "class": "softmax", "from": "data:source", "target": "classes", "loss": "ce"
+        },
+        'output': {
+          'class': 'choice', 'target': "classes", 'beam_size': 3, 'from': "output_prob", "input_type": "prob",
+          "initial_output": 0,
+        },
+        "output_is_not_blank": {
+          "class": "compare", "from": "output", "value": 0, "kind": "not_equal", "initial_output": True
+        },
+      },
+    }
+  }
+  config = Config({
+    "debug_print_layer_output_template": True,
+    "extern_data": {
+      "data": {"dim": 3},
+      "classes": {"sparse": True, "dim": 5},
+    }
+  })
+  from test_TFNetworkLayer import make_feed_dict
+  with make_scope() as session:
+    net = TFNetwork(config=config)
+    net.construct_from_dict(net_dict)
+    out = net.get_default_output_layer().output
+    out_v = session.run(out.placeholder, feed_dict=make_feed_dict(net.extern_data))
+    assert isinstance(out_v, numpy.ndarray)
+
+
 def test_rec_explicit_lstm():
   net_dict = {
     "lstm": {"class": "rec", "from": "data", "unit": {