Repository: rwth-i6/returnn

Issue:
`InvalidArgumentError: Retval[0] does not have value` in loss computation
I am getting a strange error related to loss computation i think. This also happen when computing loss scores on dev.

Example:
- Error with CTC loss: https://gist.github.com/mmz33/b40fb4099790f0048ece9553c1fab6ea
- Error with CL loss: https://gist.github.com/mmz33/96cb48cc034c1221add2aa4c542c6019

Config with CTC loss: https://gist.github.com/mmz33/17ff9794a98ab0fee02d341e48411cb1

Error:
```
InvalidArgumentError: Retval[0] does not have value
```
Looks related to this: https://github.com/tensorflow/tensorflow/issues/32106

If I checkout RETURNN with older version before this commit below, the issue does not occur:

```
commit e819b63be89fd6dae77e7d40950a8d990031b6a3
Author: Albert Zeyer <albzey@gmail.com>
Date:   Fri Feb 11 11:16:48 2022 +0100

    flatten_with_seq_len_mask cache (#944)
```


Discussion:
It’s really strange. It's probably non-deterministic and this change (#944) might just have introduced it randomly. Not sure if this i really related to the loss. From other similar reports, it sounds like it is an issue with TF variables + control flow logic.

In #961, I try to update to control flow V2. In some other error reports (https://github.com/tensorflow/tensorflow/issues/32106) it is reported that this should fix such errors.

Introducing control flow V2 has its own problems (see the process in #961) and I'm not really sure about it anymore.

I wonder if there is another way to fix this problem here.

Resolving patch:
diff --git a/returnn/tf/util/basic.py b/returnn/tf/util/basic.py
--- a/returnn/tf/util/basic.py
+++ b/returnn/tf/util/basic.py
@@ -1947,7 +1947,7 @@ def flatten_with_seq_len_mask(x, seq_lens, batch_dim_axis=None, time_dim_axis=No
   cache = get_flatten_with_seq_len_mask_cache(x, seq_lens, batch_dim_axis, time_dim_axis)
   if cache.has_cache():
     return cache.get_cache()
-  with tf.name_scope("flatten_with_seq_len_mask"):
+  with same_control_flow_ctx([x, seq_lens]), tf.name_scope("flatten_with_seq_len_mask"):
     seq_lens = check_input_ndim(seq_lens, 1)
     # If not (batch,time,...s...), transform.
     if batch_dim_axis != 0 or time_dim_axis != 1:


Test patch:
diff --git a/tests/test_TFEngine.py b/tests/test_TFEngine.py
--- a/tests/test_TFEngine.py
+++ b/tests/test_TFEngine.py
@@ -1570,6 +1570,57 @@ def make_net_dict(task):
   engine.finalize()
 
 
+def test_ctc_bn_train_eval():
+  # https://github.com/rwth-i6/returnn/issues/962
+  from returnn.datasets.generating import TaskNumberBaseConvertDataset
+  input_dim, output_dim = 2, 8
+  train_data = TaskNumberBaseConvertDataset(input_base=input_dim, output_base=output_dim, num_seqs=2)
+  train_data.init_seq_order(epoch=1)
+  dev_data = TaskNumberBaseConvertDataset(input_base=input_dim, output_base=output_dim, num_seqs=2)
+  dev_data.init_seq_order(epoch=1)
+
+  def make_net_dict():
+    """
+    :rtype: dict[str,dict[str]]
+    """
+    return {
+      "source": {
+        "class": "linear", "n_out": 10, "from": "data",
+      },
+      "bn": {
+        "class": "batch_norm", "from": "source",
+        "momentum": 0.1, "epsilon": 1e-3, "update_sample_only_in_training": False, "delay_sample_update": False,
+        "masked_time": True,
+      },
+      "encoder": {"class": "copy", "from": "bn"},
+      "ctc": {
+          "class": "softmax",
+          "from": "encoder",
+          "target": "classes",
+          "loss": "ctc",
+          "loss_opts": {"beam_width": 1, "use_native": True},
+      },
+    }
+
+  config = Config()
+  config.update({
+    "model": "%s/model" % _get_tmp_dir(),
+    "batch_size": 1000,
+    "max_seqs": 2,
+    "extern_data": {"data": {"dim": input_dim, "sparse": True}, "classes": {"dim": output_dim, "sparse": True}},
+    "network": make_net_dict(),
+    "search_output_layer": "decision",
+    "debug_print_layer_output_template": True
+  })
+  _cleanup_old_models(config)
+  engine = Engine(config=config)
+  print("Train...")
+  engine.init_train_from_config(config=config, train_data=train_data, dev_data=dev_data)
+  engine.train()
+
+  engine.finalize()
+
+
 def check_train_and_search_two_targets(net_dict):
   """
   Tests training and search for network architectures having two targets ("classes_0", "classes_1")