Repository: rwth-i6/returnn

Issue:
Speaker tags in SprintDataset
Currently there is no support to use speaker labels via the `ExternSprintDataset`.

In combination with 80d0a3845ca60ad78d49b25beeb5e22342b44ca7 I figured out that replacing
``` python
    self.num_outputs["speaker_name"] = (256, 1)
```
by
```python
    if "speaker_name" in self.target_maps:
      self.num_outputs["speaker_name"] = (len(self.target_maps["speaker_name"]), 1)
    else:
      self.num_outputs["speaker_name"] = (256, 1)
```
in https://github.com/rwth-i6/returnn/blob/master/returnn/datasets/sprint.py#L140
would be sufficient to get working embeddings.

However there are some issues here:
 - `speaker_name` is then containing indices instead of the utf8 string, which is counter-intuitive
 - It seems that `target_maps` can only be used for `speaker_name` and for nothing else, as the corresponding entry in `self.num_outputs` will always be wrong, thus causing a crash at some point. Also, there is no test case using `target_maps`.
 
 So my question would here be: should I simply proceed with the small fix (takes no time, would work for my use case) or should we refactor this to be better designed, meaning removing the possibly useless/broken `target_maps` parameter and code, and introducing a consistent and documented `speaker_map` parameter that will create a new output called `speaker_classes` (consistent naming with with `orth_classes`) based on the map file.


Discussion:
`target_maps` is probably also only rarely used. This seems to be very old code. I don't understand why you think this is only for `speaker_name`. This can be used for any data with fixed number of classes. But you are right that it might need some fixed depending on how this is used, e.g. fixing `num_outputs`.

But I wonder now, I think some people made use of this, e.g. Markus Kitza for speaker adaptation, or maybe also @vieting or Tobias Menne. How does their setup look like?

I'm quite sure that there are some setups using `target_maps`. Maybe only older ones.

But in any case, yes, the `len(self.target_maps["speaker_name"])` change should be pushed, and we could later just handle other cases.

> I don't understand why you think this is only for `speaker_name`

No this is not what I meant, I meant after the fix it would work only for `speaker_name`, but for nothing else. So basically what you confirmed. At least from Markus setups I know that they did not use this (the i-vector extraction code is independent from RETURNN training).

Ok, I will push the change.

Resolving patch:
diff --git a/returnn/datasets/sprint.py b/returnn/datasets/sprint.py
--- a/returnn/datasets/sprint.py
+++ b/returnn/datasets/sprint.py
@@ -55,7 +55,9 @@ def __init__(self, target_maps=None, str_add_final_zero=False, input_stddev=1.,
                suppress_load_seqs_print=False,
                **kwargs):
     """
-    :param dict[str,str|dict] target_maps: e.g. {"speaker": "speaker_map.txt"}
+    :param dict[str,str|dict] target_maps: e.g. {"speaker_name": "speaker_map.txt"}, with "speaker_map.txt" containing
+      a line for each expected speaker. The indices will be given by the line index. Note that scalar
+      content (e.g. single index) will automatically get a time axis added with the length of the audio frames.
     :param bool str_add_final_zero: adds e.g. "orth0" with '\0'-ending
     :param float input_stddev: if != 1, will divide the input "data" by that
     :param str|list[str]|((str)->str)|None orth_post_process: :func:`get_post_processor_function`, applied on orth
@@ -137,7 +139,10 @@ def set_dimensions(self, input_dim, output_dim):
     if self.orth_vocab:
       self.num_outputs["orth_classes"] = (self.orth_vocab.num_labels, 1)
     self.num_outputs["orth"] = (256, 1)
-    self.num_outputs["speaker_name"] = (256, 1)
+    if self.target_maps and "speaker_name" in self.target_maps:
+      self.num_outputs["speaker_name"] = (len(self.target_maps["speaker_name"]), 1)
+    else:
+      self.num_outputs["speaker_name"] = (256, 1)
     self._base_init()
     # At this point, we are ready for data. In case we don't use the Sprint PythonSegmentOrdering
     # (SprintInterface.getSegmentList()), we must call this at least once.


Test patch: