Repository: rwth-i6/returnn

Issue:
Data dtype inconsistent in demo-hyper-param-tuning.config
In the last check, `Data` has been assigned its dtype according to the `"sparse"` attribute. `"int32"` if not sparse, `"float32"` if sparse. However, in this case the data is of type `"int64"`. I noticed that dtype is assigned or returned regardless of the true dtype in many points of the code. For istance, in the `__init__` of `Data` or in the `get_data_dtype` functions. I don't know if this is the result of a design choice to enforce these types or it just happened. 
If this is a design choice, the issue can be solved with a quick fix that converts the data from `Task12AXDataset` to the correct types (they are just 0 and 1, so it's not a problem).
```
Network layer topology:
  extern data: data: Data(shape=(None, 9), batch_shape_meta=[B,T|'time:var:extern_data:data',F|9]), classes: Data(shape=(None,), dtype='int32', sparse=True, dim=2, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes'])
  used data keys: ['classes', 'data']
  layers:
    layer source 'data' #: 9
    layer source 'data:classes' #: 2
    layer rec 'fw0' #: 20
    layer softmax 'output' #: 2
net params #: 2442
net trainable params: [<tf.Variable 'fw0/rec/W:0' shape=(9, 80) dtype=float32_ref>, <tf.Variable 'fw0/rec/W_re:0' shape=(20, 80) dtype=float32_ref>, <tf.Variable 'fw0/rec/b:0' shape=(80,) dtype=float32_ref>, <tf.Variable 'output/W:0' shape=(20, 2) dtype=float32_ref>, <tf.Variable 'output/b:0' shape=(2,) dtype=float32_ref>]
Unhandled exception <class 'AssertionError'> in thread <_MainThread(MainThread, started 140021292939008)>, proc 24887.

Thread current, main, <_MainThread(MainThread, started 140021292939008)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
  File "rnn.py", line 11, in <module>
    line: main()
    locals:
      main = <local> <function main at 0x7f593371c048>
  File "/home/mdigangi/bin/returnn/returnn/__main__.py", line 642, in main
    line: execute_main_task()
    locals:
      execute_main_task = <global> <function execute_main_task at 0x7f5933712ea0>
  File "/home/mdigangi/bin/returnn/returnn/__main__.py", line 535, in execute_main_task
    line: tuner.work()
    locals:
      tuner = <local> <returnn.tf.hyper_param_tuning.Optimization object at 0x7f5834019550>
      tuner.work = <local> <bound method Optimization.work of <returnn.tf.hyper_param_tuning.Optimization object at 0x7f5834019550>>
  File "/home/mdigangi/bin/returnn/returnn/tf/hyper_param_tuning.py", line 553, in work
    line: _IndividualTrainer(optim=self, individual=population[0], gpu_ids={0}).run()
    locals:
      _IndividualTrainer = <global> <class 'returnn.tf.hyper_param_tuning._IndividualTrainer'>
      optim = <not found>
      self = <local> <returnn.tf.hyper_param_tuning.Optimization object at 0x7f5834019550>
      individual = <not found>
      population = <local> [<returnn.tf.hyper_param_tuning.Individual object at 0x7f583403dda0>, <returnn.tf.hyper_param_tuning.Individual object at 0x7f583403dc88>, <returnn.tf.hyper_param_tuning.Individual object at 0x7f583403dc50>, <returnn.tf.hyper_param_tuning.Individual object at 0x7f583403d860>, <returnn.tf.hyper_pa..., len = 30
      gpu_ids = <not found>
      run = <not found>
  File "/home/mdigangi/bin/returnn/returnn/tf/hyper_param_tuning.py", line 655, in run
    line: trainer = Runner(engine=engine, dataset=train_data, batches=batches, train=True)
    locals:
      trainer = <not found>
      Runner = <global> <class 'returnn.tf.engine.Runner'>
      engine = <local> <returnn.tf.engine.Engine object at 0x7f59212df2b0>
      dataset = <not found>
      train_data = <local> <StaticDataset 'dataset_id140020785477224' epoch=1>
      batches = <local> <returnn.engine.batch.BatchSetGenerator object at 0x7f58871bc2b0>
      train = <not found>
  File "/home/mdigangi/bin/returnn/returnn/tf/engine.py", line 79, in __init__
    line: engine.network.extern_data.check_matched_dataset(
            dataset=dataset, used_data_keys=engine.network.get_used_data_keys())
    locals:
      engine = <local> <returnn.tf.engine.Engine object at 0x7f59212df2b0>
      engine.network = <local> <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>
      engine.network.extern_data = <local> <ExternData data={'data': Data(name='data', shape=(None, 9), batch_shape_meta=[B,T|'time:var:extern_data:data',F|9]), 'classes': Data(name='classes', shape=(None,), dtype='int32', sparse=True, dim=2, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes'])}>
      engine.network.extern_data.check_matched_dataset = <local> <bound method ExternData.check_matched_dataset of <ExternData data={'data': Data(name='data', shape=(None, 9), batch_shape_meta=[B,T|'time:var:extern_data:data',F|9]), 'classes': Data(name='classes', shape=(None,), dtype='int32', sparse=True, dim=2, available_for_inference=False, batch_shape_meta...
      dataset = <local> <StaticDataset 'dataset_id140020785477224' epoch=1>
      used_data_keys = <not found>
      engine.network.get_used_data_keys = <local> <bound method TFNetwork.get_used_data_keys of <TFNetwork 'root' train=<tf.Tensor 'globals/train_flag:0' shape=() dtype=bool>>>
  File "/home/mdigangi/bin/returnn/returnn/tf/network.py", line 122, in check_matched_dataset
    line: assert data.dtype == data_dtype, "key %r dtype mismatch. %s" % (key, base_err_msg)
    locals:
      data = <local> Data(name='classes', shape=(None,), dtype='int32', sparse=True, dim=2, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes'])
      data.dtype = <local> 'int32'
      data_dtype = <local> dtype('int64'), len = 0
      key = <local> 'classes', len = 7
      base_err_msg = <local> "<StaticDataset 'dataset_id140020785477224' epoch=1> num_outputs {'data': (9, 2), 'classes': (2, 1)} vs <ExternData data={'data': Data(name='data', shape=(None, 9), batch_shape_meta=[B,T|'time:var:extern_data:data',F|9]), 'classes': Data(name='classes', shape=(None,), dtype='int32', sparse=True, ..., len = 390
AssertionError: key 'classes' dtype mismatch. <StaticDataset 'dataset_id140020785477224' epoch=1> num_outputs {'data': (9, 2), 'classes': (2, 1)} vs <ExternData data={'data': Data(name='data', shape=(None, 9), batch_shape_meta=[B,T|'time:var:extern_data:data',F|9]), 'classes': Data(name='classes', shape=(None,), dtype='int32', sparse=True, dim=2, available_for_inference=False, batch_shape_meta=[B,T|'time:var:extern_data:classes'])}>
```


Discussion:

Resolving patch:
diff --git a/returnn/datasets/generating.py b/returnn/datasets/generating.py
--- a/returnn/datasets/generating.py
+++ b/returnn/datasets/generating.py
@@ -288,7 +288,7 @@ def generate_seq(self, seq_idx):
     input_seq = self.generate_input_seq(seq_len)
     output_seq = self.make_output_seq(input_seq)
     features = class_idx_seq_to_1_of_k(input_seq, num_classes=len(self._input_classes))
-    targets = numpy.array(output_seq)
+    targets = numpy.array(output_seq, dtype='int32')
     return DatasetSeq(seq_idx=seq_idx, features=features, targets=targets)
 
 
diff --git a/returnn/util/basic.py b/returnn/util/basic.py
--- a/returnn/util/basic.py
+++ b/returnn/util/basic.py
@@ -1250,7 +1250,7 @@ def class_idx_seq_to_1_of_k(seq, num_classes):
   :rtype: np.ndarray
   """
   num_frames = len(seq)
-  m = np.zeros((num_frames, num_classes))
+  m = np.zeros((num_frames, num_classes), dtype='float32')
   m[np.arange(num_frames), seq] = 1
   return m
 


Test patch: