Repository: espnet/espnet

Issue:
CTC Segmentation Questions
I try to modify tedlium2/align1  CTCsegment example to another example like egs/aishell
but report error as following
I have no idea to solve this index error.

```2020-09-28 01:37:54,391 (asr_align:208) INFO: Decoding device=cuda
2020-09-28 01:38:09,148 (asr_align:245) INFO: (1/14326) Aligning BAC009S0724W0121
2020-09-28 01:38:10,293 (ctc_segmentation:66) INFO: CTC segmentation of 19 chars to 4.20s audio (105 indices).
2020-09-28 01:38:10,296 (ctc_segmentation:140) WARNING: IndexError: Backtracking was not successful, the window size might be too small.
2020-09-28 01:38:10,296 (ctc_segmentation:145) WARNING: Increasing the window size to: 16000
2020-09-28 01:38:10,298 (ctc_segmentation:140) WARNING: IndexError: Backtracking was not successful, the window size might be too small.
2020-09-28 01:38:10,299 (ctc_segmentation:145) WARNING: Increasing the window size to: 32000
2020-09-28 01:38:10,301 (ctc_segmentation:140) WARNING: IndexError: Backtracking was not successful, the window size might be too small.
2020-09-28 01:38:10,301 (ctc_segmentation:145) WARNING: Increasing the window size to: 64000
2020-09-28 01:38:10,303 (ctc_segmentation:140) WARNING: IndexError: Backtracking was not successful, the window size might be too small.
2020-09-28 01:38:10,303 (ctc_segmentation:148) ERROR: Maximum window size reached.
2020-09-28 01:38:10,304 (ctc_segmentation:149) ERROR: Check data and character list!
Traceback (most recent call last):
  File "/mnt/nas2/*****/espnet/tools/venv/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/mnt/nas2/*****/espnet/tools/venv/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/nas2/*****/espnet/espnet/bin/asr_align.py", line 277, in <module>
    main(sys.argv[1:])
  File "/mnt/nas2/*****/espnet/espnet/bin/asr_align.py", line 180, in main
    ctc_align(args, device)
  File "/mnt/nas2/*****/espnet/espnet/bin/asr_align.py", line 260, in ctc_align
    config, lpz, ground_truth_mat
  File "/mnt/nas2/*****/espnet/tools/venv/lib/python3.7/site-packages/ctc_segmentation/ctc_segmentation.py", line 119, in ctc_segmentation
    est_stay_prob = table[t, c] - table[t - 1, c]
IndexError: index -106 is out of bounds for axis 0 with size 105 
```


I try to run original tedlium2/align1 CTCsegment example.
Another question when stage 3: Alignments using CTC segmentation it cost a lot of memory. (run on  tedlium2/align1, align_set is test)
It will out of memory when I use gpu(TITAN RTX)
I try to use cpu run stage 3, it cost over 200G RAM, and out of memory too. 
If reduce test set to only 1 utt  stage 3 will success.
How can I Alignments all align_set ?

Thanks you!



Discussion:
**IndexError for backtracking:**
this usually means that text and audio data are misaligned. More precisely, the optimal alignment of an utterance is not within the window size.

This error happens if the blank character was not set correctly (or similar characters). Use the option `--use-dict-blank` of `asr_align.py` to automatically set this parameter from the model dictionary. Also,  check your character list - the same dictionary that you used to train your model should also be used for inputting the label sequence.

Furthermore, in your case you also might adapt the window parameters. The parameters for min and max window size were set so that the length of each utterance (in terms of frames of CTC outputs) fits into that window. The default window is 8000, your audio file is much shorter. You may reduce this value to improve alignment speed.

**Regarding memory consumption:**
For aligning, an inference of the full audio file is done on the GPU. Here, the memory consumption depends on how much memory your model consumes to infer the audio file. Depends on model size, subsampling etc.

The RAM memory consumption (CPU) depends on primarily (1) the size of the audio file times text length and (2) to what extent your model uses subsampling. That's because all of the CTC layer outputs are stored for CTC segmentation. If your audio file is very large, you will run into large RAM memory consumption. To reduce the memory consumption, e.g. use a model with subsampling in the encoder.

In the CTC segmentation paper, we aligned audio files of up to several hours. The memory consumption of CTC activations alone is roughly ~8 GB for one hour audio, but only ~1.8 GB with 4x subsampling (assumed default settings).
Our default models in most recipes are based on transformer (self-attention), which requires N^2 memory and is not adequate for such purposes (where N is the subsampled length). I recommend you to try it with BLSTM models (usually espnet1 recipe also provides a BLSTM config file). We actually prepared a BLSTM model when we apply CTC segmentation to other recipes.
Thanks a lot.
I use BLSTM model for tedlium2/align1 CTCsegment example success!
I am training a BLSTM model for aishell try to avoid IndexError now.

@s920128, please let us know here if the BLSTM model works.

@lumaku, if the issue comes from self-attention, I think it would be better to add such a note in README.md, and add some warning when we use transformer or conformer models in utils/ctc_align_wav.sh. 

Resolving patch:
diff --git a/espnet/bin/asr_align.py b/espnet/bin/asr_align.py
--- a/espnet/bin/asr_align.py
+++ b/espnet/bin/asr_align.py
@@ -206,6 +206,17 @@ def ctc_align(args, device):
         preprocess_args={"train": False},
     )
     logging.info(f"Decoding device={device}")
+    # Warn for nets with high memory consumption on long audio files
+    if hasattr(model, "enc"):
+        encoder_module = model.enc.__class__.__module__
+    elif hasattr(model, "encoder"):
+        encoder_module = model.encoder.__class__.__module__
+    else:
+        encoder_module = "Unknown"
+    logging.info(f"Encoder module: {encoder_module}")
+    logging.info(f"CTC module:     {model.ctc.__class__.__module__}")
+    if "rnn" not in encoder_module:
+        logging.warning("No BLSTM model detected; memory consumption may be high.")
     model.to(device=device).eval()
     # read audio and text json data
     with open(args.data_json, "rb") as f:
@@ -237,7 +248,13 @@ def ctc_align(args, device):
     char_list = train_args.char_list
     if args.use_dict_blank:
         config.blank = char_list[0]
-    logging.debug(
+        logging.info(f"Blank char was set to >{config.blank}<")
+    else:
+        logging.info(f"Blank char >{config.blank}< (align) >{char_list[0]}< (model)")
+        if config.blank != char_list[0]:
+            logging.error("Blank char mismatch; this can result in an IndexError.")
+            logging.error("Pass the parameter --use-dict-blank to asr_align.py")
+    logging.info(
         f"Frame timings: {config.frame_duration_ms}ms * {config.subsampling_factor}"
     )
     # Iterate over audio files to decode and align


Test patch: