Repository: espnet/espnet

Issue:
Vertical attention during training
I am using ESPNet 1 ljspeech-like dataset for pretrained weights and am getting some attention graphs like this in training which I haven't seen before:
![individualImage](https://user-images.githubusercontent.com/62999745/89983349-f8349200-dca9-11ea-8f46-2ae8b7300890.png)

I recently updated to latest commit and I know there are lots of ESPNet2 changes so could this be version related?
I init from pretrained tacotron v4 with BCE-pos-weight 2.0
Vertical attention during training
I am using ESPNet 1 ljspeech-like dataset for pretrained weights and am getting some attention graphs like this in training which I haven't seen before:
![individualImage](https://user-images.githubusercontent.com/62999745/89983349-f8349200-dca9-11ea-8f46-2ae8b7300890.png)

I recently updated to latest commit and I know there are lots of ESPNet2 changes so could this be version related?
I init from pretrained tacotron v4 with BCE-pos-weight 2.0


Discussion:
Basically such changes would not cause this issue, but it might be.
It would be great if you try it with several different versions (e.g., v08, v07), and identify the issue due to the version or your ljspeech-like dataset-specific problems.
@sw005320 I just try on v0.8.0 and the attention graphs look normal.
Thanks. Then, it would be a bug. @kan-bayashi, could you check this issue?
How did you get this attention plot?
Is this a plot during training?
Yes, from images tab in tensorboard, even after 11k steps.
I use torch 1.4.0 espnet1 commit b2d1d446 and two GPU with v4 tacotron and pretrained model
Thanks. Could you attach the normal attention plot for the same data in v0.8.0?
(I checked diff between v0.8.0 and v0.9.0 but I could not find the change in espnet1 tts and Tacotron2 part.)
Sorry I did not save exact same data but here is a 0.8.0 graph after some small steps. I find these graphs helpful to track down utts with bad attention (there is sadly a limit to num-save-attention I think but I have set to 32 here).
![individualImage](https://user-images.githubusercontent.com/62999745/90045556-c9e2a100-dd01-11ea-800d-04da2ff2d1aa.png)

Thanks.
Could you try to remove `[::-1]` here?
https://github.com/espnet/espnet/blob/5410797ebc8c4db040a3c0e23eb47051ababe684/espnet/asr/asr_utils.py#L115
Yes that seems to have fixed it.
![pre](https://user-images.githubusercontent.com/62999745/90048820-96564580-dd06-11ea-9b24-c84d3230001a.png)
(different data but was wrong before change)
![post](https://user-images.githubusercontent.com/62999745/90048827-98200900-dd06-11ea-9273-30f4fcd27dd6.png)

Thank you for your checking.
It really helps us!

@hirofumi0810 Is this change for ASR or ST?
https://github.com/espnet/espnet/commit/8eae4ff81f2708dc3552ae863a1b97b40a782a9a#diff-405f988e9d432ff8a43112abd8094233
Thanks. If I could suggest it would be helpful to be able to see all attention plots in tensorboard if I set num-save-attention high. It is nice to check problematic input files.
Basically such changes would not cause this issue, but it might be.
It would be great if you try it with several different versions (e.g., v08, v07), and identify the issue due to the version or your ljspeech-like dataset-specific problems.
@sw005320 I just try on v0.8.0 and the attention graphs look normal.
Thanks. Then, it would be a bug. @kan-bayashi, could you check this issue?
How did you get this attention plot?
Is this a plot during training?
Yes, from images tab in tensorboard, even after 11k steps.
I use torch 1.4.0 espnet1 commit b2d1d446 and two GPU with v4 tacotron and pretrained model
Thanks. Could you attach the normal attention plot for the same data in v0.8.0?
(I checked diff between v0.8.0 and v0.9.0 but I could not find the change in espnet1 tts and Tacotron2 part.)
Sorry I did not save exact same data but here is a 0.8.0 graph after some small steps. I find these graphs helpful to track down utts with bad attention (there is sadly a limit to num-save-attention I think but I have set to 32 here).
![individualImage](https://user-images.githubusercontent.com/62999745/90045556-c9e2a100-dd01-11ea-800d-04da2ff2d1aa.png)

Thanks.
Could you try to remove `[::-1]` here?
https://github.com/espnet/espnet/blob/5410797ebc8c4db040a3c0e23eb47051ababe684/espnet/asr/asr_utils.py#L115
Yes that seems to have fixed it.
![pre](https://user-images.githubusercontent.com/62999745/90048820-96564580-dd06-11ea-9b24-c84d3230001a.png)
(different data but was wrong before change)
![post](https://user-images.githubusercontent.com/62999745/90048827-98200900-dd06-11ea-9273-30f4fcd27dd6.png)

Thank you for your checking.
It really helps us!

@hirofumi0810 Is this change for ASR or ST?
https://github.com/espnet/espnet/commit/8eae4ff81f2708dc3552ae863a1b97b40a782a9a#diff-405f988e9d432ff8a43112abd8094233
Thanks. If I could suggest it would be helpful to be able to see all attention plots in tensorboard if I set num-save-attention high. It is nice to check problematic input files.

Resolving patch:
diff --git a/espnet/nets/pytorch_backend/tacotron2/encoder.py b/espnet/nets/pytorch_backend/tacotron2/encoder.py
--- a/espnet/nets/pytorch_backend/tacotron2/encoder.py
+++ b/espnet/nets/pytorch_backend/tacotron2/encoder.py
@@ -24,11 +24,9 @@ class Encoder(torch.nn.Module):
     """Encoder module of Spectrogram prediction network.
 
     This is a module of encoder of Spectrogram prediction network in Tacotron2,
-    which described in `Natural TTS
-    Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`_.
-    This is the encoder which converts either a
-    sequence of characters or acoustic features
-    into the sequence of hidden states.
+    which described in `Natural TTS Synthesis by Conditioning WaveNet on Mel
+    Spectrogram Predictions`_. This is the encoder which converts either a sequence
+    of characters or acoustic features into the sequence of hidden states.
 
     .. _`Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions`:
        https://arxiv.org/abs/1712.05884
@@ -54,7 +52,7 @@ def __init__(
 
         Args:
             idim (int) Dimension of the inputs.
-            :param str input_layer: input layer type
+            input_layer (str): Input layer type.
             embed_dim (int, optional) Dimension of character embedding.
             elayers (int, optional) The number of encoder blstm layers.
             eunits (int, optional) The number of encoder blstm units.
@@ -133,10 +131,9 @@ def forward(self, xs, ilens=None):
         """Calculate forward propagation.
 
         Args:
-            xs (Tensor): Batch of the padded sequence.
-                Either character ids (B, Tmax)
-                    or acoustic feature (B, Tmax, idim * encoder_reduction_factor).
-                Padded value should be 0.
+            xs (Tensor): Batch of the padded sequence. Either character ids (B, Tmax)
+                or acoustic feature (B, Tmax, idim * encoder_reduction_factor). Padded
+                value should be 0.
             ilens (LongTensor): Batch of lengths of each input batch (B,).
 
         Returns:
diff --git a/espnet/tts/pytorch_backend/tts.py b/espnet/tts/pytorch_backend/tts.py
--- a/espnet/tts/pytorch_backend/tts.py
+++ b/espnet/tts/pytorch_backend/tts.py
@@ -488,7 +488,6 @@ def train(args):
         data = sorted(
             list(valid_json.items())[: args.num_save_attention],
             key=lambda x: int(x[1]["output"][0]["shape"][0]),
-            reverse=True,
         )
         if hasattr(model, "module"):
             att_vis_fn = model.module.calculate_all_attentions


Test patch: