Unable to reproduce results in Emformer

Rachel_Zhang: <p>Hello everyone,<br>
I tried to reproduce the results in <a href="https://github.com/pytorch/audio/tree/99b5ef5ca0c3a48d8b703d0e1260aafe3334e844/examples/asr/emformer_rnnt" rel="noopener nofollow ugc">streaming ASR (Must-C)</a>, which shows the WER should be:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>WER</th>
</tr>
</thead>
<tbody>
<tr>
<td>dev</td>
<td>0.190</td>
</tr>
<tr>
<td>tst-COMMON</td>
<td>0.213</td>
</tr>
<tr>
<td>tst-HE</td>
<td>0.186</td>
</tr>
</tbody>
</table>
</div><p>However, following the training and eval scripts listed in the link above (except the slurm), I got the loss becomes <code>inf</code> from the second epoch:</p>
<blockquote>
<p>Epoch 2, global step 10770: ‘Losses/val_loss’ reached inf (best 72.69429), saving model to ‘pytorch/audio/examples/asr/emformer_rnnt/experiments_gradClip5.0/checkpoints/epoch=2-step=10770.ckpt’ as top 5<br>
Epoch 2, global step 10770: ‘Losses/train_loss’ reached inf (best 169.92964), saving model to ‘pytorch/audio/examples/asr/emformer_rnnt/experiments_gradClip5.0/checkpoints/epoch=2-step=10770-v1.ckpt’ as top 5</p>
</blockquote>
<p>Later, I removed <code>--gradient-clip-val 5.0</code> from the training script, and get a very large WER:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>WER</th>
</tr>
</thead>
<tbody>
<tr>
<td>dev</td>
<td>0.385</td>
</tr>
<tr>
<td>tst-COMMON</td>
<td>0.389</td>
</tr>
<tr>
<td>tst-HE</td>
<td>0.352</td>
</tr>
</tbody>
</table>
</div><p>The training log this time shows the model loss becomes <code>nan</code> from the 8-th epoch, but the  streaming ASR (Must-C) link above shows the model can be converged in the 55-th epoch (as it uses <code>epoch=55-step=106679.ckpt</code> for eval).</p>
<p>Quite confused about the <code>nan</code> and <code>inf</code> of loss. Can someone help to show me the correct script or environment? Thank you very much.</p>
<p>================================================</p>
<p>My env:<br>
GPU V100, cuda 10.2, 8 gpus and 1 node in training<br>
python 3.7.6<br>
torch ‘1.11.0+cu102’<br>
torchaudio ‘0.11.0+cu102’</p>
<p>my training script:</p>
<pre><code class="lang-auto">$PYTHON global_stats.py --model-type mustc --dataset-path $MUSTC_DATA

CUDA_VISIBLE_DEVICES=${CUDA_CARD} \
    $PYTHON train.py \
    --model-type mustc \
    --exp-dir ./experiments \
    --dataset-path $MUSTC_DATA \
    --num-nodes 1 \
    --gpus 8 \
    --global-stats-path ./global_stats.json \
    --sp-model-path ./spm_bpe_500.model \
    --debug
</code></pre>
<p>the MUST-C dataset I used:</p>
<pre><code class="lang-auto">$ wc -l *_asr.tsv
    1419 dev_asr.tsv
  225278 train_asr.tsv
    2588 tst-COMMON_asr.tsv
     601 tst-HE_asr.tsv
  229886 total
</code></pre>
<p>Others:<br>
To avoid GPU OOM, I changed the <a href="https://github.com/pytorch/audio/blob/99b5ef5ca0c3a48d8b703d0e1260aafe3334e844/examples/asr/emformer_rnnt/mustc/lightning.py#L160" rel="noopener nofollow ugc"><code>max_token_limit</code> from 100</a> to 50 in lightning.py:</p>
<pre><code class="lang-auto">        dataset = CustomDataset(MUSTC(self.mustc_path, subset="train"), 50, 20)
</code></pre>
<p>Waiting for your kind help, thank you.</p>

nateanl: <p>Hi <a class="mention" href="/u/rachel_zhang">@Rachel_Zhang</a>, thanks for sharing the info. I also met nan gradient issue when I wrote the script. It is due to large <code>vocab_size</code> when training the sentencepiece model. By decreasing it to 500 resolves the issue.</p>
<p>Could you share how you compute the global_stats.json and spm_bpe_500.model? Thanks!</p>

Rachel_Zhang: <p>Hi <a class="mention" href="/u/nateanl">@nateanl</a> , Thank you for your quick reply.<br>
<a href="https://drive.google.com/drive/folders/1iHpseosqy8MsQXjGPG_BmvFbtC6CLOva?usp=sharing" rel="noopener nofollow ugc">Here</a> shows my global_stats.json and spm_bpe_500.model.</p>
<p>Scripts for generating them:</p>
<pre><code class="lang-auto">$PYTHON global_stats.py --model-type mustc --dataset-path $MUSTC_DATA
$PYTHON mustc/train_spm.py --mustc-path $MUSTC_DATA
</code></pre>
<p>The default <code>vocab_size</code> is 500 and I haven’t changed it.</p>

nateanl: <p>Thanks! The global_stats.json and sentencepiece model should be fine. In your <code>$MUSTC_DATA</code> directory, is the dataset listed as:</p>
<pre><code class="lang-auto">en-de
  en-de/data
    en-de/data/train
    en-de/data/dev
    en-de/data/tst-COMMON
    en-de/data/tst-HE
</code></pre>

Rachel_Zhang: <p>Thank you for reply.<br>
Exactly.</p>
<pre><code class="lang-auto">$ pwd
…/speech_trans/dataset/MUST_C/en-de/data

$ tree . -L 2
.
|-- dev
|   |-- h5
|   |-- txt
|   `-- wav
|-- train
|   |-- h5
|   |-- txt
|   `-- wav
|-- tst-COMMON
|   |-- h5
|   |-- txt
|   `-- wav
`-- tst-HE
    |-- h5
    |-- txt
    `-- wav

$ wc -l */txt/*.en    
    1423 dev/txt/dev.en
  229703 train/txt/train.en
    2641 tst-COMMON/txt/tst-COMMON.en
     600 tst-HE/txt/tst-HE.en
  234367 total
</code></pre>
<p>Further, could you help me to debug, whether changing max_token_limit from 100 to 50 would lead to such collapse? (I only have V100-16G for now.)</p>

nateanl: <p>Sure. I changed <code>max_token_limit</code> to 50 on my side. It’s been running for 3 epochs and the loss is normal. Could you share the lightning log and the provide your environment information? You can get it by</p>
<pre><code class="lang-bash">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre>

Rachel_Zhang: <p>Sure. I’ve put the environment (<code>env_res.txt</code>) and lightning log (<code>experiment*.log</code>) <a href="https://drive.google.com/drive/folders/1iHpseosqy8MsQXjGPG_BmvFbtC6CLOva?usp=sharing" rel="noopener nofollow ugc">here</a>, waiting for your help to debug:)</p>

nateanl: <p>Thanks! From your <code>env_res.txt</code>. There are two versions of pytorch (pytorch 1.7.1 and torch 1.11.0). Also the cudatoolkit version (9.2) doesn’t match pytorch build (10.2).<br>
Could you try to uninstall and update pytorch, torchaudio, and cudatoolkit to make the versions consistent, then re-run the experiment to see if the nan loss issue can be resolved?</p>

nateanl: <p>Hi <a class="mention" href="/u/rachel_zhang">@Rachel_Zhang</a>, have you resolved the gradient issue?</p>