Repository: espnet/espnet

Issue:
The scoring script is incorrect if any pruning is done on the dictionary or in the case of word models
Hi @sw005320 and other developers,

The command `json2trn.py ${dir}/data.json ${dic} --num-spkrs ${num_spkrs} --refs ${dir}/ref.trn --hyps ${dir}/hyp.trn` in the `score_sclite.sh` replaces all units in reference that are not in the dictionary with unk and then scores it.

This is wrong because you are modifying the reference transcription to match your dictionary. Let's say I train a model with dictionary with only 1 term "Mr" and the hypothesis is 
"Mr unk unk" and reference is "Mr John Doe". This would give a WER of 0% because the reference would be modified to "Mr unk unk".  


Discussion:
Thanks. You're right. Let me try to fix it.

Resolving patch:
diff --git a/utils/json2trn.py b/utils/json2trn.py
--- a/utils/json2trn.py
+++ b/utils/json2trn.py
@@ -58,21 +58,25 @@ def convert(jsonf, dic, refs, hyps, num_spkrs=1):
         ref_file = codecs.open(refs[ns], 'w', encoding="utf-8")
 
         for x in j['utts']:
-            # hyps
+            # recognition hypothesis
             if num_spkrs == 1:
                 seq = [char_list[int(i)] for i in j['utts'][x]['output'][0]['rec_tokenid'].split()]
             else:
                 seq = [char_list[int(i)] for i in j['utts'][x]['output'][ns][0]['rec_tokenid'].split()]
+            # In the recognition hypothesis, the <eos> symbol is usually attached in the last part of the sentence
+            # and it is removed below.
             hyp_file.write(" ".join(seq).replace('<eos>', '')),
             hyp_file.write(" (" + j['utts'][x]['utt2spk'].replace('-', '_') + "-" + x + ")\n")
 
-            # ref
+            # reference
             if num_spkrs == 1:
-                seq = [char_list[int(i)] for i in j['utts'][x]['output'][0]['tokenid'].split()]
+                seq = j['utts'][x]['output'][0]['token']
             else:
-                seq = [char_list[int(i)] for i in j['utts'][x]['output'][ns][0]['tokenid'].split()]
-            ref_file.write(" ".join(seq).replace('<eos>', '')),
-            ref_file.write(" (" + j['utts'][x]['utt2spk'].replace('-', '_') + "-" + x + ")\n")
+                seq = j['utts'][x]['output'][ns][0]['token']
+            # Unlike the recognition hypothesis, the reference is directly generated from a token without dictionary
+            # to avoid to include <unk> symbols in the reference to make scoring normal.
+            # The detailed discussion can be found at https://github.com/espnet/espnet/issues/993
+            ref_file.write(seq + " (" + j['utts'][x]['utt2spk'].replace('-', '_') + "-" + x + ")\n")
 
         hyp_file.close()
         ref_file.close()


Test patch:
diff --git a/test_utils/test_json2trn.py.bats b/test_utils/test_json2trn.py.bats
new file mode 100644
--- /dev/null
+++ b/test_utils/test_json2trn.py.bats
@@ -0,0 +1,62 @@
+#!/usr/bin/env bats
+
+setup() {
+    [ ! -z $LC_ALL ] && export LC_ALL="en_US.UTF-8"
+
+    utils=$(cd $BATS_TEST_DIRNAME/..; pwd)/utils
+    tmpdir=$(mktemp -d testXXXXXX)
+
+    cat << EOF >> $tmpdir/valid.hyp
+A B C (spk1-uttid1)
+A <unk> <unk> (spk2-uttid2)
+EOF
+
+    cat << EOF >> $tmpdir/valid.ref
+A B C (spk1-uttid1)
+A D E (spk2-uttid2)
+EOF
+
+    cat << EOF > $tmpdir/data.json
+{
+    "utts": {
+        "uttid1": {
+            "output": [
+                {
+                    "rec_tokenid": "3 4 5",
+                    "token": "A B C"
+                }
+            ],
+            "utt2spk": "spk1"
+        },
+        "uttid2": {
+            "output": [
+                {
+                    "rec_tokenid": "3 1 1",
+                    "token": "A D E"
+                }
+            ],
+            "utt2spk": "spk2"
+        }
+    }
+}
+EOF
+
+    cat << EOF > $tmpdir/dict
+<unk> 1
+<space> 2
+A 3
+B 4
+C 5
+EOF
+
+}
+
+teardown() {
+    rm -rf $tmpdir
+}
+
+@test "json2trn.py" {
+    python $utils/json2trn.py $tmpdir/data.json $tmpdir/dict --refs $tmpdir/ref --hyps $tmpdir/hyp
+    diff $tmpdir/ref $tmpdir/valid.ref
+    diff $tmpdir/hyp $tmpdir/valid.hyp
+}