Package: kaldilm
arpa2fstPython wrapper for kaldi'sarpa2fst.Installationkaldilmcan be installed using eithercondaorpip.Using condaconda install -c k2-fsa -c conda-forge kaldilmUsing pippipinstallkaldilmIn case it doesn't work usingpip install(you can't import_kaldilm), something
likely failed during the compilation of the native part of this library.
The following steps will show you a more verbose log that can help diagnose the issue:# Remove the broken version firstpipuninstallkaldilm
pipinstall-v--no-cache-dirkaldilmTo test thatkaldilmis installed successfully, run:$ python3 -m kaldilm --helpIt should display the usage information ofkaldilm.Please create an issueon GitHubif you encounter any problems while installingkaldilm.UsageFirst, let us see the usage information of kaldi's arpa2fst:kaldi/src/lmbin$./arpa2fst
./arpa2fst

ConvertanARPAformatlanguagemodelintoanFST
Usage:arpa2fst[opts]<input-arpa><output-fst>e.g.:arpa2fst--disambig-symbol=#0 --read-symbol-table=data/lang/words.txt lm/input.arpa G.fstNote:Whencalledwithoutswitches,theoutputG.fstwillcontain
anembeddedsymboltable.Thisiscompatiblewiththewayaprevious
versionofarpa2fstworked.

Options:--bos-symbol:Beginningofsentencesymbol(string,default="
")--disambig-symbol:Disambiguator.Ifprovided(e.g.#0), used on input side of backoff links, and 
 and 
 are replaced with epsilons (string, default = "")--eos-symbol:Endofsentencesymbol(string,default="
")--ilabel-sort:Ilabel-sorttheoutputFST(bool,default=true)--keep-symbols:StoresymboltablewithFST.SymbolsalwayssavedtoFSTifsymboltablesareneitherreadorwritten(otherwisesymbolswouldbelostentirely)(bool,default=false)--max-arpa-warnings:MaximumwarningstoreportonARPAparsing,0todisable,-1toshowall(int,default=30)--read-symbol-table:Useexistingsymboltable(string,default="")--write-symbol-table:Writegeneratedsymboltabletoafile(string,default="")kaldilmuses the same arguments as kaldi's arpa2fst:$python3-mkaldilm--helpprintsusage: Python wrapper of kaldi's arpa2fst [-h] [--bos-symbol BOS_SYMBOL]
                                          [--disambig-symbol DISAMBIG_SYMBOL]
                                          [--eos-symbol EOS_SYMBOL]
                                          [--ilabel-sort ILABEL_SORT]
                                          [--keep-symbols KEEP_SYMBOLS]
                                          [--max-arpa-warnings MAX_ARPA_WARNINGS]
                                          [--read-symbol-table READ_SYMBOL_TABLE]
                                          [--write-symbol-table WRITE_SYMBOL_TABLE]
                                          [--max-order MAX_ORDER]
                                          input_arpa [output_fst]

positional arguments:
  input_arpa            input arpa filename
  output_fst            Output fst filename. If empty, no output file is
                        created.

optional arguments:
  -h, --help            show this help message and exit
  --bos-symbol BOS_SYMBOL
                        Beginning of sentence symbol (default = "
")
  --disambig-symbol DISAMBIG_SYMBOL
                        Disambiguator. If provided (e.g., #0), used on input
                        side of backoff links, and 
 and 
 are replaced
                        with epsilons (default = "")
  --eos-symbol EOS_SYMBOL
                        End of sentence symbol (default = "
")
  --ilabel-sort ILABEL_SORT
                        Ilabel-sort the output FST (default = true)
  --keep-symbols KEEP_SYMBOLS
                        Store symbol table with FST. Symbols always saved to
                        FST if symboltables are neither read or written
                        (otherwise symbols would be lost entirely) (default =
                        false)
  --max-arpa-warnings MAX_ARPA_WARNINGS
                        Maximum warnings to report on ARPA parsing, 0 to
                        disable, -1 to show all (default = 30)
  --read-symbol-table READ_SYMBOL_TABLE
                        Use existing symbol table (default = "")
  --write-symbol-table WRITE_SYMBOL_TABLE
                        (Write generated symbol table to a file (default = "")
  --max-order MAX_ORDER
                        Maximum order (inclusive) in the arpa file is used to
                        generate the final FST. If it is -1, all ngram data in
                        the file are used.If it is 1, only unigram data are
                        used.If it is 2, only ngram data up to bigram are
                        used.Default is -1.It has one extra argument--max-order, which is not present in kaldi's arpa2fst.Example usageSuppose you have an arpa fileinput.arpawith the following content:\data\
ngram 1=4
ngram 2=2
ngram 3=2

\1-grams:
-5.234679	a -3.3
-3.456783	b
0.0000000	
 -2.5
-4.333333	


\2-grams:
-1.45678	a b -3.23
-1.30490	
 a -4.2

\3-grams:
-0.34958	
 a b
-0.23940	a b 


\end\and the word symbol table iswords.txt:<eps> 0
a 1
b 2
#0 3

 4

 5Note: Numbers in the arpa file arelog10(p), while numbers on arcs
in OpenFst are-log(p)and it islog(p)ink2.log(10) = 2.3026log10(p)plog(p)note-5.2346790.000006-12.053294log(p) = log10(p) * log(10), -12.053294 = 2.3026 * (-5.234679)-3.3000000.000501-7.598531-3.4567830.000349-7.9595370.0000001.0000000.000000-2.5000000.003162-5.756463-4.3333330.000046-9.977868-1.4567800.034932-3.354360-3.2300000.000589-7.437350-1.3049000.049556-3.004643-4.2000000.000063-9.670856-0.3495800.447116-0.804938-0.2394000.576235-0.551239Caution: All symbols with ID >= the ID of #0 are set to<eps>during compiling HLG.
Seehttps://github.com/k2-fsa/icefall/blob/243fb9723cb82287ec5a891155ab9e0bc304740d/egs/librispeech/ASR/local/compile_hlg.py#L103If IDs of
and
are less than that of#0, the resulting HLG is problematic.You can use the following code to convert it into an FST.3-gramThis uses all n-gram data inside the arpa file.python3 -m kaldilm \
    --read-symbol-table="./words.txt" \
    --disambig-symbol='#0' \
    ./input.arpa > G_fst.txtThe resultingG_fst.txtis shown in the following3	5	1	1	3.00464
3	0	3	0	5.75646
0	1	1	1	12.0533
0	2	2	2	7.95954
0	9.97787
1	4	2	2	3.35436
1	0	3	0	7.59853
2	0	3	0
4	2	3	0	7.43735
4	0.551239
5	4	2	2	0.804938
5	1	3	0	9.67086which can be visualized ink2usingimportk2withopen('G_fst.txt')asf:G=k2.Fsa.from_openfst(f.read(),acceptor=False)G.labels_sym=k2.SymbolTable.from_file('words.txt')G.aux_labels_sym=k2.SymbolTable.from_file('words.txt')#G.labels[G.labels >= 3] = 0 # convert symbols with ID >= ID of #0 to epsG.draw('G.svg',title='G')G.svgis shown below:1-gramIt uses only uni-gram data inside the arpa file
since--max-order=1is used.python3 -m kaldilm \
    --read-symbol-table="./words.txt" \
    --disambig-symbol='#0' \
    --max-order=1 \
    ./input.arpa > G_uni_fst.txtThe generatedG_uni_fst.txtis3	0	3	0	5.75646
0	1	1	1	12.0533
0	2	2	2	7.95954
0	9.97787
1	0	3	0	7.59853
2	0	3	0which can be visualized ink2usingwithopen('G_uni_fst.txt')asf:G=k2.Fsa.from_openfst(f.read(),acceptor=False)G.labels_sym=k2.SymbolTable.from_file('words.txt')G.aux_labels_sym=k2.SymbolTable.from_file('words.txt')#G.labels[G.labels >= 3] = 0 # convert symbols with ID >= ID of #0 to epsG.draw('G_uni.svg',title='G_uni')G_uni.svgis shown below:What's morePlease refer tohttps://github.com/k2-fsa/icefall/blob/master/egs/librispeech/ASR/prepare.shfor howkaldilmis used inicefall.