nbest-lattice

nbest-lattice

NAME

nbest-lattice - rescore N-best lists and lattices

SYNOPSIS

nbest-lattice [-help] option ...

DESCRIPTION

nbest-lattice rescores N-best lists or optimizes word-level recognition scores (as opposed to sentence-level scores). There are two rescoring modes. In N-best word error minimization mode, the program computes the posterior expected word error for each hypothesis relative to all hypotheses in the N-best list, choosing the one with the lowest value.

In lattice word error minimization mode, the program constructs a word lattice from all the N-best hypotheses and extracts the path with the lowest expected word error. This is similar to N-best word error minimization but allows hypotheses not contained in the N-best list. A variant of this mode uses a word ``mesh'' instead of a word lattice, in which all hypotheses are aligned into a grid of word positions, and one is allowed to chose a word from each grid position, thus allowing an even greater number of potential hypotheses.

Each filename argument can be an ASCII file, or a compressed file (name ending in .Z or .gz), or ``-'' to indicate stdin/stdout.

OPTIONS

-help
Print option summary.
-debug level
Controls the amount of output (the higher the level, the more). At level 1, the expected word error counts for the chosen hypotheses are printed. At level 2, the word posterior probabilities are printed in addition (only for lattice mode, similar to -dump-posteriors).
-wer
Chooses N-best word error minimization mode.
-lattice-wer
Chooses lattice word error minimization mode (the default).
-use-mesh
Choose the variant of lattice mode that uses word meshes instead of simple lattices.
-deletion-bias D
Causes the probabilities of deletions to be biased by a factor D in doing mesh-based word error minimization. This controls the trade-off between insertion and deletion errors. The default is 1 (no bias).
-rescore file
Reads the N-best list from file. The N-best list can be in any of the formats described in nbest-format(5).
-nbest file
A synonym for -rescore.
-write-nbest file
Outputs the N-best list to a file, after sorting and processing (for validation purposes).
-nbest-files file-list
Rescores multiple N-best lists whose filenames are read from file-list.
-max-nbest n
Limits the number of hypotheses read from each N-best list to the first n.
-max-rescore m
Only choose among the top m hypotheses when optimizing word error. This is convenient to limit computation for long N-best lists. The cutoff is made after reading all hypotheses (subject to -max-nbest) and reordering them according to the posterior probabilities.
The worst-case time taken in N-best error minimization is proportional to m times n, where n is the length of the N-best list (or the value given to -max-nbest). However, in practice the average time per sentence is independent of m, so this option is usually not necessary.
-posterior-prune threshold
Don't process N-best hypotheses whose cumulative posterior probability is below threshold. This is another strategy to speed up the algorithm.
-no-reorder
Process N-best hypotheses in the order in which they appear. By default, hypotheses are first sorted by their aggregate scores.
-rescore-lmw lmw
Sets the language model weight used in combining the language model log probabilities with acoustic log probabilities (only relevant if separate scores are given in the N-best input).
-rescore-wtw wtw
Sets the word transition weight used to weight the number of words relative to the acoustic log probabilities (only relevant if separate scores are given in the N-best input).
-posterior-scale scale
Divide the total weighted log score by scale when computing normalized posterior probabilities. This controls the peakedness of the posterior distribution. The default value is whatever was chosen for -rescore-lmw, so that language model scores are scaled to have weight 1, and acoustic scores have weight 1/lmw.
-posterior-amw amw
Sets the acoustic model weight for computing posteriors; the default is 1. This and the next two options allow posteriors to be computed using a different weighting than that used in ranking and reordering the hypotheses.
-posterior-lmw lmw
Sets the language model weight for computing posteriors. The default is to use whatever was specified for -rescore-lmw.
-posterior-wtw wtw
Sets the word transition weight for computing posteriors. The default is to use whatever was specified for -rescore-wtw.
-vocab file
Read the N-best list vocabulary from file. This option is mostly redundant since words found in the N-best input are implicitly added to the vocabulary.
-tolower
Map vocabulary to lowercase, eliminating case distinctions.
-multiwords
Split multiwords (words joined by '_') into their components when reading N-best lists.
-noise noise-tag
Designate noise-tag as a vocabulary item that is to be ignored in aligning hypotheses with each other (the same as the -pau- word). This is typically used to identify a noise marker.
-noise-vocab file
Read several noise tags from file, instead of, or in addition to, the single noise tag specified by -noise.
-keep-noise
Do not remove pause or noise tokens from hypotheses. The default is to preserve noise tags but still eliminate pauses.
-nbest-errors references
Compute the N-best error (minimum word error) of the N-best list read with -nbest. Error is computed relative to all the hypotheses contained in a separate N-best list given by references. Pause and noise tokens (as specified with -noise) in the N-best list are ignored.
-dump-posteriors
Output posterior probabilities of all N-best hypotheses instead of choosing the best hypothesis. In N-best mode, only the posterior probability for each hypothesis is output. In lattice mode, the hyp posterior is followed by word posterior probabilities for each (non-pause, non-noise) token in the hypothesis. The -max-rescore option limits the number of hypotheses per N-best list processed.
-dump-errors
Output word correctness indicators for all N-best hypotheses instead of choosing the best hypothesis. For each hypothesis, a line is output containing first the total number of errors and the list of indicators of whether the corresponding word is correct, substituted or inserted relative to the reference string. The location of deleted words is also indicated by a corresponding marker. The -max-rescore option limits the number of hypotheses per N-best list processed.
-reference w1 w2 ...
Specifies a reference word string for -dump-errors. Additionally, in -use-mesh mode, the reference words are recorded in the word mesh and can be output with -write, indicating which word in each alignment position is the correct one.

The following options only affect lattice mode.

-read file
Reads an initial lattice from file, to be merged with additional paths constructed from the N-best hypotheses.
-write file
Writes the resulting N-best lattice to file.
-prime-lattice
Start building the lattice with the best hypothesis obtained from N-best error minimization. This produces slightly better alignments and sometimes lower error rates. The default is to start with the top-scoring hypothesis.
-prime-with-1best
Similar to -prime-lattice, but uses the top-ranked sentence hypothesis for priming. (Experience shows that -no-reorder -prime-lattice gives best results.)
-no-merge
Build a lattice from the N-best hypotheses without merging edges (string/lattice alignment). This creates a lattice with one disjoint path per hypothesis, and is useful mainly for debugging purposes.
-lattice-errors references
Compute the lattice error (minimum word error) of the lattice read with -read or built with -nbest. Error is computed relative to all the hypotheses contained in a separate N-best list given by references.

SEE ALSO

ngram(1), nbest-optimize(1), nbest-scripts(1), nbest-format(5).
A. Stolcke, Y. Konig, and M. Weintraub, ``Explicit Word Error Minimization in N-best List Rescoring,'' Proc. Eurospeech, 163-166, 1997.
The ``word meshes'' used here are equivalent to the ``confusion networks'' described in: L. Mangu, E. Brill, and A. Stolcke, ``Finding Consensus Among Words: Lattice-based Word Error Minimization.'' Proc. Eurospeech, vol. 1, 495-498, 1999.

BUGS

Only a subset of functions is available when processing multiple N-best lists.
The options for word and lattice error computation are not consistent: -dump-errors takes the reference on the command line, whereas -lattice-errors and -nbest-errors use a separate N-best list.

AUTHOR

Andreas Stolcke <stolcke@speech.sri.com>.
Copyright 1996-2001 SRI International