Package: rustfst-python
RustfstRustPythonRust implementation of Weighted Finite States Transducers.Rustfst is a library for constructing, combining, optimizing, and searching weighted
finite-state transducers (FSTs). Weighted finite-state transducers are automata where
each transition has an input label, an output label, and a weight.
The more familiar finite-state acceptor is represented as a transducer
with each transition's input and output label equal. Finite-state acceptors
are used to represent sets of strings (specifically, regular or rational sets);
finite-state transducers are used to represent binary relations between pairs of
strings (specifically, rational transductions). The weights can be used to represent
the cost of taking a particular transition.FSTs have key applications in speech recognition and synthesis, machine translation,
optical character recognition, pattern matching, string processing, machine learning,
information extraction and retrieval among others. Often a weighted transducer is used to
represent a probabilistic model (e.g., an n-gram model, pronunciation model). FSTs can be
optimized by determinization and minimization, models can be applied to hypothesis sets
(also represented as automata) or cascaded by finite-state composition, and the best
results can be selected by shortest-path algorithms.OverviewFor a basicexamplesee the section below.Some simple and commonly encountered types of FSTs can be easily
created with the macro [fst] or the functionsacceptorandtransducer.For more complex cases you will likely start with theVectorFsttype, which will be imported
in the [prelude] along with most everything else you need.VectorFst<TropicalWeight>corresponds
directly to the OpenFSTStdVectorFst, and can be used to load
its files usingreadorread_text.Because "iteration" over an FST can mean many different things,
there are a variety of different iterators.  To iterate over state
IDs you may usestates_iter, while to
iterate over transitions out of a state, you may useget_trs.  Since it is common to
iterate over both, this can be done usingfst_iterorfst_into_iter.  It
is also very common to iterate over paths accepted by an FST,
which can be done withpaths_iter, and as a convenience
for generating text,string_paths_iter.
Alternately, in the case of a linear FST, you may retrieve the
only possible path withdecode_linear_fst.Note that iterating over paths is not the same thing as finding
theshortestpath or paths, which is done withshortest_path(for a single path)
orshortest_path_with_config(for N-shortest paths).For the complete list of algorithms, see the [algorithms] module.You may now be wondering, especially if you have previously used
such linguist-friendly tools aspyfoma, "what if I just want
totransduce some text???"  The unfriendly answer is that
rustfst is a somewhat lower-level library, designed for
implementing things like speech recognizers.  The somewhat more
helpful answer is that you would do this by constructing anacceptorfor your input, which you willcomposewith atransducer, thenprojectthe resultto itsoutput, and finallyiterate over the pathsin
the resulting FST.ReferencesImplementation heavily inspired from Mehryar Mohri's, Cyril Allauzen's and Michael Riley's work :Weighted automata algorithmsThe design principles of a weighted finite-state transducer libraryOpenFst: A general and efficient weighted finite-state transducer libraryWeighted finite-state transducers in speech recognitionThe API closely resembles that of OpenFST, with some
simplifications and changes to make it more idiomatic in Rust, notably
the use ofTrinstead ofArc.  SeeDifferences fromOpenFSTfor more information.Exampleuseanyhow::Result;userustfst::prelude::*;userustfst::algorithms::determinize::{DeterminizeType,determinize};userustfst::algorithms::rm_epsilon::rm_epsilon;fnmain()->Result<()>{// Creates a empty wFSTletmutfst=VectorFst::<TropicalWeight>::new();// Add some stateslets0=fst.add_state();lets1=fst.add_state();lets2=fst.add_state();// Set s0 as the start statefst.set_start(s0)?;// Add a transition from s0 to s1fst.add_tr(s0,Tr::new(3,5,10.0,s1))?;// Add a transition from s0 to s2fst.add_tr(s0,Tr::new(5,7,18.0,s2))?;// Set s1 and s2 as final statesfst.set_final(s1,31.0)?;fst.set_final(s2,45.0)?;// Iter over all the paths in the wFSTforpinfst.paths_iter(){println!("{:?}",p);}// A lot of operations are available to modify/optimize the FST.// Here are a few examples :// - Remove useless states.connect(&mutfst)?;// - Optimize the FST by merging states with the same behaviour.minimize(&mutfst)?;// - Copy all the input labels in the output.project(&mutfst,ProjectType::ProjectInput);// - Remove epsilon transitions.rm_epsilon(&mutfst)?;// - Compute an equivalent FST but deterministic.fst=determinize(&fst)?;Ok(())}Differences from OpenFSTHere is a non-exhaustive list of ways in which Rustfst's API
differs from OpenFST:The default epsilon symbol is<eps>and not<epsilon>.Functions and methods follow Rust naming conventions,
e.g.add_staterather thanAddState, but are otherwise mostly
equivalent, except that:Transitions are calledTrand notArc, becauseArchas a
rather different and well-established meaning in Rust, and rustfst
uses it (std::sync::Arc, that is) to reference-count symbol
tables.  All associated functions also usetr.Final states are not indicated by a final weight ofzero.  You
can test for finality usingis_final, andfinal_weightreturns an [Option].  This
requires some care when converting OpenFST code.Transitions can be accessed directly as a slice rather than requiring
an iterator.Semiring operations are expressed as plain old methods rather
than strange C++ things.  So writew1.plus(w2)rather thanPlus(w1, w2), for instance.Weights have in-place operations for ⊕
(plus_assign) and ⊗
(times_assign).Most of the type aliases (which would be trait aliases in Rust) such
asStdArc,StdFst, and so forth, are missing, but type inference
allows us to avoid explicit type arguments in most cases, such as
when calling [Tr::new], for instance.State IDs are unsigned, with [NO_STATE_ID] used for a missing value.
They are also 32 bits by default (presumably, 4 billion states
is enough for most applications).  This means you must take care to
cast them to [usize] when using them as indices, and vice-versa,
preferably checking for overflowsSymbol IDs are also unsigned and 32-bits, with [NO_LABEL] used
for a missing value.Floating-point weights are not generic, so are always single-precision.Benchmark with OpenFSTI did a benchmark some time ago on almost every linear fst algorithm and compared the results withOpenFst. You can find the results here :Benchmark at the C++ levelBenchmark at the CLI levelSpoiler alert:Rustfstis faster on all those algorithms 😅DocumentationThe documentation of the last released version is available here :https://docs.rs/rustfstRelease processUse the scriptupdate_version.shto update the version of every package.PushPush a new tag with the prefixrustfst-vExample :./update_version.sh0.9.1-alpha.6
gitcommit-am"Release 0.9.1-alpha.6"gitpush
gittag-arustfst-v0.9.1-alpha.6-m"Release rustfst 0.9.1-alpha.6"gitpush--tagsOptionally, if this is a major release, create a GitHub release in the UI.Projects contained in this repositoryThis repository contains two main projects:rustfstis the Rust re-implementation.Crate available on crates.iohereDocumentation available on docs.rshererustfst-pythonis the python binding ofrustfst.Package available on PypihereDocumentation available on Github PageshereLicenseLicensed under either ofApache License, Version 2.0 (LICENSE-APACHE orhttp://www.apache.org/licenses/LICENSE-2.0)MIT license (LICENSE-MIT) orhttp://opensource.org/licenses/MIT)at your option.ContributionUnless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.