Package: rhasspy-nlu
Rhasspy Natural Language UnderstandingLibrary for parsing Rhasspy sentence templates, doing intent recognition, and generating ARPA language models.RequirementsPython 3.7Installation$gitclonehttps://github.com/rhasspy/rhasspy-nlu
$cdrhasspy-nlu
$./configure
$make
$makeinstallRunning$bin/rhasspy-nlu<ARGS>Parsing Sentence TemplatesRhasspy voice commands are stored in text files formatted like this:[Intent1]this is a sentencethis is another sentence[Intent2]a sentence in a different intentYou can parse these into a structured representation withrhasspynlu.parse_iniand then convert them to a graph usingrhasspynlu.intents_to_graph:importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[LightOn]turn on [the] (living room lamp | kitchen light){name}""")graph=rhasspynlu.intents_to_graph(intents)The result is adirected graphwhose states are words and edges are input/output labels.You can pass anintent_filterfunction toparse_inito returnTruefor only the intent names you want to parse.
Additionally, a function can be provided for thesentence_transformargument that each sentence will be passed through (e.g., to lower case).Template SyntaxSentence templates are based on theJSGFstandard. The following constructs are available:Optional wordsthis is [a] test- the word "a" may or may not be presentAlternativesset color to (red | green | blue)- either "red", "green", or "blue" is possibleTagsturn on the [den | playroom]{location} light- named entitylocationwill be either "den" or "playroom"Substitutionsmake ten:10 coffees- output will be "make 10 coffees"turn off the: (television | tele):tv- output will be "turn off tv"set brightness to (medium | half){brightness:50}- named entitybrightnesswill be "50"Rulesrule_name = rule bodycan be referenced as<rule_name>Slots$slotwill be replaced by a list of sentences in thereplacementsargument ofintents_to_graphRulesNamed rules can be added to your template file using the syntax:rule_name=rule bodyand then reference using<rule_name>. The body of a rule is a regular sentence, which may itself contain references to other rules.You can refrence rules from different intents by prefixing the rule name with the intent name and a dot:[Intent1]rule=a testthis is <rule>[Intent2]rule=this is<rule> <Intent1.rule>In the example above,Intent2uses its local<rule>as well as the<rule>fromIntent1.SlotsSlot names are prefixed with a dollar sign ($). When callingintents_to_graph, thereplacementsargument is a dictionary whose keys are slot names (with$) and whose values are lists of (parsed)Sentenceobjects. Each$slotwill be replaced by the corresponding list of sentences, which may contain optional words, tags, rules, and other slots.For example:importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetColor]set color to $color""")graph=rhasspynlu.intents_to_graph(intents,replacements={"$color":[rhasspynlu.Sentence.parse("red | green | blue")]})will replace$colorwith "red", "green", or "blue".Intent RecognitionAfter converting your sentence templates to a graph, you can recognize sentences. Assuming you have a.inifile like this:[LightOn]
turn on [the] (living room lamp | kitchen light){name}You can recognize sentences with:frompathlibimportPathimportrhasspynlu# Load and parseintents=rhasspynlu.parse_ini(Path("sentences.ini"))graph=rhasspynlu.intents_to_graph(intents)rhasspynlu.recognize("turn on living room lamp",graph)will return a list ofRecognitionobjects like:[
    Recognition(
        intent=Intent(name='LightOn', confidence=1.0),
        entities=[
            Entity(
                entity='name',
                value='living room lamp',
                raw_value='living room lamp',
                start=8,
                raw_start=8,
                end=24,
                raw_end=24,
                tokens=['living', 'room', 'lamp'],
                raw_tokens=['living', 'room', 'lamp']
            )
        ],
        text='turn on living room lamp',
        raw_text='turn on living room lamp',
        recognize_seconds=0.00010710899914556649,
        tokens=['turn', 'on', 'living', 'room', 'lamp'],
        raw_tokens=['turn', 'on', 'living', 'room', 'lamp']
    )
]An empty list means that recognition has failed. You can easily convertRecognitionobjects to JSON:...importjsonrecognitions=rhasspynlu.recognize("turn on living room lamp",graph)ifrecognitions:recognition_dict=recognitions[0].asdict()print(json.dumps(recognition_dict))You can also pass anintent_filterfunction torecognizeto returnTrueonly for intent names you want to include in the search.TokensIf your sentence is tokenized by something other than whitespace, pass the list of tokens intorecognizeinstead of a string.Recognition FieldsTherhasspynlu.Recognitionobject has the following fields:intent- arhasspynlu.Intentinstancename- name of recognized intentconfidence- number for 0-1, 1 being suretext- substituted input textraw_text- input textentities- list ofrhasspynlu.Entityobjectsentity- name of recognized entity ("name" in(input:output){name})value- substituted value of recognized entity ("output" in(input:output){name})tokens- list of words invaluestart- start index ofvalueintextend- end index ofvalueintext(exclusive)raw_value- value of recognized entity ("input" in(input:output){name})raw_tokens- list of words inraw_valueraw_start- start index ofraw_valueinraw_textraw_end- end index ofraw_valueinraw_text(exclusive)recognize_seconds- seconds taken forrecognizeStop WordsYou can pass a set ofstop_wordstorecognize:rhasspynlu.recognize("turn on that living room lamp",graph,stop_words=set(["that"]))Stop words in the input sentence will be skipped over if they don't match the graph.Strict RecognitionFor faster, but less flexible recognition, setfuzzytoFalse:rhasspynlu.recognize("turn on the living room lamp",graph,fuzzy=False)This is at least twice as fast, but will fail if the sentence is not precisely present in the graph.Strict recognition also supportsstop_wordsfor a little added flexibility. If recognition withoutstop_wordsfails, a second attempt will be made usingstop_words.ConvertersValue conversions can be applied during recognition, such as converting the string "10" to the integer 10. Following a word, sequence, or tag name with "!converter" will run "converter" on the string value duringrecognize:importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetBrightness]set brightness to (one: hundred:100)!int""")graph=rhasspynlu.intents_to_graph(intents)recognitions=rhasspynlu.recognize("set brightness to one hundred",graph)assertrecognitions[0].tokens[-1]==100Converters can be applied to tags/entities as well:importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetBrightness]set brightness to (one:1 | two:2){value!int}""")graph=rhasspynlu.intents_to_graph(intents)recognitions=rhasspynlu.recognize("set brightness to two",graph)assertrecognitions[0].tokens[-1]==2The following default converters are available inrhasspynlu:int - convert to integerfloat - convert to realbool - convert to booleanlower - lower-caseupper - upper-caseYou may override these converters by passing a dictionary to theconvertersargument ofrecognize. To supply additional converters (instead of overriding), useextra_converters:importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetBrightness]set brightness to (one:1 | two:2){value!myconverter}""")graph=rhasspynlu.intents_to_graph(intents)recognitions=rhasspynlu.recognize("set brightness to two",graph,extra_converters={"myconverter":lambda*values:[int(v)**2forvinvalues]})assertrecognitions[0].tokens[-1]==4Lastly, you can chain converters together with multiple "!":importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetBrightness]set brightness to (one:1 | two:2){value!int!cube}""")graph=rhasspynlu.intents_to_graph(intents)recognitions=rhasspynlu.recognize("set brightness to two",graph,extra_converters={"cube":lambda*values:[v**3forvinvalues]})assertrecognitions[0].tokens[-1]==8ARPA Language ModelsYou can computengram countsfrom arhasspynlugraph, useful for generatingARPA language models. These models can be used by speech recognition systems, such asPocketsphinx,Kaldi, andJulius.importrhasspynlu# Load and parseintents=rhasspynlu.parse_ini("""[SetColor]set light to (red | green | blue)""")graph=rhasspynlu.intents_to_graph(intents)counts=rhasspynlu.get_intent_ngram_counts(graph,pad_start="
",pad_end="
",order=3)# Print counts by intentforintent_nameincounts:print(intent_name)forngram,countincounts[intent_name].items():print(ngram,count)print("")will print something like:SetColor
('
',) 3
('set',) 3
('
', 'set') 3
('light',) 3
('set', 'light') 3
('
', 'set', 'light') 3
('to',) 3
('light', 'to') 3
('set', 'light', 'to') 3
('red',) 1
('to', 'red') 1
('light', 'to', 'red') 1
('green',) 1
('to', 'green') 1
('light', 'to', 'green') 1
('blue',) 1
('to', 'blue') 1
('light', 'to', 'blue') 1
('
',) 3
('red', '
') 1
('green', '
') 1
('blue', '
') 1
('to', 'red', '
') 1
('to', 'green', '
') 1
('to', 'blue', '
') 1OpengrmIf you have theOpengrmcommand-line tools in yourPATH, you can userhasspynluto generate language models in theARPA format.Thegraph_to_fstandfst_to_arpafunctions are used to convert between formats. Callingfst_to_arparequires the following binaries to be present in yourPATH:fstcompile(fromOpenFST)ngramcountngrammakengrammergengramprintngramreadExample:# Convert to FSTgraph_fst=rhasspynlu.graph_to_fst(graph)# Write FST and symbol text filesgraph_fst.write("my_fst.txt","input_symbols.txt","output_symbols.txt")# Compile and convert to ARPA language modelrhasspynlu.fst_to_arpa("my_fst.txt","input_symbols.txt","output_symbols.txt","my_arpa.lm")You can now usemy_arpa.lmin any speech recognizer that accepts ARPA-formatted language models.Language Model MixingIf you have an existing language model that you'd like to mix with Rhasspy voice commands, you will first need to convert it to an FST:rhasspynlu.fst_to_arpa("existing_arpa.lm","existing_arpa.fst")Now when you callfst_to_arpa, make sure to provide thebase_fst_weightargument. This is a tuple with the path to your existing ARPA FST and a mixture weight between 0 and 1. A weight of 0.05 means that the base language model will receive 5% of the overall probability mass in the language model. The rest of the mass will be given to your custom voice commands.Example:rhasspynlu.fst_to_arpa("my_fst.txt","input_symbols.txt","output_symbols.txt","my_arpa.lm",base_fst_weight=("existing_arpa.fst",0.05))Command Line UsageTherhasspynlumodule can be run directly to convertsentences.inifiles into JSON graphs or FST text files:python3-mrhasspynlusentences.ini>graph.jsonYou can pass multiple.inifiles as arguments, and they will be combined. Adding a--fstargument will write out FST text files instead:python3-mrhasspynlusentences.ini--fstThis will output three files in the current directory:fst.txt- finite state transducer as textfst.isymbols.txt- input symbolsfst.osymbols.txt- output symbolsThese file names can be changed with the--fst-text,--fst-isymbols, and--fst-osymbolsarguments, respectively.Compile to a binary FST usingfstcompile(fromOpenFST) with:fstcompile\--isymbols=fst.isymbols.txt\--osymbols=fst.osymbols.txt\--keep_isymbols=1\--keep_osymbols=1\fst.txt\out.fstWord PronunciationsMethods for loading and using phonetic pronunciation dictionaries are provided inrhasspynlu.g2p("g2p" stands for "grapheme to phoneme").Dictionaries are expected in the same format as theCMU Pronouncing Dictionary, which is simply one word per line with whitespace separating words and phonemes:yawn Y AO N
test T EH S T
say S EY
who HH UW
bee B IY
azure AE ZH ER
read R EH D
read(2) R IY DWhen multiple pronunciations are available for a word (like "read" in the previous example), a(N)can be suffixed to the word.You can load a phonetic dictionary into a Python dictionary withrhasspynlu.g2p.read_pronunciations:importrhasspynlu.g2pwithopen("/path/to/file.dict","r")asdict_file:pronunciations=rhasspynlu.g2p.read_pronunciations(dict_file)assertpronunciations=={"yawn":[["Y","AO","N"]],"test":[["T","EH","S","T"]],"say":[["S","EY"]],"who":[["HH","UW"]],"bee":[["B","IY"]],"azure":[["AE","ZH","ER"]],"read":[["R","EH","D"],["R","IY","D"]],}Seevoice2json profilesfor pre-built phonetic dictionaries.Guessing PronunciationsTherhasspynlu.g2p.guess_pronunciationsfunction usesPhonetisaurusand a pre-trained grapheme to phoneme model to guess pronunciations for unknown words. You will needphonetisaurus-applyin your$PATHand the pre-trained model (g2p.fst) available:importrhasspynlu.g2pguesses=rhasspynlu.g2p.guess_pronunciations(["moogle","ploop"],"/path/to/g2p.fst",num_guesses=1)print(list(guesses))# Something like: [#   ('moogle', ['M', 'UW', 'G', 'AH', 'L']),#   ('ploop', ['P', 'L', 'UW', 'P'])# ]Seevoice2json profilesfor pre-trained g2p models.Sounds Like PronunciationsRhasspy NLU supports an alternative way of specifying word pronunciations. Instead of specifying phonemes directly, you can describe how a word should be pronounced by referencing other words:unknown_word1 known_word1 [known_word2] ...
...For example, the singerBeyoncésounds like a combination of the words "bee yawn say":beyoncé bee yawn sayTherhasspynlu.g2p.load_sounds_likefunction will parse this text and, when given an existing pronunciation dictionary, generate a new pronunciation:importioimportrhasspynlu.g2p# Load existing dictionarypronunciations=rhasspynlu.g2p.read_pronunciations("/path/to/file.dict")sounds_like="""beyoncé bee yawn say"""withio.StringIO(sounds_like)asf:rhasspynlu.g2p.load_sounds_like(f,pronunciations)print(pronunciations["beyoncé"])# Something like: [['B', 'IY', 'Y', 'AO', 'N', 'S', 'EY']]You may reference a specific pronunciation for a known word using theword(N)syntax, whereNis 1-based. Pronunciations are loaded in line order, so the order is predictable. For example,read(2)will reference the second pronunciation of the word "read". Without an(N), all pronunciations found will be used.Phoneme LiteralsYou can interject phonetic chunks into these pronunciations too. For example, the word "hooiser" sounds like "who" and the "-zure" in "azure":hooiser who /Z 3/Text between slashes (/) will be interpreted as phonemes in the configured speech system.Word SegmentsIf a grapheme-to-phoneme alignment corupus is available, segments of words can also be used for pronunciations. Using the "hooiser" example above, we can replace the phonemes with:hooiser who a>zure<This will combine the pronunciation of "who" from the current phonetic dictionaries (base_dictionary.txtandcustom_words.txt) and the "-zure" from the word "azure".The brackets point>at<the segment of the word that you want to contribute to the pronunciation. This is accomplished using a grapheme-to-phoneme alignment corpus generated withphonetisaurusand a pre-built phonetic dictionary. In thea>zure<example, the word "azure" is located in the alignment corpus, and the output phonemes from the phonemes "zure" in it are used.importioimportrhasspynlu.g2p# Load existing dictionarypronunciations=rhasspynlu.g2p.read_pronunciations("/path/to/file.dict")# Example alignment corpus:# a}AE z}ZH u|r}ER e}_alignment=rhasspynlu.g2p.load_g2p_corpus("/path/to/g2p.corpus")sounds_like="""hooiser who a>zure<"""withio.StringIO(sounds_like)asf:rhasspynlu.g2p.load_sounds_like(f,pronunciations,g2p_alignment=alignment)print(pronunciations["hooiser"])# Something like [["HH", "UW", "ZH", "ER"]]Seevoice2json profilesfor g2p alignment corpora.