English:
[0001] The present invention relates to speech processing applications, and more specifically to precompiled search networks as used in speech processing applications such as automatic speech recognition systems and text-to-speech systems.
[0002] Speech recognition systems try to determine the semantic meaning of a speech input. Typically, probabilistic finite state speech models are used to determine a sequence of words that best corresponds to the speech input. One common example is an automotive application which allows control of applications within the passenger cabin such as navigational systems, audio, systems, climate control, etc. The recognition task in such an application represents a combination of predetermined elements such as user commands and lists of streets in a city, along with dynamically determined elements such as lists of mp3 songs, names from a personal digitized address book (e.g., from a smartphone), and navigational points of interest. For such applications, it is desirable to have compact memory efficient search networks with fast low latency performance.
[0003] The recognition task includes multiple different knowledge levels, from the acoustic forms of fundamental speech sounds known as phonemes, to the phoneme sequences that form words in a recognition vocabulary, to the word sequences that form phrases in a recognition grammar. One powerful and convenient way to organize the recognition task is based on the use of finite state machines such as finite state acceptors (FSA) and finite state transducers (FST). For convenience, the following discussion uses the term FST, though not in any particularly rigorous sense but rather in a generic inclusive way that also applies in many respects to FSAs and other specific forms of finite state machines. Figure 1 shows a specific example of one simple FST that parses a set of word level symbols: {AAA, BA, AAB} ; see e.g., Jurafsky & Martin, Speech and incorporated herein by reference.
[0004] Implementing speech processing applications on mobile platforms such as smartphones is especially challenging in view of the limited resources that are available. Some optimizations may be implemented to improve speed, accuracy, and/or use of limited memory. Among other things, FSTs allow for a single unified representation of acoustic information (HMM set), word pronunciations, language model, and recognition grammar. Due to this unified representation, computationally inexpensive computations can be performed during traversal of the search network while other computationally expensive optimizations can be offloaded to a precompilation stage. The four most widely used optimizations for static compilation of FSTs include determinization, minimization, weight and label pushing; see e.g., Mohri, Minimization of Sequential Transducers, (5th Annual Symposium on Combinatorial Pattern Matching 1994); incorporated herein by reference.
[0005] Despite determinization, existing FST optimizations are global, i.e. they need to access a FST as a whole and therefore can't be applied in speech applications that incrementally modify the search network. Some theoretical aspects of incrementally adding sentences to a word-level FST are described more fully in Crochemore and Giambruno, On-Line Construction Of A Small Automaton For A Finite Set Of Words, The Prague Stringology Conference 2009; incorporated herein by reference.
[0006] Embodiments of the present invention are directed to methods of incrementally modifying a word-level finite state transducer (FST) for adding and removing sentences. A prefix subset of states and arcs in the FST is determined that matches a prefix portion of the sentence. A suffix subset of states and arcs in the FST is determined that matches a suffix portion of the sentence. A new sentence can then be added to the FST by appending a new sequence of states and arcs to the FST corresponding to a remainder of the sentence between the prefix and suffix. An existing sentence can be removed from the FST by removing any arcs and states between the prefix subset and the suffix subset. The resulting modified FST is locally efficient but does not satisfy global optimization criteria such as minimization.
[0007] The FST may be a speech application FST and the sentence may be dynamically added or removed from the FST during online operation of the speech application. To remove an existing sentence, the steps may be repeated one or more times in a partial match process ("fuzzy removal") to remove multiple instances of the existing sentence from the FST. A hash table may be used for determining the prefix subset and/or the suffix subset, for example, a special dual-use hash table. The global optimization criteria may include FST minimization criteria.
[0008] Figure 1 shows a simple example of one specific FST.
[0009] Figure 2 A-C shows an example of adding a sentence to an FST according to one embodiment of the present invention.
[0010] Figure 3 A-C shows an example of adding a new state to an existing FST for shared suffixes.
[0011] Figure 4 A-B shows an example of the global non-minimality of the FST.
[0012] Figure 5 A-B shows an example of removing a sentence from an existing FST using an exact match process.
[0013] Figure 6 A-B shows an example of a partial match process for removing a sentence from an existing FST.
[0014] Figure 7 shows a basic architecture for an embodiment of a system using a modified FST arrangement according to one embodiment of the present invention.
[0015] Various embodiments of the present invention are directed to techniques for to efficiently allow for dynamic modifications of FSTs using local optimizations slightly weaker than global optimizations; for example, adding and removing of sentences to a word-level FST based on incremental local minimization. The property of local minimization is a weaker version of minimization that only applies to suffix paths on which all states have a fan-out of not more than 1 arc. Given a sentence of length N, embodiments of the present invention add to/remove from an FST based on an amortized complexity of 0(N), i.e., it takes not more than c*N steps with c being constant to the implementation, but not N2steps. This can be used to incrementally compose FSTs that decode speech into text almost as efficient as globally optimized FSTs and also help reduce memory consumption for (offline) static compilation of FSTs. Although the following discussion is presented in terms of a speech recognition application, the ideas can be extended to any application that needs to incrementally build time or space efficient FSTs such as text-to-speech, text processing, etc.
[0016] Figure 2 A-C shows an example of adding a sentence to an FST according to one embodiment of the present invention with linear time complexity based on sharing a prefix and suffix of the new sentence with those already present in the existing FST. ( B: Again as explained above, the formal distinction between FSA and FST is actually rather trivial in this context, where the input/output arc label pairs from an FST are given a single label to simplify the figures). Fig. 2A shows a simple existing word-level FST for the sentence ABC. To add a new sentence ADC to this FST, first a prefix subset of states and arcs in the FST is determined that matches a prefix portion of the new sentence. In this specific case, the existing prefix subset for adding the new sentence ADC would be determined to be the existing states 0 and 1 and their connecting arc A, Fig. 2B. Next a suffix subset of states and arcs in the FST is determined that matches a suffix portion of the new sentence. In the case of the new sentence ADC, the suffix subset would be the existing states 2 and 3 and their connecting arc C. The new sentence can then be added to the FST by appending a new sequence of states and arcs to the FST corresponding to a remainder of the sentence between the prefix and suffix. In the case of adding ADC, this can be done by adding the arc D between states 1 and 2, Fig. 2C.
[0018] Figure 3 A-C shows another example of adding a new state to an existing FST for shared suffixes. Fig. 3A shows an existing FST for the sentence AAA. To add a new sentence BA, the suffix A can be shared by using state 2 and adding the new arc B from state 0 to state 2, Fig. 3B. To then add another new sentence AAB, a new state 4 is added that receives the prefix subset AA, the existing arc A from state 2 to state 3 is replicated from state 4 to state 3 (parsing existing sentence AAA) and a new arc 4B3 is added to complete the new path AAB, Fig. 3C.
[0019] While such an approach will be locally efficient, at some point, the growing FST becomes globally non-minimal. For example, if the modified FST in Fig. 3C then has another new sentence added, BB, the existing B prefix will be identified and a new arc B added from state 2 to state 3, Fig. 4A. But this new FST is not globally minimized because state 4 is not longer required and could be removed as shown in Fig. 4B which still correctly parses all and only AAA, BA, AAB, and BB. But the ease of the process outweighs the small incremental loss of global non-minimality.
[0020] In addition to adding sentences to an existing FST, an existing sentence also can be removed from the FST by removing any arcs and states between the prefix subset and the suffix subset; e.g. by a "free" list for later use. A hash table may be useful in connection with such a removal process, for example, a prefix hash table can quickly access start states of potential matches. So: p -> 0(p) accesses to a hash table that matches input label p. Only obsolete arcs need to be removed.
[0021] One specific process for removing a sentence from an existing FST may be based on an exact match approach as shown in Figure 5 A-B. Fig. 5A shows an existing FST that parses the sentences AAA and BA. To remove a sentence, any remaining prefixes and/or suffixes first must be determined and retained. For example, to remove the sentence AAA (and retain BA), there is no prefix to retain, but the A suffix of BA is identified and retained, so the suffix subset (2, A, 3) is retained (that is, states 2 and 3 and arc (edge) A between them: 2— > 3), while the remainder of the path for AA is removed: state 1 and the arcs 0— > 1 and 1 ^ 2 (the (-1) as shown in Fig. 5B).
[0023] Figure 7 shows a basic architecture for an embodiment of a system using a modified FST arrangement according to one embodiment of the present invention, in this case a speech application FST which dynamically adds and remove sentences during online operation of the speech application, that is, the system can be modified at run time. Compiler 703 performs a run time expansion of the incrementally minimized word-level FST 701 (i.e., the language model) with the HMM unit sequences 702 (i.e., the acoustic models) to create a HMM-unit-to-word FST 704 (on which the compiler 703 also does determinization and local minimization) for speech recognition of unknown speech inputs in a speech application. And while the HMM-unit-to-word FST 704 is incrementally constructed as described above (i.e., during recognition) rather than strictly globally minimized, it still is compact and efficient enough to be beneficial for commercial use and additional computations are avoided. In some embodiments, run time expansion may be an optional step that may or may not be performed. The HMM-unit-to-word FST can also be directly constructed using incremental minimization. However, due to various factors such as lengthy pronunciations (and the same may be used more than once) this approach may require more overhead for the additional/removal steps. Thus, specific embodiments may for that reason choose to dynamically expand the transcriptions during recognition.
[0024] Embodiments of the invention may be implemented in whole or in part in any conventional computer programming language. For example, preferred embodiments may be implemented in a procedural programming language (e.g., "C") or an object oriented programming language (e.g., "C++", Python). Alternative embodiments of the invention may be implemented as pre-programmed hardware elements, other related components, or as a combination of hardware and software components.
[0025] Embodiments can be implemented in whole or in part as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, electrically erasable ROM like "Flash ROM", or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g. , a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software (e.g., a computer program product).
[0026] Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention.

中文:
[0001]本发明涉及语音处理应用系统,并且更具体地,涉及在诸如自动语音识别系统和文-语转换系统的语音处理应用中使用的预编译的搜索网络。
[0002]语音识别系统试图确定语音输入的语义含义。典型地,概率有限状态语音模型用于确定与语音输入最对应的词序列。一个常见的例子是汽车应用,其允许控制乘客车厢内的应用,诸如导航系统、语音、空调控制等等。诸如该设备中的识别任务表示诸如用户命令和城市内街道的列表的预先确定的元素,以及诸如mp3歌曲列表、来自个人数字化地址簿(例如,来自智能电话)的姓名和感兴趣的导航点的动态确定的元素的组合。对于这些应用,预期具有带有快速低延迟性能的紧凑存储有效搜索网络。
[0003]该识别任务包括多个不同的知识等级,从基本语音声音的声学形式,诸如所熟知的音素(phoneme),到形成识别词汇中单词的音素序列,到形成识别语法中短语的单词序列。组织识别任务的一种有效且方便的方式是基于有限状态机的使用,诸如有限状态接受器(FSA)和有限状态转换机(FST)。为了便利,下述讨论中使用术语FST,然而,不是任何特定的严格意义,而是以通用的包含的方式,其也能够应用在FSA和其他特定形式的有限状态机的许多方面。附图1示出了一个简单FST的特定的示例,所述FST解析单词等级符号集合:{AAA,BA,AAB};参见例如Jurafsky&Martin编写的Speech and Language Processing(语音和语言处理),Prentice Hall2001,第33-83页,ISBN0-13-095069-6;其通过引用被包含在此。
[0004]在诸如智能电话的移动平台上实施语音处理系统特别受到有限的可用资源的挑战。实施一些优化以提高速度、精确率和/或有限存储器的使用。除此之外,FST允许声学信息(HMM集合)、单词发音、语言模型和识别语法的单独的统一的表示。由于该统一表示,导致在遍历搜索网络期间,可以执行计算花费不大的计算,而其他计算花费大的优化能够被卸下到预编译阶段进行。FST的静态编译的优化使用的最广泛的四个包括确定化、最小化、权重和标签推送;参见例如Mohri编写的Minimization of Sequential Transducers(序列转换机的最小化),(5th Annual Symposium on Combinatorial Pattern Matching1994,关于组合模式匹配的第5年度论文集中发表,1994年);通过引用被包含在此。
[0005]不管确定化,现有的FST优化是全局的,也就是说,需要作为一个整体来访问FST,并且因而,不能在渐进修改搜索网络的语音应用中应用。渐进将句子添加到单词级FST的一些理论性方法在Crochemore和Giambruno编写的On-Line Construction Of A small Automaton for A Finite Set of Words(用于有限单词集合的小型机器人的在线建立),出版于Prugue Stringology Conference2009中更加全面地被描述;通过引用被包含在此。
[0006]本发明的实施例是关于通过增加和删除句子来渐进修改单词级有限状态转换机(FST)的方法。确定在FST中与句子的前缀部分匹配的状态和弧的前缀子集。确定在FST中与句子的后缀部分匹配的状态和弧的后缀子集。然后通过将状态和弧的新序列附加到与前缀和后缀之间的句子的剩余部分相对应的FST,将新句子添加到FST。通过将前缀子集和后缀子集之间的任何状态删除,可以从FST中删除现有句子。产生的修改的FST是局部有效的,但是并不满足全局优化标准,诸如最小化标准。
[0007]FST可以是语音应用FST,并且在语音应用的在线操作期间能够动态地从FST中添加或从FST中删除句子。为了删除现有句子,在部分匹配处理(“模糊删除”)中可以将各步骤重复一次或多次,以从FST中删除现有句子的多个实例。能够使用哈希表来确定前缀子集和/或后缀子集,例如,特别的双向使用的哈希表。全局优化标准可以包括FST最小化标准。
[0008]图1示出了一个特定FST的一个简单示例。
[0009]图2A-C示出了根据本发明的一个实施例,将句子添加到FST中的示例。
[0010]图3A-C示出了将新状态添加到现存FST中用于共享后缀的示例。
[0011]图4A-B示出了FST的全局非最小性的示例。
[0012]图5A-B示出了使用精确匹配处理从现有FST中删除句子的示例。
[0013]图6A-B示出了用于从现有FST中删除句子的部分匹配处理的示例。
[0014]图7示出了根据本发明的一个实施例,使用修改的FST设置的系统的实施例的基本架构。
[0015]本发明的各种实施例是针对使用稍劣于全局优化的局部优化有效地允许FST的动态修改的技术;例如,基于渐进的局部最小化将句子添加到单词级FST或从单词级FST中删除句子。局部最小化的性能是最小化的弱化版本,其只应用于后缀路径,在其上所有状态具有不超过一个弧的扇出(fan-out)。给定长度是N的句子,基于O(N)的折算复杂度(amortized complexity),也就是说,实施时需要花费不超过c*N个步骤,其中c是常数,而不是N2个步骤,本发明的具体实施例将句子添加到FST或从FST中删除句子。这可以被用于渐进编写FST,其与全局优化的FST几乎一样地有效地将语音解码为文本,并且同样帮助减少了用于FST的(离线)静态编译所耗费的存储器。尽管下面的讨论是关于语音识别应用的,但是该方法可以扩展到任何需要渐进建立时间或空间有效的FST的应用,例如,文语转换,文本处理,等等。
[0016]图2A-C示出了根据本发明的一个实施例,将句子添加到FST的示例,其中基于共享新句子的前缀和后缀和现有的FST中已存在的那些,其具有线性时间复杂度(NB:再次如上面所解释的,在本文中,FSA和FST之间形式上的区别实际上是微不足道的,其中为了简化图,来自FST的输入/输出弧标签对被给定单一的标签)。图2A示出了句子ABC的简单的现有的单词级FST。为了将新句子ADC添加到该FST,首先,确定FST中状态和弧的前缀子集与该新句子的前缀部分匹配。在该特定情况中,添加新的句子ADC的现有的前缀子集可以被确定为,如图2B所示的,现有状态0和1以及它们的连接弧A。接下来,确定FST中状态和弧的后缀子集与该新句子后缀部分匹配。在新句子ADC的情况下,后缀子集应该是现有的状态2和3以及它们的连接弧C。然后通过将状态和弧的新序列添加至前缀和后缀之间句子的剩余部分相对应的FST中,可以将该新句子添加至FST中。在添加ADC的情况下,通过在状态1和2之间添加弧D来完成上述,如图2C所示。
[0018]图3A-C示出了将新的状态添加到现有FST用于共享后缀的另一个示例。图3A示出了句子AAA的现有FST。为了添加新句子BA,通过使用状态2并添加从状态0到状态2的新弧B来共享后缀A,如图3B所示。然后,为了添加另一个新句子AAB,添加状态4,其接收前缀子集AA,从状态2到状态3的现有弧A从状态2到状态3被重复(解析现有的句子AAA),并且添加新弧4B3,以完成新路径AAB,如图3C所示。
[0019]在一些点处,尽管该方法是局部有效的,但是增长的FST变得全局非最小。例如,如果图3C中已修改的FST要添加另一个新句子BB,那么将确认现有的B前缀,以及从状态2到状态3添加的新弧B,如图4A所示。但是,该新FST不是全局最小化的,因为状态4将不再需要,并且应该被删除,如图4B所示,其仍然能够正确解析所有以及仅AAA,BA,AAB和BB。但是,处理的容易性要超过全局非最小化的小的渐进损失。
[0020]除了向现有FST添加句子之外,通过删除前缀子集和后缀子集之间的任何弧和状态;例如,通过用于以后使用的“空”列表,还可以从FST中删除现有句子。在与该删除处理关联地,哈希表是有用的,例如,前缀哈希表能够快速访问潜在匹配的起始状态。这样:p→O(p)访问哈希表,其匹配输入标签p。只有废弃的弧才需要被删除。
[0021]从现有FST中删除句子的一个特定处理可以基于精确匹配的方法来实现,如图5A-B所示。图5A示出了解析句子AAA和BA的现有FST。为了删除句子,首先,任何剩余的前缀和/或后缀必须被确定和保留。例如,为了删除句子AAA(和保留BA),没有保留前缀,但是BA对应的A后缀被确认和保留,因此后缀子集(2,A,3)被保留(即,状态2和状态3以及它们之间的弧(边)A:同时删除AA的路径的剩余部分:状态1和弧和(如图5B所示的(-1))。
[0023]图7示出了根据本发明的一个实施例,使用修改的FST设置的系统的实施例的基本架构,在语音应用FST情形下,在语音应用的在线操作期间能够动态添加和删除句子,也就是说,在运行期间可以修改该系统。在语音应用中,编译器703通过HMM单元序列702(即,声学模型)执行渐进最小化单词级FST701(即,语言模型)的运行时间扩展来创建HMM单元-到-单词FST704(在其上编译器703也执行确定化和局部最小化)用于未知的语音输入的语音识别。并且,尽管HMM单元-到-单词FST704如上述内容(即,在识别期间)被渐进创建而不是严格意义上全局最小化,但是它对于商业应用仍然是足够紧凑和有效的,并且避免了附加的计算。在一些实施例中,运行时间扩展可以是可选的步骤,其可以被执行或不被执行。HMM-单元-到-单词FST也可以直接使用渐进最小化被创建。但是,由于诸如冗长的发音(和相同的使用超过一次)的各种因素,导致该方法可能需要更多的开销用于添加的/删除的步骤。因而,由于此原因,特定具体实施例选择在识别期间动态扩展转换本(transcription)。
[0024]可以通过任何传统的计算机编程语言全部或部分地实现本发明的实施例。例如,可以通过面向过程的编程语言(例如,“C”)或面向对象编程语言(例如,“C++”、Python)实现优选的实施例。本发明的可替换实施例被实现为预编程的硬件元件,或其他相关组件,或硬件和软件组件的组合。
[0025]实施例全部或部分地被实现为在计算机系统中使用的计算机程序产品。该实施可以包括一系列计算机指令,其或固定在有形介质上,诸如计算机可读介质(例如,软盘、CD-ROM、ROM,像“闪存ROM”的电可擦除ROM,或者固定盘)或经由调制解调器或其他接口设备可传送到计算机系统,诸如经过介质连接到网络的通信适配器。该介质或是一有形介质(例如,光的或模拟通信传输线路)或使用无线技术实现的介质(例如,微波的,红外的或其他传输技术)。该一系列计算机指令包括实施在此这里先前所描述的与该系统有关的全部或部分功能。本领域技术人员应该理解,这些计算机指令可以通过很多编程语言被编写成很多程序语言以用于配合许多计算机架构或操作系统使用。此外,这些指令可以存储在任何存储器设备中,诸如半导体、磁的、光的或其他存储器设备上,并且可以使用通过任何通信技术被发送,诸如光的、红外的、微波的或其他传送传输技术。可以预想到预期,这样的一个计算机程序产品可以被分发作为一可移动被分布于具有所附的打印或电子文档(例如,套装软件(shrink wrapped software))的可移动介质,预加载有计算机系统(例如,在系统ROM或固定盘上),或通过网络(诸如,因特网或万维网)从服务器或电子公示板分发。当然,本发明一些实施例可以被实现为软件(例如,计算机程序产品)和硬件两者的组合。本发明的其它实施例仍然被全部实现为硬件,或全部实现为软件(例如,计算机程序产品)。
[0026]尽管此处已公开了本发明的各种示例性实施例,但对本领域技术人员显而易见的是,在不背离本发明的真实保护范围的情况下,可以做出各种改变和修改以完成本发明的一些优点。