Patent Description:
This disclosure relates generally to the field of sign language recognition (SLR) and, more specifically, to systems and methods for SLR including Hidden Markov Models (HMMs) and the modeling of non-dominant-hand features in sign language recognition.

Sign languages are natural languages mainly used by hard-of-hearing (HoH) people. While spoken languages use voices to convey meaning, sign languages utilize hand shape, orientation and movement, sometimes with the aid of facial expression and body movement, to express thoughts. Modern research has proved that sign languages are true languages and can be the first languages of HoH people. <NPL> and <NPL> disclose aspects of sign language recognition.

The World Health Organization estimates that over <NUM>% of the global population (including <NUM> million children) has disabling hearing loss. The technology of SLR may lead to products that can benefit the HoH communities in various ways. However, compared with SR, SLR has received relatively limited research efforts and is immature yet. Introducing the mature SR techniques into SLR may lead to substantial progresses in SLR.

The tasks of SLR and SR are similar in the sense that they both attempt to transfer a sequence of input signals into a sequence of text words. While the input signals for SR are voice signals, those for SLR could be sensor signals from wearable device and/or image sequences from a camera. The similarity indicates that the SR frameworks can also be applicable for SLR. Though, certain strategies need to be used to handle special characteristics of sign languages.

One major characteristic of sign language is that in the signals of a sign phrases, there are transition parts of signals between every two sequential signs, before the first sign and after the last sign. In previous works for small-vocabulary continuous SLR, the transition parts are normally ignored in modeling. However for larger-vocabulary continuous SLR, modeling transition parts explicitly has been proven to be beneficial. The previous works that explicitly model transition signals require transition models structurally different from sign models and significantly increase the complexity of recognition. Consequently, improvements to SLR systems and methods that improve the recognition of sign language gestures for sign languages with a wide range of vocabulary sizes would be beneficial.

For sign languages, there are other special characteristics related to the non-dominant hand. The first characteristic is that the dominant hand and the non-dominant hand are not equally important for sign languages. For example, frequently used sign-language words normally only involve the dominant hand in signing. The second one is that for the dominant-hand-only signs, the non-dominant-hand signals may not be stable and are influenced by the neighborhood signs or even additional actions. Strategies to handle these characteristics related to the non-dominant hand are thus beneficial for sign language recognition.

The method includes generation of explicit models for the transition signals before, between, and after signs within a HMM framework. In traditional speech recognition, an HMM based framework trains HMMs for phonemes as well as for the short pause between words and the silence between phrases. The speech recognition HMM then connects the phoneme HMMs into word HMMs and connects the word HMMs with the short pause/silence HMMs into a decoding network. Recognition is conducted by searching for the path with the highest probability in the decoding network given the input signals. <FIG> illustrates a prior art decoding network <NUM> for SR along with the HMM structures.

The HMM framework for SLR differs from existing SR frameworks. The system generates an HMM framework for SLR through (<NUM>) training a HMM for each gesture that forms an individual sign; (<NUM>) training a transition HMM for the transition signals between two continuous signs within a larger phrase, a start HMM for the transition signals before the first sign of sign phrase begins, and an end HMM after the last sign of a phrase finishes; and (<NUM>) connecting the sign and transition HMMs into phrases with the transition/st/end HMMs into a decoding network similar to that of SR, as illustrated in <FIG>. In this way, the system generates an HMM-based recognition framework for the task of SLR. The SLR framework generates explicit models of transition signals to reduce the influence from the neighborhood signs on the modeling of signs, which improves the recognition accuracy.

The SLR system also incorporates features that are extracted from the dominant hand and non-dominant hand into the training of HMMs. While traditional speech recognition comes from the single voice of a user, the input of the SLR system comes from two channels, which correspond to the movements of the left and right hands. One characteristic of sign language is the usages of dominant and non-dominant hands are unbalanced. The dominant hand is used in most signs, while the non-dominant hand appears less frequently and mainly plays an assistant role compared with the dominant hand. This characteristic indicates that incorporating dominant-hand and non-dominant-hand information in a distinguishing way may be beneficial. The system optionally employs two embodiments to only keep key information from the non-dominant hand in modeling and thus make the SLR system focus more on the dominant hand. In one embodiment, the system only includes features from the non-dominant hand related to the hand global angles in modeling.

Experiment results indicate that compared with equally including left/right hand information, the first approach effectively improve the recognition accuracy while the second approaches provides comparable accuracy. One additional advantage is that using the two approaches, the non-dominant-hand device (e.g., digital gloves) to collect sign signals can be substantially simplified, only requiring one or several sensors to get the hand global angles during sign movements.

The SLR systems and methods described herein also model the non-dominant hand variances for dominant-hand-only signs. The non-dominant hand variance models are designed to handle the special characteristic of sign language that the non-dominant hand inputs of dominant-hand-only signs are influenced by the neighborhood signs or even additional actions. The system design includes a Tying-with-Transition (TWT) solution to reduce the impact of this phenomenon on recognition performance. One embodiment of a training and sign language recognition method ties the non-dominant-hand related parameters of the transition model with those of the models for dominant-hand-only signs, and updates the decoding network by including the tied model as an alternative option for each dominant-hand-only sign model in the decoding network if this sign follows a double-hand sign. The advantage of adopting this solution is an increase in the robustness of the SLR system.

As mentioned in the last two paragraphs of the last section, previous works in SLR either did not explicitly model transition signals or modeled transition signals with the need to introduce complex recognition procedures. Prior art SLR systems do not distinguish between the left and right hand gestures as described herein. The SLR systems and methods described herein differ from previous works.

In one embodiment, a training method for automated sign language recognition has been developed. The invention referring to the training method for automated sign language recognition is as defined in claim <NUM>.

In another embodiment, a method for automated sign language recognition has been developed. The invention referring to the method for automated sign language recognition is as defined in claim <NUM>.

In another embodiment, a system, as defined in claim <NUM>, for automated sign language recognition has been developed. The system includes an input device configured to receive input corresponding to a plurality of hand movements and postures from a user corresponding to a sequence of signs, an output device, a memory, and a processor operatively connected to the input device, the output device, and the memory. The processor is configured to receive an input from the input device based on a plurality of hand movements and postures of a user that correspond to a sequence of sign, extract a plurality of features from the input corresponding to the plurality of hand movements and postures, identify a start of the sequence of signs in the input based on a first set of features in the plurality of features and a first Hidden Markov Model (HMM) stored in the memory, identify a first sign in the input based on a second set of features in the plurality of features and a second HMM stored in the memory, and generate an output with the output device corresponding to the first sign from the input.

For the purposes of promoting an understanding of the principles of the embodiments disclosed herein, reference is now be made to the drawings and descriptions in the following written specification. No limitation to the scope of the subject matter is intended by the references. The present disclosure also includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles of the disclosed embodiments as would normally occur to one skilled in the art to which this disclosure pertains.

As used herein, the terms "sign phoneme" or more simply "phoneme" are used interchangeably and refer to a gesture, handshape, location, palm orientation, or other hand posture using one or two hands that corresponds to the smallest unit of a sign language. Each sign in a predetermined sign language includes at least one sign phoneme, and as used herein the term "sign" refers to a word or other unit of language that is formed from one or more sign phonemes. In some instances a single sign phoneme corresponds to a single word, while in other instances multiple phonemes form a single word depending upon the conventions of the sign language. Many signed words can use the same phoneme or different phonemes more than once in a sequence to form a single word or express an idea in the context of a larger sentence or phrase. As used herein, the term "transition" refers to the movements of the hands that are made between individual signs, such as between sequences of sign phonemes that form individual words or between the phonemes within a single sign. Unlike a sign phoneme, a transition movement is not a typical unit of a sign language that has independent meaning, but transition movements still provide important contextual information to indicate the end of a first word or phoneme and the beginning of a second word or phoneme. As used herein, the term "st/end" refers to hand movements and postures of the user for starting and ending a sequence of signs, including predetermined sequences of signs that are used in a training process for sign language recognition and for non-predetermined sequences of signs that an automated sign language recognition system detects during operation. As used herein, the term "phrase" refers to any sequence of signs that include some form of start and end hand motions along with motions for at least one individual sign. In phrases that include multiple signs, a transition hand movement separates individual signs within the phrase. Phrases can correspond to any sequence of interrelated signs such as signs corresponding to multiple words in a complete sentence or shorter word sequence. A phrase can also include, for example, sequences of numbers or individual letters in a predetermined sign language.

As used herein, the term "dominant hand" refers to a hand of the user that performs a greater number of movements when producing signs. The term "non-dominant hand" refers to the other hand that performs fewer movements. Many sign language conventions specify that dominant hand performs most of the movements to form signs while the non-dominant hand performs fewer gestures to form the signs. In many instances the dominant hand is the right hand of the user while the non-dominant hand is the left hand, although the roles of the hands may be reversed for some signers, such as left-handed signers, or for different sign language conventions where the left hand is the dominant hand for forming signs.

As described below, a sign language recognition (SLR) system records data based on the hand gestures of a user who produces one or more signs in at least one form of recognized sign language. The system receives input using, for example, gloves that include motion and position sensors or from recorded video data of the hands while the user performs sign gestures. The system also includes one or more digital processing devices, data storage devices that hold a system of Hidden Markov Models (HMMs) that are used to recognize sign language gestures, and an output device. The digital processing devices analyze recorded data from the gestures and identify the meanings of words in the sign language from one or more hand gestures that correspond to different sign phonemes and with reference to transitions movements as the user repositions his or her hand to form different sign phonemes in one or more words. In some embodiments, the system also distinguishes between left-hand and right-hand gestures and analyzes the movements of both hands since some forms of sign language use the left and right hands for different purposes. The output device generates a text or audio translation of recognized signs, and the output can be provided to other humans for translation or additional computing systems receive the input to enable automated sign language recognition as a human machine interface.

<FIG> depicts an embodiment of a system <NUM> that performs automated sign language recognition. The system <NUM> includes a processor <NUM>, memory <NUM>, one or more input devices <NUM>, and one or more output devices <NUM>. The processor <NUM> is, for example, a digital microprocessor that includes one or more central processing unit (CPU) cores and optionally one or more graphical processing unit (GPU) units, digital signal processors (DSPs), field programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs) for processing data from the input devices <NUM> and generating output using the output devices <NUM>. In one embodiment of the system <NUM> that is described in more detail below, the input devices <NUM> include two gloves that the user <NUM> wears while producing sign language input. The gloves include accelerometers and gyroscope sensors that provide input data tracking the position, movement, orientation, and hand shape of both the dominant hand and non-dominant hand of the user <NUM> as the user <NUM> performs different sequences of signs.

The memory <NUM> includes one or more volatile memory devices such as dynamic or static random access memory (RAM) and one or more non-volatile memory devices such as a magnetic disk or solid state storage device that store program instructions <NUM>, model data, and other forms of data for operation of the system <NUM>. In the embodiment of <FIG>, the memory <NUM> includes a sign language phrase model <NUM> that is formed from three classes of different Hidden Markov Models (HMMs) including a phrase st/end (st/end) model <NUM>, individual sign models <NUM>, and a sign transition model <NUM>. The combined HMMs <NUM> - <NUM> form a sign language recognition decoding network <NUM>. The overall sign language phrase recognition model <NUM> uses the individual HMMs <NUM> - <NUM> in the decoding network <NUM> to identify the beginning and end of phrases, individual signs within a phrase, and transitions between signs in multi-sign phrases in features that are extracted from the input received from the input devices <NUM>. The sign language phrase recognition model in the memory <NUM> also includes a language model <NUM> that converts identified signs from the sign language phrase recognition model <NUM> into phrases/sentences for one or more languages, such as Chinese or American Sign Language phrases/sentences. The memory <NUM> also stores training data <NUM> that the processor <NUM> uses to generate the HMMs in the sign language phrase recognition model <NUM>.

During operation, a user <NUM> provides sign language input to the system <NUM> using the input device <NUM>. In a training mode, the user <NUM> provides sign inputs to the input devices <NUM> for one or more predetermined sequences of signs that the processor <NUM> stores with the training data <NUM> in the memory <NUM>. As described in more detail below, the processor <NUM> uses the predetermined structure of the sign sequence and observed features that are extracted from the training input data to train the multiple HMMs for the phrase st/end model <NUM>, individual sign models <NUM>, and sign transition model <NUM>. In a recognition mode, the user <NUM> provides sign inputs to the system <NUM> using the input devices <NUM> and the system <NUM> uses the trained HMMs <NUM> - <NUM> along with the language model <NUM> to identify sequences of signs based on the hand movements and postures of the user <NUM>. The system <NUM> generates an output using the output devices <NUM> to, for example, generate an audio output in a predetermined language based on the identified signs in each phrase. In other embodiments, the output device <NUM> is a display screen or network adapter that displays or transmits, respectively, a text representation of the identified signs. In still other embodiments, the system <NUM> receives the signs as command inputs and the processor <NUM> executes stored program instructions <NUM> based on the identified command inputs to produce an output using one or more of the output devices <NUM>.

The sign language recognition (SLR) system <NUM> trains HMMs and builds a decoding network in a similar way as conventional HMM-based speech recognition system. The SLR system views every unique basic sign as a sign phoneme, and trains a HMM for it. Each sign word can thus be modeled by connecting the HMMs of the component sign phonemes, just like speech words are modeled by connecting voice phoneme HMMs. The decoding network is constructed by connecting sign word models using a language model, which can be grammars, n-grams, or other types of language model. The special treatment needed for SLR lies in the modeling of transition. The system trains a HMM, called the transition model, to model the transition signals between neighboring signs. A start (st) HMM and an end HMM are also trained to model the transition signals in a phrase before the first sign and after the last sign respectively. The system then modify the decoding network by inserting the transition model between neighboring sign phonemes, inserting the start model before phrase start nodes and inserting the end model after the phrase end nodes. <FIG> illustrates the decoding network along with the HMM structures in the system <NUM> in more detail. In <FIG>, an HMM-based SLR decoding network <NUM> incorporates all of the HMMs, including sign HMMs <NUM>, the transition HMM <NUM>, and the st/end HMM <NUM>. All of these HMMs adopt a left-to-right HMM structure. The HMM decoding network <NUM> uses a fixed number of states within the sign HMM <NUM> to encode each of the potential sets of features for different sign phonemes. The numbers of states are tuned for the sign HMM <NUM>, the transition HMM <NUM>, and the st/end HMMs <NUM>, respectively.

<FIG> is a block diagram of a training process <NUM> for training HMMs that identify features corresponding to hand movements and postures for each sign, transition between signs, and start and end hand movements and postures for predetermined sequences of signs corresponding to multiple phrases in a training process. As set forth below, a description of the process <NUM> performing an action or function refers to the operation of a processor to execute stored program instructions to perform the function or action in association with other components in the automated sign language recognition system. The process <NUM> is described in conjunction with the system <NUM> of <FIG> for illustrative purposes.

Process <NUM> begins as the system <NUM> receives predetermined sequences of signs in a sign language from the input device <NUM> including generated sensor data from both the dominant and non-dominant hands of the user <NUM> or other users who perform the training process (block <NUM>). In some embodiments, each sequence of sign language inputs includes a set of phonemes that correspond to a single word in a predetermined sign language, but the inputs also include start and end features from the user and transitions between sign phonemes. <FIG> depicts an example of an input with a sequence of individual signs that form one predetermined sequence during the training process. In some embodiments of the process <NUM>, each of the signs in <FIG> forms one input sequence to the training process <NUM>. Since signs are formed from individual phonemes, the training process <NUM> does not require input for all possible words and phrases in the language, but the system <NUM> receives a representative sample of the sign phonemes, st/end signals, and transitions during the process <NUM> to enable recognition of sign language words that are not directly included in the training data inputs. In <FIG> the sequence <NUM> includes a phrase formed from four individual signs for the words "at" (sign <NUM>), "where" (sign <NUM>), and two signs that specify the word "register" (signs <NUM> and <NUM>). In addition to the hand movements and/or postures that form the signs, the system <NUM> also receives input data for the hand movements and postures that the user performs at the beginning and end of the predetermined sequence along with transition movements for the hands that occur between the signs in the sequence. As used herein, the terms "hand posture" or more simply "posture" refer to the orientation and shape of the hand, including either or both of the palm and fingers, that conveys information in sign phonemes along with the motion of the hand and fingers on the hand. During the process <NUM> the system <NUM> receives multiple inputs for a wide range of phrases from one or more users to provide a set of training data that not only provide training information for the recognition of individual signs, but further include input data corresponding to the st/end hand motions for phrases and for transitions between signs within the phrases.

During the process <NUM>, the system <NUM> collects training inputs that include a certain number of samples are collected for each of the sign words in the vocabulary, including both signs that include a single phoneme and multi-phoneme signs. To make a full use of the captured data, the system <NUM> classifies the input data in several segments, i.e., the component signs, preceded by the start part (i.e., from the beginning to the start of the first sign), followed by the end part (i.e., from the finishing point of the last sign to the end), and the transition parts. Each sign phoneme model can thus be trained on all the segments of the focused signs in the training data, while the transition, start, and end models can be trained on the corresponding segments, respectively. The main advantages of this data collection methodology is as follows: (i) when the vocabulary size increases, the need in training data increases only linearly; (ii) every word in the vocabulary, including uncommon words, can be robustly trained; and (iii) it is feasible to label word samples to provide reliable segmentation information for training, which is especially variable for noisy real-world data.

As described above, in one embodiment the system <NUM> uses gloves with sensors that record acceleration and hand angle position information for the dominant and non-dominant hands of the user. <FIG> depicts examples of gloves <NUM> that are worn during the process <NUM> and during subsequent sign language recognition processing. The gloves <NUM> include multiple sensors, such as micro-electromechanical systems (MEMS) sensors that record the altitude, orientation, and acceleration of different portions of the dominant and non-dominant hands of the user as the user performs hand movements/postures to form the signs in the predetermined sequence. <FIG> depicts a more detailed view of one embodiment of the sensors <NUM> in a glove for a right hand. The sensors <NUM> include multiple sensors 604A and 604B that generate data for the position and movement of the phalanges (fingers) with the sensors 604A on the fingers and 604B on the thumb. The sensors <NUM> also include sensors <NUM> that generate data for the position and movement of the metacarpal region (palm) of the hand and a sensor <NUM> that measures the position and movement of the carpal region (wrist) of the hand. While <FIG> depicts a sensor arrangement for the right-hand, the system <NUM> includes a glove with a similar sensor arrangement for the left-hand as well.

Referring again to <FIG>, the process <NUM> continues as the system <NUM> performs feature extraction to identify the features that correspond to individual signs, sign transitions, and the st/end movements in the input corresponding to the predetermined sequence of sign phonemes (block <NUM>). The feature extraction process enables the system <NUM> to identify characteristics of the sensor data for the hand movement/posture inputs in a uniform manner to train the HMMs <NUM> - <NUM> for subsequent sign language recognition operations. In some embodiments, the system <NUM> receives manually inputted timestamps that categorize different sets of features for hand movements and postures that correspond to the st/end of the input, to different predetermined sign phonemes in each input sequence, and to the transitions between sign phonemes. Additionally, the system <NUM> optionally records time information for the predetermined sequence, which includes recorded data for hand movements/postures that occur in a predetermined order, to distinguish between the different sets of hand movements and postures for the feature extraction process. As described in more detail below, the extracted features include both finger related (FR) features that correspond to the movements and shapes of individual fingers and to global angle (GA) features that correspond to the overall angles and movements of the hands. In some embodiments the system <NUM> extracts different sets of features for the input data from the dominant hand and the non-dominant hand. The extracted features for both hands form the basis of training data to train the HMMs <NUM> - <NUM> in the system <NUM>.

For the dominant and non-dominant hands, the feature set can be grouped into two categories: finger related and global angle related, which are referred to as FR features and GA features, respectively, herein. The FR features include the closeness of each finger, the openness of the thumb, and the deltas (i.e., the differences between the current and the previous frames) of these features. The GA features refer to the posture of the hand and are derived from the global polar angles, x, y and z, for a hand. The features include trigonometric functions of the angles and the delta of the angles for the hand as the hand moves to different positions and orientations to produce signs, inter-phoneme transitions, and phrase start/end gestures and postures.

Given a series of sampled angular velocity values given by sensors at a time ti as <MAT>, and the angle rotated in each sample period can be estimated by <MAT>. If the rotation quaternion is qs,i, and given some initial attitude quaternion, ps, then the current orientation after M accumulated rotations is <MAT> where <MAT> and <MAT>. The processor <NUM> identifies the rotation and angular position of both the dominant and non-dominant hands using the Hamilton product of the quaternions based on the different sensor inputs and features that are generated at different times as the system <NUM> receives input from the user <NUM>. In the process <NUM>, the system <NUM> uses the quaternions identified by angular velocity and acceleration to represent accumulated rotation and positions for the angles of both hands in three-dimensional spaces, such as the three-dimensional space in which the hands move to produce the training sequences, in a simple manner that avoids gimbal lock and since quaternion operations are mathematically non-commutative, which accurately represents the physical characteristics of the rotational movements of the hands of the user.

<FIG> depicts components of the quaternions that the process <NUM> uses to extract the features. <FIG> depicts three spatial axes of movement xs, ys, and zs for the hand of the user <NUM> while forming sign phonemes or transitions. The axes xg, yg, and zg indicate a predetermined absolute coordinate system and the orientation of the hand changes within the absolute coordinate system. The sensors in the input device gloves <NUM> that the user <NUM> wears to provide input to the system <NUM> are In <FIG>, the downward direction of gravity serves as an initial attitude unit quaternion, ps. The user <NUM> hangs his or her arms with hands and fingers extended downward in parallel to the direction of gravity (shown as gs in <FIG>) in the direction of the quaternion, ps at the beginning of each training sign input sequence, to enable each initial quaternion ps in the input data to be aligned with the single reference quaternion for gravity without requiring complex calibration for the individual bones in the fingers and other portions of the hands of the user.

Given a sampled acceleration of <MAT>, the system <NUM> estimates the gravity using the mean of the first k samples, <MAT>, and ps is the unit quaternion of <MAT>. The system <NUM> estimates the directions of the finger bones pointing in a global coordinate space, as <MAT>, where q<NUM> is the rotation from the initial sensor coordinate to the global coordinate according to the sensor coordinate.

As mentioned above, the system <NUM> extracts two types of features from the input data. The first type of feature corresponds to features of the hand shape (finger related, or FR), and the second type of feature corresponds to the hand direction (global angle of hand, or GA). In one embodiment, the FR feature for the hand shape is the cosine distance between the directions of adjacent finger bones, <MAT>. For instance, if s = '2c' and r = '2b', as depicted in the hand configuration of <FIG>, then As,r describes a rotation about the proximal interphalangeal joint (the second far end joint of the index finger). The system <NUM> extracts the FR features for each finger. In some embodiments, the system <NUM> only extracts the FR features for individual fingers only in the dominant hand while extracting the GA features for both the dominant and non-dominant hands as described in more detail below. In other embodiments, the system <NUM> extracts both the FR and GA features for both the dominant and non-dominant hands.

To extract the GA features for the direction of the hand, more specifically the direction of the palm, the system <NUM> identifies the mean of the directions of sensors that are associated with the palm of the hand, such as sensors '2a', '3a', '4a' and '5a' in <FIG>. The system <NUM> extracts a feature that fully describes the direction of the hand in space based on two perpendicular directions, a first direction along the palm plan and a second direction along the normal of the palm plan. Additionally, the system <NUM> optionally identifies velocity and acceleration features of the finger bones or of the palm of the hand by identifying multiple rotational positions over time. The system <NUM> uses the quaternion representation of <MAT> including multiple measurements from the sensors generated at series of predetermined sampling times instead of the single measurement ps to estimate the accumulated rotation of either or both of the dominant and non-dominant hands.

Process <NUM> continues as the system <NUM> uses the extracted feature data from multiple predetermined sequences of sign inputs to train the HMMs <NUM> - <NUM> (block <NUM>). The processor <NUM> generates the HMM <NUM> for the features that are extracted from the input data corresponding to the start end hand movements and postures, the HMM <NUM> for the features that are extracted from the input data corresponding to the sign hand movements and postures, and the HMM <NUM> for the features that are extracted from the input data corresponding to the transition hand movements and postures. To generate each of the HMMs, the system <NUM> performs a training process to generate the HMMs and the system <NUM> based on training inputs from the user <NUM> and the predetermined sign input sequences in the training data <NUM>.

For sign languages, a word may contain one or more sign phonemes. The system <NUM> generates the HMM <NUM> for the sign phonemes to accurately model words based on three requirements. First, words have to be modeled in order to evaluate the feature probability, P(X|W). Second, language models have to be used to calculate the prior sentence probability, P(W). Third, the word models and language models have to be combined to form networks of HMM states in which the decoding for the most likely word string W* in the search space Γ can be performed. During the process <NUM>, the processor <NUM> performs the training process to generate the HMM <NUM> for the sign phonemes based on the observed features for the signs conditioned upon the signs in the predetermined training sequences of signs. The processor <NUM> similarly performs the training process for the transitions <NUM> based on the observed features for the transitions conditioned upon the transitions between signs in the predetermined training sequences of signs and the training process for the phrase st/end hand movements <NUM> based on the observed features for the st/end hand movements conditioned upon the st/end hand movements in the predetermined training data <NUM>.

The system <NUM> uses the extracted features from the feature data as a plurality of different feature vectors that form the inputs for the training processes of each of the separate HMMs including the st/end HMM <NUM>, sign HMM <NUM>, and transition HMM <NUM>. The feature vectors include selected features from both the dominant hand and the non-dominant hand. The feature vectors include features from both hands that are recorded concurrently to map the movements of both hands that occur concurrently while the hands of the user <NUM> move to form signs and st/end or transition movements.

In one embodiment, the system <NUM> trains the sign phoneme models <NUM> based on the segments of the corresponding signs. Alternative training methods are based on the detailed properties of the start and end segments, which may vary based on different contexts for receiving training phrases. To normalize the start and end of each phrase during the training process, one training process embodiment require the signers to put both hands down before and after signing each phrase. The start segment for each phrase contains static hand position and posture signals followed by movement signals, while every end segment contains movement signals which may or may not be followed by static signals. The movement signals can be viewed as a type of transition. During the process <NUM>, the system <NUM> labels each start segment into a static segment and a transition segment, and label each end segment into a transition segment and a static segment, if it exists. The training process for all of the HMMs in the decoding network <NUM> includes the start and end segments with both the static segments for hand posture and the dynamic segments for hand movement. The system <NUM> generates the start model HMM <NUM> by connecting the static model with the transition model. The end model HMM is includes two alternatives, one connecting the transition model with the static model, and the other only including the transition model. In the system <NUM>, the model HMMS <NUM> - <NUM> are diagonal covariance Gaussian mixtures, and the system <NUM> re-estimates the HMMs iteratively after initialization as well as after splitting the Gaussian mixtures.

The training process implicitly includes Gaussian splitting that occurs within different sets of hand motions in the separate HMMs during the iterative process described above. For example, through the standard HMM training procedure of iterative re-estimation and Gaussian splitting, the transition model may contain multiple Gaussian mixture components in each state and thus implicitly perform classification on the transition hand movements and postures using the transition HMM <NUM> that occur between different signs that are encoded in the separate HMM <NUM>. During process <NUM> the system <NUM> trains a "universal" transition HMM <NUM> that can identify the transitions that occur between multiple signs and does not require the system <NUM> to be trained to recognize tightly connected two-sign and three-sign sequences to accurately recognize sign language input. The st/end HMM <NUM> also servers a similar role to enable the system <NUM> to identify the features that correspond to the start and end of each phrase.

Process <NUM> continues as the processor <NUM> stores the separate HMMs <NUM> - <NUM> in the memory <NUM> (block <NUM>). As described in more detail below, during a sign language recognition process the system <NUM> uses the trained HMMs to recognize sequences of signs that a user inputs to the system <NUM> using the input devices <NUM>. The trained HMMs can of course recognize sequences of signs that correspond to the training data, but the system <NUM> is also configured to recognize a wide range of sign language sequences that do not exactly match the training sign sequences since the processor <NUM> uses the language model <NUM> to build a decoding network <NUM> that includes the separate HMMs to recognize individual signs and the transitions between signs with many different combinations that are not necessarily included in the training data directly.

In another embodiment of the process <NUM>, the system <NUM> performs a different feature extraction process and HMM generation process for the processing of blocks <NUM> and <NUM> in the process <NUM>. In the other embodiment, the system <NUM> employs a Tying-with-Transition (TWT) solution to enhance the robustness of the sign language recognition system towards the dominant-hand-only sign phoneme that immediately follows a double-hand sign in a given sign-sequence phrase. For dominant-hand-only signs, the non-dominant hand should involve no movements by definition, but in reality, may move due to the neighboring signs or additional action.

In the TWT embodiment, the system <NUM> separates the dominant-hand and non-dominant-hand features for extraction during the processing of block <NUM> in two data streams for HMM modeling. All the HMM models <NUM>-<NUM> are then trained using the same training procedures described above based on the two data streams, which results in two sets of Gaussian mixtures in each HMM state for the dominant-hand and non-dominant-hand related data streams respectively. Thus, in the TWT embodiment the processor <NUM> generates the HMM models <NUM> for signs based on a first portion of the first set of features from the training inputs corresponding to hand movements and postures for a dominant hand of the user that form the sign phonemes during the training process and based on a portion of the second set of features from the training data corresponding to the hand movements and postures of a non-dominant hand of the user to perform transitions during the training process.

The system <NUM> generates an additional enhanced HMM model for each of the dominant-hand-only sign model based on the trained HMM models for both hands by replacing the non-dominant-hand related Gaussian mixtures in the focused sign model with those non-dominant-hand related Gaussian mixtures in the transition model <NUM>. Note that in this way, the non-dominant-hand variances are thus covered for the dominant-hand-only signs by the enhanced sign models. Both the enhanced dominant-hand-only sign models and the originally trained models (i.e., st/end model <NUM>, sign models <NUM>, and transition model <NUM>) are utilized in block <NUM> for the TWT solution. The decoding network <NUM> is modified to incorporate the enhanced dominant-hand-only sign models based on the neighboring signs. If a sign in the decoding network <NUM> is a dominant-hand-only sign and the preceding sign in the same path involves the non-dominant hand, this sign model in the decoding network <NUM> is replaced with a confusion set of two parallel models, which include the corresponding enhanced model and the original model for this dominant-hand-only sign. The recognition procedure based on this modified decoding network remains the same.

<FIG> depicts a process <NUM> for automated sign language recognition. As set forth below, a description of the process <NUM> performing an action or function refers to the operation of a processor to execute stored program instructions to perform the function or action in association with other components in the automated sign language recognition system. The process <NUM> is described in conjunction with the system <NUM> of <FIG> for illustrative purposes.

Process <NUM> begins as the system <NUM> receives dominant hand and non-dominant hand input from the user <NUM> via the input devices <NUM> (block <NUM>). As described above, in one embodiment the input devices are the gloves <NUM> that include sensors to record the hand movements and postures for the user <NUM> as the user <NUM> performs hand movements and postures corresponding to a sequence of signs.

Process <NUM> continues as the system <NUM> extracts features from the input data (block <NUM>). The system <NUM> performs the feature extraction processing of block <NUM> in substantially the same manner as the feature extraction that is described above in conjunction with the processing of block <NUM> in the process <NUM>. For example, the processor <NUM> extracts the same FR and GA features for the input data from the dominant hand and extracts the GA features and optionally the FR features for the input data from the non-dominant hand in the same manner as in the process <NUM> to produce feature vectors in the recognition process in the same manner as during the earlier training process.

Process <NUM> continues as the system <NUM> analyzes the extracted features from the input to identify a start of a sequence of at least one sign that forms a phrase (block <NUM>). In the system <NUM>, the processor <NUM> uses the st/end HMM <NUM> in the sign language recognition model <NUM> to identify features that correspond to the hand movements and postures associated with beginning a new sequence of signs. The processor <NUM> uses the same sets of extracted features for recognition using the HMM models in the decoding network <NUM> as are used during the training process of <FIG>, such as sets of FR and GA features for the dominant hand and GA features for the non-dominant hand or the FR and GA features of the dominant hand in combination with the transition features of the non-dominant hand in the TWT embodiment. As indicated in <FIG>, the HMMs <NUM> - <NUM> are linked to each other so after detection of the start of a sequence, the processor <NUM> analyzes the next set of features that occur in the time interval following the detection of the initial start feature with a high probability weighting toward the features corresponding to the phonemes in a given sign (block <NUM>). The processor <NUM> uses the HMM <NUM> to identify the next sign in the sequence based on the next set of features that are extracted from the input sequence.

During process <NUM>, the sign input data from the user <NUM> includes either a transition to a new sign or an input that signifies the end of a phrase. For example, in a multi-sign phrase the user performs a series of signs with transitions between the signs until the phrase is completed, while in a single-sign phrase the user produces an end of phrase motion after the sign. During process <NUM>, the processor <NUM> uses both the transition HMM <NUM> and st/end HMM <NUM> to identify a set of features that precede and follow a sign in the input data (block <NUM>). In some embodiments, the system <NUM> interprets an entire phrase of sign inputs as a whole, and the processor <NUM> uses the HMM network <NUM> in the sign language recognition model <NUM> to identify input based on the maximum likelihoods of word sequences based not only on a sequence of words extending forward in time but also backward in time once the system <NUM> has received the entire phrase. The processor <NUM> identifies a path using the HMM network <NUM> and a maximum a posteriori (MAP) estimation process with a Bayes decision formulation to identify the sign phonemes and sequences of signs in the trained HMMs that have the greatest likelihood of matching the extracted features from the input data to determine if a set of features corresponds to a transition between signs or the end of a phrase and determines the transition or end of phrase. Additionally, in some configurations the processor <NUM> also identifies the separation in time between the detection of features, which may indicate the likelihood of the features corresponding to a brief transition with a short time interval between signs or the end of a phrase with a somewhat longer time interval between signs. If the system <NUM> detects a transition but not the end of the sequence in the phrase (block <NUM>), then the process <NUM> continues as the system <NUM> identifies the next sign in the sequence (block <NUM>).

The processing of blocks <NUM> - <NUM> continues until the processor <NUM> identifies features that correspond to the end of the sequence of signs in the phrase (block <NUM>). The recognition procedure of the detected phrase (block <NUM>) simultaneously generates multiple hypothesized sign sequences, which are the top-ranked paths with the highest P(X|W) P(W) values (as in the formula above) in the decoding network <NUM> given the inputs related to the phrase, using the standard HMM decoding technology. During the process <NUM> the system <NUM> performs a maximum a posteriori (MAP) process with a Bayes decision formulation to identify the paths in the HMM decoding network <NUM> for recognition of the sign input from the user <NUM>. The system <NUM> conducts sign language recognition based on the following formula: <MAT> Where X = {x<NUM>, x<NUM>,. , xT} is a sequence of T observed feature vectors corresponding to the unknown input sentence, W={w<NUM>, w<NUM>,. , wM} is any sequence of M signs into which the input may be transcribed, and Γ is the set of all allowable phrases representing the search space of the language model (e.g., grammars). As described above, a sign phoneme here refers to the smallest units that bear linguistic meaning in a predetermined sign language.

In the system <NUM>, the memory <NUM> stores HMMs in the decoding network <NUM> and includes all the recognized sign-language phonemes along with st/end and transition gestures, for recognition of phrases that are supported by the language model for inputs from the user <NUM>, which can be grammars, n-gram models, or other types of language model. The hypothesized sign sequence with the highest P(X|W) P(W) value among the multiple hypotheses generated is used as the result of the sign language recognition procedure for the detected phrase. Block <NUM> further translates that sign sequence from a predetermined sign language into one or more different spoken or written languages such as Chinese or English. To ensure a proper grammatical output, the translation procedure may include reordering the words in the output from the actual order of the signs as entered by the user <NUM> or adding additional articles, conjunctions, prepositions, and other words that are not directly included in the sign language to the final output.

After receiving one or more phrases, the system <NUM> generates an output corresponding to the identified words from the sign language input (block <NUM>). As described above, in some embodiments the processor <NUM> generates synthesized speech for the words in the output using a loudspeaker or other audio output device <NUM>. In this mode, the system <NUM> can produce audio output for recipients who can hear but are unable to understand the signs from the user <NUM>. In another configuration, the output device <NUM> includes a display device that generates a text output corresponding to the signs from the user <NUM>. In still another configuration, the system <NUM> either performs a command based on the sign inputs from the user <NUM>. For example, in some embodiments the system <NUM> is further connected to a home automation system or other system that normally receives speech input, but in the configuration of <FIG> the system can accept sign language input to provide similar functionality to users who communicate via sign language instead of spoken words.

Claim 1:
A training method for automated sign language recognition, the method comprising:
receiving, with an input device, a plurality of training inputs corresponding to a plurality of hand movements and postures, of a user, that correspond to a sequence of signs; the receiving further comprising:
receiving, with the input device, a first plurality of inputs corresponding to a plurality of movements of a dominant hand of the user; and
receiving, with the input device, a second plurality of inputs corresponding to a plurality of movements of a non-dominant hand of the user,
extracting, with the processor, a plurality of features from the training inputs corresponding to the hand movements and postures; the extracting further comprising:
extracting a first plurality of features corresponding to finger related features in the dominant hand from the first plurality of inputs; and
extracting a second plurality of features corresponding to global angle features in the non-dominant hand from the second plurality of inputs,
generating, with the processor, a first Hidden Markov Model to identify a start of the sequence of signs based on a first set of features in the plurality of features from the training inputs corresponding to the hand movements and postures;
generating, with the processor, a second Hidden Markov Model to identify additional signs based on a second set of features in the plurality of features from the training inputs corresponding to the hand movements and postures;
storing, with the processor, the first Hidden Markov Model, and the second Hidden Markov Model in a memory for recognition of said start of the sequence of signs and said additional signs received from the input device; and
generating and then storing in said memory, with the processor, a third Hidden Markov Model based on the first plurality of features from the training inputs and the second plurality of features from the training inputs for identifying transitions between signs in the sequences of signs.