Patent Abstract:
A method and related system, computer program product and device for interactively tracking oral reading of text from a document includes recording audio for a sentence read by a user and determining when the user has reached the last word of the sentence. The method also includes providing visual feedback to the user reading on a sentence by sentence level to indicate a current location in the passage.

Full Description:
BACKGROUND  
       [0001]     Reading software tends to focus on reading skills other than reading fluency. A few reading software products claim to provide benefit for developing reading fluency. One component in developing reading fluency is developing rapid and correct recognition and pronunciation of words included in a passage.  
       SUMMARY  
       [0002]     According to an aspect of the present invention, a method for interactively tracking oral reading of text from a document includes recording audio for a sentence read by a user and determining when the user has reached the last word of the sentence. The method also includes providing visual feedback to the user reading on a sentence by sentence level to indicate a current location in the passage.  
         [0003]     Embodiments can include one or more of the following.  
         [0004]     The method can include determining where the user is within the sentence. Determining where the user is can include determining the current word the user is reading. The method can include assessing the quality of the user&#39;s reading on a word-by-word basis. The method can include providing pronunciation and timing indications on a word-by-word basis. Determining where the user is within the sentence can include recognizing the user re-starting reading of the sentence, recognizing the user repeating words, recognizing the user skipping words, and/or recognizing the user skipping reading of sentences or parts thereof. The text sequence that is treated as a sentence by the tutoring software can be a portion of a full syntactic sentence. The sentence-by-sentence visual feedback can be provided by displaying the current sentence text in a first color and the surrounding text in a second color. The sentence after the current sentence can be displayed in a third color. The sentence after the current sentence can be displayed in the same color as the current sentence as the user gets close to the end of the current sentence. The method can include switching a language model to the next sentence when the end of a sentence is reached.  
         [0005]     In another aspect, the invention includes a computer program product residing on a computer readable medium. The computer program product includes instructions for causing an electrical device to record audio for a sentence read by a user, determine, using speech recognition processing to convert the audio to a text file, when the user has reached the last word of the sentence, and provide visual feedback to the user reading on a sentence by sentence level to indicate a current location in the passage.  
         [0006]     Embodiments can include one or more of the following.  
         [0007]     The computer program product can include instructions for causing an electrical device to determine where the user is within the sentence. The computer program can include instructions for causing an electrical device to assess the quality of the user&#39;s reading on a word-by-word basis. The computer program product can include instructions for causing an electrical device to provide pronunciation and timing indications on a word-by-word basis. The computer program product can include instructions for causing an electrical device to display the current sentence text in a first color and the surrounding text in a second color.  
         [0008]     In a further aspect, the invention includes a device configured to record audio for a sentence read by a user, determine, using speech recognition processing to convert the audio to a text file, when the user has reached the last word of the sentence, and provide visual feedback to the user reading on a sentence by sentence level to indicate a current location in the passage.  
         [0009]     Embodiments can include one or more of the following.  
         [0010]     The device can be configured to determine where the user is within the sentence. The device can be further configured to assess the quality of the user&#39;s reading on a word-by-word basis. The device can be further configured to provide pronunciation and timing indications on a word-by-word basis. The device can be further configured to display the current sentence text in a first color and the surrounding text in a second color.  
         [0011]     The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
     
    
     DESCRIPTION OF DRAWINGS  
       [0012]      FIG. 1  is a block diagram of a computer system adapted for reading tutoring.  
         [0013]      FIG. 2  is a block diagram of a network of computer systems.  
         [0014]      FIG. 3  is a screenshot of a passage for use with the reading tutor software.  
         [0015]      FIG. 4  is a block diagram of inputs and outputs to and from the speech recognition engine or speech recognition process.  
         [0016]      FIG. 5  is a flow chart of a location tracking process.  
         [0017]      FIG. 6  is a flow chart of visual and audio interventions.  
         [0018]      FIGS. 7A and 7B  are portions of a flow chart of an intervention process based on elapsed time.  
         [0019]      FIG. 8  is a screenshot of a set up screen for the tutor software.  
         [0020]      FIG. 9  is a flow chart of environmental weighting for a word based on a reader&#39;s location in a passage.  
         [0021]      FIG. 10  is a block diagram of word categories.  
         [0022]      FIG. 11  is a table of exemplary glue words.  
         [0023]      FIGS. 12A and 123  are portions of a flow chart of a process using word categories to assess fluency.  
         [0024]      FIG. 13  is a screenshot of a passage. 
     
    
     DETAILED DESCRIPTION  
       [0025]     Referring to  FIG. 1 , a computer system  10  includes a processor  12 , main memory  14 , and storage interface  16  all coupled via a system bus  18 . The interface  16  interfaces system bus  18  with a disk or storage bus  20  and couples a disk or storage media  22  to the computer system  10 . The computer system  10  would also include an optical disc drive or the like coupled to the bus via another interface (not shown). Similarly, an interface  24  couples a monitor or display device  26  to the system  10 . Other arrangements of system  10 , of course, could be used and generally, system  10  represents the configuration of any typical personal computer. Disk  22  has stored thereon software for execution by a processor  12  using memory  14 . Additionally, an interface  29  couples user devices such as a mouse  29   a  and a microphone/headset  29   b,  and can include a keyboard (not shown) to the bus  18 .  
         [0026]     The software includes an operating system  30  that can be any operating system, speech recognition software  32  which can be an open source recognition engine or any engine that provides sufficient access to recognizer functionality, and tutoring software  34  which will be discussed below. A user would interact with the computer system principally though mouse  29   a  and microphone/headset  29   b.    
         [0027]     Referring now to  FIG. 2 , a network arrangement  40  of such systems  10  is shown. This configuration is especially useful in a classroom environment where a teacher, for example, can monitor the progress of multiple students. The arrangement  40  includes multiple ones of the systems  10  or equivalents thereof coupled via a local area network, the Internet, a wide-area network, or an Intranet  42  to a server computer  44 . An instructor system  45  similar in construction to the system  10  is coupled to the server  44  to enable an instructor and so forth access to the server  44 . The instructor system  45  enables an instructor to import student rosters, set up student accounts, adjust system parameters as necessary for each student, track and review student performance, and optionally, to define awards.  
         [0028]     The server computer  44  would include amongst other things a file  46  stored, e.g., on storage device  47 , which holds aggregated data generated by the computer systems  10  through use by students executing software  34 . The files  46  can include text-based results from execution of the tutoring software  34  as will be described below. Also residing on the storage device  47  can be individual speech files resulting from execution of the tutor software  34  on the systems  10 . In other embodiments, the speech files being rather large in size would reside on the individual systems  10 . Thus, in a classroom setting, an instructor can access the text-based files over the server via system  45 , and can individually visit a student system  10  to play back audio from the speech files if necessary. Alternatively, in some embodiments the speech files can be selectively uploaded to the server  44 .  
         [0029]     Like many complex skills, reading depends on an interdependent collection of underlying knowledge, skills, and capabilities. The tutoring software  34  fits into development of reading skills based on existence of interdependent areas such as physical capabilities, sensory processing capabilities, and cognitive, linguistic, and reading skills and knowledge. In order for a person to learn to read written text, the eyes need to focus properly and the brain needs to properly process resulting visual information. A person learning to read should also possess basic vocabulary and language knowledge in the language of the text, such as may be acquired through oral language experience or instruction in that language, as well as phonemic awareness and a usable knowledge of phonics. In a typical classroom setting, a person should have the physical and emotional capability to sit still and “tune out” distractions and focus on a task at hand. With all of these skills, knowledge, and capabilities in place, a person can begin to learn to read with fluency and comprehension and, through such reading, to acquire the language, vocabulary, information, and ideas of texts.  
         [0030]     The tutor software  34  described below, while useful for students of reading in general, is specifically designed for the user who has developed proper body mechanics and sensory processing and has acquired basic language, alphabet, and phonics skills. The tutor software  34  can develop fluency by supporting frequent and repeated oral reading. The reading tutor software  34  provides this frequent and repeated supported oral reading, using speech recognition technology to listen to the student read and provide help when the student struggles and by presenting records of how much and how accurately and fluently the student has read. In addition, the reading tutor software  34  can assist in vocabulary development by providing definitions of words in the built-in dictionary, by keeping track of the user&#39;s vocabulary queries, and by providing assistance that may be required to read a text that is more difficult than the user can easily read independently. The tutor software  34  can improve reading comprehension by providing a model reader to which the user can listen, and by assisting with word recognition and vocabulary difficulties. The reading tutor  34  can also improve comprehension by promoting fluency, vocabulary growth, and increased reading. As fluency, vocabulary, and reading experience increase, so does reading comprehension. which depends heavily on reading fluency. The software  34  can be used with persons of all ages including children in early though advanced stages of reading development.  
         [0031]     Referring now to  FIG. 3 , the tutor software  34  includes passages such as passage  47  that are displayed to a user on a graphical user interface. The passages can include both text and related pictures. The tutor software  34  includes data structures that represent a passage, a book, or other literary work or text. The words in the passage are linked to data structures that store correct pronunciations for the words so that utterances from the user of the words can be evaluated by the tutor software  34 . The speech recognition software  32  verifies whether a user&#39;s oral reading matches the words in the section of the passage the user is currently reading to determine a user&#39;s level of fluency.  
         [0032]     Referring to  FIG. 4 , the speech recognition engine  32  in combination with the tutor software  34  analyzes speech or audio input  50  from the user, and generates a speech recognition result  66 . The speech recognition engine  32  uses an acoustic model  52 , a language model  64 , and a pronunciation dictionary  70  to generate the speech recognition result  66 .  
         [0033]     The acoustic model  52  represents the sounds of speech (e.g., phonemes). Due to differences in speech for different groups of people or individual users, the speech recognition engine  32  includes multiple user acoustic models  52  such as an adult male acoustic model  54 , an adult female acoustic model  56 , a child acoustic model  58 , and a custom acoustic model  60 . In addition, although not shown in  FIG. 4 , acoustic models for various regional accents, various ethnic groups, or acoustic models representing the speech of users for which English is a second language could be included. A particular one of the acoustic models  52  is used to process audio input  50 , identify acoustic content of the audio input  50 , and convert the audio input  50  to sequences of phonemes  62  or sequences of words  68 .  
         [0034]     The pronunciation dictionary  70  is based on words  68  and phonetic representations. The words  68  come from the story texts or passages, and the phonetic representations  72  are generated based on human speech input or models. Both the pronunciation dictionary  70  and the language model  64  are derived from the story texts to be recognized. For the pronunciation dictionary  70 , the words are taken independently from the story texts. In contrast, the language model  64  is based on sequences of words from the story texts or passages. The recognizer uses the language model  64  and the pronunciation dictionary  70  to constrain the recognition search and determine what is considered from the acoustic model when processing the audio input from the user  50 . In general, the speech recognition process  32  uses the acoustic model  52 , a language model  64 , and a pronunciation dictionary  70  to generate the speech recognition result  66 .  
         [0035]     Referring to  FIG. 5 , a process  80  for tracking a user&#39;s progress through the text and providing feedback to the user about the current reading location in a passage (e.g., a passage as shown in  FIG. 2 ) is shown. As the student reads the passage, the tutor software  34  guides the student through the passage on a sentence-by-sentence basis using sentence-by-sentence tracking. In order to provide sentence-by-sentence tracking, a passage is displayed  82  to the user. The sentence-by-sentence tracking provides  84  a visual indication (e.g., changes the color of the words, italicizes, etc.) for an entire sentence to be read by the user. The user reads the visually indicated portion and the system receives  86  the audio input. The system determines  88  if a correct reading of the indicated portion has been received. The portion remains visually indicated  90  until the speech recognition obtains an acceptable recognition from the user. After the sentence has been completed, the visual indication progresses  92  to a subsequent (e.g., the next) sentence or clause. In some embodiments, the visual indication may progress to the next sentence before the user completes the current sentence, e.g. when the user reaches a predefined point in the first sentence. Sentence-by-sentence tracking can provide advantages over word-by-word tracking (e.g., visually indicating only the current word to be read by the user, or ‘turning off’ the visual indication for each word as soon as it has been read correctly). Word-by-word tracking may be more appropriate in some situations, e.g., for users who are just beginning to learn to read. However, sentence-by-sentence tracking can be particularly advantageous for users who have mastered a basic level of reading and who are in need of developing reading fluency and comprehension. Sentence-by-sentence tracking promotes fluency by encouraging students to read at a natural pace without the distraction of having a visual indication change with every word. For example, if a child knows a word and can quickly read a succession of multiple words, word-by-word tracking may encourage the user to slow his or her reading because the words may not be visually indicated at the same rate as the student would naturally read the succession of words. Sentence-by-sentence feedback minimizes the distraction to the user while still providing guidance as to where s/he should be reading within the passage.  
         [0036]     In order to provide sentence-by-sentence tracking, sentence transitions or clause transitions are indicated in the software&#39;s representation of the passage. These transitions can be used to switch the recognition context (language model) and provide visual feedback to the user. The tracking process  80  aligns the recognition result to the expected text, taking into account rules about what words the tutor software recognizes and what words can be skipped or misrecognized (as described below).  
         [0037]     While the tutor software  34  is described as providing visual feedback based on a sentence level, other segmentations of the passage are possible and can be treated by the system as sentences. For example, the tutor software can provide the visual indication on a phrase-by-phrase basis, a clause-by-clause basis, or a line-by-line basis. The line-by-line segmentation can be particularly advantageous for poetry passages. Phrase-by-phrase and clause-by-clause segmentation can be advantageous in helping the student to process the structure of long and complex sentences.  
         [0038]     In some embodiments, in addition to the visual indication of the portion of the passage currently being read, a visual indication is also included to distinguish the portions previously read by the user from the portions not yet completed. For example, the previously read portions could be displayed in a different color or could be grayed. The difference in visual appearance of the previously read portions can be less distracting for the user and help the user to easily track the location on the screen.  
         [0039]     In some embodiments, the highlighting can shift as the user progresses in addition to changing or updating the highlighting or visual indication after the recognition of the completion of the sentence. For example, when the user reaches a predetermined transition point within one sentence the visual indication may be switched off for the completed part of that sentence and some or all of the following sentence may be indicated.  
         [0040]     As described above, the location of a student&#39;s reading within the passage is visually indicated to the user on a sentence-by-sentence basis. However, the system tracks where the user is on a word-by-word basis. The location is tracked on a word-by-word basis to allow the generation of interventions. In general, interventions are processes by which the application assists a user when the user is struggling with a particular word in a passage. It also tracks on a word-by-word basis so as to allow evaluation, monitoring and record-keeping of reading accuracy and fluency, and to generate reports to students and teachers about same.  
         [0041]     The tutor software  34  provides multiple levels of interventions, for example, the software can include a visual intervention state and audio intervention state, as shown in  FIG. 6 . When the tutor software  34  does not receive a valid recognition on an expected word after a specified duration has elapsed, the tutor software  34  intervenes  106  by applying a visual indication to the expected word. For example, a yellow or other highlight color may be applied over the word. Words in the current sentence that are before the expected word may also be turned from black to gray to enable the user to quickly identify where he/she should be reading. The user is given a chance to self-correct or re-read the word. The unobtrusive nature of the visual intervention serves as a warning to the student without causing a significant break in fluent reading. If the tutor software  34  still fails  108  to receive an acceptable recognition of the word, an audio intervention takes place  110 . A recording or a synthesized version of the word plays with the correct pronunciation of the word and the word is placed  114  on a review list. Alternatively, a recording indicating “read from here” may be played, particularly if the word category  190  indicates that the word is a short common word that the user is likely to know. In this case, the user is likely struggling with a subsequent, more difficult word or is engaged in extraneous vocalization, so likewise the software may not place the word on a review list depending on the word category (e.g. if the word is a glue word  194 ). The tutor software  34  gives the student the opportunity to re-read the word correctly and continue with the current sentence. The tutor software  34  determines if a valid recognition for the word has been received and if so, proceeds  102  to a subsequent word, e.g., next word. If a valid recognition is not received, the software will proceed to the subsequent word after a specified amount of time has elapsed.  
         [0042]     As described above, the reading tutor software  34  provides visual feedback to the user on a sentence-by-sentence basis as the user is reading the text (e.g. the sentence s/he is currently reading will be black and the surrounding text will be gray). This user interface approach minimizes distraction to the user compared to providing feedback on a word-by-word basis (e.g., having words turn from black to gray as s/he is recognized). With the sentence-by-sentence feedback approach, however, it can be desirable to non-disruptively inform the user of the exact word (as opposed to sentence) where the tutor software expects the user to be reading. The software may need to resynchronize with the user due to several reasons. For example, the user may have read a word but stumbled or slurred the word and the word was not recognized, the application may have simply misrecognized a word, the user may have lost his/her place in the sentence, the user may have said something other than the word, and the like. It can be preferable to provide an intervention to help to correct such errors, but a full intervention that plays the audio for the word and marks the word as incorrect and puts the word on the review list may not be necessary. Thus, a visual intervention allows the user or the application to get back in synchronization without the interruption, distraction, and/or penalty of a full intervention on the word.  
         [0043]     As described above, there will be a time gap from the time that a valid recognition is received for one (previous) word, during which a valid recognition for the expected (next) word has not yet been received. If there is no relevant previous word, there will be a time gap from the time the current utterance (i.e. audio file or audio buffer) was initiated, during which the expected word has not yet been received. This time gap can become significant or large for a number of reasons, e.g. a user may pause during the reading of a passage because s/he does not know the expected word, the user may mispronounce or skip the expected word, or the recognition engine may not correctly identify the expected word in the audio stream. The tutor software  34  can provide an intervention based on the length of time elapsed since the previous word, or since the start of the audio buffer or file, during which the tutor software  34  has not yet received a valid recognition for the expected word.  
         [0044]     Referring to  FIG. 7 , a process  130  for determining an intervention based on an elapsed amount of time or a pause is shown. Process  130  includes initializing  132  a timer, e.g., a software timer or a hardware timer can be used. The timer can be initialized based on the start of a silence (no voice input) period, the start of a new audio buffer or file, the completion of a previous word, or another audio indication. The timer determines  136  a length of time elapsed since the start of the timer. Process  130  determines  140  if the amount of time on the timer since the previous word is greater than a threshold. If the time is not greater than the threshold, process  130  determines  138  if valid recognition has been received. If a valid recognition has not been received, process  130  returns to determining the amount of time that has passed. This loop is repeated until either a valid recognition is received or the time exceeds the threshold. If a valid recognition is received (in response to determination  138 ), process  130  proceeds  134  to a subsequent word in the passage and re-initializes  132  the timer. If the time exceeds the threshold, process  130  provides  142  a first/visual intervention. For example, the tutor software highlights the word, changes the color of the word, underlines the word, etc.  
         [0045]     After providing the visual intervention, process  130  determines  144  an amount of time since the intervention or a total time. Similar to the portion of the process above, process  130  determines  148  if the amount of time on the timer is greater than a threshold. This threshold may be the same or different than the threshold used to determine if a visual intervention is needed. If the time is not greater than the threshold, process  130  determines  150  if a valid recognition has been received. If input has not been received, process  130  returns to determining  148  the amount of time that has passed. This loop is repeated until either a valid recognition is received or the time exceeds the threshold. If a valid recognition is received (in response to determination  148 ), process  130  proceeds  146  to a subsequent word in the passage and re-initializes  132  the timer. If the time exceeds the threshold, process  130  provides  152  an audio intervention.  
         [0046]     After providing the audio intervention, process  130  determines  156  an amount of time since the intervention or a total time and determines  148  if the amount of time is greater than a threshold (e.g., a third threshold). This threshold may be the same or different from the threshold used to determine if a visual intervention or audio intervention is needed. If the time is not greater than the threshold, process  130  determines  158  if a valid recognition has been received. If input has not been received, process  130  returns to determining  160  the amount of time that has passed. This loop is repeated until either a valid recognition is received or the time exceeds the threshold. If a valid recognition is received (in response to determination  160 ), process  130  proceeds  154  to a subsequent word in the passage and re-initializes  132  the timer. If the time exceeds the threshold, process  130  proceeds  162  to a subsequent word in the passage, but the word is indicated as not receiving a correct response within the allowable time period.  
         [0047]     In some embodiments, the visual intervention state and the full audio intervention state are used in combination. A visual intervention is triggered after a time-period has elapsed in which the tutor software  34  does not recognize a new sentence word. The “visual intervention interval” time period can be about 1-3 seconds, e.g., 2 seconds as used in the example below. However, the interval can be changed in the application&#39;s configuration settings (as shown in  FIG. 8 ). For example, if the sentence is “The cat sat” and the tutor software  34  receives a recognition for the word “The”, e.g., 0.9 seconds from the time the user starts the sentence, no intervention will be triggered for the word “The” since the time before receiving the input is less than the set time period. However, if 2.0 seconds elapses from the time the software received a recognition for “The”, during which the tutor software does not receive a recognition for the word “cat” the tutor software  34  triggers a visual intervention on the word “cat”″ (the first sentence word that has not been recognized). For the visual intervention, words in the current sentence which are prior to the intervened word are colored gray. The word that triggered the visual intervention (e.g. cat) is colored black and additionally has a colored (e.g., yellow) oval “highlight” overlaid over the word. The remainder of the sentence is black. Other visual representations could, however, be used.  
         [0048]     From the point of view of speech recognition, a new recording (starting with .“cat”) starts with the visually intervened word and the tutor software re-synchronizes the recognition context (language model) so that the recognizer expects an utterance beginning with the intervened word.  
         [0049]     If the user reads the word that has received visual intervention successfully before the audio intervention is triggered, the intervened word is coded, e.g., green, or correct unless the word is a member of a certain word category. For example if the word is a target word, it can be coded in a different color, and/or placed on a review list, indicating that the word warrants review even though it did not receive a full audio intervention. If the user does not read the word successfully, a full audio intervention will be triggered after a time period has elapsed. This time period is equal to the Intervention Interval (set on a slider in the application, e.g., as shown in  FIG. 8 ) minus the visual intervention interval. The time periods before the visual intervention and between the visual intervention and the full intervention would be a minimum of about 1-5 seconds so that these events do not trigger before the user has been given a chance to say a complete word. The optimum time period settings will depend upon factors including the reading level of the text, the word category, and the reading level, age, and reading rate of the user. If the Intervention Interval is set too low (i.e. at a value which is less than the sum of the minimum time period before the visual intervention, and the minimum time period between the visual intervention and the full intervention), the visual intervention state will not be used and the first intervention will be an audio intervention.  
         [0050]     Referring to  FIG. 8 , a screenshot  170  of a user interface for setting speech recognition characteristics for the tutor software  34  is shown. The speech recognition screen  170  allows a user or administrator to select a particular user (e.g., using selection boxes  171 ) and set speech recognition characteristics for the user. The user or administrator can select an acoustic model by choosing between acoustic models included in the system by selecting one of the acoustic model boxes  172 . In addition, the user can select a level of pronunciation correctness using pronunciation correctness continuum or slider  173 . The use of a pronunciation correctness slider  173  allows the level of accuracy in pronunciation to be adjusted according to the skill level of the user. In addition, the user can select an intervention delay using intervention delay slider  174 . The intervention delay slider  174  allows a user to select an amount of time allowed before an intervention is generated.  
         [0051]     As described above, speech recognition is used for tracking where the user is reading in the text. Based on the location in the text, the tutor software  34  provides a visual indication of the location within the passage where the user should be reading. In addition, the speech recognition can be used in combination with the determination of interventions to assess at what rate the user is reading and to assess if the user is having problems reading a word. In order to maximize speech recognition performance, the tutor software dynamically defines a “recognition configuration” for each utterance (i.e. audio file or buffer that is processed by the recognizer).  
         [0052]     A new utterance will be started when the user starts a new sentence or after a visual intervention or audio intervention. The recognition configuration includes the set of items that can be recognized for that utterance, as well as the relative weighting of these items in the recognizer&#39;s search process. The search process may include a comparison of the audio to acoustic models for all items in the currently active set. The set of items that can be recognized may include expected words, for example, the words in the current sentence, words in the previous sentence, words in the subsequent sentence, or words in other sentences in the text. The set of items that can be recognized may also include word competition models. Word competition models are sequences of phonemes derived from the word pronunciation but with one or more phonemes omitted, or common mispronunciations or mis-readings of words. The set of recognized sounds include phoneme fillers representing individual speech sounds, noise fillers representing filled pauses (e.g. “um”) and non-speech sounds (e.g. breath noise).  
         [0053]     For some recognition items in the active set, for example phoneme fillers, the relative weighting of these items is independent of prior context (independent of what has already been recognized in the current utterance, and of where the user started in the text). For other items, the relative weighting of items is context-dependent, i.e. dependent on what was recognized previously in the utterance and/or on where the user was in the text when the utterance started.  
         [0054]     The context-dependent weighting of recognition items is accomplished through language models. The language models define the words and competition models that can be recognized in the current utterance, and the preferred (more highly weighted) orderings of these items, in the recognition sequence. Similar to a statistical language model that would be used in large-vocabulary speech recognition, the language model  64  defines the items (unigrams—a single word), ordered pairs of items (bigrams—a two word sequence), and ordered triplets of items (trigrams—a three word sequence) to be used by the recognition search process. It also defines the relative weights of the unigrams, bigrams, and trigrams which is used in the recognition search process. Additionally, the language model defines the weights to be applied when recognizing a sequence (bigram or trigram) that is not explicitly in the language model. However, unlike a statistical language model, the language model  64  is not based on statistics derived from large amounts of text. Instead it is based on the sequence of words in the text and on patterns of deviation from the text that are common among readers.  
         [0055]     Referring to  FIG. 9 , the language model generation process  177  takes the current text  178  that the user is reading and divides it into segments  179 . In one embodiment, each segment includes the words in a single sentence and one or more words from the following sentence. In other implementations, the segment could be based on other units such as paragraph, a page of text, or a phrase. The unigram, bigram, and trigram word sequences and corresponding weights are defined  180  based on the sequence of words in the sentence, and the word competition models for those words. The language model generation process uses rules about which words in the sentence may be skipped or not recognized in oral reading (based on word category). The speech recognition process selects the language model to use based on where the user is reading in the text  186  (e.g., the process selects the language model for the current sentence). The recognition process adjusts the probability or score of recognition alternatives currently being considered in the recognition search based on the language model  185 . Once the user starts an utterance, the “prior context” used by the language model to determine weightings comes from recognition alternatives for the utterance up until that point. For example, if the sentence is “The cat sat on the mat” and a recognition alternative for the first part of the utterance is “The cat”, then the weightings provided by the language model will typically prefer a recognition for “sat” as the next word over other words in the sentence.  
         [0056]     At the very start of the utterance however, no prior context from the recognizer is yet available. In this case, the tutor software uses the prior context based on where the user was in the text at the start of this utterance. This “initial recognition context” information is also included in the language model. Therefore, if the user just received an intervention on “sat” and is therefore starting an utterance with that word, the initial recognition context of “the cat” (the preceding text words) will mean that the weightings applied will prefer recognition for “sat” as the first word of the utterance.  
         [0057]     There are multiple ways that the recognizer configuration is dynamically changed to adjust to both the current text that is being read, and the current user. The language model  64  is sentence-based and is switched dynamically  186  each time the user enters a new sentence. The “initial recognition context” is based on the precise point in the text where the current utterance was started. In addition, the “pronunciation correctness slider” can control many aspects of the relative weighting of recognition items, as well as the content of the language model, and this setting can be changed either by the user or by the teacher during operation. Weightings or other aspects of recognition configuration that can be controlled include the relative weighting of sequences including word competition models in the language model, the relative weighting of word sequences which are explicitly in the language model (represented in bigrams and trigrams) vs. sequences which are not, and the content of the language model. The content of the language model is chosen based on how competition models are generated, what word sequences are explicitly in the language model and how s/he are weighted relative to one another. The “pronunciation correctness slider” setting may also control the relative weighting of silence, noise, or phoneme filler sequences vs. other recognition items.  
         [0058]     In the current implementation, the language model includes the words in the current sentence and one or more words from the subsequent sentence (up to and including the first non-glue word in the subsequent sentence). The subsequent sentence words are included to help the tutor software  34  determine when the user has transitioned from the current sentence into the next sentence, especially in cases where the reader does not pause between sentences.  
         [0059]     Referring to  FIG. 10 , a set of word classifications or categories  190  is shown. The word categories can have different settings in the speech recognition and tutor software  34 . The settings can be used to focus on particular words or sets of words in a passage. Word categories  190  include target words  192 , glue words  194 , and other words  196 . Words in a passage or story are segmented into one or more of these categories or other word categories according to his or her type as described below. Based on the category, the acoustic match confidence score may be used to determine the color coding of the word and whether the word is placed on a review list. For example, if the passage is focusing on a particular set of words to expand the student&#39;s vocabulary, a higher acoustic confidence match score may be required for the words in the set.  
         [0060]     Glue words  194  include common words that are expected to be known by the student or reader at a particular level. The glue words  194  can include prepositions, articles, pronouns, helping verbs, conjunctions, and other standard/common words. A list of common glue words  194  is shown in  FIG. 11 . Since the glue words  194  are expected to be very familiar to the student, the tutor software and speech recognition engine may not require a strict acoustic match confidence on the glue words  194 . In some examples, the software may not require any recognition for the glue words  194 . The relaxed or lenient treatment of glue words  194  allows the reader to focus on the passage and not be penalized or interrupted by an intervention if a glue word is read quickly, indistinctly, or skipped entirely.  
         [0061]     Target words  192  also can be treated differently than other words in the passage. Target words  192  are the words that add content to the story or are the new vocabulary for a passage. Since the target words are key words in the passage, the acoustic match confidence required for the target words  192  can be greater than for non-target words. Also, the word competition models may be constructed or weighted differently for target words. In addition, the target words  192  may be further divided into multiple sub-classifications, each sub-classification requiring different treatment by the speech recognizer and the tutoring software.  
         [0062]     Additional word categories may also be defined, such as a category consisting of words which the user has mastered based on the user&#39;s past reading history. For example, the time gap measurement may not be used to color code words or place words on the review list if the words are in the mastered word category. Instead, if the time gap measurement for the mastered word exceeds a threshold, it will be used as an indication that the user struggled with a different word in the sentence or with the overall interpretation of the sentence.  
         [0063]     Words in a text can be assigned to a word category based on word lists. For example, words can be assigned to the glue word category if the are on a list such as the common glue word list ( FIG. 11 ), assigned to the mastered word category if s/he are on a list of words already mastered by that user, and assigned to a target word category if s/he are in a glossary of new vocabulary for a passage. However, to be more effective, word categorization can also take into account additional factors such as the importance of a word to the meaning of a particular sentence, the lesson focus, and the reading level of the user and of the text. Therefore a word may be assigned to a particular category (e.g. the glue word category) in one sentence or instance, and the same word may assigned to a different category in another sentence or instance, even within the same text.  
         [0064]     Referring to  FIG. 12 , a process  200  related to the progression of a reader through a story is shown. For the location of the user within the story, the speech recognition software determines  202  the word category for the next or subsequent word in the passage. The speech recognition software determines  204  if the word is a target word.  
         [0065]     The speech recognition software  32  receives  208  audio from the user and generates a recognition sequence corresponding to the audio. If a valid recognition for an expected word is not received, the software will follow the intervention processes outlined above, unless the word is a glue word. If the word is a glue word, a valid recognition may not be required for the word. In this example, the speech recognition software receives  210  audio input including the expected glue word or a subsequent word and proceeds  216  to a subsequent word.  
         [0066]     If a valid recognition for the expected word is received, and the word is not a glue word, the tutor software analyzes additional information obtained from the speech recognition sequence. The software measures  222  and  224  if there was a time gap exceeding a predetermined length prior to or surrounding the expected word. If there is such a time gap, the word is placed  220  on a review list and coded a color to indicate that it was not read fluently. Typically this color is a different color from that used for ‘correct’ words (e.g. green), and also different from the color used to code words that have received an audio intervention (e.g. red). In addition, if the word is a target word, the software analyzes the acoustic match confidence  214  that has been generated for the word. The acoustic match confidence is used to determine if the audio received from the user matches the expected input (as represented by the acoustic model for that word) closely enough to be considered as a correct pronunciation. The speech recognition software determines  218  if the acoustic match confidence for the particular target word is above a predefined level. If the match confidence is not above the level, the word is placed on a review list  220  and coded a color to indicate that it was not read correctly or fluently. After determining the coding of the word, the tutor software  34  proceeds  226  to the subsequent word.  
         [0067]     While in the above example, only target words were evaluated using acoustic match confidence, other words in the glue word category or other word category could also be evaluated using acoustic match confidence. The implementation of word categories may include additional different treatment of words and may include more or fewer word categories  190 . In addition, the treatment of different categories of words can be controlled dynamically at the time the software is run. As described above, the tutor software  34  generates a list of review words based on the student&#39;s reading of the passage. A word may also be placed on the review list for reasons not directly related to the student&#39;s reading of the passage, for example if the student requested a definition of the word from the tutor software, the word could be placed on the review list. The review list can include one or more classifications of words on the review list and words can be placed onto the review list for multiple reasons. The review list can be beneficial to the student or to an administrator or teacher for providing feedback related to the level of fluency and specific difficulties for a particular passage. The review list can be used in addition to other fluency assessment indications such as number of total interventions per passage or words per minute. In some embodiments, the list of review words can be color-coded (or distinguished using another visual indication such as a table) based on the reason the word was included in the review list. For example, words can be included in the review list if an acoustic match confidence for the word was below a set value or if the user struggled to say the word (e.g., there was a long pause prior to the word). Words can also be placed on the review list if the user received a full audio intervention for the word (e.g., if the tutor software did not receive a valid recognition for the word in a set time, or the user requested an audio intervention for that word). Words that have been included on the review list due an audio intervention can be color coded in a one color while words placed on the review list based on the analysis of a valid recognition for the word (either time gaps associated with the word, or acoustic match confidence measurements) can be color coded in a second color.  
         [0068]     Referring to  FIG. 13 , in addition to color coding words on a review list, the words can also be color coded directly in the passage as the student is reading the passage. For example, in passage  323  shown on screenshot  230  the word  234  ‘huge’ is coded in a different manner than the word  236  ‘wolf.’ The first color-coding on word  234  is related to a pause exhibited in the audio input between the word ‘what’ and the word ‘huge’. The second color-coding on word  236  is related to the user receiving an audio intervention for the word  236 . Both words  234  and  236  would also be included on a list of review words for the user.  
         [0069]     While the language models and sentence tracking have been described above based on a sentence, other division points within a passage could be used. For example, the language models and sentence-by-sentence tracking could be applied to sentence fragments as well as to complete sentences. For example, s/he could use phrases or lines as the “sentence.” For example, line-by-line type sentence-by-sentence tracking can be useful to promote fluency in poetry reading. In addition, tracking sentences by clauses or phrases can allow long sentences to be divided and understood in more manageable linguistic units by the user. In some embodiments, single words may be used as the unit of tracking. Furthermore, the unit of tracking and visual feedback need not be the same as the unit of text used for creating the language models. For example, the language models could be based on a complete sentence whereas the tracking could be phrase-by-phrase or word-by-word.  
         [0070]     A number of embodiments of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, the system can provide support to people who are learning to read a second language. The system can support people who are learning to read in a language other than English, whether as a first or second language. The system can have a built-in dictionary that will explain a word&#39;s meaning as it is used in the text. The built-in dictionary can provide information about a word&#39;s meaning and usage in more than one language including, for example, the language of the text and the primary language of the user. Accordingly, other embodiments are within the scope of the following claims.

Technology Classification (CPC): 6