Patent ID: 12260011

DETAILED DESCRIPTION

The present disclosure relates to machine learning systems and methods for sensory augmentation using gaze tracking and emotional prediction techniques, as discussed in detail below in connection withFIGS.1-5.

FIG.1is a diagram illustrating the system10of the present disclosure. Generally, the system10monitors a reader's progress through a given text using gaze tracking systems to generate one or more sensory augmentation routines using emotional prediction techniques to accompany the text as a reader progresses therethrough, elevating the reader's experience, enhancing reading comprehension, and increasing reading speed. As described in greater detail herein, sensory augmentation routines can include, for example, ambient music, contextual sounds, contextual lighting effects, and the like, which are selected to compliment a predicted emotional timbre of the text being read by a reader. As shown inFIG.1, the system10includes a display device12, an image capture device or vision system14, and an emotion processing unit (“EPU”)16. The system10can also include, or be in communication with, other systems such as a remote server/“cloud”-based computing platform18, an audio output device20(e.g., headphones, a stereo system, etc.), a multimedia system22(e.g., an A/V system or home theatre), and one or more 3rdparty smart devices and/or services24(e.g., voice-enabled speakers, smart lighting systems, connected home appliances, streaming music and video services, etc.), and combinations thereof.

The display device12can be a smartphone, a tablet computer, a laptop computer, a desktop computer, or any other device capable of displaying the text to the reader. It should be understood that the image capture device14can be any image or video capture device or vision system with sufficient resolution and/or processing power to capture movements of the reader's eyes such that gaze tracking can be performed. Additionally, the image capture device14need not be a standalone device or system and can be integrated with the display device12, for example, but not limited to, a front-facing camera of a smartphone of tablet computer. As described in greater detail in connection withFIG.2, the EPU can be a cloud-based service, a computer, a processor, an application specific integrated circuit (“ASIC”) device, or a combination thereof, and is configured to receive information from one or more of the display device and the image capture device and generate the sensory augmentation routines using emotional prediction techniques. For example, the EPU can be an emotion processing unit described by U.S. Pat. No. 10,424,318, the entire disclosure of which is hereby expressly incorporated by reference. Importantly, the EPU (as disclosed in the '318 patent) generates its own emotional state (e.g., an emotional state of the EPU itself) based on inputs, which is used by the system10to predict a future emotional state of a person.

The devices of the system10can communicate with each other over a network26, which could include, but is not limited to, the Internet. Of course, as known to one of ordinary skill in the art, the network26can provide for communication between the devices of the system10using one or more of wired (e.g., RS485, ethernet, USB, serial, etc.), wireless (e.g., Wi-Fi, Bluetooth, ZigBee, Z-Wave, cellular, thread, etc.), and direct communication protocols and combinations thereof. While the foregoing discussion references network26, it shall be understood that the present system can be a self-contained system that does not include network connectivity or cloud communication capabilities. For example, in such a system, the display device12the image capture device14, and the EPU16could be included in a single device, or could be directly connected to one or more of the remaining devices of the system10by way of a serial connection or any other suitable direct communication protocols.

FIG.2is a diagram illustrating additional hardware and software components capable of being utilized to implement the system10of the present disclosure. As shown, the EPU16could be in communication with one or more databases40(e.g., music databases, user information databases, etc.), which could be stored, for example, in server/cloud platform18, and could receive external data42from one or more devices or systems, such as, but not limited to the display device12and the image capture device14. The EPU16executes system code44to generate the one or more sensory augmentation routines to accompany text as a reader progresses therethrough using gaze tracking and emotional prediction algorithms. As discussed above, the EPU could include, but is not limited to, a personal computer, a laptop computer, a tablet computer, a smart telephone, a server, and/or a cloud-based computing platform.

The system code44(i.e., non-transitory, computer-readable instructions) can be stored on a computer-readable medium and is executable by the EPU, or in combination with one or more additional computer systems. The system code44could include various custom-written software modules that carry out the steps/processes discussed herein, and could include, but is not limited to, a text recognition module46, a gaze tracking module48, an emotion determination module50, a sensory augmentation routing generation module52, and a communications module54. The system code44could be programmed using any suitable programming language including, but not limited to, C, C++, C#, Java, Python, or any other suitable language. Additionally, the system code44could be distributed across multiple computer systems in communication with each other over a communications network, and/or stored and executed on a cloud computing platform and remotely accessed by a computer system in communication with the cloud platform. The system code44could communicate with the one or more databases40and the external data sources42, which could be located within the same computer system as the system code44, or on one or more other computer systems in communication with the system code44.

Still further, the EPU16could be embodied as a customized hardware component such as a field-programmable gate array (“FPGA”), application-specific integrated circuit (“ASIC”), embedded system, or other customized hardware component without departing from the spirit or scope of the present disclosure. It should be understood thatFIG.2is only one potential configuration, and the system10of the present disclosure can be implemented using a number of different configurations.

FIG.3is a flowchart illustrating the overall process steps100carried out by the system10of the present disclosure for generating sensory augmentation based on a reader's predicted emotional response to a given portion of text. In step102, the system10determines the reader's current location in the text. As should be apparent to those of skill in the art, the system10can receive information from the display device12related to the text currently displayed on the display device, can receive information from the image capture device14related to the reader's eye movements and positions (e.g., gaze tracking), and can then determine the portion of the text currently being read. In step104, the system10identifies a subsequent location in the text (e.g., based on a current reading rate of X words per minute). In step106, the system10processes the text at the subsequent location (e.g., using optical character recognition, or the like, if necessary) and predicts an emotional response that the reader will have in response to reading the text at the subsequent location. For example, the EPU16can utilize artificial intelligence (“AI”) and/or machine learning algorithms to predict an emotional response based on a given word, sentence, or passage. As such, the system10can anticipate the feelings that a reader will have in advance of reaching the subsequent location. In step108, the system10generates a sensory augmentation routine based on the predicted emotional response. The sensory augmentation routine can include, for example, music or lighting selected to match the predicted emotional response. For example, if the system10predicts that a given passage will induce a feeling of happiness, the system can generate a sensory augmentation routine that includes ambient music that is upbeat and seamlessly adjusts to amplify the feeling of happiness. According to another example, if a main character of a story is in peril, the system10can generate a sensory augmentation routine having an atmosphere of low, sustained musical tension that can automatically rise to highlight a thrilling passage. According to another example, the system10can generate a sensory augmentation routine that includes an ambient lighting scene that matches the predicted emotional response (e.g., bright warm white ambient light for feelings of happiness and dark red ambient light for feelings of tension). In step110, the system10stores the generated sensory augmentation routine (e.g., to database40) for subsequent retrieval and playback. In step112, the system10determines if the reader has reached the subsequent location in the text.

If a positive determination is made in step112, the process proceeds to step114, where the system10initiates playback of the stored sensory augmentation routine. For example, the system10can initiate playback of a sensory augmentation routine including ambient music or sounds on the audio output device20, or can negotiate with a 3rdparty streaming music service (e.g., Spotify, Apple Music, etc.) to begin playback. In another example, the system10can initiate playback of a sensory augmentation routine including ambient lighting effects by instructing a 3rdparty smart lighting system (e.g., Philips Hue) to initiate a lighting scene with the ambient lighting effects. If a negative determination is made in step112, the process returns to step102, where the process steps are repeated until the system10determines that the reader has reached the subsequent location in the text. In step116, the system10determines if the reader has arrived at the end of the text (e.g., using gaze tracking). If a negative determination is made in step116, the process returns to step102and the process steps are repeated. If a positive determination is made in step116, the process ends.

FIG.4is a flowchart illustrating the overall process steps200carried out by the system10of the present disclosure for generating a sensory augmentation routine based on contextual indicators identified in a given portion of text. It should be understood that process steps100and200are not mutually exclusive and can be executed by the system10concurrently. In step202, the system10determines the reader's current location in the text. As discussed above, the system10can receive information from the display device12related to the text currently displayed on the display device, can receive information from the image capture device14related to the reader's eye movements and positions (e.g., gaze tracking), and can then determine the portion of the text currently being read. In step204, the system10identifies a subsequent location in the text (e.g., based on a current reading rate of X words per minute). In step206, the system10processes the text at the subsequent location (e.g., using optical character recognition, or the like, if necessary) and identifies one or more contextual indicators at a subsequent location. As such, the EPU16can utilize artificial intelligence (“AI”) and/or machine learning algorithms to identify a given word, sentence, or passage that conveys a particular sound, ambience, or context. For example, the system10could identify contextual indicators, such as, but not limited to a “crash,” “thunder and lightning,” a description of a sunset, and the like. According to another example, the system10could identify a particular composition of music that is being played within the context of a story. In step208, the system10generates a sensory augmentation routine based on the identified contextual indicators. The sensory augmentation routine can include, for example, music or lighting selected to match the identified contextual indicators. For example, if the system10identifies the phrase “car crash,” the sensory augmentation routine can include an audio recording of a car crash. Likewise, if the system10identifies the phrases “raining,” or “thundering,” the sensory augmentation routine can include an audio recording of a rainstorm or thunderstorm, respectively. Furthermore, if the system10identifies the phrases “raining,” or “thundering,” the sensory augmentation routine can include darkened ambient lighting effects and/or brief flashes of bright white lighting to convey darkened skies and/or lightening, respectively. According to another example, if the system10identifies a passage that describes a sunset, the system10can generate a sensory augmentation routine that includes ambient lighting effects (e.g., colors) based on the description of the sunset. In yet another example, if the system10identifies a passage where a particular composition of music is being played, the sensory augmentation routine can include the same composition of music.

In step210, the system10stores the generated sensory augmentation routine (e.g., to database40) for subsequent retrieval and playback. In step212, the system10determines if the reader has reached the subsequent location in the text. If a positive determination is made in step212, the process proceeds to step214, where the system10initiates playback of the stored sensory augmentation routine. For example, the system10can initiate playback of a sensory augmentation routine including ambient sounds or music on the audio output device20, or can negotiate with a 3rdparty streaming music service (e.g., Spotify, Apple Music, etc.) to begin playback. In another example, the system10can initiate playback of a sensory augmentation routine including ambient lighting effects by instructing a 3rdparty smart lighting system (e.g., Philips Hue) to initiate a lighting scene with the ambient lighting effects. If a negative determination is made in step212, the process returns to step202, where the process steps are repeated until the system10determines that the reader has reached the subsequent location in the text. In step216, the system10determines if the reader has arrived at the end of the text (e.g., using gaze tracking). If a negative determination is made in step216, the process returns to step202and the process steps are repeated. If a positive determination is made in step216, the process ends.

FIG.5is a flowchart illustrating the overall process steps300carried out by the system10of the present disclosure for generating a sensory augmentation routine based on the reading speed, or “flow,” of a reader. It should be understood that process steps300are not mutually exclusive with process steps100and200and can be executed by the system10concurrently therewith. In step302, the system10determines if the reader is currently reading, e.g., via gaze tracking systems, as described herein. If a negative determination is made in step302, the process proceeds to step304, where the system10stops playback of any active sensory augmentation routines and the process then returns to step302. As such, the system10is continuously monitoring the reader to determine if the text is actively being read. If a positive determination is made in step302, the process proceeds to step306, where the system10determines the current reading speed of the reader. As should be apparent to those of skill in the art, the system10can receive information from the display device12related to the text currently displayed on the display device, can receive information from the image capture device14related to the reader's eye movements and positions (e.g., gaze tracking), and can then determine the reader's current reading speed. In step308, the system10determines if the current reading speed is above a predetermined threshold (e.g., 250 words per minute). According to some embodiments, the system10can continuously adjust the predetermined threshold as the reader's speed and comprehension increases. For example, if the reader's speed is determined to be above the threshold (e.g., 250 words per minute), the system10can increase the reading speed threshold (e.g., to 255 words per minute) in a future reading session. Conversely, if the reader's speed is determined to be below the threshold (e.g., 250 words per minute), the system10can decrease the reading speed threshold (e.g., to 245 words per minute) in a future reading session. If a positive determination is made in step308, the process proceeds to step310, where the system10generates a sensory augmentation routine based on an enhanced reading speed. For example, the system10can generate a sensory augmentation routine in accordance with the process100, described in connection withFIG.3, and can increase the tempo, or beats per minute, thereof. If a negative determination is made in step308, the process proceeds to step312, where the system10generates a sensory augmentation routine based on a reduced reading speed. For example, the system10can generate a sensory augmentation routine in accordance with the process100, and can decrease the tempo, or beats per minute, thereof. The process then proceeds to step314, where the system10initiates playback of the sensory augmentation routine. For example, the system10can initiate playback of a sensory augmentation routine including ambient sounds or music on the audio output device20, or can negotiate with a 3rdparty streaming music service (e.g., Spotify, Apple Music, etc.) to begin playback. In another example, the system10can initiate playback of a sensory augmentation routine including ambient lighting effects by instructing a 3rdparty smart lighting system (e.g., Philips Hue) to initiate a lighting scene with the ambient lighting effects.

Accordingly, the system10of the present disclosure can adjust the speed of the music, sound, lighting, or other sensory augmentation routine component to create a flow depending on the speed of reading. The system10can also detect how fluently a reader reads, and can generate sensory augmentation routines that initiate playback of certain musical compositions (e.g., upbeat or high-tempo tracks) only when a particular reading speed level is achieved. As such, the system of the present disclosure can create a resonance between the emotion in the text, the flow or speed of a reader, and sound, lighting, or other sensory augmentation components. Furthermore, sensory augmentation routines including music, sound, lighting effects, and the like create harmonies that link the cognition of the text and the emotional flow of the reader. This enables a reader to master his or her reading fluency rate, encourages the reader to reach a flow state, and enhances reading comprehension. Additionally, flow has been shown to lead to improved performance. For example, researchers have found that flow can enhance performance in a wide variety of areas, including, but not limited to, teaching, learning, athletics, and artistic creativity. Flow can also lead to advanced learning and skill development. Because achieving flow indicates proficiency with a certain skill (e.g., reading speed and comprehension), the reader must continually seek new challenges and information in order to maintain this state, which the system of the present disclosure can provide by continuously encouraging the reader to improve their reading speed and comprehension (e.g., by allowing access to particular music, sounds, and lighting effects only when higher levels of reading speed and comprehension are achieved).

It is noted that the systems/methods disclosed herein can also be utilized to generate other sensory augmentation routines/outputs. For example, the system could be configured to provide a neural link to a person's brain, and based on the person's predicted emotional state, conduct deep brain stimulation using the neural link. In such circumstances, instead of playing music in response to the person's predicted emotional state (e.g., from reading text), the system can induce a particular mood or sensory effect in the person using deep brain stimulation. Further, the system could be extended for use with augmented reality (AR), such that the system generates an AR scene or environment for a person in response to the person's predicted emotional state. Still further, the system can be applied to other types of inputs (beyond text), such as movies, music, images, sounds, sensor inputs, and other inputs. Additionally, the system could track where a user is looking within an image or a video, and based on the identified location within the image or the video, provide a sensory augmentation tailored to information obtained from the identified location. For example, if a user is looking at a knife, or a victim, or a killer in a video or image, a specific sensory augmentation (e.g., fear, dread, terror, and associated music and/or other sensory input) could be generated by the system.

Having thus described the system and method in detail, it is to be understood that the foregoing description is not intended to limit the spirit or scope thereof. It will be understood that the embodiments of the present disclosure described herein are merely exemplary and that a person skilled in the art can make any variations and modification without departing from the spirit and scope of the disclosure. All such variations and modifications, including those discussed above, are intended to be included within the scope of the disclosure.