Patent Description:
Reference is also made to the following prior art document <CIT>.

Some examples involve multiple participants connected together over a network for a virtual interaction, such as a remote meeting. A remote meeting as used herein is intended to refer to an interaction between at least two participants where not all of the participants are located at the same physical location (i.e., at least one of the participants is remotely located). The participants of a remote meeting may use a portable or non-portable computing device, such as, but not limited to, a personal computer, a desktop computer, a laptop computer, a notebook computer, a network computer, a personal digital assistant (PDA), a mobile device, a hand-held device, or any other suitable computing device. Some examples involve at least one presenter and a multiple number of participants connected together over a network, such as, the Internet. It may be noted that the presenter is a "participant" in the context of a remote meeting of this nature, where he or she is interacting with other "participants".

Some examples are directed to applying gaze detection to determine participant attention information in a remote meeting, and providing feedback to a presenter of the remote meeting. Gaze angle information of meeting participants is determined using a gaze-sensitive interface (e.g., webcam) associated with a computing device of each participant. Some examples rely on the images obtained from consumer level webcams and deal with unconstrained environments, which may include different head poses, variable illumination, as well as other factors. Some examples extract information from these images, such as detecting facial landmarks, head rotation, eye location, and head angle. Eye information is determined and input to a convolutional neural network (CNN) to extract features, which are used as input for a machine learning prediction module to determine the eye gaze angle. Based on the eye gaze angle information, regions of interest in the content being presented are identified and provided as feedback to the presenter.

<FIG> is a diagram illustrating a remote meeting system <NUM> according to one example. The system <NUM> involves multiple remote participants <NUM>(<NUM>)-<NUM>(<NUM>) (collectively referred to as remote participants <NUM>) with associated remote computing devices <NUM>(<NUM>)-<NUM>(<NUM>), respectively, and a presenter participant <NUM> with an associated presenter computing device <NUM>. The computing devices <NUM> and <NUM> are communicatively coupled to each other via network <NUM>. Each of the computing devices <NUM>/<NUM> includes a remote meeting application (e.g., Lync, Skype, Webex, Google Hangouts), and generates video and audio streams during the remote meeting, which are sent to the network <NUM>, and then provided to each of the other computing devices <NUM> and <NUM>.

Computing devices <NUM> and <NUM> may include a personal computer, a desktop computer, a personal digital assistant (PDA), a mobile device, a hand-held device, or other type of computing device. The network <NUM> may be a wired network, a wireless network, or a combination of wired and wireless networks. In some examples, network <NUM> is a computer network, which may include a private network, such as an intranet, or a public network, such as the Internet. System <NUM> may also be implemented using a cloud computing architecture.

The presenter participant <NUM> communicates with the remote participants <NUM> over network <NUM> for a virtual interaction (e.g., a remote meeting). Presenter participant <NUM> uses presenter computing device <NUM> to transmit presentation content <NUM> (e.g., slides, text, images, video, etc.) to network <NUM>. Remote participants <NUM>(<NUM>)-<NUM>(<NUM>) use remote computing devices <NUM>(<NUM>)-<NUM>(<NUM>), respectively, to receive the transmitted presentation content <NUM> from the network <NUM> and display the received presentation content <NUM>. In some examples, remote computing devices <NUM>(<NUM>)-<NUM>(<NUM>) determine eye gaze angle based feedback information <NUM> for their associated remote participants <NUM>(<NUM>)-<NUM>(<NUM>), respectively, and transmit that information <NUM> to presenter computing device <NUM> via network <NUM> during the remote meeting.

<FIG> is a block diagram illustrating one example of a remote computing device <NUM> for the remote meeting system <NUM> shown in <FIG>. Remote computing device <NUM> includes at least one processor <NUM>, a memory <NUM>, input devices <NUM>, output devices <NUM>, display <NUM>, and camera <NUM>. Processor <NUM>, memory <NUM>, input devices <NUM>, output devices <NUM>, display <NUM>, and camera <NUM> are communicatively coupled to each other through communication link <NUM>. The camera <NUM> can be embedded within the frame of the display <NUM>, mounted along at least one of the edges of the display <NUM>, or mounted in a suitable location in the room in which the display <NUM> is located.

Input devices <NUM> include a keyboard, mouse, data ports, and/or other suitable devices for inputting information into device <NUM>. Output devices <NUM> include speakers, data ports, and/or other suitable devices for outputting information from device <NUM>.

Processor <NUM> includes a Central Processing Unit (CPU) or another suitable processor. In one example, memory <NUM> stores machine readable instructions executed by processor <NUM> for operating device <NUM>. Memory <NUM> includes any suitable combination of volatile and/or nonvolatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory. These are examples of non-transitory computer readable storage media. The memory <NUM> is non-transitory in the sense that it does not encompass a transitory signal but instead is made up of at least one memory component to store machine executable instructions for performing techniques described herein.

Memory <NUM> stores remote meeting application <NUM>, eye gaze angle estimation module <NUM>, and eye gaze angle processing and feedback generation module <NUM>. Processor <NUM> executes instructions of remote meeting application <NUM>, eye gaze angle estimation module <NUM>, and eye gaze angle processing and feedback generation module <NUM> to perform the techniques described herein. It is noted that some or all of the functionality of remote meeting application <NUM>, eye gaze angle estimation module <NUM>, and eye gaze angle processing and feedback generation module <NUM> may be implemented using cloud computing resources.

Remote meeting module <NUM> allows the user of remote computing device <NUM> to participate in a remote meeting, and view the presentation content <NUM> (<FIG>) for the remote meeting on display <NUM>. During the remote meeting, camera <NUM> captures images of the user of computing device <NUM>, which are provided to eye gaze angle estimation module <NUM>. Based on the captured images of the user, the eye gaze angle estimation module <NUM> continually estimates a current eye gaze angle of the user during the remote meeting. Eye gaze angle processing and feedback generation module <NUM> receives and processes the estimated eye gaze angle data generated by module <NUM>, and generates feedback information (e.g., eye gaze angle based feedback information <NUM>, shown in <FIG>) that is transmitted to the presenter computing device <NUM>.

<FIG> is a block diagram illustrating one example of a presenter computing device <NUM> for the remote meeting system <NUM> shown in <FIG>. Presenter computing device <NUM> includes at least one processor <NUM>, a memory <NUM>, input devices <NUM>, output devices <NUM>, display <NUM>, and camera <NUM>. Processor <NUM>, memory <NUM>, input devices <NUM>, output devices <NUM>, display <NUM>, and camera <NUM> are communicatively coupled to each other through communication link <NUM>. The camera <NUM> can be embedded within the frame of the display <NUM>, mounted along at least one of the edges of the display <NUM>, or mounted in a suitable location in the room in which the display <NUM> is located.

Processor <NUM> includes a Central Processing Unit (CPU) or another suitable processor. In one example, memory <NUM> stores machine readable instructions executed by processor <NUM> for operating device <NUM>. Memory <NUM> includes any suitable combination of volatile and/or nonvolatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory. These are examples of non-transitory computer readable media. The memory <NUM> is non-transitory in the sense that it does not encompass a transitory signal but instead is made up of at least one memory component to store machine executable instructions for performing techniques described herein.

Memory <NUM> stores remote meeting application <NUM>, feedback processing module <NUM>, and presentation content <NUM>. Processor <NUM> executes instructions of remote meeting application <NUM> and feedback processing module <NUM> to perform the techniques described herein. It is noted that some or all of the functionality of remote meeting application <NUM> and feedback processing module <NUM> may be implemented using cloud computing resources.

Remote meeting module <NUM> allows the user of remote computing device <NUM> to participate in a remote meeting, and present the presentation content <NUM> for the remote meeting to remote participants <NUM>. The presentation content <NUM> may be viewed by the presenter on display <NUM>. During the remote meeting, the presentation content <NUM> is presented to the remote participants <NUM>, and feedback processing module <NUM> processes the eye gaze angle based feedback information <NUM> received from the remote participants. In some examples, feedback processing module <NUM> provides indications on display <NUM> to identify regions of interest of the presentation content <NUM> based on the received feedback information <NUM>.

<FIG> is a flow diagram illustrating a method <NUM> of performing eye gaze angle estimation in a remote meeting according to one example. The following example is not according to the invention and is present for illustration purposes only. In one example, remote computing devices <NUM> (<FIG>) may perform method <NUM>. At <NUM> in method <NUM>, an image of a participant <NUM> in a remote meeting is captured. The image may be captured by camera <NUM> (<FIG>). At <NUM>, a head location of the participant <NUM> in the captured image is estimated by estimation module <NUM>. At <NUM>, a face in the captured image is detected by estimation module <NUM>. At <NUM>, left and right eyes are detected in the detected face by estimation module <NUM>. At <NUM>, a first set of features is extracted from the detected left eye by estimation module <NUM> using a first convolutional neural network trained with left eye information. At <NUM>, a second set of features is extracted from the detected right eye by estimation module <NUM> using a second convolutional neural network trained with right eye information. Features are data used by machine learning models to learn. At <NUM>, the extracted first set of features is used as an input to a first machine learning prediction module that is part of estimation module <NUM>, and which outputs a first estimated eye gaze angle value. At <NUM>, the extracted second set of features is used as an input to a second machine learning prediction module that is part of estimation module <NUM>, and which outputs a second estimated eye gaze angle value. At <NUM>, the first and second estimated eye gaze angle values are combined by estimation module <NUM> using a mean (average) of the two values to generate a final estimated eye gaze angle value. Using both eyes and combining their estimated gaze angles with a mean between them increases the accuracy of the output compared to a solution that uses a single eye. At <NUM>, feedback information is generated by module <NUM> based on the final estimated eye gaze angle value, and is provided to a presenter of the remote meeting.

Method <NUM> may be performed for each participant <NUM> in a remote meeting to determine a current eye gaze angle value for each such participant <NUM>. Method <NUM> may also be continually repeated to provide to presenter <NUM> continuous updates regarding the focus of attention of each participant <NUM>. Temporal attention information of overall participants <NUM> during the remote meeting is also generated and provided to presenter <NUM>.

In method <NUM> according to one example, the CNN is used to extract relevant features, and a further machine learning prediction module is used to output the estimated gaze angle. Some examples of method <NUM> use high performance convolutional-based networks such as "VGG" CNN architecture, which was developed by the Oxford University Visual Geometry Group and provides better feature extraction power than, for example, the AlexNet architecture. Other examples of method <NUM> may use other deep neural network architectures.

By tracking eye movements of the participants <NUM>, system <NUM> (<FIG>) can determine if an individual is facing away from the computing device <NUM> or has closed his or her eyes. System <NUM> can also determine if overall participants' eye movements have increased or decreased. In some examples, immediate feedback is provided to the presenter computing device <NUM> during the remote meeting and includes an identification of the part or parts of the presentation content <NUM> currently receiving the most attention from the participants <NUM>, which allows the presenter <NUM> to adapt the speech accordingly (e.g., increasing or slowing the pace of the presentation, or checking if anyone has questions, or changing other characteristics of the presentation). By knowing the specific focus of attention or where specific individuals (or groups of individuals) pay attention during a remote meeting, the presenter <NUM> can personalize content to maximize the attention of the specific audience the presenter <NUM> is targeting. The presenter <NUM> may also use this feedback after the remote meeting in order to adapt and enrich subsequent presentations.

System <NUM> may also evaluate the attention of participants <NUM> along the time axis of the presentation. A temporal series includes the flow of time as another layer of information, and allows the visualization of different information and patterns.

<FIG> is a diagram illustrating the providing of feedback to a presenter <NUM> in a remote meeting according to one example. Presentation content <NUM> is displayed on the presenter computing device <NUM> during a remote meeting. In the illustrated example, the presentation content <NUM> includes a plurality of slides <NUM>(<NUM>)-<NUM>(<NUM>) (collectively referred to as slides <NUM>) that are presented over time. Slide <NUM>(<NUM>) includes slide title <NUM>, text <NUM>, and image <NUM>. Slide <NUM>(<NUM>) includes slide title <NUM>, image <NUM>, and text <NUM>. Slide <NUM>(<NUM>) includes slide title <NUM> and text <NUM>.

During the presentation of the slides <NUM>, eye gaze angle based feedback information <NUM> for each participant <NUM> is received by presenter computing device <NUM>. In the illustrated example, based on the received eye gaze angle based feedback information <NUM>, presenter computing device <NUM> provides an indication on the displayed presentation content of the current focus of attention of each participant <NUM>. For example, indicator <NUM> may represent the focus of attention of a first participant <NUM>(<NUM>), and indicator <NUM> may represent the focus of attention of a second participant <NUM>(<NUM>). As shown by the indicator <NUM> in <FIG>, the first participant <NUM>(<NUM>) focused on the text <NUM> in slide <NUM>(<NUM>), the image <NUM> in slide <NUM>(<NUM>), and the text <NUM> in slide <NUM>(<NUM>). As shown by the indicator <NUM> in <FIG>, the second participant <NUM>(<NUM>) focused on the image <NUM> in slide <NUM>(<NUM>), the image <NUM> in slide <NUM>(<NUM>), and the slide title <NUM> in slide <NUM>(<NUM>).

In some examples, the positions of the indicators <NUM> and <NUM> are continually updated during the remote meeting based on the received eye gaze angle based feedback information <NUM> to provide the presenter <NUM> with immediate feedback regarding the current focus of attention of participants <NUM>.

By evaluating where most of the gazes were targeted at, the presenter <NUM> can prioritize certain segments of his or her material, change the location of certain images and reorganize text to minimize loss of attention, or to increase visibility of content deemed more relevant. The gaze information can be visualized by different metrics, including a heat map that informs through a color gradient which regions were of most interest to participants on a given moment of the presentation.

<FIG> is a diagram illustrating the display of presentation content <NUM> with heat map type indications <NUM> according to one example. The heat map type indications <NUM> are generated based on the eye gaze angle based feedback information <NUM>, and are overlaid on the presentation content <NUM> displayed on the presenter computing device <NUM>. The indications <NUM> identify specific regions of interest in the displayed presentation content <NUM>, as well as the intensity of participant interest. The varying intensities may be represented by different colors, and a corresponding map legend <NUM> may be displayed that includes colors that vary from the left end of the legend <NUM> (lowest intensity) to the right end of the legend <NUM> (greatest intensity).

System <NUM> may also evaluate which participants <NUM> had the best focus during the remote meeting or suffered the most interruptions while watching a given presentation. By combining the feedback information <NUM> from multiple participants <NUM>, the presenter <NUM> can identify the moment of the meeting where everyone was most focused or more dispersed.

<FIG> is a diagram illustrating a graph of eye movement intensity over time for two participants of a remote meeting according to one example. Graph <NUM> represents eye movement intensity for a first participant <NUM>(<NUM>), and graph <NUM> represents eye movement intensity for a second participant <NUM>(<NUM>). The vertical axis in graphs <NUM> and <NUM> represents eye movement intensity, and the horizontal axis in graphs <NUM> and <NUM> represents time. During time periods <NUM> and <NUM>, there is a relatively high level of eye movement intensity for both participants <NUM>(<NUM>) and <NUM>(<NUM>), which indicates that the specific content being displayed at this time may be a distraction candidate. In contrast, during time periods <NUM> and <NUM>, for example, the eye movement intensity for the participants <NUM>(<NUM>) and <NUM>(<NUM>) is lower, potentially indicating a higher level of attentiveness. Graph <NUM> also includes an indication <NUM> of an average level of eye movement intensity for the first participant <NUM>(<NUM>), and graph <NUM> includes an indication <NUM> of an average level of eye movement intensity for the second participant <NUM>(<NUM>).

<FIG> is a flow diagram illustrating a method <NUM> of providing feedback to a presenter in a remote meeting. At <NUM> in method <NUM>, images of remote participants in the remote meeting are captured using cameras associated with computing devices that display content presented by the presenter. At <NUM>, eye gaze angle information for at least one of the remote participants is determined based on the captured images. At <NUM>, at least one region of interest in the displayed content is identified based on the eye gaze angle information. At <NUM>, feedback is provided to the presenter including an indication of the identified at least one region of interest. The determining, identifying, and providing are performed by at least one processor.

In method <NUM>, determining eye gaze angle information includes detecting a left eye and a right eye of the at least one remote participant in the captured images, and extracting features from the detected left eye and the detected right eye using at least one convolutional neural network. The at least one convolutional neural network in method <NUM> may comprise a plurality of convolutional neural networks. The method <NUM> may include estimating, with a machine learning prediction module, at least one eye gaze angle value based on the extracted features. Extracting features in method <NUM> further includes extracting a first set of features from the detected left eye using a first convolutional network trained with left eye information, and extracting a second set of features from the detected right eye using a second convolutional network trained with right eye information. The method <NUM> further includes estimating, with a first machine learning prediction module, a first eye gaze angle value based on the extracted first set of features, and estimating, with a second machine learning prediction module, a second eye gaze angle value based on the extracted second set of features. The method <NUM> further includes calculating a mean of the first eye gaze angle value and the second eye gaze angle value to determine a final estimated eye gaze angle value. The method <NUM> may further include providing an indication to the presenter of the current focus of attention of each of the remote participants. The method <NUM> may further include generating a heat map type indication during the remote meeting that identifies to the presenter regions of the displayed presentation content that are of most interest to the remote participants. The method <NUM> may further include generating a graph of eye movement intensity over time for the at least one remote participant based on the eye gaze angle information.

Another example is directed to a system that includes a display to display content presented by a presenter of a remote meeting, and a camera to capture images of a remote participant in the remote meeting. The system further includes at least one processor to: determine eye gaze angle values for the remote participant based on the captured images; identify at least one region of interest in the displayed content based on the eye gaze angle values; and output feedback to the presenter during the remote meeting including an indication of the identified at least one region of interest.

The system may comprise a portable computing device, wherein the camera is integrated into the portable computing device. The at least one processor may detect an eye of the remote participant in the captured images, extract features from the detected eye using a convolutional neural network, and estimate, with a machine learning prediction module, an eye gaze angle value based on the extracted features.

Yet another example is directed to a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: generate a display of presentation content remotely provided by a presenter in a remote meeting; receive images of a remote participant in the remote meeting; determine eye gaze angle information for the remote participant based on the received images; identify at least one region of interest in the displayed presentation content based on the eye gaze angle information; and generate feedback information to be provided to the presenter including an indication of the identified at least one region of interest.

The non-transitory computer-readable storage medium may further store instructions that, when executed by the least one processor, cause the at least one processor to: detect an eye of the remote participant in the received images; extract features from the detected eye using a convolutional neural network; and estimate, with a machine learning prediction module, an eye gaze angle value based on the extracted features.

Some examples disclosed herein provide valuable instant personalized feedback information from each participant <NUM> to a presenter <NUM> in a remote meeting using low cost components (e.g., a webcam). Some examples may rely on ordinary laptop computer cameras, without additional hardware or environment setup, and provide a cost effective solution for gaze attention detection in remote meetings. Some examples disclosed herein do not use specialized hardware (e.g., virtual reality headsets or depth cameras) to perform attention tracking, and do not involve a special setup space, such as a minimum distance from the hardware, illumination conditions, etc. Such additional specialized hardware increases the cost of a solution, and can limit the portability and practicality of the solution, as well as possibly cause discomfort for participants. Some examples disclosed herein may involve participants that are at a variety of different geographic locations, as opposed to a solution that involves all users being physically present in the same meeting room and under the same illumination constraints and that restricts the number of participants based on the room size, capture device or by other factors.

Some examples use image processing convolutional neural network models that are efficient in automatically detecting relevant features for the classification/regression task of determining an eye gaze angle. Some examples capture each participant's eye gaze angle, and provide custom feedback, such as information indicating the attention of a specific participant during the remote meeting, or an identification of which slide (or sections of a specific slide) received more attention from the participants during a remote meeting.

Claim 1:
A method (<NUM>) of providing feedback to a presenter (<NUM>) in a remote meeting, comprising:
capturing images of remote participants (<NUM>) in the remote meeting using cameras associated with computing devices (<NUM>) that display content (<NUM>) presented by the presenter (<NUM>);
determining eye gaze angle information for at least one of the remote participants (<NUM>) based on the captured images;
identifying at least one region of interest in the displayed content (<NUM>) based on the eye gaze angle information;
providing feedback (<NUM>) to the presenter (<NUM>) including an indication of the identified at least one region of interest; and
wherein the determining, identifying, and providing are performed by at least one processor
wherein the at least one region of interest in the displayed content (<NUM>) based on the eye gaze angle information includes an identification of the part or parts of the presentation content (<NUM>) receiving the most attention from the participants (<NUM>),
wherein temporal attention information of overall participants (<NUM>) during the remote meeting is generated and provided to presenter (<NUM>),
wherein determining eye gaze angle information comprises:
detecting a left eye and a right eye of the at least one remote participant (<NUM>) in the captured images;
extracting features from the detected left eye and the detected right eye using at least one convolutional neural network;
wherein extracting features further comprises:
extracting a first set of features from the detected left eye using a first convolutional network trained with left eye information;
extracting a second set of features from the detected right eye using a second convolutional network trained with right eye information,
estimating, with a first machine learning prediction module, a first eye gaze angle value based on the extracted first set of features;
estimating, with a second machine learning prediction module, a second eye gaze angle value based on the extracted second set of features, and
calculating a mean of the first eye gaze angle value and the second eye gaze angle value to determine a final estimated eye gaze angle value.