Patent Publication Number: US-9851789-B2

Title: Information processing technique for eye gaze movements

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-071251, filed on Mar. 31, 2014, the entire contents of which are incorporated herein by reference. 
     FIELD 
     This invention relates to an information processing technique based on movement of eye positions on a display screen. 
     BACKGROUND 
     When an eye tracker (this is provided, for example, above or below a display screen) is a high-end model, it is possible to presume that a user in front of the display screen reads displayed sentences along them or gives a sentence area a once-over based on measured eye gaze data (i.e. data of the line of sight). 
     On the other hand, when a low-end eye tracker is used, an error included in the measured eye gaze data tends to become larger than that of the high-end model. For example, even when one eye position on a typical display screen of a personal computer is detected, a true eye position is located inside of an area that includes the one eye position and is about ⅙ of the display screen. 
     In such a case, even when the user reads a document along the sentences, it is not expected that coordinates of the eye position obtained from the measured eye gaze data are included in the sentence area. Moreover, it is difficult to accurately determine, according to the arrangement of the measured eye gaze data in time series, whether or not transition of eye gaze directions of a certain user is identical with transition of eye gaze directions of another user. 
     For example, if partial missing of a task to check a document or skip of reading of important portions is detected in a case where people check contents in the same document (for example, checking whether all required fields in a document are filled in or not, or whether all important points are read in a online educational document or not), it is possible to output an alarm to corresponding users. However, the low-end eye tracker cannot detect the aforementioned reading activities with high accuracy. 
     There is a technique to determine, based on the eye position data and transition speed of the eye position, whether an electronic document displayed on a display unit has been read or not. However, because the technique to determine whether a person read or not depends on the eye position, it is presupposed that the accuracy of the eye tracker is high. 
     Moreover, although there is a technique that presupposes that the eye tracker is the low-end model, it is determined based on the transition speed of the eye position and curvature obtained from temporal change of the eye positions, whether the user watches a certain area with his or her attention or not. In such a technique, there is no consideration to estimate the relationship between reading activity tendency the user of and that of other users. 
     Patent Document 1: Japanese Laid-open Patent Publication No. 2013-25656 
     Patent Document 2: Japanese Laid-open Patent Publication No. 05-56925 
     In other words, in conventional arts, there is no technique to enable to detect a user that does not perform typical movement of the eye gaze on a display screen, even when the eye tracker having low accuracy is used. 
     SUMMARY 
     An information processing method relating to this invention includes: (A) identifying, for each of plural users, transition of movement directions of an eye of the user, from movement history of the eye gaze of the user on a display screen; and (B) extracting a minor user among the plural users based on the transition of the movement directions of the eye gaze, which is identified for each of the plural users. 
     The object and advantages of the embodiment will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the embodiment, as claimed. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a diagram to explain detection results of the eye gaze by using an eye tracker having low accuracy; 
         FIG. 2  is a diagram to explain trends of eye movements; 
         FIG. 3  is a diagram to explain a substantial movement amount of the eye gaze; 
         FIG. 4  is a diagram to explain the substantial movement amount of the eye gaze; 
         FIG. 5  is a diagram depicting an outline of a system relating to an embodiment of this invention; 
         FIG. 6  is a functional block diagram of a main information processing apparatus in a first embodiment; 
         FIG. 7  is a diagram depicting a processing flow in the first embodiment; 
         FIG. 8  is a diagram depicting an example of data stored in a collected data storage unit; 
         FIG. 9  is a diagram depicting an example of data stored in the collected data storage unit; 
         FIG. 10  is a diagram depicting an example of data stored in the collected data storage unit; 
         FIG. 11  is a diagram depicting an example of data stored in the collected data storage unit; 
         FIG. 12  is a diagram depicting an example of data stored in the collected data storage unit; 
         FIG. 13  is a diagram depicting a processing flow of a processing for estimating a movement direction; 
         FIG. 14  is a diagram depicting an example of calculating the movement direction; 
         FIG. 15  is a diagram depicting an example of calculating the movement direction; 
         FIG. 16  is a diagram depicting an example of calculating the movement direction; 
         FIG. 17  is a diagram depicting an example of calculating the movement direction; 
         FIG. 18  is a diagram depicting an example of calculating the movement direction; 
         FIG. 19  is a diagram depicting an example of calculating the movement direction; 
         FIG. 20  is a diagram depicting an example of a calculated substantial movement amount; 
         FIG. 21  is a diagram depicting an example of labeling of the movement direction; 
         FIG. 22  is a diagram depicting an example of aggregating the labels of the movement directions; 
         FIG. 23  is a diagram depicting a processing flow in the first embodiment; 
         FIG. 24  is a diagram to explain a processing for determining the majority from the labels of the movement directions for plural users; 
         FIG. 25  is a diagram depicting an example of calculating a similarity in the first embodiment; 
         FIG. 26  is a functional block diagram of a main information processing apparatus in a second embodiment; 
         FIG. 27  is a diagram depicting a processing flow in the second embodiment; 
         FIG. 28  is a diagram depicting an example of an inclusive rectangle; 
         FIG. 29  is a diagram depicting another example of the inclusive rectangle; 
         FIG. 30  is a diagram depicting an example of processing results of a rectangle processing unit; 
         FIG. 31  is a diagram depicting a processing flow in the second embodiment; 
         FIG. 32  is a diagram depicting an example of calculation of similarities in the second embodiment; 
         FIG. 33  is a diagram to explain a case where a rectangle identified from the movement direction is considered; 
         FIG. 34  is a diagram depicting an example of adjustment of the substantial movement amount; 
         FIG. 35  is a diagram depicting an example of a rectangle for which a reliability is reflected; 
         FIG. 36  is a diagram depicting an example of a rectangle for which the reliability is reflected; 
         FIG. 37  is a diagram depicting an example of a rectangle for which the reliability is reflected; 
         FIG. 38  is a diagram depicting an example of a rectangle for which the reliability is reflected; 
         FIG. 39  is a diagram depicting an example of a rectangle for which the reliability is reflected; 
         FIG. 40  is a diagram depicting an example of a rectangle for which the reliability is reflected; 
         FIG. 41  is a diagram depicting an example of an inclusive rectangle in other embodiments; 
         FIG. 42  is a diagram depicting a processing flow of a rectangle setting processing in other embodiments; and 
         FIG. 43  is a functional block diagram of a computer. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     Embodiment 1 
     In this embodiment, transition of movement directions of the eye gaze is identified for each user. And a user is detected, whose transition of the movement directions of the eye gaze is different from the typical transition of the movement directions of the eye gaze. There is a possibility that such a user partially forgot the document check in a task to check whether all required fields in a document are filled in or skipped reading of important parts in online education or the like. Therefore, by outputting an alarm to such a user or an administrator, it is possible to prevent the partial omission of the document check or the skip of the reading of the important parts in online education. 
     However, as described above, when a low-end eye tracker is used, the measured eye position includes an error, and it is not preferable that the transition of the movement directions of the eye gaze is simply identified based on the measured eye positions. On the other hand, even when the low-end eye tracker is used, it is possible to identify a situation that the line of sight is moved as illustrated in (b) of  FIG. 1  from a state where the user gazed steadily in the vicinity of an area A in the display screen as illustrated in (a) of  FIG. 1 , and the state is changed to a state that the user gazes steadily in the vicinity of an area B as illustrated in (c) of  FIG. 1 . 
     In this embodiment, even in states of the steady gazes as illustrated in (a) and (c) of  FIG. 1 , trends of movement directions of the eye gaze can be extracted, and data representing the transition of the movement directions of the eye gaze is generated based on the trends of the movement directions of the eye gaze. 
     More specifically, as illustrated in  FIG. 2 , a state is assumed that 8 eye positions are detected, for example. Here, a first eye position and a second eye position are focused on. Then, when considering an error, the first eye position exists in a range of circle C, and when considering an error, the second eye position exists in a range of circle D, and when the eye position is moved from the first eye position to the second eye position, there is a possibility that the eye position moves to an area E in which the circle C overlaps with the circle D. This area E represents the trend of the movement directions, for which the error is considered into account, however, it is difficult to handle the area E as it is. Therefore, in this embodiment, the trend of the movement directions of the eye gaze is expressed by a method that will be explained later. 
     In other words, the substantial movement direction of the eye gaze is calculated based on not only an eye position to be focused on and a next eye position but also further subsequent eye positions. More specifically, a substantial movement amount is calculated by accumulating, for each direction of X-axis and Y-axis on the display screen, movement amounts from the eye position to be focused on until the reverse of the movement direction is detected within a predetermined duration of time. 
     The first eye position in  FIG. 2  is focused on, and it is determined whether or not a return of the eye gaze exists within a duration of time up to the n-th eye positions (n is a positive integer. For example, n=6). As for the X-axis direction, as illustrated in  FIG. 3 , the movement amount from the first eye position to the second eye position is dx_ 1 , and the movement amount from the second eye position to the third eye position is dx_ 2 . However, in the movement from the second eye position to the third eye position, the return of the eye gaze occurred. Then, the substantial movement amount movx_ 1  from the first eye position in the X-axis direction is represented by “dx_ 1 +dx_ 2 ”. 
     Similarly, as for the Y-axis direction, as illustrated in  FIG. 4 , the movement amount from the first eye position to the second eye position is dy_ 1 , the movement amount from the second eye position to the third eye position is dy_ 2 , and the movement amount from the third eye position to the fourth eye position is dy_ 3 . In the movement from the third eye position to the fourth eye position, a return of the eye gaze occurred. Then, the substantial movement amount movy_ 1  from the first eye position in the Y-axis direction is represented by “dy_ 1 +dy_ 2 +dy_ 3 ”. 
     By repeating such calculation for each eye position, the trend of the movement directions for each eye position is calculated. Such calculation is based on characteristics that the total sum of the movement amounts from respective eye positions within the predetermined duration of time approaches zero, although the coordinate values of the respective eye positions are not completely identical when a user watches one point on the display screen, because it is impossible to remove minute movements of the eye as the measurement results. 
     Then, by summarizing the trends of the movement directions for each predetermined time (hereinafter, referred to a time period), the transition of the movement directions of the eye gaze is determined. Then, a user who is minor among plural users is extracted based on the transition of the movement directions of the eye gaze. 
     Next, an outline of a system in this embodiment will be explained by using  FIG. 5 . 
     In the system in this embodiment, a main information processing apparatus  100  that performs a main processing in this embodiment, plural user-side information processing apparatus  300  (in this figure,  300   a  to  300   n ) and an administrator terminal  400  are connected via a network  200 . 
     The user-side information processing apparatus  300   a  includes an information display apparatus  301   a  and an information collection apparatus  302   a . The information display apparatus  301  displays data such as documents on a display device for a user, and manages data for the display history. The data such as documents may be delivered to the main information processing apparatus  100  or the like, and may be saved in the information display apparatus  301   a.    
     Furthermore, the information collection apparatus  302   a  has an eye tracker, and collects sensor data regarding the eye positions of the user on the display screen, and the like. For example, the information collection apparatus  302   a  may include a device such as a Web camera, and calculate the eye positions on the display screen from images photographed by the Web camera. As for detection techniques of the eye position, please refer to Stylianos Asteriadis et al., “Estimation of behavioral user state based on eye gaze and head pose-application in an e-learning environment”, Multimed Tools Appl, 2009 and the like. A processing to calculate eye position coordinates (in other words, coordinates of eye position) on the display screen from the sensor data may be performed in the main information processing apparatus  100  or may be performed in the information collection apparatus  302   a.    
     Moreover, the information collection apparatus  302   a  also collects history data of contents displayed on the display screen of the information display apparatus  301   a . The data collected in the information collection apparatus  302   a  is transmitted through the network  200  to the main information processing apparatus  100 . 
     The information display apparatus  301   a  and the information collection apparatus  302   a  may be integrated to form the user-side information processing apparatus  300   a . The user-side information processing apparatus  300   a  may be a mobile phone (including a smart phone), a tablet apparatus, a personal computer or the like. Furthermore, other user-side information processing apparatuses  300   b  to  300   n  has the same configuration as that of the user-side information processing apparatus  300   a.    
     The user-side information processing apparatus  300  may be prepared for each user or may be shared by plural users. 
     The administrator terminal  400  is a terminal apparatus that is operated by an administrator of a system, for example, and when data for a user who did not make the typical transition of the movement directions of the eye gaze is notified from the main information processing apparatus  100 , that data is displayed on a display device. 
     Next, a configuration example of the main information processing apparatus  100  is illustrated in  FIG. 6 . The main information processing apparatus  100  has a data collector  101 , a collected data storage unit  102 , a transition identifying unit  103 , a transition data storage unit  104 , an extraction unit  105 , an extracted data storage unit  106  and an output unit  107 . 
     The data collector  101  receives sensor data and display history data, which are transmitted from the user-side information processing apparatus  300 , and stores the received data in the collected data storage unit  102 . When the sensor data is not converted into the eye position coordinate data on the display screen, the data collector  101  also performs a data conversion processing (here, conversion means format conversion). 
     The transition identifying unit  103  has a movement direction estimating unit  1031 , and generates data representing the transition of the movement directions of the eye gaze, and stores the generated data in the transition data storage unit  104 . The movement direction estimating unit  1031  calculates a substantial movement amount explained by using  FIGS. 3 and 4 . 
     The extraction unit  105  has a majority determination unit  1051  and a minority extraction unit  1052 , and extracts data for a user who did not make the typical transition of the movement directions of the eye gaze, and stores the extracted data in the extracted data storage unit  106 . The majority determination unit  1051  determines the major transition of the movement directions of the eye gaze from the transitions of the movement directions of the eye gaze for plural users. Moreover, the minority extraction unit  1052  extracts a user who made the transition of the movement directions of the eye gaze, which is not similar to the major transition of the movement directions of the eye gaze. 
     The output unit  107  transmits data for the minor user, which is stored in the extracted data storage unit  106 , to the administrator terminal  400 , for example. Data may be outputted to other apparatus (including the user-side information processing apparatus  300 ). 
     Next, a processing in the main information processing apparatus  100  will be explained by using  FIGS. 7 to 25 . Firstly, the data collector  101  receives the sensor data and display history data from each of the user-side information processing apparatuses  300   a  to  300   n , converts the sensor data into the eye position coordinate data, and stores the converted data in the collected data storage unit  102  ( FIG. 7 : step S 1 ). 
     As described above, a document to be checked when a document checking task is performed or an online educational material in the course in case of the online education is displayed on the display screen of the information display apparatus  301 , for example. At this time, the information collection apparatus  302  simultaneously obtains sensor data from the eye tracker, further obtains data of the contents being displayed, and transmits the obtained data to the main information processing apparatus  100  in correlation with them each other. 
     The data collector  101  of the main information processing apparatus  100  receives the sensor data from the eye tracker and the data of the contents from the respective user-side information processing apparatus  300 , and stores the received data in the collected data storage unit  102 . 
       FIGS. 8 to 12  illustrate examples of the data of the contents, which are stored in the collected data storage unit  102 . Firstly,  FIG. 8  illustrates an example of the history data of window coordinates (in other words, x-y coordinates of a window). In the example of  FIG. 8 , a display time, a user ID (identifier), a screen ID, a window ID, x and y coordinates of the upper left of that window, a width and height of that window and a turn of that window in the depth direction of the screen (in other words, a turn of that window in overlapping windows) are included. Thus, it is possible to grasp what window is displayed on the display screen and which window is a foreground. 
     Furthermore,  FIG. 9  illustrates an example of relationships between the contents and the window. In an example of  FIG. 9 , a display time, a user ID, a screen ID, a window ID, a contents ID and a time section of the contents are included. According to such data, it is possible to grasp what contents are displayed on which window. The time section of the contents represents a position from the beginning of the contents, for example (e.g. time for watching, the number of pages or the like). 
       FIG. 10  illustrates an example of storage destination data of the contents. In an example of  FIG. 10 , a contents ID and an address of the storage destination (Uniform Resource Identifier (URI)) are included. This data may be stored in advance. 
     Thus, for each user, data that represents that each window displayed on the display screen is displayed in what state and which contents are displayed in that window is stored as the history data. 
     Even when the contents are displayed on a window, the window may be hidden by other windows, and the user may not watch the contents in the window to be watched. Therefore, the history data of the window coordinates ( FIG. 8 ) includes not only data representing the display position of the window and its display size but also the turn of that window in the depth direction of the screen. 
     Moreover, the eye position coordinate data obtained from the sensor data of the eye tracker is also stored in the collected data storage unit  102 .  FIG. 11  illustrates an example of the eye position coordinate data obtained as absolute coordinates on the display screen. In an example of  FIG. 11 , a measurement time, a user ID, a screen ID and x and y coordinate values are included. 
     On the other hand, when plural windows are displayed on the display screen, the eye position coordinate data within the window is calculated assuming that the window displayed foreground is watched considering the turns of the windows. In such a case, data as illustrated in  FIG. 12  is stored in the collected data storage unit  102 . In an example of  FIG. 12 , a measurement time, a user ID, a screen ID and x and y coordinate values are included. Relative coordinate values in the display screen are recorded. 
     In the following explanation, in order to simplify the explanation, an example that one window is displayed on the entire display screen, and the eye position coordinates are calculated as coordinates on the foreground window. 
     Returning to the explanation of the processing in  FIG. 7 , the transition identifying unit  103  determines whether or not the present time is a first timing for identifying the transition of the movement directions of the eye gaze (step S 3 ). For example, whether or not the present time is a predetermined timing counted by a timer, or whether or not the present time is timing when all data is received from users to be compared is determined. When the present time is not the aforementioned first timing, the processing shifts to a processing in  FIG. 23  through terminal A. 
     On the other hand, when the present time is the aforementioned first timing, the transition identifying unit  103  reads out the eye position coordinate data for the user to be processed in plural time periods to be processed from the collected data storage unit  102  (step S 5 ). 
     Then, the transition identifying unit  103  identifies one unprocessed user in the read out eye position coordinate data (step S 7 ). Then, the movement direction estimating unit  1031  performs a processing for estimating a movement direction for the identified user, and stores the processing result in the transition data storage unit  104  (step S 9 ). This processing for estimating a movement direction will be explained by using  FIGS. 13 to 19 . 
     Firstly, the movement direction estimating unit  1031  calculates, for each set of the eye position coordinates (except the last set of the eye position coordinates), difference values dx and dy with the eye position coordinates immediately after the eye position coordinates to be processed in the X-axis and the Y-axis directions ( FIG. 13 : step S 21 ). For example, as for the X-axis direction, a difference dx_ 1  between the first eye position coordinate and the second eye position coordinate, a difference dx_ 2  between the second eye position coordinate and the third eye position coordinate, a difference dx_ 3  between the third eye position coordinate and the fourth eye position coordinate, a difference dx_ 4  between the fourth eye position coordinate and the fifth eye position coordinate a difference dx_ 5  between the fifth eye position coordinate and the sixth eye position coordinate and a difference dx_ 6  between the sixth eye position coordinate and the seventh eye position coordinate are calculated. As for the Y-axis direction, the same calculation is performed. 
     Moreover, the movement direction estimating unit  1031  identifies one set of the eye position coordinates to be processed (here, k-th set of the eye position coordinates) (step S 23 ). 
     Then, the movement direction estimating unit  1031  initializes a variable r to “0” (step S 25 ). After that, the movement direction estimating unit  1031  adds r-th difference dx_(k+r) to the substantial movement amount movx_k for the eye position coordinates k to be processed (step S 27 ). 
     Then, the movement direction estimating unit  1031  determines whether or not the direction of dx_k is different from the direction of dx_(k+r), in other words, whether or not the direction of dx_(k+r) is reversed (i.e. the plus/minus sign of the dx_(k+r) is reversed) or whether or not r reached a predetermined value (which corresponds to a predetermined duration of time) (step S 29 ). 
     When conditions at this step S 29  are not satisfied, the movement direction estimating unit  1031  increments r by “1” (step S 31 ) in order to accumulatively add dx_(k+r+1) to movx_k, and the processing returns to the step S 27 . 
     On the other hand, when any condition at the step S 29  is satisfied, the substantial movement amount for the eye position coordinates to be processed this time is obtained for the X-axis direction. Then, the movement direction estimating unit  1031  initializes the variable r to “0” (step S 33 ). After that, the movement direction estimating unit  1031  adds the r-th difference dy_(k+r) to the substantial movement amount movy_k for the k-th set of the eye position coordinates to be processed (step S 35 ). 
     Then, the movement direction estimating unit  1031  determines whether or not the direction of dy_k is different from the direction of dy_(k+r), in other words, whether or not the direction of dy_(k+r) is reversed (i.e. the plus/minus sign of the dy_(k+r) is reversed) or whether or not r reached a predetermined value (which corresponds to a predetermined duration of time) (step S 37 ). 
     When conditions at this step S 37  are not satisfied, the movement direction estimating unit  1031  increments r by “1” (step S 39 ) in order to accumulatively add dy_(k+r+1) to movy_k, and the processing returns to the step S 35 . 
     On the other hand, when any condition at the step S 37  is satisfied, the substantial movement amount for the eye position coordinates to be processed this time is obtained for the Y-axis direction. Then, the movement direction estimating unit  1031  determines whether or not any eye position coordinates to be processed exist in the eye position coordinate data read out at the step S 5  (step S 41 ). 
     When there are eye position coordinates to be processed, the movement direction estimating unit  1031  increments k by “1” (step S 43 ), and the processing returns to the step S 23 . On the other hand, when there are no residual eye position coordinates, the processing returns to the calling-source processing. 
     For example, in the example of  FIG. 2 , when the upper limit value of r is “6”, movx and movy are obtained as illustrated in  FIGS. 14 to 19 .  FIG. 14  illustrates movx_ 1  and movy_ 1 .  FIG. 15  illustrates movx_ 2  and movy_ 2 .  FIG. 16  illustrates movx_ 3  and movy_ 3 .  FIG. 17  illustrates movx_ 4  and movy_ 4 .  FIG. 18  illustrates movx_ 5  and movy_ 5 .  FIG. 19  illustrates movx_ 6  and movy_ 6 . 
     Moreover, the processing results of the movement direction estimating unit  1031  are stored in the transition data storage unit  104 . For example, data as illustrated in  FIG. 20  is stored. In an example of  FIG. 20 , for each of the X-axis and Y-axis, the substantial movement amounts movx and movy are registered for each time from t=1 to t=6. The substantial movement amounts represent the directions of the X-axis and Y-axis by their plus/minus signs. As illustrated in  FIGS. 14 to 19 , in case of the X-axis direction, the positive substantial movement amount represents the right direction, and the negative substantial movement amount represents the left direction. In case of the Y-axis direction, the positive substantial movement amount represents the upward direction, and the negative substantial movement amount represents the downward direction. 
     Returning to the explanation of the processing in  FIG. 7 , the transition identifying unit  103  identifies a movement direction label of each time from the estimated movement direction calculated by the movement direction estimating unit  1031 , and stores the processing result in the transition data storage unit  104  (step S 11 ). 
     When the results as illustrated in  FIG. 20  are obtained, data as illustrated in  FIG. 21  is obtained. The right direction is represented by “R”, the left direction is represented by “L”, the upward direction is represented by “U”, and the downward direction is represented by “D”. 
     In this embodiment, when an absolute value of the substantial movement amount is equal to or less than 10, for example, it is assumed that the substantial movement amount is not detected as the error. Moreover, when the absolute value of the substantial movement amount is a value that exceeds the frame of the display screen (in the Y-axis direction, the value is equal to or greater than the height “Height”, and in the X-axis direction, the value is equal to or greater than the horizontal width “width”), it is also assumed that the substantial movement amount is not detected, because the user watches an area other than the display screen. 
     Then, in  FIG. 21 , as for t=2 and t=6 in the X-axis direction and t=5 in the Y-axis direction, the absolute values of the substantial movement amounts are equal to or less than 10. Therefore, they are excluded, and are represented by “-”. As for other lines, the labeling is performed according to the aforementioned rule. For convenience of explanation, no labeling is performed for t=7. 
     Furthermore, the transition identifying unit  103  aggregates, for each time period, the movement direction labels at each time, and stores the aggregation results in the transition data storage unit  104  (step S 13 ). For example, when the time period is set for each two seconds, it is determined whether or not there is a type of label whose occurrence frequency is equal to or greater than a threshold (e.g. 60% or more) among labels (e.g. 7 labels) included in two seconds. When it is determined that there is the aforementioned label, the label is identified as a label that is representative of that time period, and when it is determined that there is not the aforementioned label, “-”, which represents no label that is representative of that time period, is set. 
     For example, in the example of  FIG. 21 , as for the X-axis direction, a type of all of the labels is “R”. Therefore, R is selected. As for the Y-axis direction, a type of 4 labels is “D” among 5 labels, therefore, D is selected. By performing the aforementioned determination for each time period, data as illustrated in  FIG. 22  is obtained. In this example, the data as illustrated in  FIG. 21  is aggregated as transition data of the time period of 2 seconds from the time 0 to 2 seconds in  FIG. 22 . 
     Then, the transition identifying unit  103  determines whether or not there is an unprocessed user in the collected data storage unit  102  (step S 15 ). When there is an unprocessed user, the processing returns to the step S 7 . On the other hand, when there is no unprocessed user, the processing shifts to step S 51  in  FIG. 23  through terminal B. 
     Shifting to the step S 51  in  FIG. 23  through the terminal B, the extraction unit  105  determines whether or not the present time is a second timing for identifying the majority and minority (step S 51 ). For example, whether or not the present time is a predetermined timing measured by a timer or whether or not the present time is timing when data to be processed has been prepared is determined. When the present time is not the aforementioned second timing, the processing ends. 
     On the other hand, when the present time is the second timing, the extraction unit  105  reads out the eye transition data of plural users for a predetermined duration of time including plural time periods (step S 53 ). 
     Then, the majority determination unit  1051  of the extraction unit  105  determines, for each of the X-axis and Y-axis, the majority of the movement directions of the respective time period (step S 55 ). 
     The majority determination will be explained by using  FIG. 24 . In an example of  FIG. 24 , for each of the X-axis and Y-axis and for each of n users, the movement direction label for each time period is listed. Then, at this step, for each time period, the movement direction label, which is selected for the greatest number of users, is determined as the majority of the movement direction. For example, as for the time period from the time 0 to 2 seconds, the number of users for which “R” is selected is greater for the X-axis. Therefore, “R” is identified as the majority. As for the Y-axis, the number of users, for which “D” is selected, is equal to the number of users, for which “U” is selected. Therefore, it is impossible to identify the majority, and “-”, which represents no majority, is registered. When this processing is performed for each time period, the last line for the X-axis and the last line for the Y-axis are obtained. 
     After that, the minority extraction unit  1052  of the extraction unit  105  extracts one unprocessed user in the read out data (step S 57 ). Then, the minority extraction unit  1052  calculates a similarity with the majority for each time period and for the identified user (step S 59 ). 
     More specifically, for each time period, when the movement direction is identical with the movement direction of the majority, the similarity “1” is calculated, when the movement direction is not identical with the movement direction of the majority, the similarity “0” is calculated, and in case of other cases (e.g. either of the movement directions is “-”), the similarity “0.5” is calculated. In the example illustrated in  FIG. 24 , when this step is performed, the result as illustrated in  FIG. 25  is obtained for the first user “UserR1”. In this example, as for the X-axis, there is a lot of “1”, however, as for the Y-axis, “1” is not major. 
     At this step, the average value or total sum of the similarities for the X-axis and the average value or total sum of the similarities for the Y-axis may be calculated to employ them as an overall similarity. Furthermore, after the average value for each time period or the like is calculated, an overall similarity may be calculated by the average or total sum. 
     Then, the minority extraction unit  1052  determines whether or not there is an unprocessed user in the read out data (step S 61 ). When there is an unprocessed user, the processing returns to the step S 57 . On the other hand, when there is no unprocessed user, the minority extraction unit  1052  extracts users of the minority based on the calculated similarities, and stores data concerning the extracted users in the extracted data storage unit  106  (step S 63 ). For example, a user whose overall similarity is less than a threshold may be identified as the minority, or a user whose number of time periods of the similarity “1” for the X-axis or Y-axis is less than a predetermined number may be identified as the minority. 
     After that, the output unit  107  transmits data of the minor users, which is stored in the extracted data storage unit  106 , to the administrator terminal  400 , for example (step S 65 ). Accordingly, the administrator can identify users having any trouble. Alarm data may be transmitted to the user-side information processing apparatuses  300  of the minor users. 
     By repeating this processing periodically or in response to a request from the administrator or the like, users who made non-typical transition of the movement directions of the eye gaze can be identified. 
     According to this embodiment, even when the eye tracker with low accuracy is employed, the transition of the movement directions of the eye gaze is identified by extracting the movement directions of the eye gaze by a method that is appropriate for their characteristics, and it becomes possible to identify the majority and minority of the users. 
     Thus, it is possible to detect the partial omission of the document check task or the skip of the important parts in the online education, and suppress them. 
     Embodiment 2 
     In this embodiment, a method for extracting the minor users, which is different from that of the first embodiment, is employed. However, the outline of the system illustrated in  FIG. 5  is similar to that of this embodiment. 
       FIG. 26  illustrates a configuration example of a main information processing apparatus  100   b  relating to this embodiment. 
     The basic configuration of the main information processing apparatus  100   b  relating to this embodiment is similar to that of the first embodiment, and the configurations of the transition identifying unit  103   b  and the extraction unit  105   b  are different. 
     In other words, the transition identifying unit  103   b  has a rectangle processing unit  1032  in addition to the movement direction estimating unit  1031 . The rectangle processing unit  1032  performs a processing to numeralize the transition of the movement directions of the eye gaze for each time period. 
     Moreover, the extraction unit  105   b  has a similarity calculation unit  1053  and a minority extraction unit  1052   b . The similarity calculation unit  1053  calculates, for each user, a similarity between the user and another user, based on the processing results of the rectangle processing unit  1032 . 
     The minority extraction unit  1052   b  extracts the minor users based on the similarities calculated by the similarity calculation unit  1053 . 
     Next, processing contents of the main information processing apparatus  100   b  relating to this embodiment will be explained by using  FIGS. 27 to 32 . 
     Firstly, the data collector  101  receives sensor data, display history data and the like from the respective user-side information processing apparatuses  300   a  to  300   n , converts the sensor data into the eye position coordinate data, and stores the eye position coordinate data in the collected data storage unit  102  ( FIG. 27 : step S 101 ). This step is the same as the step S 1 . 
     Next, the transition identifying unit  103   b  determines whether or not the present time is a first timing for identifying the transition of the movement directions of the eye gaze (step S 103 ). This step is the same as the step S 3 . When the present time is not the aforementioned first timing, the processing shifts to a processing in  FIG. 31  through terminal C. 
     On the other hand, when the present time is the aforementioned first timing, the transition identifying unit  103   b  reads out the eye position coordinate data within plural time periods to be processed for the users to be processed from the collected data storage unit  102  (step S 105 ). This step is the same as the step S 5 . 
     Then, the transition identifying unit  103   b  identifies one unprocessed user in the read out eye position coordinate data (step S 107 ). This step is the same as the step S 7 . Then, the movement direction estimating unit  1031  performs the processing for estimating the movement direction for the identified user, and stores the processing results in the transition data storage unit  104  (step S 109 ). This processing for estimating the movement direction is the same as that illustrated in  FIG. 13 . 
     Then, the rectangle processing unit  1032  calculates a rectangular area of an inclusive rectangle that contains eye position coordinates and a movement direction at the eye position coordinates (the substantial movement amounts of the X-axis and Y-axis) and coordinate values of the central point of the inclusive rectangle (or either of them), and stores the calculated data in the transition data storage unit  104  (step S 111 ). 
     For example, in the example of  FIG. 2 , an inclusive rectangle F is identified as illustrated in  FIG. 28 . The central point of the inclusive rectangle F is a point f. Thus, in a state that the user gazes steadily on a certain area, the rectangle is almost identical to the inclusive rectangle of the eye position. However, in a time period when the eye moves largely as illustrated in (b) of  FIG. 1 , a shape of the inclusive rectangle is determined by the coordinates of the movement destination, which corresponds to the movement direction from the eye position as illustrated in  FIG. 29 . An outer frame in  FIG. 29  represents an outer shape of the display area in case of the screen illustrated in the same direction as that of (b) in  FIG. 1 . 
     Then, data as illustrated in  FIG. 30  is stored in the transition data storage unit  104 . In the example of  FIG. 30 , a rectangular area of the inclusive rectangle and the coordinates of the central point of the inclusive rectangle are stored for each user and each time period. 
     Then, the transition identifying unit  103   b  determines whether or not there is an unprocessed user in the data read out at the step S 105  (step S 113 ). When there is an unprocessed user, the processing returns to the step S 107 . On the other hand, when there is no unprocessed user, the processing shifts to a processing in  FIG. 31  through terminal D. 
     Shifting to the explanation of the processing in  FIG. 31 , the extraction unit  105   b  determines whether or not the present time is a second timing for identifying the majority and minority (step S 115 ). This step is the same as the step S 51 . When the present time is not the aforementioned second timing, the processing ends. 
     On the other hand, when the present time is the second timing, the extraction unit  105   b  reads out the eye transition data for a predetermined duration of time including plural time period for plural users (step S 117 ). 
     Then, the similarity calculation unit  1053  identifies one unprocessed user in the data read out at the step S 117  (step S 119 ). Moreover, the similarity calculation unit  1053  calculates, for each time period, a similarity with each of other users (step S 121 ). 
     For example, the similarity of the rectangular area of the inclusive rectangle is calculated as follows:
 
1−|(rectangular area of first user)−(rectangular area of second user)|/(reference area for normalization)
 
     The first user corresponds to the user identified at the step S 119 . As for the reference area for the normalization, the same value may be employed for all users, or the rectangular area of the first user may be employed. 
     Furthermore, the similarity for the central point of the inclusive rectangle is calculated by {1−(normalized distance between central points)}. The normalized distance between the central points is calculated, for example, as follows:
 
{(( x  coordinate value of first user)−( x  coordinate value of second user)) 2 +(( y  coordinate value of first user)−( y  coordinate value of second user)) 2 } 0.5 /(reference distance for normalization)
 
     The reference distance for the normalization is a diagonal distance of the screen, for example. In case of a screen having 1600*1200 pixels, the diagonal distance is 2000 pixels. 
     For example,  FIG. 32  illustrates an example that the similarity with each of other users for the time period 0-2 seconds is calculated, in the example of  FIG. 30 , for each user and for each of the rectangular area and the central point of the inclusive rectangle. As illustrated in  FIG. 32 , not only the individual similarities with each of other users but also the similarities with all other users are calculated. The similarity with all other users may be calculated as an average value of the similarities with each of other users. Moreover, the similarities for the rectangular area of the inclusive rectangle and the similarities for the central point of the inclusive rectangle are averaged to calculate the similarity with all other users. 
     Then, data as illustrated in  FIG. 32  is generated for each time period. 
     Moreover, the similarity calculation unit  1053  calculates an overall similarity with other users (step S 123 ). More specifically, the similarities with all other users, which are calculated for respective time periods, may be averaged with respect to plural users, or the minimum value may be employed among the similarities with all other users, which are calculated for respective users. Furthermore, the overall similarity may be calculated for each of the rectangular area of the inclusive rectangle and the central point of the inclusive rectangle, or one overall similarity calculated, for example, by using the average value of them may be calculated. 
     Then, the similarity calculation unit  1053  determines whether or not there is an unprocessed user in the data read out at the step S 117  (step S 125 ). When there is an unprocessed user, the processing returns to the step S 119 . 
     On the other hand, when there is no unprocessed user, the minority extraction unit  1052   b  extracts users whose overall similarity is less than a threshold as the minority, and stores the extraction results in the extracted data storage unit  106  (step S 127 ). For example, when the overall similarity for the rectangular area of the inclusive rectangle is less than a first threshold, or when the overall similarity for the central point is less than a second threshold, the user may be extracted as the minor user. Furthermore, in case where an integrated similarity of the overall similarities for the rectangular area and central point is calculated, when the integrated similarity is less than a third threshold, the user may be extracted as the minor user. 
     Furthermore, the major users may be extracted. 
     After that, the output unit  107  transmits data of the minor users, which is stored in the extracted data storage unit  106 , to the administrator terminal  400 , for example (step S 129 ). Hence, the administrator can identify problematic users. Alarm data may be transmitted to the user-side information processing apparatuses  300  of the minor users. 
     By repeating the processing periodically or in response to a request from the administrator or the like, it becomes possible to identify users who made the non-typical transition of the movement directions of the eye gaze. 
     By performing the aforementioned processing, even when using an eye tracker with low accuracy, the transition of the movement directions of the eye gaze is identified by extracting the movement directions of the eye gaze using a method corresponding to that characteristic, and the majority and the minority are identified based on estimation by similarities. 
     Other Embodiments 
     In the second embodiment, the inclusive rectangle is defined so as to contain the eye position coordinates in the time period to be focused on and the movement direction at the eye position coordinates (the substantial movement amounts for the X-axis and Y-axis directions). However, other definition may be employed. 
     For example, the inclusive rectangle is defined so as to contain the eye position coordinates and a rectangle identified by the substantial movement amount movx in the X-axis direction and the substantial movement amount movy in the Y-axis direction. This is because such a rectangle is handled as a movement range of the eye gaze. 
     As schematically illustrated in  FIG. 33 , a rectangle G that has a horizontal width movx_ 1  and a vertical width movy_ 1  is identified. Such a rectangle is generated for each eye position in the time period to be focused on, and the inclusive rectangle that contains those rectangles is identified. 
     Moreover, in case where movx_ 1  and movy_ 1  are obtained as illustrated in  FIG. 20 , the absolute value of the substantial movement amount is equal to or less than 10, it is assumed that the substantial movement amount is not detected as described above. Moreover, when the absolute value of the substantial movement amount is a value that represents the eye moves over the frame of the display screen (in the Y-axis direction, the value is equal to or greater than the height “Height” and in the X-axis direction, the value is equal to or greater than the horizontal width “width”), it is assumed that the user watches an area other than the display screen. Therefore, in this case, the substantial movement amount is excluded. 
     That is not all. When the movement that is equal to or longer than a detection error rad (a radius of a circle in  FIG. 2 ) of the eye gaze occurs, it is possible to determine the apparent movement of the eye gaze. Therefore, the processing to identify the transition of the movement direction of the eye gaze, which is performed in the embodiments, may be redundant in view of identifying the movement direction. Therefore, when a value that is equal to or greater than rad (e.g.  300 ) is calculated as movx or movy, rad may be employed. 
     Then, the results in  FIG. 20  are changed to those in  FIG. 34 . The inclusive rectangle may be identified as described in the second embodiment by employing those in  FIG. 34  or the inclusive rectangle may be identified by employing the aforementioned new definition. 
     Furthermore, the reliability of the time period may be calculated based on the number of movx and movy, which are employed in that processing. 
     In  FIG. 34 , 7 values are calculated for only 6 time intervals among time intervals, which are identified by the 8 eye positions included in the time 0-2 seconds. However, in the X-axis direction, 4 points are employed among 6 points, and in the Y-axis direction, 5 points are employed among 6 points. Therefore, the reliability of the X-axis direction is ⅔, and the reliability of the Y-axis direction is ⅚. The reliabilities of the X-axis and Y-axis may be averaged. 
     The reliability may be reflected to the size of a rectangle identified by movx and movy. For example, a rectangle to which the reliability is reflected is arranged at the center of the rectangle identified by movx and movy. 
     In the example of  FIG. 34 , in case where the reliability is ½ for convenience, a rectangle H 1  as illustrated in  FIG. 35  is obtained for a first eye position. Moreover, as illustrated in  FIG. 36 , as for a second eye position, because movx is excluded in the X-axis direction, the rectangle H 2  that is a substantial segment is obtained. As for a third eye position, a rectangle H 3  as illustrated in  FIG. 37  is obtained. Moreover, as for a fourth eye position, as illustrated in  FIG. 38 , original movx and movy are shortened, and by reflecting the reliability to the shortened movx and movy, a rectangle H 4  is obtained. As for a fifth eye position, as illustrated in  FIG. 39 , movy is excluded in the Y-axis direction. Therefore, a rectangle H 5  that is a substantial segment is obtained. Similarly, as illustrated in  FIG. 40 , as for a sixth eye position, movx is excluded in the X-axis direction. Therefore, a rectangle H 6  that is a substantial segment is obtained. 
     Then, an inclusive rectangle J illustrated in  FIG. 41  is obtained. In this example, the same results as that of  FIG. 28  is obtained, however, the same results are not obtained for all cases. 
     A setting processing of the rectangle using the reliability is summarized in  FIG. 42 . This processing is executed by the rectangle processing unit  1032  as a portion of the step S 111 . 
     For example, the rectangle processing unit  1032  identifies one unprocessed time period (step S 201 ). Then, the rectangle processing unit  1032  excludes substantial movement amounts that satisfy an exclusion condition among the substantial movement amounts calculated for respective eye positions included in the identified time period, and shortens the substantial movement amounts that is equal to or greater than the threshold (the aforementioned rad) to the threshold (step S 203 ). 
     The rectangle processing unit  1032  calculates a reliability by a ratio of the number of remaining substantial movement amounts to the number of substantial movement amounts (step S 205 ). Furthermore, the rectangle processing unit  1032  adjusts the substantial movement amount according to the reliability (step S 207 ). Then, the rectangle processing unit  1032  locates a rectangle corresponding to the adjusted substantial movement amount on the center of the rectangle identified by the substantial movement amount before the adjustment (step S 209 ). The location of the rectangle is a mere example, and the rectangle may be located according to the eye position. 
     After that, the rectangle processing unit  1032  determines whether or not there is an unprocessed time period (step S 211 ). When there is an unprocessed time period, the processing returns to the step S 201 . On the other hand, when there is no unprocessed time period, the processing returns to the calling-source processing. 
     The respective rectangles and inclusive rectangles may be used to identify a visual area and change of the visual areas as an area in which the possibility that the eye exists is high, instead of one parameter to extract the minor users. Moreover, by recording a flow of the eye gaze of the user who belongs to the majority and minority in time series and by reproducing the record later, the visual area and the change of the visual areas are used for logging or a log confirmation method of states for the document check task and/or the online education. 
     Although the embodiments of this invention were explained, this invention is not limited to those. For example, the configuration examples of the main information processing apparatuses  100  and  100   b  do not correspond to a program module configuration. Moreover, as for the processing flows, as long as the processing results do not change, the turns of the processing may be exchanged or plural steps may be executed in parallel. 
     Furthermore, the functions of the main information processing apparatus  100  or  100   b  may be shared by plural computers instead of one computer. As described above, the functions of the user-side information processing apparatus  300  and the main information processing apparatus  100  or  100   b  may be shared. 
     In addition, the aforementioned main information processing apparatuses  100  and  100   b  are computer devices as shown in  FIG. 43 . That is, a memory  2501  (storage device), a CPU  2503  (processor), a hard disk drive (HDD)  2505 , a display controller  2507  connected to a display device  2509 , a drive device  2513  for a removable disk  2511 , an input unit  2515 , and a communication controller  2517  for connection with a network are connected through a bus  2519  as shown in  FIG. 43 . An operating system (OS) and an application program for carrying out the foregoing processing in the embodiment, are stored in the HDD  2505 , and when executed by the CPU  2503 , they are read out from the HDD  2505  to the memory  2501 . As the need arises, the CPU  2503  controls the display controller  2507 , the communication controller  2517 , and the drive device  2513 , and causes them to perform necessary operations. Besides, intermediate processing data is stored in the memory  2501 , and if necessary, it is stored in the HDD  2505 . In this embodiment of this technique, the application program to realize the aforementioned functions is stored in the computer-readable, non-transitory removable disk  2511  and distributed, and then it is installed into the HDD  2505  from the drive device  2513 . It may be installed into the HDD  2505  via the network such as the Internet and the communication controller  2517 . In the computer as stated above, the hardware such as the CPU  2503  and the memory  2501 , the OS and the necessary application programs systematically cooperate with each other, so that various functions as described above in details are realized. 
     The aforementioned embodiments are outlined as follows: 
     An information processing apparatus relating to this embodiment includes: (A) an identifying unit to identify, for each of plural users, transition of movement directions of an eye gaze of the user, from movement history of the eye gaze of the user on a display screen; and (B) an extraction unit to extract a minor user among the plural users based on the transition of the movement directions of the eye gaze, which is identified for each of the plural users. 
     Accordingly, when a user who partially forgets a document check task or skips reading of important portions in an online education is a minor user, it is possible to extract that minor user, and further alert and/or notify the minor user. 
     Moreover, the aforementioned extraction unit may (b1) compare, for each time period, the transition of the movement directions of the eye gaze among the plural users to identify transition of movement directions of the eye gaze, which is presumed to be a majority of the plural users; and (b2) extract a user whose transition of the movement directions of the eye gaze is dissimilar to the identified transition of the movement directions of the eye gaze. By identifying a reference transition of movement directions of the eye gaze for each time period, the minority is easily identified. 
     Furthermore, the aforementioned extraction unit may (b3) calculate, for each of the plural user, a similarity by comparing the transition of the movement directions of the eye gaze of the user with transition of movement directions of eye gaze of other users; and (b4) determine, for each of the plural user and based on the similarity calculated for the user, whether the user is a minor user. By calculating the similarity for other users as a numerical value, it is possible to extract the minor user. 
     Furthermore, the aforementioned identifying unit may calculate, for each eye position, a substantial movement amount within a predetermined duration of time from the eye position. On the other hand, the aforementioned identifying unit may calculate, for each eye position, an accumulated movement amount from the eye position to an eye position for which transition of a direction is detected within a predetermined duration of time, for each axis direction on the display screen. When a processing based on a characteristic indicator value while considering a sensor with low accuracy is performed, it becomes possible to perform preferable determination. 
     Moreover, the aforementioned identifying unit may calculate, for each time period, at least either of an area of a rectangle that is identified from each eye position and each accumulated movement amount within the time period and central coordinates of the rectangle to identify the transition of the movement directions of the eye gaze. Such a rectangle characterizes the trend of the eye gaze movement. 
     In addition, the aforementioned rectangle may be a rectangle that includes a second rectangle, which has an edge that is set according to the accumulated movement amount in each axis direction, and each eye position. 
     Furthermore, the aforementioned identifying unit may (a1) determine, for each time period, whether the accumulated movement amount of each eye position for the time period is within a predetermined range; (a2) when it is determined that the accumulated movement amount is within the predetermined range, calculate a reliability by dividing a number of eye gaze positions that was used for calculation of the accumulated movement amount by a number of eye gaze positions during the time period; and (a3) locate the second rectangle that has an edge whose length is obtained by multiplying the reliability by the accumulated movement amount that is within the predetermined range. Thus, while considering the reliability representing the effectiveness of the accumulated movement amount, the similarity can be calculated. 
     Incidentally, it is possible to create a program causing a computer or processor to execute the aforementioned processing, and such a program is stored in a computer readable storage medium or storage device such as a flexible disk, CD-ROM, DVD-ROM, magneto-optic disk, a semiconductor memory such as ROM (Read Only Memory), and hard disk. In addition, the intermediate processing result is temporarily stored in a storage device such as a main memory or the like. 
     All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present inventions have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.