Patent Description:
Due to recent developments in communication technology, videoconferencing systems that connect remote locations by video and audio have been proposed. Specifically, for example, a display apparatus, a camera, a microphone, and a speaker are provided in each location, whereby video and audio acquired at one location are transmitted to the other location in real time, and output from the display apparatus and the speaker.

In a face-to-face conversation such as videoconferencing, if the lines of sight of the viewer and the partner on the display screen are not aligned, a sense of presence and the like is lost, and the efficiency of the conference or the like falls. For this reason, display apparatus with integrated cameras that achieve line-of-sight alignment have been variously proposed. For example, Patent Literature <NUM> below describes a videoconferencing system with line-of-sight alignment that installs a camera behind the screen using a half-mirror or the like. Also, Patent Literature <NUM> below describes a method of securing a camera in the center of a display screen.

However, with the technology described in Patent Literature <NUM> above, there is a problem with the configuration in that the system as a whole becomes bulky. Also, with the technology described in Patent Literature <NUM> above, there is a flaw in that the camera overlaps the screen.

Accordingly, the present disclosure proposes an information processing apparatus, an information processing method, and a program making it possible to grasp the partner's line of sight more naturally in video communication that connects remote locations.

According to the present disclosure as described above, it becomes possible to grasp the partner's line of sight more naturally in video communication that connects remote locations.

Hereinafter, the description will proceed in the following order.

An overview of an information processing system <NUM> according to an embodiment of the present disclosure will be described with reference to <FIG>. The information processing system <NUM> according to the present embodiment achieves video communication that connects remote locations by having information processing apparatus 10A and 10B in remote locations connect through a network <NUM>, and transmit and receive video and audio, and in addition, shared content and the like.

Herein, as illustrated in <FIG>, video of each remote location is captured by a camera <NUM>, but due to the influence of positional misalignment or the like between the camera <NUM> and a display <NUM>, a problem arises in which the users' lines of sight become misaligned even though the users are looking at each other appearing on the displays. The user B appearing on the display <NUM> of <FIG> actually is looking at the user A, but because of positional misalignment between the display and the camera on user B's side, the user B's line of sight appears to be looking away from the user A. Also, the user A is looking at the user B appearing on the display <NUM>, but because of positional misalignment between the display <NUM> and the camera <NUM>, the user A's line of sight appears to be looking away from the user B.

Accordingly, an information processing system according to the present embodiment makes it possible to grasp the partner's line of sight more naturally in video communication that connects remote locations. Specifically, user line-of-sight detection is performed on each information processing apparatus <NUM>, it is determined whether or not the user is looking at the partner user, and the results of such determinations are synchronized between each information processing apparatus <NUM>. Additionally, in the case in which the partner user displayed on the display is looking at the user, each information processing apparatus <NUM> executes image processing such that the partner user's line of sight appears to point towards the user. With this arrangement, natural communication like the line-of-sight alignment in the same space can be achieved in remote communication.

Next, the configuration of the information processing apparatus <NUM> according to the present embodiment will be described with reference to <FIG> is a block diagram illustrating an example of the configuration of the information processing apparatus <NUM> according to the present embodiment. As illustrated in <FIG>, the information processing apparatus <NUM> includes a control section <NUM>, an input section <NUM>, an operation interface (I/F) <NUM>, an output section <NUM>, a communication section <NUM>, and a storage section <NUM>.

The control section <NUM> functions as a computational processing apparatus and control apparatus, and controls overall operation inside the information processing apparatus <NUM> by following various programs. The control section <NUM> is realized by an electronic circuit such as a central processing unit (CPU) or a microprocessor, for example. In addition, the control section <NUM> may also include read-only memory (ROM) that stores information such as programs to use and computational parameters, as well as random access memory (RAM) that temporarily stores information such as parameters that change as appropriate. A specific functional configuration of the control section <NUM> will be described later with reference to <FIG>.

The input section <NUM> includes a function of inputting spatial information. For example, the input section <NUM> is realized by a camera <NUM>, a microphone (hereinafter designated the mic) <NUM>, and a line-of-sight sensor <NUM>. The camera <NUM> is provided on the periphery of a display <NUM>, for example, captures an image in the forward direction of the display <NUM>, and acquires a captured image (also designated "video" in this specification). The mic <NUM> is provided on the periphery of the display <NUM>, for example, and acquires nearby sound. Note that the mic <NUM> may also be a mic array that includes multiple microphones. In addition, the mic <NUM> may also be provided near the user.

The line-of-sight sensor <NUM> is provided on the periphery of the display <NUM>, for example, and tracks the line of sight of the user looking at the display <NUM> (also known as a line-of-sight tracker). For example, the line-of-sight sensor <NUM> detects the line of sight with respect to the display <NUM> from the direction of the user's head and face, the eyeballs, and the position of the pupils. Also, for example, the line-of-sight sensor <NUM> includes multiple infrared LEDs, and is capable of radiating infrared rays towards the user's eyeballs, and on the basis of the positional relationship between the position of the reflection point (light point) on the eyeballs and the pupil center points, computing the line-of-sight direction of the user. Note that the line-of-sight sensor <NUM> is not limited to a device provided on the periphery of the display <NUM>. For example, it is also possible to provide a sensor in a see-through eyeglasses-style apparatus worn by the user, and on the basis of changes in the positions of the user's eyeballs (specifically, the blacks of the eyes (including the iris portion and the pupil portion) or the like) measured from an up-close distance, detect the motion of a gaze point on the display (which may also be a display of the see-through eyeglasses-style apparatus) seen from that point of view. Additionally, it is also possible to provide a sensor in a head-mounted apparatus worn by the user, and on the basis of changes in the positions of the eyeballs measured similarly from an up-close distance, detect the motion of the gaze point on a display inside the head-mounted apparatus.

The operation I/F <NUM> receives operation input from the user, and outputs to the control section <NUM>. The operation I/F <NUM> may be realized by a keyboard, a mouse, a touch panel, buttons, switches, or the like. For example, the operation I/F <NUM> may be used to specify content to share with the partner user, perform operations on displayed content, and the like.

The output section <NUM> includes a function of presenting various information output from the control section <NUM> to the user. For example, the output section <NUM> includes the display <NUM>, a speaker <NUM>, and an indicator <NUM>. The display <NUM> displays video of the partner user (remote space) and shared content. The speaker <NUM> plays back audio of the partner user (remote space). Also, the indicator <NUM> is realized by LED illumination used as signaling, for example. For example, the indicator <NUM> is provided on the outer frame of the display <NUM>, and lights up in the case of notifying the user of something.

The communication section <NUM> connects to another information processing apparatus <NUM> through the network <NUM>, and transmits or receives data. For example, the communication section <NUM> transmits a captured image captured by the camera <NUM>, audio data collected by the mic <NUM>, and shared content data (such as reference materials for a conference) to the information processing apparatus <NUM> of a communication destination. Also, the communication section <NUM> receives video, audio data, content data, and the like from the information processing apparatus <NUM> of the communication destination.

The storage section <NUM> stores programs and parameters by which the control section <NUM> executes each function. Additionally, the storage section <NUM> according to the present embodiment may also store data transmitted or received via the communication section <NUM>.

Next, <FIG> will be referenced to describe a specific functional configuration of the control section <NUM>. <FIG> is a block diagram illustrating an example of a functional configuration of the control section <NUM> according to the present embodiment.

As illustrated in <FIG>, the control section <NUM> functions as a video/audio acquisition section 100a, an operation information acquisition section 100b, a content acquisition section 100c, a transmission control section 100d, a gaze position specification section 100e, a partner video analysis section 100f, a gaze target determination section <NUM>, a gaze information management section <NUM>, a partner video processing section 100i, a display information generation section 100j, and a gaze information presentation control section <NUM>.

The video/audio acquisition section 100a acquires a captured image (video) captured by the camera <NUM> and audio information collected by the mic <NUM>, and outputs to the transmission control section 100d. Note that the video/audio acquisition section 100a may also output to the transmission control section 100d after first performing various processing, such as noise processing and compression processing, on the acquired video and audio information.

The operation information acquisition section 100b acquires operation information input from the operation I/F <NUM>, and outputs to the transmission control section 100d.

The content acquisition section 100c acquires content information to share in the information processing system <NUM>, such as conference reference materials, and outputs to the transmission control section 100d.

The transmission control section 100d controls the transmission of video, audio data, operation information, content information, and the like to the information processing apparatus <NUM> of the communication destination from the communication section <NUM> through the network <NUM>.

The gaze position specification section 100e specifies a gaze position on the display (for example, coordinate information corresponding to the display screen), on the basis of the user's gaze detected by the line-of-sight sensor <NUM>.

The partner video analysis section 100f analyzes (image analysis, object recognition) video received from the information processing apparatus <NUM> of the communication destination via the communication section <NUM>. For example, the partner video analysis section 100f performs face recognition on a person (partner user) appearing in the video, and recognizes a region around the eyes.

The gaze target determination section <NUM> determines a gaze target of the user in the video display on the display <NUM>. For example, as illustrated in <FIG>, the gaze target determination section <NUM> compares and checks a gaze point P (specified by the gaze position specification section 100e) of the line of sight of the user A with respect to the display <NUM> to a region E (recognized by the partner video analysis section 100f) around the eyes of the user B appearing in the video displayed on the display <NUM>, and determines the gaze target of the user A. In the example illustrated in <FIG>, whether or not the user A is looking at the user B may be determined. Note that in the case in which content, such as conference reference materials, is being displayed on the display <NUM>, the gaze target determination section <NUM> determines which portion of the content the user A is gazing at.

The gaze information management section <NUM> manages the determination result by the gaze target determination section <NUM> as gaze information (management of line-of-sight status). The content of the determination result by the gaze target determination section <NUM> may be, for example, "the user A is looking at the user B", "the user A is looking at coordinates (x, y) of a materials image", or "the user A is not looking at the user B". Also, the gaze information management section <NUM> stores gaze information about the user A in the storage section <NUM>. Also, the gaze information management section <NUM> outputs gaze information about the user A to the transmission control section 100d, and synchronizes gaze information with the information processing apparatus <NUM> of the communication destination by the transmission control section 100d. Specifically, the transmission control section 100d transmits gaze information about the user A to the information processing apparatus <NUM> of the communication destination from the communication section <NUM> through the network. Note that the synchronization of user gaze information is performed in each of the information processing apparatus <NUM> with each other, and the gaze information management section <NUM> also manages gaze information about the partner user transmitted from the information processing apparatus <NUM> of the communication destination. With this arrangement, the gaze information management section <NUM> is able to grasp whether or not the partner user B is looking at the user A, for example. Furthermore, the gaze information management section <NUM> is able to determine whether or not the lines of sight of both users are aligned, on the basis of the gaze information about the user A and the user B.

The partner video processing section 100i executes predetermined image processing on the partner video, on the basis of the gaze information managed by the gaze information management section <NUM>. For example, in the case in which the gaze information indicates that the partner user is gazing at the user, the partner video processing section 100i executes image processing such that the line of sight of the partner user appearing the partner video appears to point towards the user. As described above, because misalignments in the camera position and the like cause the line of sight to become misaligned even though the partner user is looking at the user appearing on the display, in the case in which the gaze information indicates that the partner user's line of sight is pointing towards the user, the line of sight of the partner user appearing in the partner video is processed to appear to point towards the user. Specifically, for example, the positions of the blacks of the eyes, the positions of both eyes, the direction of the head (face), and the like of the partner user appearing in the partner video are processed optimally. The processing method is not particularly limited, but methods that use 2D images, 2D images based on a 3D model, a 3D model image, or the like are possible, for example. 3D model data of the partner user is generated and transmitted in the information processing apparatus <NUM> on the partner's side, for example. For example, in the case of using a 3D model image to process the direction of the partner user's head, the partner video processing section 100i acquires data of a face model of the partner user in real-time obtained from an RGB-D camera (a camera equipped with a distance sensor) provided in the information processing apparatus <NUM> on the partner's side. Next, the partner video processing section 100i is able to apply color after first rotating the face model in a predetermined direction (for example, forward), combine compositing base images (face parts of the partner user appearing in the partner video) and optimize seams, and generate images in which the direction of the partner user's face is altered in real-time.

In addition, by reciprocally performing such a line-of-sight processing process, natural communication in which the lines of sight of the user and the partner user are aligned can be achieved.

Also, in the case in which the partner user's line of sight is pointing towards the camera, if the captured partner video is displayed as-is on the display, the partner user's line of sight will appear to be pointing at oneself, and thus it is also possible for the partner video processing section 100i to execute processing to make the partner user's line of sight look away.

Additionally, the partner video processing section 100i may optimize a display system in coordination with a communication state. For example, a communication path between users whose lines of sight are aligned with each other on the basis of the gaze information may be determined to be a high priority. According to the claimed invention, coordinated display control comprising one of enlarging each other's video regions, increasing the image quality, or highlighting the display is executed. On the other hand, the video of a partner user of low priority, for whom the lines of sight of the partner side and the user are not looking at each other, may be displayed at low resolution or the like.

The display information generation section 100j generates information to display on the display <NUM>. For example, the display information generation section 100j generates a screen in which line-of-sight processed partner video generated by the partner video processing section 100i and shared content information is arranged. Also, in the case in which line-of-sight processing by the partner video processing section 100i is unnecessary, the display information generation section 100j may also generate a screen in which unmodified partner video is arranged.

The gaze information presentation control section <NUM> presents gaze information to the user on the basis of the gaze information managed by the gaze information management section <NUM>. For example, in the case in which the partner user in the video displayed on the display <NUM> is looking at the user, an indication of this is displayed on the display <NUM>, or the indicator <NUM> is made to light up.

The above specifically describes the functional configuration of the control section <NUM> according to the present embodiment. Note that the control section <NUM> according to the present embodiment is not limited to the optimization of a display system corresponding to line-of-sight alignment, and is also capable of optimizing a sound system in coordination with a communication state, for example. Specifically, by making speech clear with a loud volume between users prioritized according to line-of-sight alignment, the control section <NUM> makes it easier to hear the voice of a partner being gazed at. On the other hand, the voice of a partner of low priority may be made unclear with a quiet volume. With this arrangement, even smoother communication can be encouraged.

Next, operating processes according to the present embodiment will be described with reference to <FIG> is a sequence diagram illustrating operating processes according to the present embodiment. Herein, video communication of the user A and the user B executed between the information processing apparatus 10A and 10B will be described as an example.

As illustrated in <FIG>, first, the information processing apparatus 10A issues a video conversation connection request to the information processing apparatus 10B (step S103), obtains a connection approval from the information processing apparatus 10B (step S106), and communication between both parties is established (video conversation connection process). Note that herein, the connection request is issued from the information processing apparatus 10A, but obviously is not limited thereto, and the connection request may also be issued from the information processing apparatus 10B to the information processing apparatus 10A.

Next, the information processing apparatus 10A acquires video capturing the user A present in front of the display <NUM> (step S109), and transmits the video to the communication destination, that is, the information processing apparatus 10B (step S112). On the other hand, the information processing apparatus 10B similarly acquires video of the user B (step S115), and transmits the video to the information processing apparatus 10A (step S118).

Next, each information processing apparatus <NUM> arranges and displays the transmitted partner video on the screen (steps S119, S120). Such a video display process (steps S109 to S120) may be executed continually, and video communication between the user A and the user B may be performed in real-time. Note that herein, only "video" is illustrated, but audio data may also be transmitted/received and output together.

Next, the information processing apparatus 10A analyzes the partner video transmitted from the information processing apparatus 10B with the partner video analysis section 100f (step S121). Specifically, the information processing apparatus 10A executes image analysis, object recognition, and the like on the partner video, and recognizes the region around the eyes of the user B appearing in the partner video.

Next, the information processing apparatus 10A tracks the point of view (gaze point) of the user A on the display <NUM> with the line-of-sight sensor <NUM> (step S124). On the display <NUM>, the partner video transmitted from the information processing apparatus 10B is being displayed in real-time.

Next, the information processing apparatus 10A compares and checks the tracked point-of-view position of the user A to the region around the eyes of the user B appearing in the partner video, and determines whether or not the user A is gazing at the user B with the gaze target determination section <NUM> (step S127). Herein, a determination of whether or not the user A is looking at the user B (partner user) as a gaze target is described, but in the case in which content data such as conference reference materials is also arranged and displayed on the display <NUM>, the gaze target determination section <NUM> may also determine which part of the conference reference materials the user A is gazing at.

Next, the information processing apparatus 10A transmits gaze information to the information processing apparatus 10B (step S130).

This kind of gaze information synchronization process is also executed similarly in the information processing apparatus 10B (steps S136 to S145).

Next, <FIG> will be referenced to describe the continuation of the operating processes illustrated in <FIG>. <FIG> is a sequence diagram illustrating operating processes that follow the operating processes illustrated in <FIG>.

As illustrated in <FIG>, next, the information processing apparatus 10A determines whether or not the partner user (user B) is gazing at the user A, on the basis of the gaze information transmitted from the information processing apparatus 10B (step S151).

In the case of gazing (step S151/Yes), the partner video processing section 100i executes processing for gaze (step S154). For example, in the case in which the line of sight of the user B appearing in the partner video is misaligned as illustrated in <FIG>, the partner video processing section 100i is able to execute processing by performing image processing on the positions of the blacks of the eyes, the positions of both eyes, and the direction of the head (face) of the user B to point forward, thereby executing processing such that the line of sight of the user B appears to be pointing towards the user A facing the display. Additionally, the partner video processing section 100i may also execute a process that overlays a notification display such as "looking at user A" onto the partner video.

Next, the gaze information presentation control section <NUM> may notify the user A of the gaze state by causing the indicator to blink or the like (step S157).

On the other hand, in the case of not gazing (step S151/No), the partner video processing section 100i executes processing for no gaze (step S160). For example, the partner video processing section 100i is able to execute processing by performing image processing on the positions of the blacks of the eyes, the positions of both eyes, and the direction of the head (face) of the user B appearing in the partner video to point away from forward, thereby making it clear that the line of sight of the user B is not pointing towards the user A facing the display. Note that the partner video processing section 100i may also not execute any kind of processing in the case of no gaze.

This kind of line-of-sight processing process is also executed similarly in the information processing apparatus 10B (steps S163 to S172).

By performing the line-of-sight processing process described above in both of the information processing apparatus 10A and the information processing apparatus 10B, the lines of sight of the user A and the user B can be aligned with each other in video communication that connects remote locations, and smooth communication can be achieved.

Subsequently, if the information processing apparatus 10A issues a video conversation end request to the information processing apparatus 10B (step S175), and the request is accepted by the information processing apparatus 10B, the video conversation ends (step S178). Note that herein, the end request is issued from the information processing apparatus 10A, but obviously is not limited thereto, and the end request may also be issued from the information processing apparatus 10B to the information processing apparatus 10A.

In the embodiment described above, video communication between the two parties of the information processing apparatus 10A and the information processing apparatus 10B is described as an example, but even in video communication among three parties or more, by similarly executing the processes of gaze information synchronization and line-of-sight processing, it is possible to achieve smooth communication. Hereinafter, an embodiment in which gaze information about each user is synchronized by a processing server <NUM> will be described specifically with reference to <FIG>.

<FIG> is a diagram summarizing an information processing system <NUM> according to another embodiment of the present disclosure. As illustrated in <FIG>, the information processing system <NUM> according to another embodiment includes information processing apparatus 10A, 10B, 10C, 10D, and the processing server <NUM>. The information processing apparatus 10A, 10B, 10C, 10D and the processing server <NUM> are interconnected through the network <NUM>, and are capable of transmitting and receiving data. Note that herein, although four information processing apparatus <NUM> are used as an example, the present embodiment is not limited thereto, and in some cases gaze information about each user is synchronized by the processing server <NUM> in video communication between two parties, while in other cases, gaze information about each user is synchronized by the processing server <NUM> in video communication among three or more users.

Since the configuration of the information processing apparatus 10A to 10D is similar to the above embodiment described in <FIG> and <FIG>, details will be omitted here.

The processing server <NUM>, by transmitting gaze information received from one information processing apparatus <NUM> to other information processing apparatus <NUM>, synchronizes the gaze information of each information processing apparatus <NUM> within the system.

Specific operating processes of this kind of information processing system <NUM> according to the present embodiment will be described below with reference to <FIG>.

<FIG> is a sequence diagram illustrating operating processes according to another embodiment of the present disclosure. As illustrated in <FIG>, first, a video conversation connection process is executed among the information processing apparatus 10A to 10D (step S203). In the video conversation connection process, similarly to steps S103 to S106 of <FIG>, one information processing apparatus <NUM> issues a video conversation connection request to the other information processing apparatus <NUM>, and approval by the other information processing apparatus <NUM> causes a connection to be established.

Next, a video display process is executed among the information processing apparatus 10A to 10D (step S206). In the video display process, similarly to steps S109 to S120 in <FIG>, video of a user acquired by each information processing apparatus <NUM> is transmitted to the other information processing apparatus <NUM>, and in each information processing apparatus <NUM>, video of the partner users acquired by the other information processing apparatus <NUM> of the communication destination is displayed.

Next, a process of synchronizing gaze information among the information processing apparatus 10A to 10D is executed by the processing server <NUM>. Specifically, for example, the information processing apparatus 10A analyzes partner video (step S209), additionally tracks the point of view of the user A (step S212), compares and checks the region around the eyes of a partner user recognized by video analysis to the point-of-view position of the user A, and determines the gaze target of the user A (step S215). Herein, <FIG> will be referenced to describe the determination of the gaze target. For example, as illustrated on the left side of <FIG>, in the case in which videos of communication partners, namely the user B and a user C, are arranged on the screen of the display 1031a, the information processing apparatus 10A recognizes regions E1 and E1 around the eyes in each video, compares and checks against the point-of-view position of the user A, and determines whether the user A is gazing at the user B, gazing at the user C, or not looking at either one. Also, for example, as illustrated on the right side of <FIG>, in the case in which a materials image which is shared content and a video of the user B who is a communication partner are arranged side-by-side on the screen of the display 1031b, the information processing apparatus 10A recognizes a materials image E3 and a region E4 around the eyes of the user B, compares and checks against the point-of-view position of the user A, and determines whether the user A is gazing at the materials image, gazing at the user B, or not looking at either one.

Next, the information processing apparatus 10A transmits the determination result, that is, gaze information to the processing server <NUM> (step S218).

Next, the processing server <NUM> transmits the gaze information transmitted from the information processing apparatus 10A to the other information processing apparatus <NUM>, herein the information processing apparatus 10B, the information processing apparatus 10C, and the information processing apparatus 10D (step S221, step S224, and step S227). In this way, the processing server <NUM> executes the gaze information synchronization process. Note that although the case of transmitting gaze information transmitted from the information processing apparatus 10A to the other information processing apparatus <NUM> is described as an example herein, the processing server <NUM> similarly transmits gaze information transmitted from the information processing apparatus 10B, the information processing apparatus 10C, and the information processing apparatus 10D to the other information processing apparatus <NUM> to synchronize the gaze information within the system. These processes are executed in real-time, and each information processing apparatus <NUM> is able to recognize what the partner users are currently looking at (for example, whether or not the partner users are looking at a user appearing on the display). In addition, the processing server <NUM> is capable of managing a line-of-sight status on the basis of the gaze information transmitted from each information processing apparatus <NUM>, and synchronizing the line-of-sight status included in the gaze information with each information processing apparatus <NUM>. Herein, <FIG> illustrates a diagram explaining one example of line-of-sight status management. The example illustrated in <FIG> illustrates an example of line-of-sight status management in the case in which video communication is performed among the four parties of the user A, the user B, the user C, and the user D, for example, and content such as a materials image is shared additionally. In this case, on the screen of the display <NUM> of each information processing apparatus <NUM>, each video of the three communication partners and an image of the shared content are arranged. Next, for example, the processing server <NUM> acquires gaze information indicating that "user A is looking at user B" from the information processing apparatus 10A of the user A, and acquires gaze information indicating that "user B is looking at user A" from the information processing apparatus 10B of the user B. Also, gaze information indicating that "user C is looking at coordinates (x, y) of the shared content" is acquired from the information processing apparatus 10C of the user C, and gaze information indicating that "user D is looking at user B" is acquired from the information processing apparatus 10D of the user D. In this case, as illustrated in <FIG>, when the gaze information acquired from each information processing apparatus <NUM> is combined, the processing server <NUM> is able to recognize a line-of-sight status in which the lines of sight of the user A and the user B are aligned, while the lines of sight of the user D and the user B are not aligned. The processing server <NUM> also synchronizes such line-of-sight status in addition to the gaze information with each information processing apparatus <NUM>. For example, in addition to gaze information acquired from the other information processing apparatus <NUM>, such as "user A is looking at user B", "user B is looking at user A", and "user D is looking at user B", the processing server <NUM> transmits line-of-sight status information such as "lines of sight aligned: user A and user B", "lines of sight misaligned: user D and user B" to the information processing apparatus 10C of the user C.

The above describes a case in which the management of line-of-sight status is performed by the processing server <NUM> as an example, but the present embodiment is not limited thereto, and it is also possible to perform the management of line-of-sight status in the gaze information management section <NUM> of each information processing apparatus <NUM>.

Next, returning to <FIG>, the information processing apparatus 10A to 10D execute the line-of-sight processing process (step S230). In the line-of-sight processing process, similarly to steps S151 to S172 of <FIG>, in each information processing apparatus <NUM>, processing for gaze or processing for no gaze is executed on the basis of the synchronized gaze information. Specific screen display examples in which line-of-sight processing has been performed will be described later with reference to <FIG>.

Subsequently, the information processing apparatus 10A to 10D execute the video conversation end process (step S233). In the video conversation end process, similarly to steps S172 to S178 of <FIG>, one information processing apparatus <NUM> issues a video conversation end request to the other information processing apparatus <NUM>, and approval by the other information processing apparatus <NUM> causes the connection to be severed.

Specific screen display examples in which line-of-sight processing has been performed according to the other embodiment described above will be described with reference to <FIG>.

<FIG> is a diagram illustrating a screen display example in the information processing apparatus 10A according to the present embodiment. In the example illustrated in <FIG>, a video <NUM> of the user B, a video <NUM> of the user C, a video <NUM> of the user D, and a materials image <NUM> which is shared content are arranged side-by-side and displayed on the screen of the display 1031A of the information processing apparatus 10A.

The information processing apparatus 10A processes the line of sight of the user B of the video <NUM>, on the basis of gaze information indicating that "user B is looking at user A" in the video <NUM> of the user B. Specifically, as described above with reference to <FIG>, since a misalignment in the camera position results in a video in which the line of sight is misaligned even though the user B is looking at the user A, the information processing apparatus 10A processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user B in the video to point towards the front, and makes the line of sight of the user B of the video <NUM> appear to point towards to the user A as illustrated in <FIG>. With this arrangement, the line of sight of the user A can be aligned with the user B, and smoother video communication can be performed. Furthermore, the information processing apparatus 10A may also overlay a notification display such as "looking at user A" onto the video <NUM>.

In addition, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10A overlays a notification display such as "looking at materials" onto the video <NUM> in which the user C appears.

Also, on the basis of gaze information indicating that "user D is looking at user B", the information processing apparatus 10A processes the line of sight of the user D appearing in the video <NUM> to point in the direction where the video <NUM> in which the user B appears is arranged on the screen, as illustrated in <FIG>.

Also, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10A overlays a display or an icon indicating the gaze position at the coordinates (x, y) on the materials image <NUM>, thereby pointing out where the user C is looking in the materials.

<FIG> is a diagram illustrating a screen display example in the information processing apparatus 10B according to the present embodiment. In the example illustrated in <FIG>, a video <NUM> of the user A, a video <NUM> of the user C, a video <NUM> of the user D, and a materials image <NUM> which is shared content are arranged side-by-side and displayed on the screen of the display 1031B of the information processing apparatus 10B.

The information processing apparatus 10B processes the line of sight of the user A of the video <NUM>, on the basis of gaze information indicating that "user A is looking at user B" in the video <NUM> of the user A. Specifically, similarly to the case described above with reference to <FIG>, since a misalignment in the camera position results in a video in which the line of sight is misaligned even though the user A is looking at the user B, the information processing apparatus 10B processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user A in the video to point towards the front, and makes the line of sight of the user A of the video <NUM> appear to point towards to the user B as illustrated in <FIG>. With this arrangement, the line of sight of the user B can be aligned with the user A, and smoother video communication can be performed. Furthermore, the information processing apparatus 10B may also overlay a notification display such as "looking at user B" onto the video <NUM>.

In addition, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10B overlays a notification display such as "looking at materials" onto the video <NUM> in which the user C appears.

Also, on the basis of gaze information indicating that "user D is looking at user B", the information processing apparatus 10B processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user D appearing in the video <NUM>, and makes the line of sight of the user D of the video <NUM> appear to point towards the user B as illustrated in <FIG>.

Also, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10B overlays a display or an icon indicating the gaze position at the coordinates (x, y) on the materials image <NUM>, thereby pointing out where the user C is looking in the materials.

<FIG> is a diagram illustrating a screen display example in the information processing apparatus 10C according to the present embodiment. In the example illustrated in <FIG>, a video <NUM> of the user A, a video <NUM> of the user B, a video <NUM> of the user D, and a materials image <NUM> which is shared content are arranged side-by-side and displayed on the screen of the display 1031C of the information processing apparatus 10C.

The information processing apparatus 10C processes the lines of sight of the user A of the video <NUM> and the user B of the video <NUM>, on the basis of a line-of-sight status indicating "lines of sight aligned: user A and user B". Specifically, the information processing apparatus 10C processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user A in the video <NUM> to point in the direction of the video <NUM> of the user B arranged on the right side, and additionally, processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user B in the video <NUM> to point in the direction of the video <NUM> of the user A arranged on the left side. With this arrangement, as illustrated in <FIG>, the lines of sight of the user A and the user B may be expressed as aligned on the screen. Furthermore, the information processing apparatus 10C may also overlay a notification display such as "lines of sight aligned" onto the video <NUM> and the video <NUM>.

Also, on the basis of gaze information indicating that "user D is looking at user B", the information processing apparatus 10C processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user D appearing in the video <NUM> to point in the direction in which the video <NUM> of the user B is arranged to the left on the screen, and makes the line of sight of the user D of the video <NUM> appear to point in the direction of the user B as illustrated in <FIG>.

<FIG> is a diagram illustrating a screen display example in the information processing apparatus 10D according to the present embodiment. <FIG> is a diagram illustrating a screen display example in the information processing apparatus 10D according to the present embodiment. In the example illustrated in <FIG>, a video <NUM> of the user A, a video <NUM> of the user B, a video <NUM> of the user C, and a materials image <NUM> which is shared content are arranged side-by-side and displayed on the screen of the display 1031C of the information processing apparatus 10D.

The information processing apparatus 10D processes the lines of sight of the user A of the video <NUM> and the user B of the video <NUM>, on the basis of a line-of-sight status indicating "lines of sight aligned: user A and user B". Specifically, the information processing apparatus 10D processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user A in the video <NUM> to point in the direction of the video <NUM> of the user B arranged on the right side, and additionally, processes the positions of the blacks of the eyes, the positions of both eyes, or the direction of the head (face) of the user B in the video <NUM> to point in the direction of the video <NUM> of the user A arranged on the left side. With this arrangement, as illustrated in <FIG>, the lines of sight of the user A and the user B may be expressed as aligned on the screen. Furthermore, the information processing apparatus 10D may also overlay a notification display such as "lines of sight aligned" onto the video <NUM> and the video <NUM>.

With this arrangement, the user D understands a state in which the user B at which one's own line of sight is pointed is looking not at oneself but at the user A, and additionally, the line of sight is aligned with the user A.

In addition, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10D overlays a notification display such as "looking at materials" onto the video <NUM> in which the user C appears.

Also, on the basis of gaze information indicating that "user C is looking at coordinates (x, y) of the materials", the information processing apparatus 10D overlays a display or an icon indicating the gaze position at the coordinates (x, y) on the materials image <NUM>, thereby pointing out where the user C is looking in the materials.

As described above, an information processing system according to the present disclosure makes it possible to grasp a partner's line of sight more naturally in video communication that connects remote locations.

For example, it is also possible to create a computer program for causing hardware such as a central processing unit (CPU), read-only memory (ROM), and random access memory (RAM) built into the information processing apparatus <NUM> or the processing server <NUM> described above to exhibit the functions of the information processing apparatus <NUM> or the processing server <NUM>. In addition, a computer-readable storage medium storing the computer program is also provided.

Claim 1:
An information processing apparatus (10A) comprising:
a receiving section (<NUM>) configured to receive a captured image of a video communication partner from a communication destination apparatus (10B);
a control section (<NUM>) configured to control a display of the received captured image on a display section (<NUM>), and
a transmitting section (<NUM>) configured to transmit a captured image capturing a user looking at the display section (<NUM>) to the communication destination apparatus (10B); wherein
the control section (<NUM>) is configured to execute image processing to display the received captured image such that a line of sight of the video communication partner appearing in the received captured image appears to point towards the user looking at the display section (<NUM>), on a basis of information received from the communication destination apparatus (10B) and indicating that the video communication partner appearing in the received captured image is looking at the user being displayed on the communication destination apparatus (10B);
the control section (<NUM>) is configured to process a position of a pupil, positions of both eyes, or a direction of a head in a face image of the video communication partner appearing in the received captured image to point forward;
the control section (<NUM>) is further configured to
detect a line of sight of the user with respect to the display section to compute a gaze position on a screen of the display section (<NUM>),
compare the computed gaze position and a region around an eye of the video communication partner being displayed on the display section (<NUM>), and
determine whether or not the user is looking at the video communication partner,
on a basis of the determination, control a transmission of information indicating that the user is looking at the video communication partner to the communication destination apparatus (10B),
determine whether or not the video communication partner and the user are in a state of looking at each other, and
when determining that the video communication partner and the user are in a state of looking at each other, control a transmission of information indicating that the lines of sight of both persons are aligned to the communication destination apparatus (10B) and execute display control comprising one of enlarging the received captured image, increasing the image quality of the received captured image or highlighting the display of the received captured image.