Patent Description:
As man-machine interface, a technique has been put to actual use which identifies speech voice of user and/or detects touch panel operation by user. For example, the patent document <NUM> discloses a technique of user interface device for communicating intentions of user, in which gaze direction of user is determined by measuring the eyeball movement of user, then the certain number of blinks or gazing for a certain time is detected based on the determination result. Further, the patent document <NUM> discloses a technique of space input device, in which input is analyzed by detecting that a displayed <NUM>-dimensional virtual image is touched by user. Further, the patent document <NUM> discloses a technique of information processing device, in which user's eye contact, movement of line of sight, or reflected light that infrared rays in a direction of line of sight of user is reflected off user's eye is detected, then a displayed object is displayed as animation in a depth direction based on the detection result. Patent document <NUM> discloses an eye contact detection device according to the preamble of claim <NUM>.

Man-machine interface has been widely used from lever, brake, electrical switch, and dial etc., for conventional machine operation to writing pen, display pointer, mouse, and screen touch for recent image display device using computer. Recently, a machine using interactive device has been studied for actual use, which may output precise audio-response or behavior etc., in response to speech voice from user or human's behavior using image processing technique. For example, an interactive device as man-machine interface in smart phone, smart speaker, or robot etc., searches a precise response according to user's question based on built-in data or data stored in cloud server by connecting to cloud server over network to output the precise response.

In such interactive device, when starting up or performing a process etc., connection or relation between user and device is established by predetermined voice phrase or behavior of user or mechanical operation by user. That is, user identifies the relationship with a machine having an interactive device (smart speaker, robot, etc.) as an interface and performs so-called handshakes by recognizing the start, continuation, interruption, hold, and completion of interactions. Signs (cues) of any voices or behaviors are to be exchanged between human and device through interactive device, and the agreement of start, continuation, interruption, hold, and completion of interactions (communications) is to be recognized regardless of whether or not they are exclusive.

For example, when an user who is in an unspecified group starts a smart speaker that is interactive device, the user conveys the intention for establishing individual interaction relation between the smart speaker and the user by speaking a certain voice phrase "ooo". That is, the user stars interaction with the smart speaker by inputting a startup signal to the smart speaker. In some cases, the smart speaker answers "hello, Mr. △△△" to the user to confirm the start and establishment of mutual interaction relation. Then, questions or responses to the smart speaker is started. Further, in some cases, the smart speaker conveys the intention of establishing interaction relation to the corresponding user. Thus, when interaction between user and interactive device is performed, connection confirmation (handshake) between user and device is to be performed each time by speaking a certain voice phrase or doing physical behavior to establish mutual relation. Method or procedure for such establishment causes complicated process, slow pace of process, inconvenience and/or burden that is incompatible with service condition (such as din, noise, silent, sunshine, lighting, darkness, liquid or gaseous atmosphere), which are costs for system procedure or processing for interactive device and/or back-ground cloud system, thereby wasting system source. Further, besides the interactive device as shown in the above-described example, a method has not been put into actual use, in which a pilot for moving object such as aircraft or automobile may visually confirm instrument of operation system, or the state of operation device, control signal, pilot of target, and movement of target object, by using mechanical system. The situation in which iterative confirmation by multiple pilots or self finger-calling is required causes serious accident due to carelessness and neglect.

The present invention intends to solve such conventional problem(s). By a machine using red-eye effect of human, an eye contact detection device may be provided that achieves high-speedily, easily and surely man-machine interface. Also, the present invention intends to simplify and/or increase efficiency of interactive device, alarm device and/or identification device that use the eye contact detection device as well as a machine and system network provided with them.

The present invention provides an eye contact detection device according to claim <NUM> and an interactive device according to claim <NUM>.

According to the present invention, the presence or absence of eye contact may be determined by using light emitted from interactive device etc., and the reflected light of the emitted light reflected by the eye ground of user due to red-eye effect. Handshaking of user and interactive device may be confirmed, so that man-machine interface may be achieved easily, promptly, and economically compared to conventional techniques.

Now, embodiments of an eye contact detection device according to the present invention are described. The eye contact detection device according to the present invention is to achieve man-machine interface between humans and devices, which may be applied to any electrical devices or electrical systems such as interactive device, computer device, information terminal device, portable terminal device, and video game console, etc. In an embodiment, the eye contact detection device is used as input device to which user inputs a signal for starting up electronic devices, starting operations, or exiting operations.

In the present invention, the problem of how to avoid "red-eye effect" that is a negative factor in a field of application of photograph and image processing is rather used positively. "Red-eye effect" is a phenomenon where light is through cornea, iris, and crystalline lens of eyeball, reached at retina of eyeball, reflected off by retina, passed through crystalline lens, iris, and cornea, then emitted out of eyeball.

In the present invention, the precondition of eye contact is that "gaze direction" of a user and "gaze direction" of a device are, like human to human, matched in a straight line. A device in user's sight or a user in device's sight, in other words, merely gaze direction is not eye contact in this invention. The present invention is to detect the establishment of "relative relationship" between user and device.

In a preferred embodiment, eye contact is occurred when user performs any actions to a target device, that is, when user intends to "start" man-machine interface. In the present invention, the establishment or detection of eye contact is performed through the sequence of the following steps (<NUM>) to (<NUM>).

Characteristic matters for the detection of eye contact in a preferred embodiment of the present application are described below.

In embodiments below, examples are shown in which an eye contact detection device according to the present invention is applied to an interactive device. <FIG> is a diagram illustrating an example configuration for an interactive device according to an embodiment of the present invention. The interactive device is an electronic device or computer device having communication function with user and has functions of recognizing voice spoken by user, performing process(es) based on the recognition result, and performing audio output of the processing result. For example, when user asks a question to the interactive device, then the interactive device recognizes the question, performs process(es) for obtaining optimal answer for the question, and performs audio output of the answer for the question.

As shown in <FIG>, an interactive device <NUM> includes an eye contact detection unit <NUM> for detecting a sign caused by user's gazing, a human sensor <NUM> for detecting approaching or existence of user, an input unit <NUM> for inputting speech voice of user, a communication unit <NUM> for enabling data communication with external server etc. through network etc., an output unit <NUM> for performing audio output, a storage unit <NUM> for storing data, software, program, and application etc. required to the interactive device <NUM>, and a control unit <NUM> for controlling each components. The control unit <NUM> includes, for example, microcomputer including ROM/RAM etc., microprocessor, and/or audio/image recognition module etc., and controls each component by software, hardware, or the combination of software and hardware.

The main function of the interactive device <NUM> is to perform interaction with user by voice (sound). For example, when user speaks, the response to it is made. The input unit <NUM> includes, for example, microphone and converts a voice input signal from user to an electrical signal to provide the electrical signal to the control unit <NUM>. The control unit <NUM> recognizes user's voice based on the electrical signal and performs process(es) according to the voice recognition result. In one aspect, as shown in <FIG>, the interactive device <NUM> is in cooperation with an information provision server <NUM> through the internet to receive, for example, optimal answer to user's question from the information provision server <NUM> to perform audio output of the answer.

The eye contact detection unit <NUM> detects the presence or absence of eye contact between user and the interactive device and provides the detection result to the control unit <NUM>. The control unit <NUM> may control voice recognition process(es) according to the presence or absence of eye contact. For example, the control unit <NUM> may execute the start or completion of a process in response to the establishment of eye contact being detected.

<FIG> shows an internal configuration of the eye contact detection unit <NUM>. As shown in <FIG>, the eye contact detection unit <NUM> includes at least one light emitting element <NUM>, a driving circuit <NUM> for driving the light emitting element <NUM>, a light receiving element <NUM> for receiving the reflected light of light emitted from the light emitting element <NUM> to output an electrical signal corresponding to the received light, a detection circuit <NUM> for receiving the electrical signal output from the light receiving element <NUM> to perform a process of amplifying and/or demodulating the electrical signal, and an eye contact determination unit <NUM> for determining whether or not eye contact is established based on a detection signal output from the detection circuit <NUM>.

The light emitting element <NUM> emits, for example, light in a certain direction from an circular opening <NUM> attached to the housing of the interactive device <NUM> as shown in <FIG>. The opening <NUM> is covered with a transparent member that can pass through the wavelength of light emitted from the light emitting element <NUM>. The opening <NUM> corresponds to the eye of the interactive device <NUM>. As one example, the opening <NUM> may be in a conspicuous form or color. For example, the opening <NUM> may be in a form that imitates human's eye. As one example, when user intends to have eye contact, user acts of "gazing" the opening <NUM> as a specified object.

The light emitting element <NUM> has any light source. In one preferred example, light source may be a light-emitting diode (LED) with a relatively small divergence angle and a directivity. One or more light-emitting diode may be provided. In other words, a point light source or surface light source may be provided. Further, the wavelength of light emitted from the light emitting element <NUM> may be optional, for example, that may be visible light such as white light or red light, or infrared light (IrLED). For example, in case of white light, a blue diode may be used as light source, whose wavelength may be converted by a fluorescent material to generate RGB. Further, the light emitting element <NUM> may be a light source such as strobe and flash, or reflected light of each type of light source from the wall surface in an indoor light.

In response to an instruction from the control unit <NUM>, the driving circuit <NUM> makes the light emitting element <NUM> emit light based on a predetermined modulation manner for a predetermined period. Alternatively, the driving circuit <NUM> is started at the same time as power-on of the interactive device <NUM>, and the light emitting element <NUM> may emit light all the time or for a time period determined by a timer. The modulation manner is not specifically limited. The intensity of emitting light may be changed, the wavelength may be changed, the emitting period or lighting period may be changed, or the frequency of a driving signal for driving the light emitting element may be changed. In one aspect, when the human sensor <NUM> detects a human body, the control unit <NUM> causes the light emitting element <NUM> to illuminate light for a certain period through the driving circuit <NUM>. When user narrows his view down to the specific object (for example, the opening <NUM>) within the certain period, that is, the user acts gazing, eye contact is established.

<FIG> shows a typical state of ray of light emitted from the light emitting element <NUM>. As shown in <FIG>, the light emitting element <NUM> is implemented, for example, on a circuit board <NUM> such that an optical axis O is positioned almost at the center of the opening <NUM>. Circuit component(s) of the driving circuit <NUM> and the detection circuit <NUM> etc. may be implemented on the circuit board <NUM>.

The light emitting element <NUM> irradiates light la at a divergence angle θ in an optical axis direction in response to a driving signal modulated from the driving circuit <NUM>. Thus, the interactive device <NUM> irradiates light la as a sign from the opening <NUM>.

The light receiving element <NUM> is implemented adjacent to the light emitting element <NUM> and on the circuit board <NUM>. Since the light receiving element <NUM> is adjacent to the light emitting element <NUM>, the optical axis of the light receiving element <NUM> may be regarded as almost the same as the optical axis of the light emitting element <NUM>. The light receiving element <NUM> includes, for example, photodiode or phototransistor. As shown in <FIG>, irradiated light la is arrived at the retina of the eyeball E of user, that is, the eye ground, and then the reflected light lb reflected off at the retina caused by red-eye effect is received through the opening <NUM> by the light receiving element <NUM> which converts the receiving light to an electrical signal.

In an embodiment herein, whether or not user U makes eye contact in response to light la irradiated as a sign of the interactive device <NUM> is detected. If user U intends to make eye contact, user U looks at and gazes the opening <NUM> in response to light from the light emitting element <NUM>. When the eyeball E of user U is within the irradiation region of light la irradiated from the opening <NUM>, and the gaze direction of the irradiated light la and the gaze direction of user U are matched, the irradiated light la is reflected by the tapetum behind the retina of the eye ground of the eyeball E, then the reflected light lb is received by the light receiving element <NUM> through the opening <NUM>, as a sign from user U. When the irradiated light la is entered to the eyeball E before the pupil is closed by the iris of eye, the light is arrived at the eye ground and the retina and it's reflected light is returned straightly. Due to a lot of capillaries in the retina, light reflected at the eye ground or the retina has red color in so-called red-eye effect. Alternatively, when infrared light is irradiated as a light to be irradiated, eye in which light is reflected at the eye ground is specially shining, which is also red-eye effect. In an embodiment herein, such red-eye effect is used for detection of eye contact. Thus, the driving circuit <NUM> drives the light emitting element <NUM> in a driving manner suitable to the detection of red-eye effect of user U, and the detection circuit <NUM> receives the reflected light in a manner suitable to the driving manner of the driving circuit <NUM>.

The amount of light received by the light receiving element <NUM> peaks when the optical axis direction or gaze direction of the eyeball E of user U is matched with the optical axis O of the light emitting element <NUM>. Even if the eyeball E of user U is in the optical direction O, unless user U gazes the light-emitting window <NUM>, that is, unless the gaze direction is directed to the opening <NUM>, light reflected from the eye ground (retina) of the eyeball E of user U is deviated from the optical axis O, and consequently the reflected light lb from the eye ground of the eyeball E and the retina is deviated from the optical axis O. As a result, the amount of light received by the light receiving element <NUM> is decreased. Further, when the eyeball E of user U is away from the optical axis O (however, the eyeball E is within the irradiation range of irradiated light la), light la from the light emitting element <NUM> is not sufficiently entered into the eyeball E of user U, so that the amount of light lb reflected from the eye ground is decreased accordingly. The amount of light lb received from the eye ground is compared with a threshold as described below, and used for the detection of the presence or absence of the establishment of eye contact.

In this case, the outer diameter D of the opening <NUM> may be used to adjust the amount of light received by the light receiving element <NUM>. That is, if the outer diameter D of the opening <NUM> is decreased, the reflected light lb largely deviated from the optical axis O is shielded by the opening <NUM>, so that the reflected light lb is difficult to be received by the light receiving element <NUM>. On the other hand, if the outer diameter D is increased, the reflected light lb largely deviated from the optical axis O is easily received without being shielded by the opening <NUM>. Accordingly, the accuracy of eye contact may be adjusted by the outer diameter D of the opening <NUM>.

An analog electrical signal generated at the light receiving element <NUM> is output to the detection circuit <NUM>. The detection circuit <NUM> amplifies the analog electrical signal received from the light receiving element <NUM> as needed and further demodulates it according to a modulation manner of a driving signal. The signal processed in the detection circuit <NUM> is provided as a detection signal to the eye contact determination unit <NUM>.

When user answers with a sign (cue) in response to the sign of the light emission by the interactive device <NUM>, that is, in response to the driving circuit <NUM> irradiating modulated light for a certain period, the reflected light by red-eye effect of user that is corresponding to the irradiation of modulated light is received by the light receiving element <NUM>, and then the eye contact determination unit <NUM> determines the established of the eye contact. The determination result by the eye contact determination unit <NUM> is provided to the control unit <NUM> which performs control according to the presence or absence of eye contact.

Now, an example of operation of the eye contact detection unit <NUM> according to a first embodiment herein is described. In the first embodiment, when the human sensor <NUM> detects that user U is present near the interactive device <NUM>, the control unit <NUM> causes the driving circuit <NUM> to perform light emission of the light emitting element <NUM> only for a certain period. The reflected light lb of light la irradiated from the light emitting element <NUM> is received by the light receiving element <NUM> through the opening <NUM>, and an analog electrical signal corresponding to it is output to the detection circuit <NUM>.

The detection circuit <NUM> amplifies the analog electrical signal to binarize the amplified analog electrical signal, that is, to convert the amplified analog electrical signal to a digital signal, by using a circuit such as comparator etc. Threshold for binarization may be set by using sampling data that is extracted by actual experiments. The graph in <FIG> shows an example of the relationship between the difference of angle (vertical axis) of the optical axis O and the eyeball of user U and the amount of received light of the reflected light (horizontal axis). For example, in an experiment, an electrical signal is measured that corresponds to the amount of light received when the gaze direction of user U is deviated from the optical axis O within the irradiation range of light la of the light emitting element <NUM>, and then a relationship between the difference of angle and the amount of received light (that does not necessarily become the relationship of <FIG>) or an approximate equation is extracted. By using the relationship, the range in which eye contact is established is defined. If a range in which the difference of angle is less than S is defined as a range in which eye contact is established, an electrical signal corresponding to the amount of light R received when the difference of angle is S is set as the threshold. The detection circuit <NUM> compares an electrical signal received from the light receiving element <NUM> to the threshold. For example, a H level detection signal is output when eye contact is established, and a L level detection signal is output when not established.

The eye contact determination unit <NUM> determines the presence or absence of the establishment of eye contact based on H or L level of the detection signal output from the detection circuit <NUM> and provides the detection result to the control unit <NUM>.

Now, an example of operation according to a second embodiment herein is described. In the second embodiment, the light emitting element <NUM> is illuminated at a certain frequency for a certain period, such that the presence or absence of eye contact is determined according to the result. <FIG> shows a timing chart of signals according to the second embodiment. When the human sensor <NUM> detects that user U is present near the interactive device <NUM>, the control unit <NUM> accordingly causes the driving circuit <NUM> to generate n pieces of driving pulse signals within a certain period from time T1 to time T2, then the light emitting element <NUM> emits modulated light n times.

In response to n times light emissions (modulations), it's reflected light is received by the light receiving element <NUM> and analog electrical signals corresponding to each receiving light are output to the detection circuit <NUM>. As with the first embodiment, the detection circuit <NUM> binarizes or demodulates the analog electrical signal to output the H or L level detection signal to the eye contact determination unit <NUM>. The eye contact determination unit <NUM> counts the number of the H level detection signals to determine the presence or absence of the establishment of eye contact based on the relationship between the number P of the detected H level pulse(s) and the number of times n of light emission. For example, n/P is more than a certain value, or n-P is more than a certain value, it is determined that eye contact is established.

According to the embodiment, the presence or absence of eye contact is determined by multiple times of light receptions (demodulations) in accordance with multiple times of light emissions (modulation), which increases the accuracy of determination for the presence or absence of eye contact. Especially, when user U look at the opening <NUM> for only a moment without the intention of making eye contact, such determination method is effective.

In the above-described embodiment, the example of modulating the number of light emission of the light-emitting unit is shown. In addition to this, for example, the driving circuit <NUM> may modulate the intensity of light emission due to varying the amplitude of the driving signal or may modulate the frequency of light emission due to varying the pulse frequency of the driving signal. In this case, the detection circuit <NUM> demodulates the received light, then the eye contact determination unit <NUM> compares the modulated light with the demodulated light to determine the presence or absence of eye contact based on the comparison result. For example, when the match between them is more than the threshold, it is determined that the eye contact is established. Further, when the light emitting element <NUM> is provided with a plurality of light emitting elements that emit light with different waveforms, the driving circuit <NUM> may modulate the wavelengths of lights by sequentially driving the light emitting elements.

Now, a diagram illustrating a configuration of the eye contact detection unit <NUM> according to a third embodiment herein is described. There is a lot of capillaries in the retina of human eye. When light is irradiated to eye, light introduced from the pupil to the eye ground is reflected off and went out from the pupil. The reflected light includes lots of red-color wavelengths, which is so-called as the red eye effect. If the reflected light contains a high proportion of red-color wavelength, it is highly likely that the direction of the eye ground of user U is matched or approximate to the optical axis O. Then, in the third embodiment, the presence or absence of the establishment of eye contact is determined based on the amount of received light in which only the red-color wavelength is extracted from the light reflected off by the eyeball E of user U.

<FIG> is a diagram showing a structure of a third embodiment. As shown in <FIG>, an optical filter <NUM> which passes through light in red-color wavelength range and shields light in other wavelength ranges is attached on the front surface of the opening <NUM>. This causes only the red-color wavelength range of the reflected light reflected off by the eyeball E of user U to be received by the light receiving element <NUM>. As a result, the light receiving element <NUM> may receive the reflected light for which red-eye effect of the eyeball E of user U is taken into consideration, which increases the accuracy for determining the establishment of eye contact.

The optical filter <NUM> is not necessarily attached to the opening <NUM>. For example, it may be placed between the opening <NUM> and the light receiving element <NUM>. Further, when a condensing lens is placed in front of the light receiving element <NUM>, the optical filter may be attached to the condensing lens. Further, while the optical filter is effective when a white light source is used as light source of the light emitting element <NUM>, the optical filter may be omitted when a red diode emitting light of red-color wavelength range or an infrared diode emitting infrared beam is used.

Now, a fourth embodiment of the present invention is described. In the first embodiment, an analog electrical signal received from the light receiving element <NUM> is binarized. In the fourth embodiment, a direction α of reflected light that caused by red-eye effect of the eye ground of user U is calculated based on an analog electrical signal received from the light receiving element <NUM> to determine the presence or absence of the establishment of eye contact.

As in the second embodiment, the control unit <NUM> causes the light emitting element <NUM> to emit light n times within a certain period from time T1 to time T2 and to receive reflected light responding to the n times light emissions. The detection circuit <NUM> detects n times of directions of the eye ground α1, α2,. , αn from an integrated value or a peak value of the analog electrical signal corresponding to n times light reception. The direction of reflected light caused by red-eye effect is found by the relationship between the difference of angle and the amount of received light, as shown, for example, in <FIG>. Thus, the n pieces of directions of the eye ground detected by the detection circuit <NUM> are provided to the eye contact determination unit <NUM>.

The eye contact determination unit <NUM> determines the presence or absence of the establishment of eye contact based on fluctuation or rate of change of the n pieces of directions of the eye ground α1, α2, ···, αn. As an example, when the directions α1-αn of the eye ground are less than a certain value and the fluctuation or the rate of change of the direction α1-αn is less than a certain value, it is determined that the eye contact is established.

According to the embodiment, even when user does not keep still toward the interactive device <NUM> or user looks at the opening <NUM> without intention for establishing eye contact, the presence or absence of eye contact may be accurately determined.

While an example in which the presence or absence of eye contact for a single user is determined is described in the above-described embodiments, the interactive device <NUM> provided with multiple light-receiving elements may determine the presence or absence of eye contact for multiple users based on respective electrical signals output from the multiple light receiving elements. For example, as shown in <FIG>, when two light receiving elements 200A, 200B are provided, the presence or absence of eye contact of user U1 is determined based on an electrical signal output from the light receiving element 200A, while the presence or absence of eye contact of user U2 is determined based on an electrical signal output from the light receiving element 200B. In this case, when the optical axes of reflected lights caused by red-eye effect of users U1, U2 are matched on respective optical axes of multiple light emitting elements, eye contact is established. Light from each light emitting element may be modulated in a different modulation manner to suppress interference.

Now, a fifth embodiment of the present invention is described. In the fifth embodiments, a plurality of light receiving elements that are physically spaced are used to specify the position of user U based on the amount of light received by the respective light receiving elements to determine the presence or absence of the establishment of eye contact based on the specified position. In this case, for simplicity of explanation, two light receiving elements are illustrated. <FIG> is a plane view illustrating the light emitting element <NUM>, the light receiving element 220A, 220B, and the eyeball E of user U. Two light receiving element 220A, 220B are placed in line symmetry with respect to the optical axis O of the light emitting element <NUM>.

When light la is irradiated from the light emitting element <NUM> and the eyeball E is present within the irradiation area, reflected lights lb1, lb2 of light la are received by the light receiving elements 220A, 220B. The detection circuit <NUM> receives analog electrical signals from the light receiving elements 220A, 220B to provide the eye contact determination unit <NUM> with the gaze direction α1 detected at the light receiving element 220A and the direction α2 of the eye ground detected at the light receiving element 220B from the electrical signals.

The eye contact determination unit <NUM> determines the presence or absence of eye contact on condition that the difference of the two directions α1, α2 of eye grounds is within a certain range. That is, ideally, the directions α1, α2 of two eye grounds are equal. However, when there is a large difference therebetween, it is estimated that the accuracy of the detection is not well. Thus, when the directions α1, α2 of the eye grounds with respect to the optical axis O are less than a certain value and |α1-α2| is less than a certain value, it is determined that eye contact is established. As a variation of the fifth embodiment, the presence or absence of eye contact may be determined by n directions of eye ground that are obtained by n times light emissions (modulations) of the light emitting element <NUM>.

Now, a sixth embodiment of the present invention is described. In the sixth embodiment, as shown in <FIG>, the eye contact detection unit <NUM> further includes an imaging camera <NUM>. The eye contact determination unit <NUM> uses image data taken by the imaging camera <NUM> to determine eye contact.

The imaging camera <NUM> is placed as a fixed point camera in the interactive device <NUM> and takes images of the surroundings of the interactive device <NUM> through the opening <NUM>. Image data taken by the imaging camera <NUM> are provided to the eye contact determination unit <NUM>. The eye contact determination unit <NUM> image-recognizes face or eyeball of user on the image data to calculate the position of the coordinate of the recognized face or eyeball. In one example, the eye contact determination unit <NUM> adds a determination requirement whether or not the position of the coordinate of face or eyeball is within a predetermined area to the requirement for the determination of the presence or absence of eye contact. That is, the eye contact determination circuit <NUM> determines the presence or absence of eye contact of user based on signal(s) detected by detection circuit <NUM> and the position of the coordinate of face or eyeball extracted from image data. This increases the accuracy of the determination of the presence or absence of eye contact.

As a further example, the eye contact determination unit <NUM> may perform face authentication for user displayed on image data. In this case, the eye contact determination unit <NUM> previously stores faces to be recognized as reference data to add a determination whether or not face authentication is available based on the reference data to the requirement for the determination of the presence or absence of eye contact. That is, the presence or absence of eye contact is determined only for user for whom face authentication is available. Further, the position of eyeball for which face authentication is done may be added to the requirement. This increases the accuracy of the determination of the presence or absence of eye contact for a specific user.

As a further example, the eye contact determination unit <NUM> may perform a personal identification of user by using a pattern of the capillary of the eye ground or a feature of iris of user displayed on image data. In this case, the eye contact determination unit <NUM> previously stores the pattern of the capillary or the feature of iris as reference data to add a determination whether or not personal authentication is available based on the reference data to the requirement for the determination of the presence or absence of eye contact. That is, the presence or absence of eye contact is determined only for user for whom personal authentication is available.

As a further example, the imaging camera <NUM> may be used as human sensor. When a face of human is taken by the imaging camera <NUM>, the driving circuit <NUM> may irradiate light modulated as a sign of eye contact for a certain period. Further, the imaging camera <NUM> may be one or more. For example, a plurality of imaging cameras measure the distance to user by using stereo effect, and light is irradiated as a sign of eye contact for a certain period when user is within a certain distance.

Now, a sixth embodiment of the present invention is described. <FIG> shows an example in which the determination of eye contact in the embodiment is applied to smart speaker. As shown in <FIG>, a smart speaker <NUM> includes an eye contact unit <NUM> and a main body <NUM> electrically connected the eye contact unit <NUM>.

The eye contact unit <NUM> includes, within a housing, a light source (light emitting element) <NUM>, an optical filter <NUM>, a half mirror <NUM>, an optical filter <NUM>, and a light receiving element <NUM>. A full <NUM>-degree spherical optical system <NUM> is attached on the housing. The main body <NUM> includes a control unit <NUM> and a sound output unit <NUM>. The control unit <NUM> controls the eye contact unit <NUM> and determines the presence or absence of eye contact based on an electrical signal output from the light receiving element <NUM>.

Light emitted from the light source <NUM> is entered to the optical filter <NUM>, and the filtered light with a specified wavelength or wavelength range is entered to the half mirror <NUM>. One portion of the entered light is reflected in a direction of the optical axis C by the half mirror <NUM>. The reflected light is emitted through the full <NUM>-degree spherical optical system <NUM> to the external. The full <NUM>-degree spherical optical system <NUM> may emit light either in all directions or in a split specific direction. Light emitted from the full <NUM>-degree spherical optical system <NUM> is entered to the eye ground (retina) of the eyeball E of user. The reflected light reflected at the eye ground returns to the same path and goes on the optical axis C through the full <NUM>-degree spherical optical system <NUM>. One portion of the light passes through the half mirror <NUM>. After filtering by the optical filter <NUM>, the light is received by the light receiving element <NUM>.

As in the above-described embodiment, the control unit <NUM> determines the presence or absence of eye contact based on an electrical signal output from the light receiving element <NUM>. When eye contact is established, sounds comes out from the sound output unit <NUM>.

Thus, in the embodiment, an optical system is disposed between the eye contact unit <NUM> and the eye ball E of user, so that it is not necessarily required that the optical axis C of the eye contact unit <NUM> is placed in a straight line with the optical axis (gaze direction) of the eye ball E of user, which provides the flexibility to the optical design of the eye contact unit <NUM>. The configuration shown in <FIG> is just an example. A bidirectional optical system for incident light and reflected light may be achieved electronically, mechanically, or optically.

Now, a seventh embodiment of the present invention is described. <FIG> shows an example in which the determination of eye contact according to the embodiment is applied to a visual recognition of an instrument board (display) for airplanes, vehicles, and ships. A pair of light-emitting units (including a light emitting element and a driving circuit) and a light-receiving unit (including a light receiving element and a detection circuit) is provided on specific positions X, Y in a horizontal direction of an instrument board (display) <NUM>. A pair of a light-emitting unit and a light-receiving unit is also provided on specific positions A, B in a vertical direction, respectively.

When the gaze direction of user is scanned in a horizontal direction and red-eye effect due to the eye ball of the eyeballs Ex, Ey is detected in the specific positions X, Y within a certain time, or when the gaze direction of user is scanned in a vertical direction and red-eye effect due to the eye ball of the eyeballs Ea, Eb is detected in the specific positions A, B within a certain time, the eye contact determination unit determines that eye contact is established. Based on the determination result of eye contact, a control unit (not shown) determines that user visually recognizes the instrument board.

Thus, in the embodiment, a line of sight is scanned on a same axis. When the scan is detected as red-eye effect within a certain time, it is regarded that eye contact is established. Thus, even when user performs the behavior of scanning or moving the gaze direction, the presence or absence of eye contact may be determined.

In the above-described embodiment, an example is shown in which the light emitting element and the light receiving element are placed in the same position, that is, on the same optical axis. This is just one example and the present invention is not limited thereto. Further, the light emitting element of the present invention is not limited to a light source which has defined targets of optical axis of irradiation such as point light or spot light. A light source such as a wide horizontal ray of light like sunlight and an interior illumination for ceiling may be reflected by a wall and then reflected by the eye ground of the eyeball of user (modulated as a light emitting element), and then received at a light receiving unit in parallel to a line of sight of user (within a reference for determination or a value of error tolerance of movement of incident angle). That is, it is enough that light (with allowable movement) in parallel to an optical axis of a light receiving element (a line of sight of the eye contact detection device) is received at the eyeball of user.

Claim 1:
An eye contact detection device (<NUM>) comprising:
a light emitting means (<NUM>) for emitting light (la) from a specific area;
a light receiving means (<NUM>) for receiving reflected light (lb) of the light (la) emitted by the light emitting means (<NUM>) to output an electrical signal according to the received light; and
a determination means (<NUM>) for determining the presence or absence of eye contact of a user (U) based on the electrical signal corresponding to the light from the light emitting means (<NUM>);
wherein the determination means (<NUM>) determines the presence or absence of eye contact based on reflected light (lb), that is passed through the cornea, the iris, and the crystalline lens of the eyeball (E), arrived at the retina of the eyeball (E), reflected off at the retina, passed through the crystalline lens, the iris, and the cornea, and emitted out of the eyeball (E), the reflected light (lb) is caused by red-eye effect in which a light beam emitted from the specific area is reflected at the eye ground of the user's eye when the user's gaze direction is directed to the specific area,
characterized in that the light receiving means (<NUM>) includes two light receiving elements (220A, 220B) which are placed in symmetry with respect to the optical axis (O) of the light emitting means (<NUM>),
wherein the determination means (<NUM>) uses a relationship between a difference of angle of the user's gaze direction (α1, α2) and the optical axis (O) of the light emitting means (<NUM>) and the amount of light received by the light receiving means (<NUM>),
wherein the determination means (<NUM>) calculates two gaze directions (α1, α2) corresponding to the two light receiving elements (200A, 200B) based on the electrical signal received by the two light receiving elements (200A, 200B) in reference to the relationship, and
wherein the determination means (<NUM>) determines the presence of eye contact when the difference of the two gaze directions (α1, α2) is less than a certain value.