Image processing apparatus image processing method and program using electrodes contacting a face to detect eye gaze direction

Disclosed herein is an image processing apparatus, including: a projection section adapted to alternately project an image for a right eye and another image for a left eye; a separation section adapted to separate a projected image of the right eye image and another projected image of the left eye image from each other; a gaze detection section adapted to detect a gaze direction of an observer to which the right eye image and the left eye image are to be given; and a luminance adjustment section adapted to set a range with respect to the gaze direction to the right eye image and the left eye image and adjust a luminance value of pixels so that a portion of the right and left eye images other than the set range exhibits an invisible state.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application claims priority to Japanese Priority Patent Application JP 2010-229770 filed in the Japan Patent Office on Oct. 12, 2010, the entire content of which is hereby incorporated by reference.

BACKGROUND

The present application relates to an image processing apparatus, an image processing method and a program, and is suitable for use with, for example, a case in which a three-dimensional image is presented.

A technique is available in which an image for the right eye and an image for the left eye having a parallax therebetween are alternately given to the eyes of the observer to present a three-dimensional image to the observer (refer to Japanese Patent Laid-Open No. Hei 10-117362 and Japanese Patent Laid-Open No. 2010-193409). With the present technique, a person in a video image can look three-dimensionally, and therefore, the reality is enhanced.

SUMMARY

Incidentally, on a video image regarding a game or the like, a point of view of the user is sometimes likened and displayed as an image. In such a case as just described, as one of countermeasures for the enhancement of the reality, it is sometimes requested for a image to provide such a feeling of presence that the user actually is at the place to the user while the point of view of the user is likened as an image.

It is desirable to provide an image processing apparatus, an image processing method and a program which can enhance the reality.

According to an embodiment, there is provided an image processing apparatus including a projection section adapted to alternately project an image for a right eye and another image for a left eye, a separation section adapted to separate a projected image of the right eye image and another projected image of the left eye image from each other, a gaze detection section adapted to detect a gaze direction of an observer to which the right eye image and the left eye image are to be given, and a luminance adjustment section adapted to set a range with respect to the gaze direction to the right eye image and the left eye image and adjust a luminance value of pixels so that a portion of the right and left eye images other than the set range exhibits an invisible state.

According to another embodiment, there is provided an image processing method, including detecting a gaze direction of an observer to which an image for a right eye and another image for a left eye are to be given, setting a range with reference to the gaze direction to the right eye image and the left eye image and adjusting a luminance value of pixels so that a portion of the right and left eye images other than the set range exhibits an invisible state, alternately projecting the adjusted right eye image and left eye image, and separating a projected image of the right eye image and another projected image of the left eye image from each other.

According to a further embodiment, there is provided a program for causing a computer to execute detecting a gaze direction of the observer to which an image for a right eye and a image for a left eye are to be given, setting a range with reference to the gaze direction to the right eye image and the left eye image and adjusting a luminance value of pixels so that a portion of the right and left eye images other than the set range exhibits an invisible state, alternately projecting the adjusted right eye image and left eye image, and separating a projected image of the right eye image and another projected image of the left eye image from each other.

With the disclosed technology, the noticed region of the observer is limited around a gaze direction, and a three-dimensional image approximate to an actual view field at a dark place at night or the like can be presented to the observer. As a result, the reality can be enhanced by the image processing apparatus, image processing method and program.

DETAILED DESCRIPTION

Embodiments of the present application will be described below in detail with reference to the drawings.

2. Other Embodiments

Presentation methods of a three-dimensional image can be roughly divided into a time division method and a polarization method. In the time division method, an image for the right eye and an image for the left eye having a parallax therebetween are projected alternately along a time series, and the transmission timing for liquid crystal shutters allocated to the left and right eyes is changed over to present the three-dimensional image.

On the other hand, in the polarization method, an image for the right eye and an image for the left eye whose polarization directions are different from each other are projected alternately along a time series and a three-dimensional image is presented through lenses to which polarization directions corresponding to the left and right eyes are allocated.

As an embodiment, a picture presentation system which uses a time division method is described as an example.

1-1. Configuration of the Picture Presentation System

Referring first toFIG. 1, a picture presentation system1is shown and includes eyeglasses10and an image processing apparatus20. The eyeglasses10and the image processing apparatus20are configured such that they can transmit and receive data to and from each other by wire communication or wireless communication.

1-2. Configuration of the Eyeglasses

Referring now toFIGS. 2A,2B and3, the eyeglasses10include a frame (hereinafter referred to as eyeglass frame)11, and a liquid crystal shutter LL allocated to the left eye and a liquid crystal shutter LR allocated to the right eye are provided at predetermined positions of the eyeglass frame11.

A projection section13is mounted on the upper side of a bridge section12disposed between the liquid crystal shutters LL and LR of the eyeglass frame11. The projection section13projects an image toward a forward direction orthogonal to a liquid crystal plane. Further, a three-axis gyroscope14is built in a bridge section BLP.

An electrode (hereinafter referred to sometime as left upper electrode)15for contacting with a left upper side portion of the face with respect to the left eye and another electrode (hereinafter referred to sometime as right lower electrode)16for contacting with a right lower side portion of face with respect to the right eye are provided on the side of the eyeglass frame11which opposes to the face.

As a position of the face with which the left upper electrode15is to contact, a portion around the tail of the left eyebrow of the user who wears the eyeglass frame11is more preferable. On the other hand, as a position of the face with which the right lower electrode16is to contact, a portion below the tail of the right eyebrow of the user who wears the eyeglass frame11is preferable.

Further, an electrode (hereinafter referred to sometime as ear electrode)16for contacting a portion of the face around one ear is provided on the inner side of a temple of the eyeglass frame11. As a position of the face with which the ear electrode17is to contact, a portion of the user below a portion at which the user wears the eyeglass frame11is preferable. It is more preferable if the position is an earlobe.

It is to be noted that, if the electrodes15to17can contact with various portions of the user who wears the eyeglass frame11, then the installation form between the eyeglass frame11and the electrodes15to17is not limited to the shown form.

1-3. Configuration of the Image Processing Apparatus

Referring toFIG. 4, the image processing apparatus20is configured by connecting various kinds of hardware to a CPU (Central Processing Unit)21for controlling the entire image processing apparatus20.

In particular, a ROM (Read Only Memory)22, a RAM (Random Access Memory)23which functions as a work memory of the CPU21, an operation section24and an interface25are connected to the CPU21through a bus26.

A program for suitably using the liquid crystal shutters LL and LR, projection section13, gyroscope14and electrodes15to17to present a three-dimensional image to the user is stored in the ROM22.

The operation section24has various operation elements and provides an instruction corresponding to an operation of the operation elements by the user to the CPU21. The instruction relates to control of the three-dimensional image such as, for example, movement of a person included in a three-dimensional image or a three-dimensional image itself backwardly, forwardly, leftwardly or rightwardly, changing of a state of a three-dimensional image or the like.

The liquid crystal shutters LL and LR, projection section13, gyroscope14, and electrodes15to17as well as a recording medium MR are connected to the interface25. The recording medium MR is, for example, a Blu-ray (Blu-ray Disc Association:®) disk, a hard disk, a memory card or the like.

1-4. Picture Presentation Process

In the case where an instruction to present a video image is received, the CPU21develops the program stored in the ROM22into the RAM23. Referring now toFIG. 5, the CPU21functions, based on the program, as a video image acquisition section31, a synchronizing signal production section32, a projection controlling section33, a shutter controlling section34, a gaze detection section35, a luminance adjustment section36and an inclination adjustment section37. The CPU21thus executes a process for presenting a three-dimensional image to the user.

The video image acquisition section31acquires video image data from the recording medium MR through a wire or wireless communication path using the interface25and transmits the acquired data to the projection controlling section33. The video image data indicate a plurality of images along a time series such that an image for the right eye and an image for the left eye having a parallax therebetween make a unit.

The synchronizing signal production section32produces a signal (hereinafter referred to as synchronizing signal) for synchronization control and transmits the produced signal to the projection controlling section33, shutter controlling section34, luminance adjustment section36and inclination adjustment section37.

The projection controlling section33controls the projection section13to project a plurality of right eye images and left eye images which configure the video image data given from the video image acquisition section31along a time series in accordance with the synchronizing signal supplied thereto from the synchronizing signal production section32.

The shutter controlling section34alternately changes over opening and closing of the liquid crystal shutters LL and LR in accordance with the synchronizing signal supplied thereto from the synchronizing signal production section32such that the right eye image and the left eye image are given to the right eye and the left eye, respectively.

By the gaze detection section35, a gaze direction is detected from a potential difference of the left upper electrode15and another potential difference of the right lower electrode16from the ear electrode17.

A result of an experiment is illustrated inFIG. 6in the case, when a gaze was moved in various directions in a state in which the reference electrode was disposed on the left earlobe and the other electrodes are disposed individually at a left upper portion with respect to the left eye and a right lower portion with respect to the right eye, the potential difference between the electrodes and the reference electrode and the potential difference between the electrodes were measured.

In this measurement, the gaze was moved in order from the front to left upper, right lower, upper middle, lower middle, right upper, left lower, right middle, left middle to front, and the gaze was stopped at the individual positions for predetermined time. As seen inFIG. 4, when the gaze stops in the various directions, the sign and the magnitude are different depending upon the direction, in which the gaze stops, in regard to the potential difference between the left upper electrode and the reference electrode, the potential difference between the right lower electrode and the reference electrode and the potential difference between the left upper electrode and the right lower electrode.

It is to be noted that, since a myoelectric potential when the gaze is moved is measured, the potential differences, which correspond to an eye potential, vary by a great amount upon movement of the gaze as shown inFIG. 6. Since the myoelectric potential has a magnitude of approximately 1000 times as high as the eye potential, the myoelectric potential and the eye potential can be easily distinguished.

The gaze detection section35amplifies the potential difference between the ear electrode17which is used as the reference and the left upper electrode15and amplifies the potential difference between the ear electrode17and the right lower electrode16. Then, the gaze detection section35compares the values of the amplified potential differences and a threshold value defined as the lowest value, which is to be determined as a myoelectric potential, after every predetermined interval of time.

If both of the values of the potential difference between the ear electrode17and the left upper electrode15(the potential difference is hereinafter referred to sometime as left potential difference) and the potential difference between the ear electrode17and the right lower electrode16(the potential difference is hereinafter referred to sometime as right potential difference) are equal to or higher than the threshold value, then the gaze detection section35decides that the gaze is not to be detected.

On the other hand, if the left potential difference and the right potential difference are lower than the threshold value, then the gaze detection section35detects the gaze direction corresponding to the sign and the magnitude of the left potential difference and the right potential difference with reference to a table.

In particular, for example, determining the nine different gaze directions as a detection object, it is decided to which one of three different magnitudes of “great,” “middle” and “small” the sign and the magnitude of the left potential difference and the right potential difference belong. Then, a result of the decision and such a table which indicates a sign and a magnitude in the nine gaze directions as shown inFIG. 4are compared with each other. It is to be noted that the nine gaze directions include “front,” “upper middle,” “lower middle,” “right middle,” “left middle,” “right upper,” “right lower,” “left upper” and “left lower” directions.

If the gaze direction is detected, then the gaze detection section35produces data (hereinafter referred to sometime as gaze data) which indicates the gaze direction and transmits the produced data to the luminance adjustment section36.

In the case where the gaze data is received from the gaze detection section35, the luminance adjustment section36recognizes the right eye image and the left eye image corresponding to a point of time at which the gaze direction indicated by the gaze data is detected as an adjustment object based on the synchronizing signal supplied thereto from the synchronizing signal production section32and a built-in clock.

Then, the luminance adjustment section36sets a range with respect to the gaze direction to the right eye image and the left eye image recognized as the adjustment object and changes the luminance value of pixels in a region outside the set range to a value lower than a predetermined value so that the region outside the set range exhibits an invisible state.

Further, the luminance adjustment section36changes the luminance value of the pixels in the range so as to increase the reduction ratio for the luminance value as the distance from the center increases so that the set range gets darker gradually from the center thereof.

As a result, as seen inFIG. 7, the noticed region of the user is limited around the gaze direction and a three-dimensional image approximate to an actual view field at a dark place at night or the like can be presented.

The inclination adjustment section37acquires, as reference data, a measurement value by the gyroscope14in a state in which the eyeglass frame11is suitably worn. As an acquisition method of the reference data, for example, a method in which the reference data is read out and acquired from a ROM or the like in which it is stored in advance and another method in which a measurement value by the gyroscope14at a point of time at which wearing of the eyeglass frame11is detected is produced and acquired as the reference data are available.

If the reference data is acquired, then the inclination adjustment section37periodically compares the reference data with the measurement value outputted from the gyroscope14to detect the inclination angle with respect to the reference plane from a result of the comparison.

Here, in the case where the detected inclination angle is equal to or greater than the lowest value set for an inclination, the inclination adjustment section37recognizes the right eye image and the left eye image corresponding to a time point of the detection as the adjustment object based on the synchronizing signal supplied thereto from the synchronizing signal production section32and the built-in clock.

Then, the inclination adjustment section37corrects the inclination of the right eye image and the left eye image which are an adjustment object so that the inclination angle becomes smaller than the predetermined angle.

As a result, as seen inFIG. 8, such a situation that a projection image is inclined in response to the inclination of the head of the user who wears the eyeglass frame11is avoided and a three-dimensional image which coincides with an actual view field state is presented.

In this manner, the CPU21uses the electrodes15to17and the gyroscope14to correct the right eye image and the left eye image so as to be approximate to an actual view field at a dark place and coincide with an actual view field state to present a three-dimensional image.

1-5. Three-Dimensional Image Presentation Processing Procedure

Now, a procedure of the three-dimensional image presentation process described above is described with reference to a flow chart shown inFIG. 9. If an instruction to present a video image is received, then the CPU21starts the three-dimensional image presentation process and advances its processing to step SP1.

At step SP1, the CPU21detects the gaze direction using the electrodes15to17and detects the inclination angle with respect to the reference plane using the gyroscope14and then advances the processing to step SP2.

At step SP2, the CPU21acquires an image for the right eye and an image for the left eye in pair which configure video image data from the recording medium MR and advances the processing to step SP3.

At step SP3, the CPU21corrects the right eye image and the left eye image acquired at step SP2based on the gaze direction and the inclination angle with respect to the reference plane detected at step SP1and advances the processing to step SP4. In particular, the luminance value of the right eye image and the left eye image is corrected so that the predetermined range with respect to the gaze direction detected at step SP1is visualized, and the inclination of the right eye image and the left eye image is corrected so that the difference from the inclination angle with respect to the reference plane detected at step SP1is smaller than a predetermined amount.

At step SP4, the CPU21alternately projects the right eye image and the left eye image corrected at step SP3and changes over opening and closing of the liquid crystal shutters LL and LR in an accordance with the projection period, and advances the processing to step SP5.

At step SP5, the CPU21decides whether or not a predetermined period elapses after a detection time point at step SP1. If the predetermined period does not elapse, then the CPU21returns the processing to step SP2, at which it acquires new pair of an image for the right eye image and an image for the left eye from the recording medium MR.

On the other hand, if the predetermined period elapses after the detection time point at step SP1, then the CPU21returns the processing to step SP1, at which it detects the gaze direction and the inclination angle with respect to the reference plane again.

In this manner, the CPU21presents a three-dimensional image based on the video image data stored in the recording medium MR until an instruction for ending presentation of a video image is received.

It is to be noted that the gaze direction is not sometimes detected at step SP1due to movement of the gaze. In this instance, at step SP3, the luminance value of the right eye image and the left eye image which are an acquisition object at present is corrected similarly to the luminance value corrected for the right eye image and the left eye image acquired before the images acquired at present.

Further, if the inclination angle with respect to the reference plane is smaller than the lowest value set for a minimum inclination at step SP1, then at step SP3, the inclination of the right eye image and the left eye image which are an acquisition object at present is corrected similarly to the inclination corrected for the right eye image and the left eye image acquired before the images acquired at present.

1-6. Effect and So Fourth

The image processing apparatus20having the configuration described above detects the gaze direction of the user to which a three-dimensional image is to be presented and sets a range with respect to the gaze direction in an image for the right eye to be given to the right eye of the user and an image for the left eye to be given to the left eye of the user. Then, the image processing apparatus20adjusts the luminance value of pixels in a portion of the image for the right eye and the left eye other than the set range so that the portion exhibits an invisible state.

Accordingly, the image processing apparatus20can present a three-dimensional image whose noticed region of the user is limited around the gaze direction and which is approximate to an actual view field in a dark place at night or the like. This is specifically suitable for a case in which a screen image such as a three-dimensional image in a game or the like is perceived as a view field of the user to present a three-dimensional image.

Further, the image processing apparatus20detects the inclination angle with respect to the reference plane based on a measurement value by the gyroscope and adjusts the inclination of the right eye image and the left eye image so that the inclination angle becomes smaller than a predetermined angle.

Accordingly, the image processing apparatus20can avoid such a situation that a projection image is inclined in accordance with the inclination of the head of the user who wears the eyeglass frame11and can present a three-dimensional image which coincides with an actual view field state to the user. This is specifically suitable for a case in which the projection section13for projecting an image to be observed by the observer is provided on an apparatus such as the eyeglass frame11to be worn on the observer.

With the configuration described above, by implementing presentation of a three-dimensional image which is approximate to an actual view field or another three-dimensional image which coincides with an actual view field state, the image processing apparatus20can enhance the reality.

2. Other Embodiments

In the embodiment described above, the liquid crystal shutters LL and LR are applied as a section for separating the projection images for the right eye image and the left eye image to the corresponding eyes. However, the separation section is not limited to that in the embodiment described above. For example, a lenticular screen or the like can be applied. Further, in the case where the optical polarization method is applied in place of the time division method, a liquid crystal lens or the like serving as a polarization filter can be applied in place of the liquid crystal shutters LL and LR.

It is to be noted that, while, in the time division method, the luminance value and inclination of the right eye image and the left eye image before projection are corrected, in the light polarization method, the polarization direction of a liquid crystal lens is adjusted to correct the luminance value and inclination of the right eye image and the left eye image to be projected.

Further, in the embodiment described above, the gaze direction is detected from the potential difference between the electrodes15and16and the electrode17. However, the detection section of the gaze direction is not limited to that in the embodiment described above. For example, the gaze direction may be detected from an image obtained as a result of image pickup by an image pickup section for picking up an image of the eyes of the user to which the right eye image and the left eye image are to be given. Particularly, as a detection method of the gaze direction from an image, a method is available in which the gaze direction is detected, for example, based on a dispersion amount at the center of the eyes with respect to the center of an eye region sandwiched by lids.

Further, in the embodiment described above, the nine directions of “front,” “upper middle,” “lower middle,” “right middle,” “left middle,” “right upper,” “right lower,” “left upper” and “left lower” are applied taking the gaze direction as the detection object. However, the gaze direction of the detection object is not limited to those in the embodiment described above, and various numbers of the detection object can be applied.

Further, in the embodiment described above, the magnitude of the left potential difference and the right potential difference is classified into three stages. However, the number of stages for classifying the magnitude is not limited to that in the embodiment described above, and various numbers of the stages can be applied.

Further, in the embodiment described above, the shape of the range with reference to the gaze direction is formed as a circular shape. However, the shape of the range is not limited to the circular shape, but various shapes can be applied.

Further, in the embodiment described above, the sign and the magnitude of the left potential difference and the right potential difference and the table which indicates the sign and magnitude in the nine gaze directions are compared with each other to detect the gaze direction. However, the detection method for the gaze direction is not limited to that in the embodiment described above, and, for example, a method in which the gaze direction is directly detected by calculation from the sign and the magnitude of the left potential difference and the right potential difference or the like may be applied.

Further, in the embodiment described above, the eyeglass frame11is applied as a wearing section or apparatus which the user can wear. However, the wearing section is not limited to that in the embodiment described above. Various apparatus such as, for example, a hair band, a cap or the like can be applied.

Further, in the embodiment described above, the left upper portion with respect to the left eye and the right lower portion with respect to the right eye are applied as portions with which an electrode is to contact. However, the portion with which an electrode is to contact is not limited to the left upper portion of the left eye and the right lower portion of the right eye if the gaze in the upward and downward direction and the leftward and rightward direction can be detected. Further, while the number of the electrodes to contact with the proximity of the eyes in the embodiment described above is one electrode for each of the eyes, two or more electrodes may be used.

Further, in the embodiment described above, the program for executing the video image presentation process is stored in the ROM. However, the storage place of the program is not limited to that in the embodiment described above. Various kinds of the storage medium on the outside or the inside of the image processing apparatus20can be applied.

The present application can be applied to a game industry, a medical industry and so forth.