Patent Description:
Cameras equipped with a configuration for a line-of-sight input function (such as an eyeball image sensor for capturing a user's eye) in a finder portion have been proposed. When the user peers into the finder of a camera, usually, the distance between an eye lens that is part of the finder optical system (optical system for enabling the user peering into the finder to see an object) and the user's eyeball does not stay constant. Taking this into consideration, there is provided a light-splitting prism as part of the finder optical system for the sake of the line-of-sight input function, the optical axis of an eyeball-imaging optical system (optical system for forming an optical image of the eye on the imaging plane of an eyeball image sensor) and the optical axis of the finder optical system being made partly coincident. Japanese Patent Application Publication <CIT> discloses a video camera that detects a gaze position (position the user is looking at).

Let us assume that, for the video camera shown in Japanese Patent Application Publication <CIT>, an attempt is to be made to increase the magnification of the finder optical system or to improve the optical performance of the finder optical system, while retaining the line-of-sight input function. In this case, the presence of the light-splitting prism would necessitate enlargement of the finder portion and a significant increase in the cost. An enlargement of the finder portion and a significant cost increase could be avoided if the light-splitting prism were removed and the optical axis of the eyeball-imaging optical system and the optical axis of the finder optical system were made independent from each other.

Furthermore, <CIT> discloses a device configured to be worn on a wearer's head. The device comprises an eye-tracking camera oriented towards a first eye of the wearer for capturing eye-tracking images of the first eye, wherein a processor coupled to the eye-tracking camera identifies anatomical structures of the first eye within the eye-tracking camera images. The processor is configured for identifying reference times of external events and analyzing the eye-tracking images after the reference times to determine at least one of magnitudes of responses and reaction times of the first eye, eyelid, and pupil to the external events. Still further, <CIT> discloses an electronic device is configured to obtain an eye image in which an eye that looks at a display is captured. The electronic device performs line-of-sight detection based on the obtained eye image, and provides, based on at least one of a position of a pupil image in the eye image and a number of corneal reflection images in the eye image, a predetermined notification regarding a method of adjusting a viewing state in which the display is visually perceived.

However, if the light-splitting prism were removed and the optical axis of the eyeball-imaging optical system and the optical axis of the finder optical system were made independent from each other, it would be more difficult to capture the user's eye with the eyeball image sensor. For example, as the user' eye moves away from the eye lens, the pupil center position in the image of the eye captured by the eyeball image sensor changes, and the user's eye easily moves out of the imaging range of the eyeball image sensor.

The present invention provides a technique that makes it easier to capture the user's eye without causing an increase in apparatus size or cost.

The present invention in its first aspect provides a display apparatus as specified in claims <NUM> to <NUM>.

The present invention in its second aspect provides a finder apparatus as specified in claim <NUM>. The present invention in its third aspect provides an imaging apparatus as specified in claim <NUM>.

An embodiment of the present invention is described below.

The external appearance of a camera <NUM> (digital still camera, or lens-replaceable camera), which is an imaging apparatus according to this embodiment, is described with reference to <FIG> illustrate the external appearance of the camera <NUM>. The present invention is applicable to any electronic equipment capable of executing a line-of-sight detection operation that detects a user's line of sight (a user's gaze). Users see, for example, information such as images or characters displayed on a display device, or an optical image through an eyepiece optical system (display optical system). Such electronic equipment may include, for example, mobile phones, game machines, tablet terminals, personal computers, wristwatch-type or eyeglass-type information terminals, head mount displays, binoculars, and so on. The line of sight of the user is in other words the user's gaze position in an image or the like displayed in a display device.

<FIG> is a front perspective view, and <FIG> is a rear perspective view. As shown in <FIG>, the camera <NUM> includes a shooting lens unit 1A and a camera housing 1B. A release button <NUM>, which is an operation member that accepts a shooting operation from the user (photographer), is disposed on the camera housing 1B.

The camera housing 1B shown in <FIG> is in a most basic orientation (standard orientation), so-called a normal position state. The optical axis of the optical system of the shooting lens unit 1A in this state is defined as the Z axis, and a vertical upward axis perpendicular to the Z axis is defined as the Y axis. The axis of the right-handed system perpendicular to each of the Y axis and the Z axis is defined as the X axis.

As shown in <FIG>, on the backside of the camera housing 1B is disposed an eyepiece frame <NUM> (eyepiece portion) for the user to peer into, to see an EVF panel <NUM> contained inside the camera housing 1B as will be described later. The eyepiece frame <NUM> holds an eyepiece window <NUM> and protrudes outward (to the backside) from the camera housing 1B. On the backside of the camera housing 1B are also disposed operation members <NUM> to <NUM> that accept various operations from the user. For example, the operation member <NUM> is a touchscreen that accepts touch operations, the operation member <NUM> is an operation lever that can be pressed down in various directions, and the operation member <NUM> is a four-direction key that can be pressed in for each of four directions. The operation member <NUM> (touchscreen) includes a display panel <NUM> such as a liquid crystal panel, and provides the function of displaying images on the display panel <NUM>.

The configuration inside the camera <NUM> is described with reference to <FIG> is a block diagram illustrating the configuration inside the camera <NUM>.

Reference numeral <NUM> denotes an imaging element (image sensor) such as a CCD or CMOS sensor, for example, which is used for capturing an object. The imaging element <NUM> converts an optical image formed by the optical system of the shooting lens unit 1A on an imaging plane of the imaging element <NUM> into an electrical signal, and outputs the acquired analog image signal to an A/D converter (not shown). The A/D converter converts the analog image signal acquired by the imaging element <NUM> into a digital signal, and outputs the signal as image data.

The shooting lens unit 1A is composed of an optical system including a zoom lens, focus lens, diaphragm and so on. In the state mounted in the camera housing 1B, the lens unit guides the light from the object to the imaging element <NUM>, and forms an object image on the imaging plane of the imaging element <NUM>. A diaphragm controller <NUM>, a focus adjuster <NUM>, and a zoom controller <NUM> each receive instruction signals from a CPU <NUM> via a mounting contact <NUM>, and drive and control the diaphragm, focus lens, and zoom lens, respectively, in accordance with the instruction signals.

The CPU <NUM> equipped in the camera housing 1B reads out a control program for each of the blocks in the camera housing 1B from a ROM in a memory unit <NUM>, and deploys and executes the program in a RAM of the memory unit <NUM>. The CPU <NUM> thus controls the actions of each of the blocks in the camera housing 1B. To the CPU <NUM> are connected a line-of-sight detector <NUM>, a light measurement unit <NUM>, an autofocus detector <NUM>, a signal input unit <NUM>, a light source driver <NUM>, an eyepiece detector <NUM>, a distance calculator <NUM>, a display device driver <NUM>, and so on. The CPU <NUM> also transmits signals via the mounting contact <NUM> to the diaphragm controller <NUM>, focus adjuster <NUM>, and zoom controller <NUM> disposed inside the shooting lens unit 1A. In this embodiment, the memory unit <NUM> has the function of storing image signals from the imaging element <NUM> and a line-of-sight image sensor <NUM>.

The line-of-sight image sensor <NUM> is an imaging element such as a CCD or CMOS sensor, for example, or an eyeball image sensor used for capturing the eye of the user looking at the EVF panel <NUM>.

The line-of-sight detector <NUM> converts an output of the line-of-sight image sensor <NUM> (line-of-sight sensor image), in a state in which an eyeball image (optical image of the eyeball) is formed near the line-of-sight image sensor <NUM>, for example, into a digital signal, and sends the result to the CPU <NUM>. The CPU <NUM> extracts feature points necessary for line-of-sight detection from the line-of-sight sensor image in accordance with a predetermined algorithm to be described later, and determines the user's line of sight (gaze position on the display surface of the EVF panel <NUM>) by calculation from the positions of the feature points.

The light measurement unit <NUM> performs amplification, logarithmic compression, and A/D conversion of signals obtained from the imaging element <NUM> that also doubles as a light measurement sensor, specifically brightness signals corresponding to the brightness of an object field, and sends the results to the CPU <NUM> as object field brightness information.

The autofocus detector <NUM> converts voltages of signals from a plurality of detection elements (plurality of pixels) contained in the imaging element <NUM> and used for phase detection into digital signals, and sends the signals to the CPU <NUM>. The CPU <NUM> computes distances to the object at respective focus detection points from the signals from the plurality of detection elements. This is a known technique called imaging plane phase detection autofocus. In the case of this embodiment, as one example, there are <NUM> focus detection points on the imaging plane corresponding to <NUM> points on the image of the field of view inside the finder (display surface of the EVF panel <NUM>).

Switches SW1 and SW2 are connected to the signal input unit <NUM>. The switch SW1 is for starting light measurement, distance measurement, and line-of-sight detection operation of the camera <NUM>, and turned on by a first stroke (e.g., half press) of the release button <NUM>. The switch SW2 is for starting a shooting operation, and turned on by a second stroke (e.g., full press) of the release button <NUM>. The ON signals from the switches SW1 and SW2 are input to the signal input unit <NUM>, and sent to the CPU <NUM>. The signal input unit <NUM> also accepts operation inputs from the operation member <NUM> (touchscreen), operation member <NUM> (operation lever), and operation member <NUM> (four-direction key) shown in <FIG>.

Reference numeral <NUM> denotes an infrared LED, which is a light source that irradiates the user's eyeball with an infrared light. The light source driver <NUM> drives the infrared LED <NUM> based on signals (instructions) from the CPU <NUM>. For example, the light source driver <NUM> drives the infrared LED <NUM> to emit light with a predetermined light-emitting intensity following an instruction from the CPU <NUM>.

An image processor <NUM> performs various image processes to the image data stored in the RAM of the memory unit <NUM>. The processes includes, for example, correction of pixel defects originating from an optical system or imaging element, demosaicing, white balance adjustment, color interpolation, gamma correction, and so on.

A recording/output unit <NUM> records data including image data in a removable recording medium such as a memory card, or outputs the data to external equipment via an external interface.

Reference numeral <NUM> denotes an eyepiece detection sensor, which is for example a near infrared sensor, or a capacitive sensor. The eyepiece detector <NUM> sends the output of the eyepiece detection sensor <NUM> to the CPU <NUM>. The CPU <NUM> determines whether or not the user's eye has contacted (approached) the eyepiece frame <NUM> (eyepiece portion) from the output of the eyepiece detection sensor <NUM> (eyepiece detector <NUM>) in accordance with a predetermined algorithm.

The distance calculator <NUM> calculates the distance from the finder to the user's eyeball based on the coordinates of a corneal reflection image (image formed by regular reflection of an infrared light emitted from the infrared LED <NUM> on the cornea) in the image captured by the line-of-sight image sensor <NUM> (line-of-sight sensor image). For example, the distance calculator <NUM> calculates the distance from the rearmost plane of a display optical system <NUM> (to be described later) for the user to see the EVF panel <NUM> to the eye. The distance calculator <NUM> then transmits the calculated distance to the CPU <NUM>.

The display device driver <NUM> drives a display device <NUM> based on signals from the CPU <NUM>. For example, the display device driver <NUM> displays images of the object captured by the imaging element <NUM> and various pieces of information on the display device <NUM>. The display device <NUM> here refers to the display panel <NUM> or EVF panel <NUM>.

The configuration of the camera housing 1B is described with reference to <FIG> is a cross-sectional view of the camera housing 1B cut in the Y-Z plane made by the Y axis and Z axis shown in <FIG>, showing a diagrammatic representation of the configuration of the camera housing 1B. This is a cross-sectional view of the camera housing 1B in a normal position state as viewed from the left-hand side of the user.

A shutter <NUM> and the imaging element <NUM> are aligned along the optical axis direction of the shooting lens unit 1A in the camera housing 1B. The display panel <NUM> is provided on the backside of the camera housing 1B. The display panel <NUM> shows menus and images for operation of the camera <NUM> and for viewing and editing of the images acquired by the camera <NUM>. The display panel <NUM> is configured with a liquid crystal panel with backlighting, or organic EL panel. In an upper part of the camera housing 1B is provided an EVF unit 1C (finder apparatus, or finder module) including the EVF panel <NUM>, display optical system <NUM>, and line-of-sight detection system <NUM>. The EVF panel <NUM> is able to display the same screen as that of the display panel <NUM>, and configured with a liquid crystal panel with backlighting, or organic EL panel. The display optical system <NUM> and line-of-sight detection system <NUM> will be described in more detail later. The EVF unit 1C may be removably attached to the camera housing 1B, or not (may be fixedly attached as part of the camera housing 1B).

The configuration of the EVF unit 1C is described with reference to <FIG> is a cross-sectional view of the EVF unit 1C cut in the Y-Z plane, showing a diagrammatic representation of the configuration of the EVF unit 1C.

The EVF panel <NUM>, display optical system <NUM>, and eyepiece window <NUM> align along a display-optical-system's optical axis <NUM>, which is the optical axis of the display optical system <NUM>.

The display optical system <NUM> is disposed in front of the display surface of the EVF panel <NUM>, and normally composed of a plurality of lenses to magnify the EVF panel <NUM>. In the case of this embodiment, the display optical system <NUM> is composed of three lenses, a G1 lens <NUM>, a G2 lens <NUM>, and a G3 lens <NUM>. The number of lenses forming the display optical system <NUM> is not limited in particular and there may be four or five lenses. The G1 lens <NUM>, G2 lens <NUM>, and G3 lens <NUM> are optical lenses that transmit visible light and produced by cutting, grinding, or molding from optical glass or transparent optical plastic.

The eyepiece window <NUM> disposed further in the front of the display optical system <NUM> (opposite side from the EVF panel <NUM> across the display optical system <NUM>) is a transparent member having a portion that transmits visible light. An image displayed on the EVF panel <NUM> is enlarged by the display optical system <NUM>, and observed by the user through a transparent portion of the eyepiece window <NUM>.

The lenses that form the display optical system <NUM> and the eyepiece window <NUM> are not necessarily entirely an optically effective shape or surface (e.g., transparent surface). For example, the lenses that form the display optical system <NUM> and the eyepiece window <NUM> may have a shape for positioning or reinforcing purposes, or for the operator to grip during assembly, or a shape that provides an adhesion surface for the fixing with adhesive, or may include a lightening hole, and these parts need not be transparent. Moreover, some parts that need not be transparent from a viewpoint of optics (for example, parts that should not transmit light) may have a painted or printed surface that blocks light.

Infrared LEDs <NUM> and infrared transmission windows <NUM> are provided at the back of the eyepiece window <NUM> (on the side facing the EVF panel <NUM>). The infrared transmission window <NUM> is a window that covers the infrared LED <NUM> so that it is not visible from outside, and made of resin that absorbs visible light and transmits infrared light.

The line-of-sight detection system <NUM> is also provided at the back of the eyepiece window <NUM>. The line-of-sight detection system <NUM> includes a diaphragm <NUM>, a line-of-sight optical system <NUM>, and the line-of-sight image sensor <NUM>, these aligning along a line-of-sight-optical-system's optical axis <NUM>, which is the optical axis of the line-of-sight optical system <NUM>.

The diaphragm <NUM> is an aperture that regulates the light beams necessary for forming an image of the user's eye (eyeball <NUM>) in the line-of-sight image sensor <NUM>. In this embodiment, the diaphragm <NUM> is provided with a filter that absorbs visible light and transmits infrared light, in order to detect the light emitted from the infrared LED <NUM> and reflected by the eyeball <NUM>.

The line-of-sight optical system <NUM> is an optical system (eyeball-imaging optical system) for forming an optical image of the eyeball <NUM> on the imaging plane of the line-of-sight image sensor <NUM>, and configured with optical lenses and the like. While <FIG> illustrates one lens as the line-of-sight optical system <NUM>, the line-of-sight optical system <NUM> may include a plurality of lenses.

The line-of-sight image sensor <NUM> is an eyeball image sensor for capturing the eyeball <NUM> and outputs an image containing an infrared component (line-of-sight sensor image, for example, a captured image of the user's eye). The imaging plane of the line-of-sight image sensor <NUM> is rectangular, and so is the line-of-sight sensor image. The line-of-sight sensor image will be described in detail later with reference to <FIG>.

While the diaphragm <NUM>, line-of-sight optical system <NUM>, and line-of-sight image sensor <NUM> of the line-of-sight detection system <NUM> are separate components in this embodiment, the line-of-sight detection system <NUM> may instead be a small module camera having these components integrated as a package.

In this embodiment, the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> are nonparallel, and intersect each other at an angle <NUM>. More specifically, when the camera housing 1B takes an orientation in the normal position state (predetermined orientation), the line-of-sight detection system <NUM> is located on the lower side of the Y axis in the EVF unit 1C. The line-of-sight-optical-system's optical axis <NUM> is directed toward the display-optical-system's optical axis <NUM> that is positioned on the upper side of the Y axis (diagonally upward in the Y-Z plane).

In some conventional configurations, the display optical system <NUM> includes a light-splitting mirror or a light-splitting prism as one part thereof, with the line-of-sight-optical-system's optical axis <NUM> partly coinciding with the display-optical-system's optical axis <NUM>. Such a configuration makes it extremely difficult to improve the optical performance of the EVF unit 1C without increasing the size of the EVF unit 1C, as compared to a configuration that does not use a light-splitting mirror or a light-splitting prism. Moreover, light-splitting prisms are generally expensive and cause a cost increase. The configuration according to this embodiment has the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> arranged not in parallel, and does not use a light-splitting mirror or light-splitting prism, so that it is possible to improve the optical performance of the EVF unit 1C without causing an increase in size or cost.

In the case of this embodiment, the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> reside in the same Y-Z plane. These two optical axes need not reside in the same Y-Z plane, for example, and one of the optical axes may be offset in the direction of the X-axis. Namely, the two optical axes may be skew relative to each other.

A preferable arrangement of the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> is described with reference to <FIG> is a cross-sectional view illustrating part of <FIG>.

In <FIG>, the camera housing 1B takes an orientation in the normal position state, with the eyeball <NUM> of the user peering into the EVF unit 1C positioned on the display-optical-system's optical axis <NUM>. The eyelid <NUM> including an upper eyelid <NUM> and a lower eyelid <NUM> covers the eyeball <NUM>. The infrared LEDs <NUM> and infrared transmission windows <NUM> are disposed such as to emit an infrared light to the eyeball <NUM> each from above and below the display-optical-system's optical axis <NUM>.

The line-of-sight-optical-system's optical axis <NUM> (line-of-sight image sensor <NUM>) is disposed upward toward the eyeball <NUM> from below the display-optical-system's optical axis <NUM> (display optical system <NUM> or EVF panel <NUM>), i.e., from a direction in which there is the user's lower eyelid <NUM>. In most cases, upper eyelids <NUM> are larger and thicker than lower eyelids <NUM>. Therefore, with the line-of-sight-optical-system's optical axis <NUM> disposed upward toward the eyeball <NUM> from the lower eyelid <NUM> side, it is easier to capture the eyeball <NUM> than in the case where the line-of-sight-optical-system's optical axis <NUM> is disposed downward toward the eyeball <NUM> from the upper eyelid <NUM> side. More specifically, the occurrence of vignetting caused by the eyelid <NUM> blocking the eyeball <NUM> can be reduced when the line-of-sight image sensor <NUM> captures the eyeball <NUM>. Similarly, this arrangement can reduce the instances in which the eyelid <NUM> blocks the image of a primary light beam of the regular reflection component of infrared light emitted from the infrared LED <NUM> (corneal reflection image, Purkinje image or Purkinje reflex). The smaller the angle <NUM>, i.e., the closer the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> to parallel or coincident, the more easily the line-of-sight detection system <NUM> can capture the image of the eyeball <NUM>. Therefore, the angle <NUM> should preferably be small.

Since the camera <NUM> can be gripped in various manners when in use, the user (eyeball <NUM>) can take various orientations (relative orientations) relative to the orientation of the camera housing 1B. Therefore, the orientation of the line-of-sight-optical-system's optical axis <NUM> should preferably be set upward toward the eyeball <NUM> from the lower eyelid <NUM> side in a camera orientation (with relationships between the orientation or position of the camera housing 1B, eyeball <NUM> and eyelid <NUM>) expected to occur most frequently.

The arrangement of the line-of-sight detection system <NUM> is described in more detail with reference to <FIG> and <FIG>. <FIG> and <FIG> illustrate the line-of-sight detection system <NUM>, which is made up of a plurality of components, as one module or unit. <FIG> shows a diagrammatic view of the arrangement of the EVF panel <NUM> and line-of-sight detection system <NUM> as viewed from the eyeball <NUM>.

As shown in <FIG>, in a view from the eyeball <NUM> on the display-optical-system's optical axis <NUM>, the EVF panel <NUM> has a display surface that is a horizontal rectangle, having a lateral side 6a (a side substantially parallel to a lateral or left and right direction) longer than a vertical side 6b (a side substantially parallel to a vertical or up and down direction). The length of the lateral side 6a and the length of the vertical side 6b satisfy the following equation <NUM>. Namely, three times the length of the lateral side 6a (long side of the display surface of the EVF panel <NUM>) is four times the length of the vertical side 6b (short side of the display surface of the EVF panel <NUM>) or more. For example, the aspect ratio (length of lateral side 6a : length of vertical side 6b) of the display surface of the EVF panel <NUM> is substantially <NUM>:<NUM>.

The line-of-sight detection system <NUM> is disposed upward from below the EVF panel <NUM> having the display surface. This way, the occurrence of vignetting can be reduced as described above. Moreover, the display surface of the EVF panel <NUM> being horizontally long (with short vertical sides 6b) makes the angle <NUM> between the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> small, which makes it easier to reduce the occurrence of vignetting. Provided that the display surface of the EVF panel <NUM> has a constant area, the larger the aspect ratio (length of lateral side 6a/length of vertical side 6b), the more largely vignetting can be reduced. This is not to say that the shape of the display surface of the EVF panel <NUM> is limited to one that satisfies Equation <NUM>.

<FIG> shows a diagrammatic view of an arrangement of the G3 lens <NUM> of the display optical system <NUM> and the line-of-sight detection system <NUM> as viewed from the eyeball <NUM> (from a direction parallel to the display-optical-system's optical axis <NUM>).

The G3 lens <NUM> is an optical lens that forms part of the display optical system <NUM>, and has an optical surface necessary for displaying the EVF panel <NUM> to a larger scale, inside a lens periphery <NUM> that is a peripheral portion of a circular shape about the display-optical-system's optical axis <NUM>. The light beams necessary for displaying the EVF panel <NUM> to a larger scale need not pass through the entire optical surface of the G3 lens <NUM>, and therefore the lens periphery <NUM> may be removed (cut off) partly or entirely along a straight or curved line as required. In <FIG>, a lower part of the lens periphery <NUM> is cut off along a cut-off line <NUM> that is a straight line. The line-of-sight detection system <NUM> (line-of-sight image sensor <NUM>) is disposed such that the line-of-sight detection system <NUM> (line-of-sight image sensor <NUM>) is partly included in the cut-off region 11b surrounded by the lower part of the lens periphery <NUM> and the cut-off line <NUM>. The cut-off line <NUM> is substantially parallel to the lateral side 6a of the EVF panel <NUM> (<FIG>). This way, the line-of-sight detection system <NUM> can be disposed even closer to the display-optical-system's optical axis <NUM>, so that the line-of-sight-optical-system's optical axis <NUM> and the display-optical-system's optical axis <NUM> can be made closer to parallel (coincident). The line-of-sight detection system <NUM> (line-of-sight image sensor <NUM>) may be entirely located inside the cut-off region 11b. The cut-off line may include a straight portion substantially parallel to the lateral side 6a of the EVF panel <NUM> and a non-straight portion.

Since the lenses of the display optical system <NUM> including the G3 lens <NUM> are produced by cutting, grinding, or molding from optical glass or transparent optical plastic, the cut-off line <NUM> should preferably be a simple straight line from the viewpoints of processing cost and reliable optical performance. Nevertheless, the cut-off line <NUM> is not limited to straight lines. For example, the cut-off line <NUM> may be a curve including a straight line in part, or the cut-off line <NUM> may be a curve in its entirety. A cylindrical hole may be formed in the lens periphery <NUM> by performing a drilling (punching) process to the lens periphery <NUM>, and the line-of-sight detection system <NUM> may be disposed such that the line-of-sight-optical-system's optical axis <NUM> extends through this cylindrical hole. In this case, the cut-off line <NUM> will be a circular shape, or an elliptical shape. Cutting or molding may be performed such as to form an arcuate cut-off line <NUM>.

Further, the cut-off region may be provided to only one of the lenses of the display optical system <NUM>, or to a plurality of lenses. In <FIG>, two (G2 lens <NUM> and G3 lens <NUM>) of the three lenses of the display optical system <NUM> each have a cut-off region 10b or 11b. The line-of-sight detection system <NUM> is disposed such that the line-of-sight detection system <NUM> is partly included in the cut-off regions 10b and 11b in a view from a direction parallel to the display-optical-system's optical axis <NUM>.

A method of detecting a line of sight with the use of the line-of-sight detection system <NUM> is described with reference to <FIG>, and <FIG>. <FIG> is a schematic diagram illustrating an image captured by the line-of-sight image sensor <NUM> (line-of-sight sensor image). <FIG> illustrates a line-of-sight sensor image obtained in a condition in which an eyeball image is projected to the line-of-sight image sensor <NUM>. <FIG> is a diagram illustrating the output intensity (brightness of the line-of-sight sensor image) of the line-of-sight image sensor <NUM>. <FIG> shows the brightness information (brightness distribution) of region α of <FIG> shows a brightness distribution in the X-axis direction, wherein the X-axis direction is the horizontal direction and the Y-axis direction is the vertical direction of the line-of-sight sensor image. <FIG> shows a schematic flowchart of a line-of-sight detection operation.

The infrared light emitted from the two infrared LEDs <NUM> aligned along the X-axis direction illuminates the cornea of the user's eyeball <NUM>. Two corneal reflection images are formed by part of the infrared light reflected on the surface of cornea (two corneal reflection images corresponding to the two infrared LEDs <NUM>) near the line-of-sight image sensor <NUM> as denoted at Pd and Pe in the line-of-sight sensor image shown in <FIG>. Similarly, an optical image of the pupil of the eyeball <NUM> formed near the line-of-sight image sensor <NUM> appears as a pupil image <NUM> in the line-of-sight sensor image. An optical image of the iris of the eyeball <NUM> formed near the line-of-sight image sensor <NUM> appears as an iris image <NUM> in the line-of-sight sensor image. Hereinafter, two end portions in the X-axis direction of the pupil image <NUM> shall be referred to as pupil edge images 212a and 212b. The corneal reflection images Pd and Pe appear inside of the pupil image <NUM> or iris image <NUM>. In this embodiment, the line-of-sight-optical-system's optical axis <NUM> is directed upward from the lower side of the Y axis toward the display-optical-system's optical axis <NUM> (the line-of-sight-optical-system's optical axis <NUM> is directed upward toward the eyeball <NUM> from the lower eyelid <NUM> side), so that the pupil image <NUM> and iris image <NUM> appear as an upwardly narrowing shape.

With the start of the line-of-sight detection operation, at step S801 in <FIG>, the infrared LEDs <NUM> emit infrared light. For example, the infrared LEDs <NUM> emit the infrared light toward the user's eyeball <NUM>. An optical image of the user's eyeball illuminated with the infrared light is formed on the line-of-sight image sensor <NUM> through the line-of-sight optical system <NUM> and converted into an electrical signal. Thus a processible electrical signal of the line-of-sight sensor image is obtained.

At Step S802, the line-of-sight detector <NUM> sends an image obtained from the line-of-sight image sensor <NUM> (line-of-sight sensor image) to the CPU <NUM>.

At step S803, the CPU <NUM> determines the coordinates of points corresponding to the corneal reflection images and pupil center from the line-of-sight sensor image obtained at step S802.

As shown in <FIG>, an exceedingly high level of brightness is obtained at the coordinates Xd and Xe of the corneal reflection images Pd and Pe. In the region of the pupil image <NUM> (from the coordinate Xa corresponding to the pupil edge image 212a to the coordinate Xb corresponding to the pupil edge image 212b), an exceedingly low level of brightness is obtained except for those of the coordinates Xd and Xe. In the region of the iris image <NUM> outside the pupil image <NUM>, a brightness of a level between the above two levels of brightness is obtained. The x-coordinates Xd and Xe of the corneal reflection images Pd and Pe and x-coordinates Xa and Xb of the pupil edge images 212a and 212b can be obtained from a brightness distribution such as the one shown in <FIG>. The midpoint between the x-coordinates Xa and Xb can be calculated as an x-coordinate Xc corresponding to the pupil center c.

At step S804, the CPU <NUM> calculates an imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position (relative position) of the eyeball <NUM> relative to the line-of-sight optical system <NUM>, and can be calculated by use of a function of the distance (Xd - Xe) between the corneal reflection images Pd and Pe.

At step S805, the CPU <NUM> calculates rotation angles of the optical axis of the eyeball <NUM> relative to the line-of-sight-optical-system's optical axis <NUM>. The x-coordinate of the midpoint between the corneal reflection images Pd and Pe substantially matches the x-coordinate of the center of curvature of the cornea. Therefore, the rotation angle θx of the eyeball <NUM> in the X-Z plane can be calculated approximately by the following equation <NUM>, wherein Oc represents a standard distance from the center of curvature of the cornea to the center of the pupil. Similarly, the rotation angle θy of the eyeball <NUM> in the Y-Z plane can be calculated, too.

At step S806, the CPU <NUM> estimates the coordinates (Hx, Hy) of the user's gaze position (gaze point, the position the user's line of sight points at, the position the user is looking at) on the display surface of the EVF panel <NUM> with the use of the rotation angles θx and θy calculated at step S805.

At step S807, the CPU <NUM> stores the estimated coordinates (Hx, Hy) of the gaze position in the memory unit <NUM>, and ends the line-of-sight detection operation.

The line-of-sight sensor image is described with reference to <FIG>. The eyeball <NUM> does not stay in one position relative to the EVF unit 1C. For example, depending on whether the user wears eyeglasses or not, on user's health conditions or hair styles, or on whether the user wears a hat or not, the position of the eyeball <NUM> of the user peering in at the EVF panel <NUM> varies. <FIG> illustrates two eyeballs <NUM> located on the display-optical-system's optical axis <NUM>, the eyeball 21a being closer to the EVF panel <NUM> and the eyeball 21b being away from the EVF panel <NUM>. <FIG> show line-of-sight sensor images <NUM> of the eyeball 21a and the eyeball 21b overlapped on one another.

<FIG> shows the line-of-sight sensor image <NUM> according to this embodiment, and <FIG> shows a common line-of-sight sensor image <NUM> for comparison. Both cases of <FIG> use the same line-of-sight-optical-system's optical axis <NUM> and the line-of-sight optical system <NUM>, but the line-of-sight image sensor <NUM> is oriented differently (vertical/horizontal of the imaging plane).

The eyeball image 221a is an image of the eyeball 21a formed near the line-of-sight image sensor <NUM>, and the eyeball image 221b is an image of the eyeball 21b formed near the line-of-sight image sensor <NUM>. Not to mention, an actual line-of-sight sensor image would not show both of the eyeball image 221a and the eyeball image 221b, and would show only one of them. Reference numeral <NUM> denotes a projected optical axis, which is a straight line of the display-optical-system's optical axis <NUM> projected onto the line-of-sight sensor image <NUM> (imaging plane of the line-of-sight image sensor <NUM>).

The imaging plane of the line-of-sight image sensor <NUM> is rectangular, and so is the line-of-sight sensor image. In <FIG>, the line-of-sight sensor image <NUM> is a rectangular image having a long side 211a and a short side 211b. The length of the long side 211a is longer than the length of the short side 211b. In <FIG> (this embodiment), the projected optical axis <NUM> is substantially parallel to the long side 211a of the line-of-sight sensor image <NUM> (long side of the line-of-sight image sensor <NUM>). On the other hand, in <FIG> (comparative example), the projected optical axis <NUM> is substantially parallel to the short side 211b of the line-of-sight sensor image <NUM> (substantially perpendicular to the long side 211a).

As can be seen from a comparison between <FIG>, the eyeball image 221b is not included in the line-of-sight sensor image <NUM> in <FIG> (comparative example), whereas in <FIG> (this embodiment), the eyeball image 221b is completely included in the line-of-sight sensor image <NUM>. That is to say, while the comparative example fails to capture the eyeball 21b, this embodiment successfully captures the eyeball 21b.

When peering into the EVF unit 1C, the user adjusts the position of the eyeball <NUM> so that the eyeball center will be closer to the display-optical-system's optical axis <NUM>. Nevertheless, the position of the eyeball <NUM> on the display-optical-system's optical axis <NUM> varies as described above. According to this embodiment, the projected optical axis <NUM> being substantially parallel to the long side 211a of the line-of-sight sensor image <NUM> (long side of the line-of-sight image sensor <NUM>) enables the line-of-sight detection system <NUM> to detect the eyeball from a wider range on the display-optical-system's optical axis <NUM>. Therefore, the line-of-sight detection system can capture the eyeball <NUM> even when the eyeball <NUM> is positioned away from the eyepiece window <NUM>.

While a preferred embodiment of the present invention has been described with reference to the accompanying drawings, the present invention is not limited to the above-described embodiment. The present invention in the above-described embodiment can be interpreted as being applied to an imaging apparatus, or as being applied to a finder apparatus. The present invention can be applied to any apparatus that has a display panel, in which the distance between the display panel and an eye that looks at the display panel can vary. For example, the present invention is applicable to a display apparatus such as an HMD (Head Mount Display). In the case where the present invention is applied to an HMD, for example, the line-of-sight detection system <NUM> is used as at least one of the line-of-sight detection system for the right eye and the line-of-sight detection system for the left eye of the HMD, so that the HMD detects the line of sight of the user wearing the HMD (line of sight of at least one of the right eye and the left eye). In some HMDs, the display panel has a rectangular display surface, which is provided in an orientation such that the long side of the display surface is substantially parallel to the up and down direction when the HMD orientation is in a normal position state. In such a case, too, the line-of-sight detection system may be disposed such that the projected optical axis is substantially parallel to the long side of the line-of-sight sensor image (long side of the line-of-sight image sensor). The line-of-sight image sensor <NUM> is disposed horizontally on the left side or right side of the EVF panel <NUM> that is oriented vertically when the HMD orientation is in the normal position state, for example. Any configurations obtained by suitably modifying or changing some configurations of the above-described embodiment within the scope of the subject matter of the present invention are also included in the present invention.

The present invention makes it easier to capture the user's eye without causing an increase in apparatus size or cost.

Claim 1:
A display apparatus comprising:
a display panel (<NUM>);
a display optical system (<NUM>) for looking at the display panel (<NUM>);
an image sensor (<NUM>) for capturing an eye of an user looking at the display panel (<NUM>), the image sensor (<NUM>) having a rectangular imaging plane; and
an imaging optical system (<NUM>) for forming an optical image of the eye on the imaging plane,
wherein
the imaging optical system (<NUM>) has an optical axis (<NUM>) disposed toward the eye of the user when the user is looking at the display panel and nonparallel to an optical axis (<NUM>) of the display optical system (<NUM>), and
a projected line of the optical axis (<NUM>) of the display optical system (<NUM>) on the imaging plane is substantially parallel to a long side (211a) of the imaging plane.