System, apparatus, and method for displaying information on a head mounted display

An HMD includes an image display section that allows a user to view an image and transmits an outside scene, an evaluation section that evaluates the state of the user, and a communication section that communicates with another HMD. A control section transmits visual field data on the basis of images captured with a right camera and a left camera to the other HMD, causes the communication section to transmit notification data on the basis of a result of the evaluation performed by the evaluation section to the other HMD, and causes the image display section to perform display on the basis of data transmitted from the other HMD.

BACKGROUND

1. Technical Field

The present invention relates to a display system, a display apparatus, and a method for controlling the display apparatus, and a program.

2. Related Art

To assist a person who performs work, there has been a known case where a display apparatus called a head mounted display (HMD) worn around the person's head is used (see JP-A-2008-113317, for example). In the system described in JP-A-2008-113317, the HMD that the worker wears includes an electronic camera, and an image of a work target captured with the electronic camera of the HMD is displayed on a host PC operated by a person who gives an instruction. The person who gives an instruction looks at the image of the work target captured with the electronic camera and causes the HMD on the worker to display an alarm or any other type of information.

In the configuration of related art described above, in which the person who gives an instruction to assist the worker grasps the state of the worker from the image displayed on the PC, it has been desired that the person who assists the worker convey a large amount of information in a more understandable form.

SUMMARY

An advantage of some aspects of the invention is to enable a person, when the person assists another person using a head mounted display to perform an action, to give a large amount of information on the action in a more understandable manner.

An aspect of the invention relates to a system including a plurality of head-mounted-type display apparatus, and each of the display apparatus includes a display section that allows a user to view an image and transmits an outside scene, a communication section that communicates with another display apparatus of the plurality of display apparatus, an imaging section that performs image capturing over a range that overlaps with a visual field of the user, an evaluation section that evaluates a state of the user, and a control section, and the control section transmits visual field data on the basis of an image captured by the imaging section to the another display apparatus, causes the communication section to transmit notification data on the basis of a result of the evaluation performed by the evaluation section to the another display apparatus, and causes the display section to perform display on the basis of data transmitted from the another display apparatus.

According to the aspect of the invention, in which data on the visual fields and data on inputs can be shared among the plurality of display apparatus, a plurality of users who wear the display apparatus can share the visual fields and operation. Therefore, for example, a person who performs an action and a person who assists the action wear the head-mounted-type display apparatus for conveyance of a large amount of information on actions in a more understandable manner. Further, evaluation of the state of a user allows notification data to be transmitted at appropriate timing. A user can therefore assist another user who performs work or any other type of act or readily and accurately administer the state in which work or any other type of act is performed.

In the display system according to the aspect of the invention, the evaluation section may evaluate, on the basis of an action of the user, whether the state of the user corresponds to a state in which notification set in advance should be made, and when the evaluation section determines that the state of the user corresponds to the state in which notification should be made, the control section may cause the communication section to transmit notification data to the another display apparatus.

According to the aspect of the invention with this configuration, determination that the state of the user is the state in which notification should be made can be made, and notification data can be transmitted to another apparatus.

The display system according to the aspect of the invention may further include a recognition section that recognizes an input, and the evaluation section may evaluate the state of the user on the basis of the input recognized by the recognition section.

According to the aspect of the invention with this configuration, the each of the display apparatus can recognize an input from the user or another person to appropriately evaluate the state of the user. Further, the user or another person allows the display apparatus to recognize an input to intentionally cause the display apparatus to transmit notification data to another apparatus.

The display system according to the aspect of the invention may further include a voice processing section that detects voice, and the recognition section may recognize an input on the basis of the voice detected by the voice processing section.

According to the aspect of the invention with this configuration, each of the display apparatus can accept an input in the form of voice, whereby information can be more readily conveyed among the plurality of display apparatus.

In the display system according to the aspect of the invention, the control section may transmit data representing the input recognized by the recognition section on the basis of the voice detected by the voice processing section to the another display apparatus.

According to the aspect of the invention with this configuration, a content inputted in the form of voice to each of the display apparatus can be notified to a user who wears another display apparatus.

The display system according to the aspect of the invention may further include a motion detection section that detects motion of the display section, and the recognition section may recognize an input on the basis of the motion detected by the motion detection section.

According to the aspect of the invention with this configuration, the recognition section can accept an input according to motion of the display section or motion of the user who wears the display section.

In the display system according to the aspect of the invention, the recognition section may recognize an input on the basis of an image captured by the imaging section.

According to the aspect of the invention with this configuration, the recognition section can recognize an input on the basis of a captured image and can therefore more readily accept an input.

In the display system according to the aspect of the invention, the control section may perform a reproduction process of reproducing action guidance data containing guidance information that gives guidance on actions, order information that specifies an order in accordance with which the actions are performed, and state information that shows a state of the performed actions, cause the display section to perform display in the reproduction process and updates, on the basis of an input recognized by the recognition section, the state information in the action guidance data being reproduced, and cause the communication section to transmit the action guidance data having undergone the reproduction process to the another display apparatus.

According to the aspect of the invention with this configuration, action guidance can be given to a user who wears the display apparatus and performs an action, and data containing records on the action performed in accordance with the guidance can be taken over and used by another display apparatus.

In the display system according to the aspect of the invention, the imaging section may perform image capturing over a range that overlaps with a direction in which the user gazes.

According to the aspect of the invention with this configuration, an image in the direction in which the user gazes can be shown to another user.

Another aspect of the invention relates to a head-mounted-type display apparatus including a display section that allows a user to view an image and transmits an outside scene, a communication section that communicates with another head-mounted-type display apparatus, an imaging section that performs image capturing over a range that overlaps with a visual field of the user, an evaluation section that evaluates a state of the user, and a control section, and the control section transmits visual field data on the basis of an image captured by the imaging section to the another display apparatus, causes the communication section to transmit notification data on the basis of a result of the evaluation performed by the evaluation section to the another display apparatus, and causes the display section to perform display on the basis of data transmitted from the another display apparatus.

According to the aspect of the invention, data on the visual fields and data on inputs can be shared with another display apparatus. Therefore, for example, in a configuration in which a plurality of users wear the display apparatus, the visual fields and operation can be shared by the users. In this case, a person who performs an action and a person who assists the action wear the display apparatus for conveyance of a large amount of information on actions in a more understandable manner. Further, evaluation of the state of a user allows notification data to be transmitted at appropriate timing.

In the aspect of the invention, a head-mounted-type display apparatus including a display section that allows a user to view an image and transmits an outside scene is caused to perform image capturing over a range that overlaps with a visual field of the user, evaluate a state of the user, transmit visual field data on the basis of captured image data to another head-mounted-type display apparatus, transmit notification data on the basis of a result of the evaluation of the state of the user to the another display apparatus, and cause the display section to perform display on the basis of data transmitted from the another display apparatus.

According to the aspect of the invention with this configuration, data on the visual fields and data on inputs can be shared with another display apparatus. Therefore, for example, in a configuration in which a plurality of users wear the display apparatus, the visual fields and operation can be shared by the users. In this case, a person who performs an action and a person who assists the action wear the display apparatus for conveyance of a large amount of information on actions in a more understandable manner. Further, evaluation of the state of a user allows notification data to be transmitted at appropriate timing.

The invention can also be configured as a program executable by a computer that controls a head-mounted-type display apparatus including a display section that allows a user to view an image and transmits an outside scene. The program causes the computer to perform image capturing over a range that overlaps with a visual field of the user, evaluate a state of the user, transmit visual field data on the basis of captured image data to another head-mounted-type display apparatus, transmit notification data on the basis of a result of the evaluation of the state of the user to the another display apparatus, and cause the display section to perform display on the basis of data transmitted from the another display apparatus. The invention may also be configured as a storage medium that stores the program.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1is a schematic configuration diagram of a display system1according to an embodiment to which the invention is applied.

The display system1includes a plurality of HMDs100, and the HMDs100are connected to one another in a communicatable manner via a communication network4. Each of the HMDs100is a display apparatus that a user wears around the head and also called a head mounted display. Each of the HMDs100is an optically transmissive HMD that allows the user to not only view a virtual image but also directly view an outside scene at the same time. In the following description, a virtual image that the HMD100allows the user to view is also called a “display image” for convenience. Further, outputting image light generated on the basis of image data is also referred to as “displaying an image.”

Three HMDs100illustrated inFIG. 1are worn by different users. For example, three users wear the HMDs100and are allowed to use them at the same time. In the following description, the three HMDs100are called HMDs100A,100B, and100C for convenience of the description. The locations where the HMDs100A,100B, and100C are used are assumed to be sites A, B, and C, respectively.

In the display system1, users who perform actions wear the HMDs100B and100C, a user who instructs and administers actions wears the HMD100A, and transmission and reception of information for the instruction and administration are performed among the HMDs100A,100B, and100C. The sites A, B, and C are free from geographical restrictions as long as they are connectable to the communication network4and may be remote from each other or close to each other. In the following description, a case where the three HMDs100A,100B, and100C are connected to the communication network4is shown, but no constraint is imposed on the number of HMDs100provided in the display system1.

FIG. 2shows an example of the arrangement of the HMDs100A,100B, and100C in the display system1.

The HMDs100B and100C are arranged, for example, in the sites B and C, respectively, where workers perform work, in a work line FA in a factory. A worker UB who performs work in the site B wears the HMD100B, and a worker UC who performs work in the site C wears the HMD100C.

In the work line FA, targets OB, such as parts that are targets of work, are transported in the direction indicated by the arrows, and the workers UB and UC perform instructed work on the targets OB.

An administrator UA, who administers the work line FA, wears and uses the HMD100A in the site A, which is a site for an administrator. The administrator UA uses the display system1to assist the work performed by the workers UB and UC, administer the progress of the work, and perform other types of action.

FIG. 2shows an application example in which in a manufacturing factory or any other place, an administrator wears an HMD100, workers wear HMDs100, and the administrator assists work performed by workers, administers the progress of the work, administers the quality of a result of the work, and performs other types of action, but a target situation in which the display system1can be used is not limited to the manufacturing factory described above and the following application examples 1 to 4 can be listed.

In a work place accompanied by a warehouse, workers who move in the warehouse and pick up articles and an administrator wear HMDs100. The HMD100on each of the workers displays a guide path along which the worker moves in the warehouse to guide the worker and indicates an article to be picked up. The work administrator wears an HMD100to administer the progress of the work and perform other types of action.

In a construction site, an HMD100on a worker displays the position of a facility or any other object that the worker cannot directly view, such as an underground pipe, on the basis of design drawings to guide the worker. A construction administrator wears an HMD100to administer the progress of the construction.

A user who wears an HMD100and moves receives movement path guidance and route guidance. An administrator for the movement wears an HMD100and administers the situation of the movement of the user and the position thereof.

In a medical institute, a medical practitioner who wears an HMD100receives assistance in medical consultation, medical examination, or any other act. An assistant who wears an HMD100checks act performed by the medical practitioner, administers the progress of the act, and performs other types of action.

The communication network4is achieved by a variety of communication lines, such as a public network, a dedicated line, a wireless communication line including a mobile telephone line, a backbone communication line of any of the lines described above, or a combination thereof, and the configuration of the communication network4is not limited to a specific configuration. The communication network4may be a wide area communication network capable of connecting remote locations to each other or a LAN (local area network) installed in a specific institution or building. The communication network4may include a server device and a gateway device, a router device, or any other network apparatus that connect the variety of communication lines described above to each other. The communication network4may instead be formed of a plurality of communication lines. The HMDs100A,100B, and100C wirelessly communicate with a wireless access point (not shown) or any other device that forms the communication network4to transmit and receive data to and from each other via the communication network4.

The HMDs100A,100B, and100C will be described on the assumption that they have the same configuration. When the HMDs100A,100B, and100C are not distinguished from each other, they are collectively called the HMD100.

The HMD100includes an image display section20, which a commander who is a user wears around the head, and a control device10, which controls the image display section20. The image display section20is worn around the user's head and allows the user to view a virtual image. The control device10also functions as a controller used by the user to operate the HMD100.

FIG. 3is a descriptive diagram showing an exterior configuration of the HMD100.

The image display section20is a wearable member worn around the user's head and has a spectacle-like shape in the present embodiment. The image display section20includes a right holder21, a right display driver22, a left holder23, a left display driver24, a right optical image display section26, a left optical image display section28, a right camera61(imaging section), a left camera62(imaging section), and a microphone63. The right optical image display section26and the left optical image display section28are so disposed as to be located in front of the right and left eyes of the user who wears the image display section20. One end of the right optical image display section26and one end of the left optical image display section28are linked to each other in a position corresponding to the portion between the eyes of the user who wears the image display section20.

The right holder21is a member extending from an end ER of the right optical image display section26, which is the other end thereof, to a position corresponding to a temporal region of the user who wears the image display section20. Similarly, the left holder23is a member extending from an end EL of the left optical image display section28, which is the other end thereof, to a position corresponding to another temporal region of the user who wears the image display section20. The right holder21and the left holder23, which serve in the same manner as temples (bows) of spectacles do, hold the image display section20around the user's head.

The right display driver22and the left display driver24are disposed on opposite sides of the head of the user who wears the image display section20. The right display driver22and the left display driver24are also simply called “display drivers” in a collective manner, and the right optical image display section26and the left optical image display section28are also simply called “optical image display sections” in a collective manner.

The display drivers22and24include liquid crystal displays241and242(hereinafter referred to as “LCDs241and242”), projection systems251and252, which will be described later with reference toFIGS. 4 to 6, and other components.

The right optical image display section26and the left optical image display section28include light guide plates261and262(FIG. 4) and a light control plate20A. The light guide plates261and262are made, for example, of a light transmissive resin material and guide image light outputted from the display drivers22and24to the user's eyes. The light control plate20A is a thin-plate-shaped optical element and so disposed as to cover the front side of the image display section20, which is opposite the side where the user's eyes are present. The light control plate20A can be a plate having light transmittance of substantially zero, a nearly transparent plate, a plate that transmits light but attenuates the amount of light, a plate that attenuates or reflects light of a specific wavelength, or any of other variety of optical components. Appropriate selection of optical characteristics (such as light transmittance) of the light control plate20A allows adjustment of the amount of external light externally incident on the right optical image display section26and the left optical image display section28and hence adjustment of visibility of a virtual image. In the present embodiment, a description will be made of a case where the light control plate20A is optically transmissive enough at least to allow the user who wears the HMD100to view an outside scene. The light control plate20A also protects the right light guide plate261and the left light guide plate262and prevents the right light guide plate261and the left light guide plate262from being damaged, dirt from adhering thereto, and other defects from occurring.

The light control plate20A may be configured to be attachable to and detachable from the right optical image display section26and the left optical image display section28, or a plurality of types of light control plate20A may be exchangeably attachable. The light control plate20A may even be omitted.

The right camera61is disposed on the front surface of the HMD100and in an end portion on the side facing the right holder21. Similarly, the left camera62is disposed on the front surface of the HMD100and in an end portion on the side facing the left holder23. Each of the right camera61and the left camera62is a digital camera including a CCD, a CMOS device, or any other imaging device, an imaging lens, and other components and may be a monocular camera or a stereoscopic camera.

Each of the right camera61and the left camera62captures an image of at least part of an outside scene present on the front side of the HMD100, in other words, in the visual field direction of the user who wears the HMD100. In another expression, at least one of the right camera61and the left camera62performs image capturing over a range that overlaps with the user's visual field or in the direction thereof. More specifically, at least one of the right camera61and the left camera62performs image capturing in the direction in which the user gazes. The angle of view of each of the right camera61and the left camera62can be set at an appropriate value and is, in the present embodiment, an angle of view that covers the outside viewed by the user through the right optical image display section26and the left optical image display section28, as will be described later. Further, it is more preferable that the image capturing range of each of the right camera61and the left camera62is so set that the camera can perform image capturing over the entire visual field of the user through the light control plate20A.

Each of the right camera61and the left camera62performs image capturing under the control of an imaging control section161(FIG. 6), which is provided in a control section140, and outputs captured image data to the imaging control section161.

Distance sensors64are disposed in a portion at the boundary between the right optical image display section26and the left optical image display section28. In a state in which the user wears the image display section20, the positions of the distance sensors64are roughly at the middle between the user's eyes in the horizontal direction but above the user's eyes in the vertical direction.

The distance sensors64detect the distance to a target object under measurement located in a preset measurement direction. Each of the distance sensors64includes, for example, an LED, a laser diode, or any other light source and a light receiver that receives light emitted from the light source and reflected off the target object under measurement. In this case, the distance sensors64perform the distance measurement based on triangulation or time difference under the control of the control section140. Each of the distance sensors64may instead include a sound source that emits an ultrasonic wave and a detector that receives the ultrasonic wave reflected off a target object under measurement. In this case, the distance sensors64perform the distance measurement based on the difference in time spent until the ultrasonic wave is reflected under the control of the control section140. Each of the distance sensors64may instead include a light source and a light receiver or a sound source and a detector, and the control section140may perform the distance measurement.

The measurement direction of the distance sensors64in the present embodiment is the frontward direction from the HMD100and coincides with the image capturing direction of the right camera61and the left camera62.

FIG. 4is a key portion plan view showing the configuration of an optical system provided in the image display section20.FIG. 4shows a user's left eye LE and right eye RE for the description.

The left display driver24includes a left backlight222, which has an LED or any other light source and a diffuser plate, and the left LCD242, which is a transmissive LCD and disposed in the light path of light having exited out of the diffuser plate of the left backlight222. The left display driver24further includes the left projection system252, which includes a lens group and other components that guide image light L having passed through the left LCD242. The left LCD242is a transmissive liquid crystal panel having a plurality of pixels arranged in a matrix.

The left projection system252has a collimation lens that converts the image light L having exited out of the left LCD242into a parallelized light flux. The image light L, which is the parallelized light flux having been converted by the collimation lens, enters the left light guide plate262. The left light guide plate262is a prism having a plurality of reflection surfaces that reflect the image light L, and the image light L is reflected multiple times in the left light guide plate262and then guided to the side where the left eye LE is present. The left light guide plate262has a half-silvered mirror262A (reflection surface) formed thereon and disposed in front of the left eye LE.

The image light L reflected off the half-silvered mirror262A exits out of the left optical image display section28toward the left eye LE, and the image light L forms an image on the retina of the left eye LE and allows the user to view the image.

The right display driver22is a bilaterally symmetric display driver of the left display driver24. The right display driver22includes a right backlight221, which has an LED or any other light source and a diffuser plate, and the right LCD241, which is a transmissive LCD and disposed in the light path of light having exited out of the diffuser plate of the right backlight221. The right display driver22further includes the right projection system251, which includes a lens group and other components that guide image light L having passed through the right LCD241. The right LCD241is a transmissive liquid crystal panel having a plurality of pixels arranged in a matrix.

The right projection system251has a collimation lens that converts the image light L having exited out of the right LCD241into a parallelized light flux. The image light L, which is the parallelized light flux having been converted by the collimation lens, enters the right light guide plate261. The right light guide plate261is a prism having a plurality of reflection surfaces that reflect the image light L, and the image light L is reflected multiple times in the right light guide plate261and then guided to the side where the right eye RE is present. The right light guide plate261has a half-silvered mirror261A (reflection surface) formed thereon and disposed in front of the right eye RE.

The image light L reflected off the half-silvered mirror261A exits out of the right optical image display section26toward the right eye RE, and the image light L forms an image on the retina of the right eye RE and allows the user to view the image.

On the user's right eye RE are incident the image light L having been reflected off the half-silvered mirror261A and outside light OL having passed through the light control plate20A. On the user's left eye LE are incident the image light L having been reflected off the half-silvered mirror262A and outside light OL having passed through the light control plate20A. The HMD100thus causes the image light L, which carries an internally processed image, and the outside light OL to be superimposed on each other and incident on the user's eyes, whereby the user sees an outside scene through the light control plate20A and views the image carried by the image light L with the image superimposed on the outside scene. The HMD100thus functions as a see-through-type display apparatus.

The left projection system252and the left light guide plate262are also collectively called a “left light guide unit,” and the right projection system251and the right light guide plate261are also collectively called a “right light guide unit.” The configuration of each of the right light guide unit and the left light guide unit is not limited to the example described above and can be arbitrarily configured as long as the image light is used to form a virtual image in front of the user's eyes. For example, a diffraction grating may be used, or a semitransparent reflection film may be used.

The image display section20is connected to the control device10via a connection section40. The connection section40includes a body cord48, which is connected to the control device10, a right cord42, a left cord44, and a linkage member46. The right cord42and the left cord44are two cords into which the body cord48bifurcates. The right cord42is inserted into an enclosure of the right holder21through a lengthwise end portion AP of the right holder21and connected to the right display driver22. Similarly, the left cord44is inserted into an enclosure of the left holder23through a lengthwise end portion AP of the left holder23and connected to the left display driver24.

The linkage member46is disposed at the point where the body cord48bifurcates into the right cord42and the left cord44and has a jack for connecting an earphone plug30. A right earphone32and a left earphone34extend from the earphone plug30. The microphone63is provided in a position in the vicinity of the earphone plug30. An integrated single cord form the portion from the earphone plug30to the microphone63and bifurcates at the microphone63into two cords connected to the right earphone32and the left earphone34, respectively.

The microphone63is so disposed that a voice collection portion of the microphone63faces the directions of the user's sight lines, as shown, for example, inFIG. 3, and the microphone63collects voice and outputs a voice signal to a voice processing section187(FIG. 6). The microphone63may, for example, a monophonic microphone or a stereophonic microphone. The microphone63may be a directional microphone or an omni-directional microphone.

Each of the right cord42, the left cord44, and the body cord48may be any cord capable of transporting digital data and can be formed, for example, of a metal cable or an optical fiber. The right cord42and the left cord44may be integrated into a single cord.

The image display section20and the control device10transport a variety of signals to each other via the connection section40. Connectors (not shown) that fit into each other are provided at the end of the body cord48on the side opposite the linkage member46and at an end of the control device10. Causing the connector at the body cord48and the connector at the control device10to fit into each other and detach from each other allows the control device10and the image display section20to be connected to each other and disconnected from each other.

The control device10controls the HMD100. The control device10has a group of switches including a finalizing key11, a lighting section12, a display switch key13, a luminance switch key15, a direction key16, a menu key17, and a power switch18. The control device10further includes a trackpad14, operated by the user with a finger.

The finalizing key11detects pressing operation and outputs a signal that finalizes the content of the operation performed on the control device10. The lighting section12includes a light source, such as an LED (light emitting diode), and notifies the user of the state of action of the HMD100(whether the HMD100is powered on or off, for example) in the form of the lighting state of the light source. The display switch key13outputs a signal that instructs, for example, switching of an image display mode from one to another in accordance with pressing operation performed on the display switch key13.

The trackpad14has an operation surface that detects contact operation and outputs an operation signal according to the operation performed on the operation surface. A method for detecting operation performed on the operation surface is not limited to a specific method and can, for example, be an electrostatic method, a pressure detection method, or an optical method. The luminance switch key15outputs a signal that instructs an increase or a decrease in the luminance of an image displayed by the image display section20in accordance with pressing operation performed on the luminance switch key15. The direction key16outputs an operation signal in accordance with pressing operation performed on a key corresponding to any of the upward, downward, rightward, and leftward directions. The power switch18switches the power on/off state of the HMD100from one to the other.

FIGS. 5A and 5Bshow the configurations of key parts of the image display section20.FIG. 5Ais a perspective view of the key parts in a case where the image display section20is viewed from the user's head side, andFIG. 5Bis a descriptive diagram of the angles of view of the right camera61and the left camera62. InFIG. 5A, the right cord42and the left cord44, which are tied to the image display section20, and other components are omitted.

FIG. 5Ashows one side of the image display section20, that is, the side thereof facing the user's head, in other words, the side visible to the user's right eye RE and left eye LE. In another expression,FIG. 5Ashows the rear side of the right optical image display section26and the left optical image display section28.

In the HMD100, the half-silvered mirrors261A and262A reflect the image light to the user's eyes to form a display region, as shown inFIG. 5A. In the HMD100, the half-silvered mirrors261A and262A reflect the image light to allow the user's eyes to recognize a virtual image. Therefore, in the present embodiment, the display region is not the half-silvered mirrors261A and262A themselves but is a region where the user perceives the image light reflected off the half-silvered mirrors261A and262A. In a case where the image display section20has a configuration in which images are actually formed on the half-silvered mirrors261A and262A, the half-silvered mirrors261A and262A serve as the display region. The display region is a region which corresponds to the right LCD241and the left LCD242and where the user views images displayed on the LCDs241and242. For example, when the right LCD241and the left LCD242display images over entire regions thereof where images can be displayed (image displayable regions), the user is allowed to view an image of the size of the entire display region.

As described above, the half-silvered mirror261A, which irradiates the user's right eye RE with the image light, and the half-silvered mirror262A, which irradiates the user's left eye LE with the image light, allow the user to view a roughly rectangular display image. Further, the entire right optical image display section26and left optical image display section28including the half-silvered mirrors261A and262A transmits outside light, as described above. The user therefore views an outside scene that passes through the entire right optical image display section26and left optical image display section28and a display image formed by the image light reflected off the half-silvered mirrors261A and262A with the outside scene and the display image superimposed on each other.

The right camera61is disposed at an end portion facing the right holder21and oriented forward with respect to the image display section20, and the left camera62is disposed at an end portion facing the left holder23, as described above. The distance sensors64are disposed at the middle between the right optical image display section26and the left optical image display section28and oriented forward.

FIG. 5Bdiagrammatically shows the positions of the right camera61, the left camera62, and the distance sensors64in a plan view along with the user's right eye RE and left eye LE. InFIG. 5B, CR represents the angle of view (imaging range) of the right camera61, and CL represents the angle of view (imaging range) of the left camera62. The angles of view CR and CL shown inFIG. 5Bare angles of view in the horizontal direction, and it is noted that actual angles of view of the right camera61and the left camera62extend also in the upward/downward direction, as in the case of a typical digital camera.

The angles of view CR and CL are bilaterally symmetric with respect to the center position of the image display section20, and each of the angles of view CR and CL includes a direction passing through a point right in front of the center position of the image display section20. The angles of view CR and CL therefore overlap with each other in front of the center position of the image display section20.

For example, when a target OB is located in front of the image display section20, the target OB falls within both the angles of view CR and CL, as shown inFIG. 5B. The target OB is therefore displayed in both an image captured with the right camera61and an image captured with the left camera62. In this case, when the user gazes the target OB, the user's sight lines are directed to the target OB, as indicated by the reference characters RD and LD inFIG. 5B. In general, it is believed that the angle of field of a person is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Within these ranges, an effective field of view where a person has excellent information reception capability extends over a horizontal range of about 30 degrees and a vertical range of about 20 degrees. Further, it is believed that a stable field of fixation where a point of fixation at which a person gazes in a quick, stable manner extends over a horizontal range from about 60 to 90 degrees and a vertical range from about 45 to 70 degrees.

Therefore, when a point of fixation coincides with the target OB, the effective field of view extends over about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction, the stable field of fixation extends over about 60-90 degrees in the horizontal direction and about 45-70 degrees in the vertical direction, and the angle of field is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction, over portions on both sides of the sight lines RD and LD.

Further, an actual field of view viewed by the user who wears the HMD100through the image display section20and further through the right optical image display section26and the left optical image display section28is called an actual field of view (FOV). In the configuration of the present embodiment, the actual field of view corresponds to a field of view actually viewed by the user through the right optical image display section26and the left optical image display section28(field of view VR, which will be described later). The actual field of view is narrower than the angle of field and the stable field of fixation, which have been described with reference toFIG. 5B, but wider than the effective field of view.

Each of the right camera61and the left camera62is preferably capable of image capturing over a range wider than the user's field of view. Specifically, the sum of the angles of view CR and CL is preferably wider than at least the user's effective field of view. More preferably, the sum of the angles of view CR and CL is wider than the user's actual field of view. Still more preferably, the sum of the angles of view CR and CL is wider than the user's stable field of fixation. Most preferably, the sum of the angles of view CR and CL is wider than the user's angle of the field.

To this end, the right camera61and the left camera62are so disposed that the angles of view CR and CL overlap with each other in front of the image display section20, as shown inFIG. 5B. Each of the right camera61and the left camera62may be formed of a wide-angle camera. That is, each of the right camera61and the left camera62may include what is called a wide-angle lens as the imaging lens for image capturing over a wide angle of view. The wide-angle lens may include a lens called a super-wide-angle lens or a semi-wide-angle lens or may be a fixed-focal-length lens or a zoom lens, or each of the right camera61and the left camera62may include a lens group formed of a plurality of lenses. Further, the angle of view CR of the right camera61may not be equal to the angle of view CL of the left camera62. Moreover, the imaging direction of the right camera61is not necessarily completely parallel to the imaging direction of the left camera62. The right camera61and the left camera62only need to perform image capturing in such a way that the combination of an image captured with the right camera61and an image captured with the left camera62is wider than the user's field of view.

InFIG. 5B, the reference character64A denotes the detection direction of the distance sensors64. In the present embodiment, the distance sensors64are configured to be capable of detecting the distance from the center position of the image display section20to an object located in front thereof and, for example, detects the distance to the target OB. Since the user who wears the HMD100orients the head in a gazing direction, a target to be gazed is considered to be located in front of the image display section20. The distance sensors64, which are disposed at the center of the image display section20, can therefore detect the distance to a target gazed by the user, provided that the detection direction64A coincides with the forward direction with respect to the image display section20.

Further, sight line sensors68are disposed on the user's side of the image display section20, as shown inFIG. 5A. The sight line sensors68are provided as a pair in correspondence with the user's right eye RE and left eye LE in positions at the middle between the right optical image display section26and the left optical image display section28. The sight line sensors68are formed, for example, of a pair of cameras that captures images of the user's right eye RE and left eye LE. The sight line sensors68perform image capturing under the control of the control section140(FIG. 6), and the control section140detects, from captured image data, light reflected off the eyeball surfaces of the right eye RE and the left eye LE and images of the pupils to identify the directions of the sight lines.

FIG. 6is a functional block diagram of portions that form the HMD100.

The HMD100includes an interface125, which connects a variety of external apparatus OA, which serve as content supply sources, to the HMD100. The interface125can be an interface that supports wired connection, such as a USB interface, a micro-USB interface, and a memory card interface, and may instead be formed of a wireless communication interface. The external apparatus OA are each an image supply apparatus that supplies the HMD100with images and are, for example, a personal computer (PC), a mobile phone terminal, and a mobile game console.

The control device10includes the control section140, an input information acquisition section110, a storage section120, a transmitter (Tx)51, and a transmitter (Tx)52.

The input information acquisition section110is connected to an operation section135. The operation section135includes the trackpad14, the direction key16, the power switch18, and other components described above, and the input information acquisition section110acquires an input content on the basis of a signal inputted through the operation section135. The control device10further includes a power supply section (not shown) that supplies the portions in the control device10and the image display section20with electric power.

The storage section120is a nonvolatile storage device and stores a variety of computer programs and data associated with the programs. The storage section120may further store data on still images and motion images to be displayed in the image display section20.

The storage section120further stores setting data121. The setting data121contains a variety of setting values used by the control section140. The setting values contained in the setting data121may be values having been inputted in advance through operation of the operation section135or may be values received from any of the external apparatus OA or any other device (not shown) via a communication section117or the interface125and then stored.

The storage section120further stores a variety of data transmitted and received by the HMDs100A,100B, and100C.

FIG. 7diagrammatically shows data stored in the storage section120.

The storage section120stores the setting data121and transmission destination setting data122. The storage section120has a transmission data storage section126and a reception data storage section127as a storage region where data is stored. The transmission data storage section126and the reception data storage section127can be provided by virtual or logical partition of the storage region and therefore do not need to be classified, for example, as hardware.

The transmission data storage section126stores data transmitted by the HMD100to another HMD100. That is, data stored in the transmission data storage section126is transmitted via the communication section117to another HMD100. On the other hand, the reception data storage section127stores data received through the communication section117. The data received through the communication section117is data transmitted from another HMD100over the communication network4.

The transmission data storage section126stores visual field data131, notification data132, and action guidance data133as data to be transmitted. The reception data storage section127stores visual field data131, notification data132, and action guidance data133received through the communication section117.

The visual field data131, the notification data132, and the action guidance data133stored in the transmission data storage section126may be identical to the visual field data131, the notification data132, and the action guidance data133stored in the reception data storage section127. The data described above stored in the transmission data storage section126may differ from the data stored in the reception data storage section127in terms of header, attribute, data format, and other factors.

The visual field data131is part of image data captured with the right camera61and the left camera62, that is, data cut therefrom or image data containing the entire captured image data. That is, the visual field data131corresponds to an image of the outside that the user looks at, that is, an image of the visual field. The visual field data131may contain still image data, motion image (video image) data, and voice data. The visual field data131may instead contain voice data but no image data. The HMD100can transmit an image of the outside (visual field) that the user looks at by transmitting the visual field data131. Further, the HMD100can recreate the visual field that the user of another HMD100looks at by reproducing and displaying the visual field data131transmitted from the other HMD100.

The notification data132contains data representing a content inputted in the HMD100. The input to the HMD100is, for example, an input through operation of the control device10, a voice input using voice recognition, an input using knock operation on the image display section20, a gesture input achieved by placing a hand or any other indicating tool in the imaging ranges of the right camera61and the left camera62, as will be described later. The HMD100can transmit the input content to another HMD100by transmitting the notification data132. Further, the HMD100allows the user to grasp the content of an input in another HMD100by performing display based on the notification data132transmitted from another HMD100.

FIGS. 8A and 8Bare diagrammatic views showing an example of the configuration of the data stored in the storage section120.FIG. 8Ashows an example of the transmission destination setting data122, andFIG. 8Bshows an example of the configuration of the motion guidance data133.

The transmission destination setting data122is data that specifies the destination to which the HMD100transmits data. In the display system1shown inFIG. 1, the HMDs100A,100B, and100C can set different constraints on the destination to which data is transmitted. For example, the HMD100A is worn and used by the administrator UA, and the HMDs100B and100C are worn by the workers UB and UC. The destination to which data is transmitted can be set in accordance with whether the user of the HMD is the administrator or a worker. In the transmission destination setting data122, an ID that identifies an HMD100that is the transmission destination and the type of data to be transmitted are set in relation to each other.

FIG. 8Ashows an example of the transmission destination setting data122in the HMD100B used by the worker UB. The HMD100B can transmit the visual field data131and the notification data132to the HMD100A (ID:0001inFIG. 8A). The HMD100B can transmit only the notification data132to the HMD100C (ID:0003inFIG. 8A).

As a similar setting, for example, the transmission destination setting data122stored in the HMD100A is so set that notification data can be transmitted to the HMDs100B and100C. Further, for example, in the transmission destination setting data122stored in the HMD100C, the visual field data131and the notification data132can be transmitted to the HMD100A and the notification data132can be transmitted to the HMD100B.

The transmission destination setting data122may instead be data that makes setting of part of data stored and transmitted by the HMD100. For example, the motion guidance data133, which will be described later, can be configured not to be under constraint set by the transmission destination setting data122but to be transmittable to another HMD100in the display system1.

The motion guidance data133is data for displaying motion guidance or outputting voice to the user who wears an HMD100. The motion guidance data133contains guidance information133a, which gives guidance on the contents of actions, order information133b, which specifies the order in accordance with which the actions are performed, and state information133c, which shows the state of the performed actions, as shown inFIG. 8B.

The action performing order shown by the order information133bis a relative action order, and when the number of actions contained in the action guidance data133is one, no order needs to be set in relation thereto.

Each action contained in the action guidance data133is given an item (title). In the example shown inFIG. 8B, an item that also serves as the order information133bis given as follows: work1, work2, . . . . An action refers to an act or conduct performed by a user and may be a body motion of the user or may include thought and judgment. One action in the action guidance data133may in practice correspond to a sequence including a plurality of actions.

The guidance information133acontains a text, a still image, video images, voice, and other types of information outputted to the user who performs an action to describe or otherwise convey the content of the action.

The state information133cis data on the history of each performed action contained in the action guidance data133. For example, the state information133cis data on a progress history and a performing history of the guidance information133a. In the example shown inFIG. 8B, the state information133cis set to show “complete” representing that an action is complete, “under execution” representing a state in which an action has been initiated but has not been completed, and “not executed” representing that an action has not been initiated.

The state information133cchanges with the progress of an action performed by the user. The state information133cis updated in accordance with the progress of an action under the control of the control section140, as will be described later.

To the control section140are connected a three-axis sensor113, a GPS115, the communication section117, and a voice recognition section114. The three-axis sensor113is a three-axis acceleration sensor, and the control section140acquires a detection value from the three-axis sensor113. The three-axis sensor113allows the control section140to detect the motion of the control device10, for example, detect operation of swinging or otherwise moving the control device10. The three-axis sensor113may be replaced with a nine-axis sensor. In this case, the control section140can acquire detection values from a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis terrestrial magnetism sensor to detect, for example, the attitude, orientation, and motion, of the control device10.

The GPS115includes an antenna (not shown) and receives a GPS (global positioning system) signal to calculate the current position of the control device10. The GPS115outputs the current position and the current time determined on the basis of the GPS signal to the control section140. The GPS115may further have a function of acquiring the current time on the basis of information contained in the GPS signal to correct the time measured by the control section140.

The communication section117performs wireless data communication that complies with wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), Bluetooth (registered trademark), or any other standard.

When the external apparatus OA is wirelessly connected to the communication section117, the control section140acquires content data via the communication section117and causes the image display section20to display an image. On the other hand, when the external apparatus OA is wired to the interface125, the control section140acquires content data via the interface125and causes the image display section20to display an image. The communication section117and the interface125function as a data acquisition section DA, which acquires content data from the external apparatus OA.

The control section140includes a CPU (not shown) that executes a program, a RAM (not shown) that temporarily stores the program executed by the CPU and data, and a ROM (not shown) that stores a basic control program executed by the CPU and data in a nonvolatile manner. The control section140reads and executes the computer programs stored in the storage section120to function as an operating system (OS)150and an image processing section160. The control section140further functions as the imaging control section161, an input recognition section162(recognition section), an AR display control section163, a state detection section164, a communication control section165, an evaluation section166, the voice processing section187, and a display control section190.

The image processing section160acquires an image signal contained in a content. The image processing section160separates a vertical sync signal. VSync, a horizontal sync signal HSync, and other sync signals from the acquired image signal. Further, the image processing section160produces a clock signal PCLK, for example, by using a PLL (phase locked loop) circuit (not shown) in accordance with the cycles of the separated vertical sync signal VSync and horizontal sync signal HSync. The image processing section160converts the analog image signal from which the sync signals are separated into a digital image signal, for example, by using an A/D conversion circuit (not shown). The image processing section160then stores the converted digital image signal as image data on a target image (Data inFIG. 6) in the RAM in the control section140on a frame basis. The image data is, for example, RGB data.

The image processing section160may perform, as required, resolution conversion in which the resolution of the image data is converted into resolution suitable for the right display driver22and the left display driver24. The image processing section160may further perform image adjustment in which the luminance and chroma of the image data are adjusted, 2D/3D conversion in which 2D image data is created from 3D image data or 3D image data is created from 2D image data.

The image processing section160transmits the clock signal PCLK, the vertical sync signal VSync, the horizontal sync signal HSync, and the image data Data stored in the RAM via the transmitters51and52. Each of the transmitters51and52functions as a transceiver for serial transportation between the control device10and the image display section20. The image data Data transmitted via the transmitter51is also called “image data for the right eye,” and the image data Data transmitted via the transmitter52is also called “image data for the left eye.”

The display control section190produces control signals that control the right display driver22and the left display driver24, and the control signals control the right display driver22and the left display driver24to cause them to produce and output image light. Specifically, the display control section190controls a right LCD control section211to cause it to drive the right LCD241or not and controls a right backlight control section201to cause it to drive the right backlight221or not. The display control section190further controls a left LCD control section212to cause it to drive the left LCD242or not and controls a left backlight control section202to cause it to drive the left backlight222or not.

The voice processing section187acquires a voice signal contained in the content, amplifies the acquired voice signal, and outputs the amplified voice signal to the right earphone32and the left earphone34. The voice processing section187further acquires voice collected through the microphone63and converts the collected voice into digital voice data. The voice processing section187may perform preset processing on the digital voice data.

The image display section20includes the right camera61, the left camera62, and the distance sensors64described above. The image display section20further includes an interface25, the right display driver22, the left display driver24, the right light guide plate261as the right optical image display section26, the left light guide plate262as the left optical image display section28, a nine-axis sensor66, and the sight line sensors68.

The nine-axis sensor66(motion detection section) is a motion sensor that detects acceleration (three axes), angular velocity (three axes), and terrestrial magnetism (three axes). The control section140can detect motion of the head of the user who wears the image display section20around the head on the basis of a detection value from the nine-axis sensor66. For example, the control section140can estimate the magnitude of inclination of the image display section20and the orientation of the inclination on the basis of a detection value from the nine-axis sensor66.

The interface25includes a connector to which the right cord42and the left cord44are connected. The interface25outputs the clock signal PCLK, the vertical sync signal VSync, the horizontal sync signal HSync, and the image data Data transmitted from the transmitters51and52to corresponding receivers (Rx)53and54. The interface25further outputs the control signals transmitted from the display control section190to the corresponding receivers53,54, right backlight control section201, or left backlight control section202.

The interface25is an interface that connects the right camera61, the left camera62, the distance sensors64, the nine-axis sensor66, and the sight line sensors68to the control section140. Image data captured with the right camera61and the left camera62, a result of detection performed by the distance sensors64, a result of detection of acceleration (three axes), angular velocity (three axes), and terrestrial magnetism (three axes) from the nine-axis sensor66, and a result of detection performed by the sight line sensors68are sent to the control section140via the interface25.

The right display driver22includes the right backlight221, the right LCD241, and the right projection system251described above. The right display driver22further includes the receiver53, the right backlight (BL) control section201, which controls the right backlight (BL)221, and the right LCD control section211, which drives the right LCD241.

The receiver53operates as a receiver corresponding to the transmitter51and performs serial transportation between the control device10and the image display section20. The right backlight control section201drives the right backlight221on the basis of the inputted control signal. The right LCD control section211drives the right LCD241on the basis of the clock signal PCLK, the vertical sync signal VSync, the horizontal sync signal HSync, and the image data Data for the right eye, which are inputted via the receiver53.

The left display driver24has the same configuration as that of the right display driver22. The left display driver24includes the left backlight222, the left LCD242, and the left projection system252described above. The left display driver24further includes the receiver54, the left backlight control section202, which drives the left backlight222, and the left LCD control section212, which drives the left LCD242.

The receiver54operates as a receiver corresponding to the transmitter52and performs serial transportation between the control device10and the image display section20. The left backlight control section202drives the left backlight222on the basis of the inputted control signal. The left LCD control section212drives the left LCD242on the basis of the clock signal PCLK, the vertical sync signal VSync, the horizontal sync signal HSync, and the image data Data for the left eye, which are inputted via the receiver54.

The right backlight control section201, the right LCD control section211, the right backlight221, and the right LCD241are also collectively referred to as a right “image light generation unit.” Similarly, the left backlight control section202, the left LCD control section212, the left backlight222, and the left LCD242are also collectively referred to as a left “image light generation unit.”

The imaging control section161controls the right camera61and the left camera62to cause them to perform image capturing for captured image data acquisition. The imaging control section161may cause only one of the right camera61and the left camera62to perform image capturing or may cause both the right camera61and the left camera62to perform image capturing.

The input recognition section162detects and recognizes an input to the HMD100. The input recognition section162detects operation performed on the operation section135on the basis of a signal inputted from the input information acquisition section110. Further, when the voice processing section187analyzes voice collected through the microphone63, and a result of the analysis shows that the voice is a voice command set in advance, the input recognition section162detects the input of the voice command. Further, when the input recognition section162detects an image of an indicating tool, such as the user's hand, from images captured with the right camera61and/or the left camera62and determines that the position, orientation, or motion of the indicating tool corresponds to a gesture set in advance, the input recognition section162detects the gesture input. Further, when the input recognition section162determines that a pattern of detection signals from the nine-axis sensor66provided in the image display section20corresponds to operation of knocking the image display section20, the input recognition section162detects that the knock operation on the image display section20as an input. Further, when the input recognition section162detects an image of an input marker, such as a two-dimensional code or a barcode, from images captured with the right camera61and/or the left camera62, the input recognition section162detects the marker reading input. When the input recognition section162detects an indicating tool or a marker from image data captured with the right camera61and/or the left camera62, the input recognition section162may detect that the indicating tool or marker corresponding to the direction of the user's sight lines detected with the sight line sensors68.

The HMD100may include a foot switch (not shown) operated by the user with a foot. The foot switch may be wired to the control section140or the operation section135or may be connected to the communication section117over wireless communication, such as Bluetooth (registered trademark). In this case, the input recognition section162detects operation of the foot switch and recognizes the operation as an input.

The AR display control section163reads data stored in the storage section120and controls the image processing section160and the display control section170to cause the image display section20to display an image or a text. The AR display control section163may AR-display, 3D-display (stereoscopically display), or 2D-display (planarly display) the image or the text in a position that coincides with the position of a target in an actual space present in the user's field of view. Further, the AR display control section163may control the voice processing section187on the basis of the data read from the storage section120to cause the right earphone32and the left earphone34to output voice.

Specifically, the AR display control section163displays a still image or video images in the visual field of a user of another HMD100on the basis of the visual field data131stored in the reception data storage section127. Voice may be further outputted in this process.

The AR display control section163further reads the action guidance data133in the reception data storage section127and displays a still image, motion images, a text, or any other type of information that gives guidance on an action on the basis of the action guidance data133. In this process, the AR display control section163may display the contents of guidance information133acorresponding to a plurality of actions in the form of a list in which the contents of information are arranged in accordance with the order specified by the order information133b. The AR display control section163may instead display the contents of guidance information133aby switching one content to another or one set of a preset number of contents to another set in accordance with the order specified by the order information133b.

The state detection section164detects the state of a performed action (work) corresponding to the guidance information133ain the action guidance data133during a period for which the AR display control section163performs display based on the action guidance data133.

The performed action state detection performed by the state detection section164is performed, for example, by analysis of image data captured with the right camera61and/or the left camera62. In this case, the state detection section164detects at least one of a target on which an action is performed, a gadget or a tool used to perform the action, the user's body part, or any other object from the captured image data. The state detection section164generates data on the state of the performed action on the basis of the position, size, color, shape, and other factors of the target on which the action is performed, the gadget or the tool used to perform the action, the user's body part, or any other object.

Instead, in the state in which the AR display control section163displays an item in the action guidance data133, the state detection section164may detect the state of a performed action corresponding to the item on the basis of an input detected by the input recognition section162.

When the input recognition section162detects an indicating tool or a marker from image data captured with the right camera61and/or the left camera62, a characteristic quantity set in advance may be used. The same holds true for the case where the state detection section164detects the position, size, color, shape, and other factors of a target on which an action is performed, a gadget or a tool used to perform the action, the user's body part, or any other object from image data captured with the right camera61and/or the left camera62. The characteristic quantity is data used to detect an image from a captured image and is a characteristic quantity or any other parameter of the image. For example, to detect a target that is an object, the setting data contains a characteristic quantity representing the color, shape, or any other characteristic of a captured image of the object. In this case, the input recognition section162and the state detection section164extract an image of the object from image data on an outside scene image, calculate a characteristic quantity of the extracted image of the object, and compare the calculated characteristic quantity with the characteristic quantity contained in the setting data121and verify the degree of coincidence. When the comparison shows close similarity or complete coincidence, the object in the image extracted from the outside scene image can be recognized as the target. When the setting data121contains a plurality of characteristic quantities of a target, the input recognition section162and the state detection section164can detect the target from an outside scene image on the basis of the plurality of characteristic quantities and recognize the target.

How the input recognition section162and the state detection section164recognize a target and an indicating tool from an image is not limited to the method for recognizing an image of the target on the basis of a characteristic quantity of the image as described above. For example, the user may indicate a target or an indicating tool, for example, from objects contained in an outside scene image to select the target or the indicating tool. In this case, the indication performed by the user may be voice indication, and the voice processing section187converts the voice collected through the microphone63into a text, which allows the input recognition section162and the state detection section164to acquire information for recognition and identification of the target. For example, when voice that specifies a characteristic of a target in a captured image, such as the color or the shape of the target, is converted into a text, the input recognition section162and the state detection section164detect an image corresponding to the specified characteristic from the captured image and recognize the image. A method for inputting information on a target can be a method using an input detectable by the input recognition section162, as described above.

The evaluation section166evaluates the state of a user of the HMD100. More specifically, the evaluation section166evaluates whether or not the state of the user is a state in which the notification data132should be transmitted from the HMD100to another HMD100. The reference in accordance with or the condition under which it is determined that the notification data132should be transmitted is set in advance and stored, for example, in the storage section120as the setting data. For example, the reference in accordance with which the determination is made can be a situation in which the input recognition section162has detected and recognized a predetermined input. In this case, when the input recognition section162has recognized an input corresponding to a condition set in advance, the evaluation section166determines that the user's state is the state in which the notification data132should be transmitted.

The reference or the condition used by the evaluation section166for the evaluation is not limited to a content relating to an input recognized by the input recognition section162. For example, the evaluation section166may evaluate whether or not motion of the user's head or body corresponds to preset motion on the basis of a detection value from the nine-axis sensor66. Instead, the evaluation section166may evaluate whether or not motion of the user's sight lines corresponds to preset motion on the basis of a detection value from the sight line sensors68. Still instead, the HMD100may be provided with a vital sensor, such as an electromyograph (not shown), a pulse measurement apparatus (not shown), a blood pressure measurement apparatus (not shown), or a blood oxygen level measurement apparatus (not shown), and the evaluation section166may perform the evaluation on the basis of a measurement value or a detection value from the vital sensor. In any of these cases, even when the user has no intention of transmitting the notification data132, the state of the user's body can be evaluated. For example, when the user desires to transmit the notification data132but cannot issue an input recognizable by the input recognition section162, the evaluation section166can evaluate the user's state.

Further, the evaluation section166may learn an input recognized by the input recognition section162, detection values from the nine-axis sensor66and the sight line sensors68, and a detection value from the vital sensor and set or change the reference in accordance with or the condition under which the user's state is evaluated. The learning may be autonomous learning by providing the evaluation section166with an AI (artificial intelligence) engine. The evaluation section166may instead optimize the evaluation reference or condition in accordance with an algorithm set by the evaluation section166in advance.

In the example shown inFIG. 2, when a problem occurs in the work performed by the worker UB, it is appropriate that the worker UB transmits the notification data132to the administrator UA or the worker UC, but it is conceivable that the worker UB is absorbed in solving the problem and does not operate the HMD100. In this case, evaluation of the state of the worker UB allows not only detection of occurrence of a problem in the work line and identification of the location where the problem has occurred but also assistance or other countermeasures taken by another worker or the administrator. Subsequent transmission of the notification data132and images captured with the right camera61and/or the left camera62allows the administrator UA or the worker UC other than the worker UB, who is the person in trouble, to quickly grasp the situation of the problem having occurred.

The communication control section165controls the communication section117to cause it to perform wireless communication with a wireless access point (not shown) or any other device connected to the communication network4.

When the plurality of HMDs100that form the display system1are located in positions close to one another, that is, when the plurality of HMDs100are located in a range over which the communication section117of one of the HMDs100can transmit and receive a wireless signal to and from the communication section117of another HMD100, direct wireless communication between the communication sections117may be performed without via the communication network4.

FIGS. 9A and 9Bare flowcharts showing the action of the display system1.

The following description will be made of the case where the administrator UA, who administers work, wears and uses the HMD100A, the worker UB wears and uses the HMD100B, and the worker15C wears and uses the HMD100C, as shown inFIGS. 1 and 2.

FIG. 9Ashows the action of the HMD100B, andFIG. 9Bshows the action of the HMD100A.

To start work, an action start instruction is inputted to the HMD100A (step S31), and the action start instruction is inputted also to the HMD100B (step S11).

The HMD100A transmits the action guidance data133stored in advance in the transmission data storage section126(step S32), and the HMD100B receives the action guidance data133and stores it in the reception data storage section127(step S12).

The AR display control section163of the HMD100B starts, for example, displaying a still image, motion images, a text, or any other type of information and/or outputting voice on the basis of the action guidance data133in the reception data storage section127(step S13). The worker UB, who wears the HMD100B, can therefore perform work on the target OB (FIG. 2) in accordance with guidance based on the action guidance data133.

The input recognition section162of the HMD100B starts image capturing by using the right camera61and/or the left camera62and voice detection by using the voice processing section187when the work starts (step S14).

The HMD100B uses the function of the input recognition section162to generate the visual field data131from image data captured with the right camera61and/or the left camera62, stores the visual field data131in the transmission data storage section126, and transmits the visual field data131to the HMD100A (step S15). The cycle of the generation and transmission of the visual field data131is specified by data contained in the action guidance data133or the setting data121.

The input recognition section162evaluates whether or not an input has been detected (step S17). When an input has been detected (step S17: YES), the input recognition section162generates notification data132corresponding to the content of the input, stores the notification data132in the transmission data storage section126, transmits the notification data132to the HMD100A (step S18), and transitions to step S19. When a result of the evaluation shows that no input has been detected (step S17: NO), the input recognition section162transitions to step S19.

In step S19, the state detection section164evaluates whether or not the action state has changed (step S19). When the action state has changed (step S19: YES), for example, when an action has been initiated or when an action has been completed, the state detection section164updates the state information133cin the action guidance data133stored in the reception data storage section127(step S20) and transitions to step S21. When the action state has not changed (step S19: NO), the state detection section164transitions to step S21.

When the visual field data131has been transmitted from the HMD100B to the HMD100A, the HMD100A receives the visual field data131and stores it in the reception data storage section127(step S34). The AR display control section163of the HMD100A outputs a still image, motion images, voice, or any other type of information corresponding to the visual field of the worker UB who wears the HMD100B on the basis of the received visual field data131(step S35).

When the HMD100B has transmitted the notification data132to the HMD100A, the HMD100A receives the notification data132and stores it in the reception data storage section127(step S36). The AR display control section163of the HMD100A outputs a text, a still image, motion images, voice, or any other type of information representing the content of the input in the HMD100B on the basis of the received notification data132(step S37).

The input recognition section162of the HMD100A evaluates whether or not an input has been detected (step S38). When an input has been detected (step S38: YES), the input recognition section162generates notification data132corresponding to the content of the input, stores the notification data132in the transmission data storage section126, transmits the notification data132to the HMD100B (step S39), and transitions to step S40. When a result of the evaluation shows that no input has been detected (step S38: NO), the input recognition section162transitions to step S40.

In step S39, the input recognition section162of the HMD100A may transmit the notification data132to the plurality of HMDs100B and100C at the same time. In this case, the same notification data132may be transmitted to the plurality of HMDs100B and100C, or different pieces of notification data132may be generated and transmitted to the plurality of HMDs100B and100C. The transmission destination may be specified in accordance with an input from the administrator UA whenever the transmission is performed or may be determined in accordance with the transmission destination setting data122.

In step S23, the control section140of the HMD100B evaluates whether or not an end condition has been satisfied (step S23). The end condition is satisfied, for example, when all actions contained in the action guidance data133received in step S12have been completed or when the input recognition section162has detected an end instruction input. When the end condition has not been satisfied (step S23: NO), the control section140returns to step S15. When the end condition has been satisfied (step S23: YES), the control section140terminates the present procedure. At the time of termination of the procedure, the control section140may duplicate the action guidance data133stored in the reception data storage section127, store the duplicated action guidance data133in the transmission data storage section126, and transmit the duplicated action guidance data133to the HMD100A. Further, the control section140may transmit notification data132that notifies the HMD100A of the action termination.

In step S40, the control section140of the HMD100A evaluates whether or not an end condition has been satisfied (step S40). The end condition is satisfied, for example, when the HMD100B has notified the HMD100A of action termination or when the input recognition section162has detected an end instruction input. When the end condition has not been satisfied (step S40: NO), the control section140returns to step S34. When the end condition has been satisfied (step S40: YES), the control section140terminates the present procedure. At the time of termination of the procedure, the control section140may transmit notification data132that notifies the termination to the other HMDs100contained in the display system1. In this case, an HMD100other than the HMD100A used by the administrator UA may be set to terminate any action when the HMD100receives the notification data132that notifies the termination from the HMD100A.

In the flowchart shown inFIG. 9A, the HMD100B carries out the process of detecting an input (step S17), the process of detecting a change in the action state (step S19), the process of receiving the notification data132(step S21), and the process of evaluating the termination (step S23) sequentially on a step basis, but the invention is not necessarily configured this way. The action of each of the steps described above may be performed as an interruption process. That is, the control section140of the HMD100B starts operating in step S11and is then ready for the input detection (step S17), the detection of a change in the action state (step S19), the reception of the notification data132(step S21), and the instruction of the termination (step S23). When an input or an action is detected, the action of the corresponding one of the steps S17, S19, S21, and S23may be initiated as an interruption process. The same holds true for the steps S34, S38, S39, and S40inFIG. 9B.

FIGS. 10A and 10Bare flowcharts showing the action of the display system1and show an action in which data on actions are taken over between the HMDs100B and100C used by the workers UB and UC who perform the actions.FIG. 10Ashows the action of the HMD100B, andFIG. 10Bshows the action of the HMD100C.FIGS. 10A and 10Bshow a case where an action is taken over from the HMD100B to the HMD100C.

The action inFIGS. 10A and 10Bis performed during a period for which the HMD100B performs an action in accordance with the action guidance data133, that is, during a period between steps S13to S23inFIG. 9A.

The input recognition section162of the HMD100B detects an input of an action takeover instruction (step S51) and then detects an input of an action takeover request to an HMD100that takes over the action (step S52). The input recognition section162generates notification data132on the basis of the content of the input relating to the action takeover request, stores the notification data132in the transmission data storage section126, and transmits the notification data132to the HMD100C (step S53).

The AR display control section163of the HMD100C receives the notification data132from the HMD100B, stores the notification data132in the reception data storage section127(step S61), and outputs a text, a still image, motion images, voice, or any other type of information on the basis of the received notification data132(step S62).

The control section140of the HMD100B duplicates the action guidance data133stored in the reception data storage section127, stores the duplicated action guidance data133in the transmission data storage section126, and transmits the duplicated action guidance data133to the HMD100C (step S54). The action guidance data133is data having been stored in the reception data storage section127and updated in accordance with the progress of the action in the HMD100B.

The control section140of the HMD100C receives the action guidance data133and stores it in the reception data storage section127(step S63). The AR display control section163produces a notification of the reception of the action guidance data133, transmits the notification to the HMD100B (step S64), and starts, for example, displaying a still image, motion images, a text, or any other type of information and/or outputting voice on the basis of the received action guidance data133(step S65). At this point, the AR display control section163outputs, in accordance with the state information133ccontained in the received action guidance data133, the guidance information133aformed of an action under execution or an action not having been performed. The worker UC, who wears the HMD100C, can therefore follow the action having been performed by the worker UB, who wears the HMD100B, and perform an action on the target OB (FIG. 2) in accordance with the guidance based on the action guidance data133.

The control section140of the HMD100B receives the notification relating to the reception of the action guidance data133from the HMD100C (step S55) and terminates the present procedure. At this point, the control section140of the HMD100B may delete the action guidance data133stored in the reception data storage section127. The control section140of the HMD100B may also delete the action guidance data133stored in the transmission data storage section126. Since the action guidance data133is therefore not redundantly held by a plurality of HMDs100, a situation in which actions are mixed up can be avoided.

The input of the action takeover instruction and the action takeover request in steps S11and S12may be performed in the HMD100A. In this case, notification data132on the action takeover instruction is transmitted from the HMD100A to the HMDs100B and100C, and the HMDs100B and100C may start the action inFIGS. 10A and 10Bin accordance with the notification data132.

FIGS. 11A to 11Cshow examples of display in the display system.FIG. 11Ashows an example in which the HMDs100B and100C perform display for the workers UB and UC, andFIG. 11Bshows another example in which the HMDs100B and100C perform display for the workers UB and UC.FIG. 11Cshows an example in which the HMD100A performs display.

InFIGS. 10A to 10C, reference character VR denotes the field of view of a user (administrator UA, workers UB and UC), and reference character V1denotes a region in which an HMD100displays an image and allows the user to view the image, that is, the display region of the image display section20. The display region V1is, for example, located roughly at the center of the user's field of view VR and narrower than the field of view VR. The size of the display region V1may instead be equal to the size of the field of view VR, and the size and the position of the display region V1are not limited to those in the examples shown inFIGS. 11A to 11C.

InFIG. 11A, the AR display control section163displays a list SR, which lists the contents of the guidance information133ain the action guidance data133. The list SR is a list in which the contents of the guidance information133acontained in the action guidance data133are arranged in accordance with the order specified by the order information133b. The list SR may display the titles and contents of the of guidance information133a. Instead, the order specified by the order information133bmay be displayed.

In the list SR, check boxes CH are disposed in correspondence with the positions where the contents of guidance information133aare displayed. Each of the check boxes CH, when it is checked, indicates that the corresponding action in the action guidance data133has been completed. When the state detection section164detects that an action has been completed, the AR display control section163changes the display state of the corresponding check box CH to a state showing action complete. In response to the change, the state detection section164may update the state information133cin the action guidance data133in the reception data storage section127. When the number of contents of guidance information133acontained in the action guidance data133is one, the list SR contains the content of the one action and the check box CH associated therewith.

The list SR can instead have a configuration with no check box CH. In this case, the AR display control section163may delete, from the list SR, an item corresponding to an action detected by the state detection section164that the action has been completed. Instead, the color or brightness displayed in the list SR may be changed.

The example shown inFIG. 11Bis a case where the contents of the guidance information133ain the action guidance data133is displayed one by one or on a preset number basis. In this example, the contents of the guidance information133aare displayed one by one in the display region V1and in a balloon-shaped display section D. The edge of the display section D may be drawn in the form of an arrow that points an action target position of the target OB. Instead, a marker M may be displayed in a position on which an action corresponding to the guidance information133adisplayed in the display section D is performed. In the example shown inFIG. 11B, when the state detection section164detects that an action corresponding to the guidance information133adisplayed in the display section D is completed, the AR display control section163displays the following guidance information133acontained in the action guidance data133in the display section D. The itemized actions in the guidance information133aare thus sequentially displayed in the display section D in accordance with the order in accordance with which the actions are performed.

The display position and display form of each of the list SR and the display section D can be changed as appropriate. For example, each of the list SR and the display section D may be displayed in a position where the list SR or the display section D overlaps with the target OB on which an action corresponding to an item in the action guidance data133is performed. However, displaying the list SR or the display section D in a position where the list SR or the display section D does not overlap with the target OB is advantageous in that the list SR or the display section D does not prevent the action and allows the action to be readily performed while the item is viewed. Instead, the list SR or the display section D may be displayed in a 3D (stereoscopic) form or as a 2D (planar) image. Still instead, the list SR or the display section D may be displayed only by one of the right display driver22and the left display driver24.

InFIGS. 11A and 11B, a pointer P, which indicates the direction of the user's sight lines detected with the sight line sensors68, may be displayed in the display region V1.

The example shown inFIG. 11Cis an example in which the HMD100A displays still images based on the visual field data131transmitted from the HMDs100B and100C. In the display region V1, a visual field display VW1based on the visual field data131from the HMD100B and a visual field display VW2based on the visual field data131from the HMD100C are displayed side by side. In the visual field displays VW1and VW2, IDs representing the HMDs100B and100C, which are transmission sources, are displayed in correspondence therewith. In this case, the administrator UA can view the visual fields of the workers UB and UC at the same time.

Further, in the visual field displays VW1and VW2are displayed the pointer P indicating the direction of the sight lines of the worker UB and the pointer P indicating the direction of the sight lines of the worker UC, respectively. The pointer P can be displayed when the visual field data131contains data representing the position where the pointer P is displayed.

As described above, in the display system1, among the plurality of HMDs100, the HMD100A, which has been set in advance as the HMD100for an administrator, can display the visual fields of the other HMDs100B and100C in the form of images. Further, the content of an input or a content corresponding to the input in each of the HMDs100B and100C is transmitted to the HMD100A in the form of the notification data132, and the HMD100A can output the notification data132. Therefore, in the display system1, the action guidance data133allows assistance of actions performed by a plurality of performers who perform the actions and grasp of situations in which the actions are performed.

The input to the HMD100can be performed in the form of voice, as described above, and a gesture can be detected for GUI operation of the HMD100. Further, input through a foot switch and knock operation performed on the image display section20are also allowed. Input contents corresponding to the input to a foot switch and the input by the knock operation may be set in advance in the input recognition section162. For example, when the foot switch is turned on, a message set in advance by the setting data121may be transmitted as the notification data132. Further, for example, knock operation performed on the image display section20may trigger transmission of image data captured with the right camera61and the left camera62to the HMD100A as the notification data132along with a message or after preset image processing is performed on the image data. In this case, the foot switch, the knock operation, or any other input operation allows emergency contact with the administrator UA, a request for assistance, or any other action. Further, the transmission destination in the case where input through the foot switch, input of the knock operation, or any other type of input is detected may be set by the transmission destination setting data122. In this case, for example, in the event of an emergency, an emergency message can be transmitted from one HMD100to the other HMDs100for display of the emergency message.

The assistance and administration of the workers (performers) using the plurality of HMDs100can be administered by the HMD100A in the administrator's site A provided in a separate room or at a remote location, whereby the point at which a performer's gaze is directed and the content of an action of the performer can be checked. The work in the work line FA can be quickly checked, for example, by measurement of lost time for each performer who wears an HMD100and discovery of occurrence of a defect due to work failure. In this case, lost time in the work line FA can be shortened, and the overall throughput can be improved.

Further, causing the plurality of HMDs100to provide output based on the action guidance data133allows work assistance images to be displayed as teaching images for unification of work order and synchronization of workers with one another. When a performer needs to be replaced, work can be taken over between HMDs100, as described with reference toFIGS. 10A and 10B, whereby the same work order can be taken over.

Further, making use of the display system1allows evaluation of whether or not the worker who uses any of the HMDs100among the HMDs100connected to one another via the communication network4is idle. The administrator UA can therefore issue information that allows efficient supply of a necessary member to an idle worker's site.

Further, the function of transmitting the notification data132from the HMD100A to each of the HMDs100B and100C and causing the HMD100to output the notification can be used to cause the image display section20to display an indicator as a sign that prompts supply of a necessary member. Exchange of information among the wearers of the HMDs100as described above allows smooth supply of parts, work takeover, and other types of operation, whereby a series of work can be continuously maintained.

FIGS. 12A to 12Dare descriptive diagrams showing a specific application example of the invention. In the specific application example, a plurality of workers UB, UC, and UD wear HMDs100, an administrator UA wears an HMD100, and the workers UB, UC, and UD work under the administration of the administrator UA, as in the work line shown inFIG. 2. The HMD100worn by the administrator UA is called an HMD100A, and the HMDs100worn by the workers UB, UC, and UD are called HMDs100B,100C, and100D, respectively. Each of the HMDs100A,100B,100C, and100D has the configuration of the HMD100described above.

When a trouble has occurred in the work performed by the worker UB, and the evaluation section166of the HMD100B has determined that the state of the worker UB is the state in which notification data132should be transmitted, the notification data132is transmitted from the HMD100B to the HMD100A (step ST1), as shown inFIG. 12A. The destination to which the notification data132is transmitted is set by the transmission destination setting data122described above and is, for example, a group including the HMDs100A,100C, and100D. The notification data132can notify the administrator UA and the workers UC and UD of the occurrence of a problem with the worker UB.

Subsequently, the administrator UA operates the HMD100A to cause it to transmit notification data132containing an instruction to the HMD100C (step ST2), as shown in FIG.12B. Instead, the worker UC operates the HMD100C to cause it to transmit notification data132on assistance offer to the HMD100A (step ST3). The mutual transmission and reception of the notification data132allow determination of replacement of the worker UB with the worker UC.

Thereafter, the administrator UA operates the HMD100A to cause it to transmit notification data132representing that the worker UC replaces the worker UB to the HMDs100B and100D (step ST4), as shown inFIG. 12C.

The worker UB then leaves the work line, and the worker UC moves to the work site in place of the worker UB. At this point, the worker UC provides an input to the HMD100C to cause it to transmit notification data132representing completion of the replacement to the HMDs100A and100D.

When the worker UB leaves, action guidance data may be transmitted from the HMD100B to the HMD100C in a period between the state inFIG. 12Cand the state inFIG. 12D.

As described above, application of the invention allows quick discovery of time lost by a single user in in-line work and occurrence of a defect due to work failure, whereby lost time in the entire in-line work can be shortened, and the overall throughput can be improved. Each worker can readily and quickly share and check the work of the worker and action of conduct of the worker with another HMD wearer for improvement in reliability of team work and conduct. Further, the users who wear the HMDs100can share information, and an HMD100can share information with the other HMDs100in parallel to one another, in a loop form, or in series. An administrator is not necessarily stationed in a vicinity of a work line and may be stationed in any position where the administrator can administer the work line, and the administrator s HMD100can further monopolistically collect and administer information. For example, setting the transmission destination setting data122and the reference or condition of evaluation made by the evaluation section166allow proper use of the range over which the information is shared and the procedure in accordance with which information is shared depending on TPO (use application).

As described above, the display system1according to the embodiment to which the invention is applied includes a plurality of HMDs100, which are head-mounted-type display apparatus. Each of the HMDs100includes the image display section20, which allows a user to view an image and transmits an outside scene, and the communication section117, which communicates with the other HMDs100. Each of the HMDs100further includes the right camera61and the left camera62, which perform image capturing over a range that overlaps with the user's visual field, the evaluation section166, which evaluates the user's state, and the control section140. The control section140transmits the visual field data131on the basis of the captured images to another HMD100and transmits the notification data132in accordance with a result of the evaluation performed by the evaluation section166to another HMD100via the communication section117. The control section140further causes the image display section20to perform display on the basis of data transmitted from another HMD100. In the display system1, data on the visual fields and data on inputs can therefore be shared among the plurality of HMDs100. As a result, a plurality of users who wear the HMDs100can share the visual fields and operation. For example, a person who performs an action and a person who assists the action wear the HMDs100, which are head-mounted-type display apparatus, for conveyance of a large amount of information on actions in a more understandable manner. A user can therefore assist another user who performs work or any other type of act or readily and accurately administer the state in which work or any other type of act is performed.

Since the input recognition section162recognizes an input on the basis of voice detected by the voice processing section187, the HMD100can accept an input in the form of voice, whereby information can be more readily conveyed among a plurality of HMDs100.

The control section140transmits data representing an input recognized by the input recognition section162on the basis of voice detected by the voice processing section187to another HMD100. A content inputted in the form of voice to an HMD100can therefore be notified to a user who wears another HMD100.

The HMD100includes the input recognition section162, which recognizes an input, and the evaluation section166evaluates the state of the user on the basis of an input recognized by the input recognition section162. The HMD100can therefore recognize an input from the user or another person to appropriately evaluate the state of the user. Further, the user or another person allows the HMD100to recognize an input to intentionally cause the HMD100to transmit notification data to another apparatus.

The HMD100further includes the nine-axis sensor66, which detects motion of the image display section20, and the input recognition section162may instead recognize an input on the basis of motion detected with the nine-axis sensor66. In this case, the input recognition section162can accept an input according to motion of the image display section20or motion of the user who wears the image display section20.

The input recognition section162may instead recognize an input on the basis of images captured with the right camera61and the left camera62. In this case, the input recognition section162can more readily accept an input.

The control section140carries out a reproduction process of reproducing the action guidance data133including the guidance information133a, which gives guidance on actions, the order information133b, which specifies the order in accordance with which the actions are performed, and the state information133c, which shows the state of the performed actions. In the reproduction process, the control section140causes the image display section20to display a text, a still image, motion images, or any other type of information. The control section140updates, on the basis of an input recognized by the input recognition section162, the state information133cin the action guidance data133being reproduced and causes the communication section117to transmit the action guidance data133having undergone the reproduction process to another HMD100. As a result, action guidance can be given to a user who wears an HMD100and performs an action, and data containing records on the action performed in accordance with the guidance can be taken over and used by the other HMD100.

The invention is not limited to the configuration of the embodiment described above and can be implemented in a variety of aspects to the extent that they do not depart from the substance of the invention.

In the embodiment described above, the configuration in which a user views an outside scene that passes through the display section is not limited to the configuration in which the right optical image display section26and the left optical image display section28transmit outside light. For example, the invention is also applicable to a display apparatus that displays an image but does not allow a user to view an outside scene. Specifically, the invention is applicable to a display apparatus that displays images captured with the right camera61and/or the left camera62, an image and a CG produced on the basis of the captured images, video images based on prestored video data or externally inputted video data, and other types of image. An example of a display apparatus of this type may include a display apparatus that does not allow a user to view an outside scene or what is called a closed-type display apparatus. Further, a display apparatus that does not perform AR display, MR display, or VR display but displays externally inputted video data or an analog video signal is, of course, an apparatus to which the invention is applied.

Further, for example, the image display section20may be replaced with an image display section worn, for example, as a cap or any other image display section worn based on another method. A display section that displays an image in correspondence with a user's left eye and a display section that displays an image in correspondence with the user's right eye only need to be provided. Moreover, the display apparatus according to the embodiment of the invention may, for example, be configured as a head mounted display incorporated in an automobile, an airplane, and other vehicles. Further, for example, the display apparatus may be configured as a head mounted display built in a helmet or other body protection gears. In this case, a portion that determines the position relative to a user's body and a portion that is positioned relative to the positioning portion can be a portion worn by the user.

Further, in the embodiment described above, the description has been made of the case where the image display section20is separated from the control device10and they are connected to each other via the connection section40. The control device10and the image display section20can instead be integrated with each other, and the integrated unit can be worn around a user's head.

The control device10may be a notebook computer, a tablet computer, or a desktop computer. Instead, the control device10may, for example, be a mobile electronic apparatus including a game console, a mobile phone, a smartphone, and a mobile media player, or any other dedicated apparatus. Further, the control device10may be configured to be separate from the image display section20, and a variety of signals may be transmitted and received between the control device10and the image display section20over wireless communication.

Further, for example, the configuration that generates image light in the image display section20may be a configuration including an organic EL (organic electro-luminescence) display and an organic EL control section. Moreover, an LCOS (liquid crystal on silicon) device (LCoS is a registered trademark), a digital micromirror device, or any other device can be used as the configuration that generated image light.

The “display section” used herein corresponds to a configuration that outputs image light, and output of image light from the HMD100is called “display” in the following description.

The embodiment described above illustrates the configuration in which the right and left image light generation units described with reference toFIGS. 5A and 5Bgenerate image light and the right optical image display section26and the left optical image display section28shown inFIG. 3radiate the image light toward the user's right and left eyes to cause the image light to be incident on the user's right and left eyes. The configuration of the “display section” is not limited to the configuration described above. That is, any configuration that radiates the image light other than the configuration shown inFIG. 3andFIGS. 5A and 5Bcan be used. For example, in the configuration of the present embodiment, the “right light guide unit” and the “left light guide unit” having the half-silvered mirrors261A and262A output the image light toward the user's eyes. As the configuration that generates image light, the right backlight221and the left backlight222as well as the right LCD241and the left LCD242are provided. The “display section” does not require this configuration as an essential portion.

For example, image light generated by a mechanism in which one or both of the right display driver22and the left display driver24of the image display section20are built may be reflected off a reflection mechanism provided on the user's side of the image display section20, that is, the side facing the user's eyes and outputted toward the user's eyes. The reflection mechanism can, for example, be a sweep system using a MEMS (micro electro mechanical systems) mirror. That is, a configuration in which a sweep system having a MEMS mirror that sweeps the light outputted from the image light generation units is provided and the light swept by the sweep system is directly incident on the user's eyes may be employed. Further, the image display section20may be provided with an optical member on which a virtual image is formed by the light swept by the sweep system. The optical member uses the light swept with the MEMS mirror to form a virtual image. In this case, when the MEMS mirror sweeps light, a virtual image is formed on a virtual image formation plane, and the user captures the virtual image with the eyes to view (recognize) the image. The optical part in this case may be a part that guides light by reflecting the light multiple times, for example, the right light guide plate261and the left light guide plate262in the embodiment described above, or may be a half-silvered surface.

The sweep system is not limited to the configuration including a MEMS mirror. The mechanism that generates image light may also instead be a laser light source that emits laser light. For example, the invention is also applicable to a laser-retina-projection-type head mounted display. That is, a configuration in which a light output section may include a laser light source and an optical system that guides the laser beam from the laser light source to user's eyes may be employed. In this configuration, the laser beam is caused to be incident on each of the user's eyes, and the laser beam is swept over the retina to form an image on the retina, so that the user is allowed to view the image.

Instead, in place of the virtual formation plane that receives the swept light, a diffraction grating may be used to guide the image light to the user's eyes. That is, the configuration in which the image light is guided through an optical member is not necessarily employed, and a configuration having only a function of guiding the image light toward the user's eyes by refracting and/or reflecting the image light.

In the configuration in which a sweep system having a MEMS or any other component, changing the angle at which the sweep system is attached to the image display section20allows the position where the user views an image, that is, the position where an image is displayed to be changed. Therefore, in the process of changing the image display position in the embodiment described above, the angle of the sweep system may be changed instead of changing the position where images are displayed in the right LCD241and the left LCD242.

As the optical system that guides image light to the user's eyes, an employable configuration includes an optical member that transmits external light externally incident on the display apparatus and allows the external light along with image light to be incident on the user's eyes. Another usable optical system may be an optical member that is disposed in a position in front of the user's eyes and overlaps with part of the visual field of the user or coincides with the entire visual field of the user.

In the embodiment described above, the configuration in which the half-silvered mirrors261A and262A form virtual images on part of the right optical image display section26and the left optical image display section28, which are located in front of the user's eyes is illustrated. The invention is not limited to the configuration described above, and a configuration in which an image is displayed in a display region that occupies the entire or majority of the right optical image display section26and the left optical image display section28may be employed. In this case, the process of reducing the size of an image may be included in the action of changing the position where the image is displayed.

Further, the optical element in the invention are not limited to the right light guide plate261and the left light guide plate262having the half-silvered mirrors261A and262A and only needs to be an optical part that causes image light to be incident on the user's eyes. Specifically, a diffraction grating, a prism, or a holography display section may be used.

At least part of the functional blocks shown inFIG. 6may be achieved by hardware or hardware and software cooperating with each other, and the configuration formed of independent hardware resources shown inFIG. 6is not necessarily employed. The program executed by the control section140may be stored in the storage section120or a storage device in the control device10, or a program stored in an external device may be acquired via the communication section117or the interface125and executed. Among the configurations formed in the control device10, only the operation section135may be formed as an independent user interface (UI). Further, the configurations formed in the control device10may be redundantly formed in the image display section20. For example, the control section140shown inFIG. 6may be formed both in the control device10and the image display section20, and the control section140formed in the control device10and the CPU formed in the image display section20may perform different functions.

The entire disclosure of Japanese Patent Application No. 2015-135194, filed Jul. 6, 2015 is expressly incorporated by reference herein.