Display system, information processing device, display control method of display system

A display system includes a projector, an HMD, and an information processing device. The information processing device includes a processing device storage unit configured to store three-dimensional map data in which setting information in which a display position of an object is set and positional information indicating a position of a display surface are registered, and image data of an object image, and a processing control unit configured to select whether to display the object image on the projector or to display the object image on the HMD, based on a positional relationship between a position of the HMD notified from the HMD and the display position of the object, and transmit the image data of the object image to the projector or the HMD being selected.

The present application is based on, and claims priority from JP Application Serial Number 2020-060698, filed Mar. 30, 2020, the disclosure of which is hereby incorporated by reference herein in its entirety.

BACKGROUND

1. Technical Field

The present disclosure relates to a display system, an information processing device, and a display control method of the display system.

2. Related Art

In the related art, a head-mounted display apparatus that displays an image through an external scene so as to be visually recognizable together with the external scene has been known. Since the head-mounted display apparatus allows the external scene to be visually recognizable while displaying the image, a mechanism for operating the head-mounted display apparatus with another display device has been proposed.

For example, a transmission-type display device disclosed in JP-A-2016-31373 acquires data about a display image displayed by an external display device, and captures the display image with a camera. The display device detects, from a captured image by the camera, that a part of the display image displayed by the external display device is blocked by an obstacle and becomes visually unrecognizable or less likely to be visually recognized. The display device displays the part of the display image and allows the display image to be visually recognizable.

The head-mounted display apparatus tends to increase a frame rate of an image in order to allow a user wearing the head-mounted display apparatus to visually recognize the image without an uncomfortable feeling, and a load on the head-mounted display apparatus tends to increase. Further, the head-mounted display apparatus superimposes an image on a real space for display. However, when a position in the real space on which the image is superimposed is too far from a user, the intended image may not be visually recognizable by the user. For example, in a case in which a stereoscopic image is displayed by the head-mounted display apparatus, when a position in the real space on which the stereoscopic image is superimposed is too far from the user, there is a problem that the user cannot visually recognize the stereoscopic image.

SUMMARY

An aspect that solves the problem described above is a display system including a display device configured to display an object image on a display surface, and a head-mounted display apparatus mounted on a head of a user and configured to display an object image so as to be visually recognizable together with an external scene, where the display system refers to map data in which a position in a real space associated with the object image and a position of the display surface in the real space are recorded, and acquires the position in the real space associated with the object image and the position of the display surface in the real space, and the display system displays the object image on any one of the display device and the head-mounted display apparatus, based on a position of the head-mounted display apparatus notified from the head-mounted display apparatus, the acquired position in the real space associated with the object image, and the acquired position of the display surface in the real space.

Another aspect that solves the problem described above is an information processing device communicably coupled to a display device configured to display an object image on a display surface, and a head-mounted display apparatus mounted on a head of a user and configured to display an object image so as to be visually recognizable together with an external scene, and the information processing device includes a first storage unit configured to store map data in which a position in a real space associated with the object image and a position of the display surface in the real space are recorded, and a control unit configured to display the object image on any one of the display device and the head-mounted display apparatus, based on a position of the head-mounted display apparatus notified from the head-mounted display apparatus, the position in the real space associated with the object image, and the position of the display surface in the real space.

Another aspect that solves the problem described above is a display control method of a display system including a display device configured to display an object image on a display surface, and a head-mounted display apparatus mounted on a head of a user and configured to display an object image so as to be visually recognizable together with an external scene, and the display control method of a display system includes receiving, from the head-mounted display apparatus, notification information that notifies a position of the head-mounted display apparatus, referring to map data in which a position in a real space associated with the object image and a position of the display surface in the real space are recorded, and acquiring the position in the real space associated with the object image and the position of the display surface in the real space, and displaying the object image on any one of the display device and the head-mounted display apparatus, based on the position of the head-mounted display apparatus, the acquired position in the real space associated with the object image, and the acquired position of the display surface in the real space.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings.

FIG. 1is a diagram illustrating a schematic configuration of a display system1.

The display system1includes a projector100, a head-mounted display apparatus200, and an information processing device300. The projector100corresponds to a display device of the present disclosure. The information processing device300is coupled to the projector100and the head-mounted display apparatus200so as to be able to perform data communication.FIG. 1illustrates an example in which the information processing device300and the projector100are coupled to each other with a cable in a wired manner and the information processing devices300and the HMD200are coupled to each other in a wireless manner, but the information processing device300and the projector100may be coupled to each other in a wireless manner. Hereinafter, the head-mounted display apparatus200is referred to as the HMD200. HMD is an abbreviation for Head Mounted Display.

FIG. 1illustrates one projector100and one information processing device300, and illustrates an HMD200A and an HMD200B in total of two as the HMD200, but the number of the projector100, the HMD200, and the information processing device300is arbitrary. For example, the information processing device300may be configured to be coupled to a plurality of projectors100and a plurality of HMDs200. Further, a plurality of information processing devices300coupled to the plurality of projectors100and the plurality of HMDs200may be provided. In the following description, the HMD200A and the HMD200B may be collectively referred to as the HMD200. The HMD200A corresponds to a first head-mounted display apparatus of the present disclosure, and the HMD200B corresponds to a second head-mounted display apparatus of the present disclosure.

The projector100generates imaging light based on content data363supplied from the information processing device300, and projects the generated imaging light onto a display surface10. The display surface10may be a curtain-type screen, or an outer wall of a building, a wall surface of a room, a flat surface of an installed object, and the like may serve as the display surface10. Further, the display surface10is not limited to a flat surface, and may be a curved surface and a surface having irregularities.

The HMD200includes an image display unit210mounted on a head of a user U, and a control device230that controls the image display unit210. The HMD200causes an image based on the content data363received from the information processing device300to be displayed by the image display unit210. The HMD200is a transparent type display device that causes the user U wearing the image display unit210on the head to visually recognize a virtual image, and also allows the user U to also visually recognize an external scene directly.

Further, the control device230may be a device specially provided for the HMD200, and may be configured to couple an external device such as a smartphone via a connection device coupled to the image display unit210. When the control device230is configured to couple the external device via the connection device, the connection device executes processing of operating various sensors included in the image display unit210, and causing an image to be displayed on the image display unit210, based on video data and display data input from the external device.

The information processing device300is a device that supplies the content data363to the projector100and the HMD200. More specifically, the information processing device300selects whether to display an image based on the content data363on the HMD200or to to display the image on the projector100, based on positional information of the HMD200notified from the HMD200. The information processing device300transmits the content data363to the HMD200or the projector100being selected, and causes the image based on the content data363to be displayed.

The HMD200and the projector100display an object image500as an example of the image based on the content data363. The object image500may be, for example, a stationary object such as a table, a desk, furniture, and a house, or a moving body such as an animal and a person.

The object image500displayed by the HMD200is displayed as a virtual object. The virtual object is not present in the real space, but is an image that is superimposed on the real space by being visually recognized by the user U through the image display unit210, and makes the user U feel as if the virtual object is present in the real space. The projector100causes the object image500to be displayed on the display surface10.

FIG. 2is a block configuration diagram illustrating a configuration of the projector100.

The configuration of the projector100will be described with reference toFIG. 2.

The projector100includes a communication unit110, an image processing unit120, a frame memory130, a driving unit140, a projection unit150, and a PJ control unit160.

The communication unit110includes a conductive connector and an interface circuit conforming to a prescribed communication standard, and transmits and receives data to and from the information processing device300coupled thereto via a cable. When the communication unit110receives the content data363from the information processing device300, the communication unit110outputs the received content data363to the PJ control unit160.

Image data is input to the image processing unit120from the PJ control unit160. The image data is data included in the content data363. The image processing unit120develops, in the frame memory130, the image data input from the PJ control unit160. The image processing unit120executes, on the image data developed in the frame memory130, image processing such as, for example, resolution conversion processing or resizing processing, distortion aberration correction, shape correction processing, digital zoom processing, and an adjustment of a shade and brightness of an image. Of course, the image processing unit120can also execute the plurality of image processing described above in combination. When the image processing is completed, the image processing unit120generates a display signal corresponding to the image data developed in the frame memory130, and outputs the generated display signal to the driving unit140.

The driving unit140drives a light modulation device151, based on the display signal input from the image processing unit120. Further, the driving unit140turns on and turns off a light source and adjusts zooming and focus of a projection optical system according to control of the PJ control unit160.

The projection unit150includes the light modulation device151and a projection optical system153. The light modulation device151includes three liquid crystal panels provided for each color of red, green, and blue, for example. The light modulation device151is driven by the driving unit140, and light transmittance of the three liquid crystal panels is changed. Specifically, the light transmittance of the three liquid crystal panels is changed to light transmittance corresponding to a display signal. Light emitted by the light source passes through the light modulation device151, and thus imaging light corresponding to the display signal is generated. The projection optical system153includes an optical element such as a lens and a mirror, and forms an image of the imaging light generated by the light modulation device151on the display surface10.

The PJ control unit160includes a PJ storage unit161and a processor163.

The PJ storage unit161includes a memory such as, for example, a ROM, a RAM, and a flash memory. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory.

The PJ storage unit161stores a program executed by the processor163and various kinds of data processed by the processor163in a non-volatile manner.

The PJ control unit160controls each unit of the projector100by the processor163executing a program. The PJ control unit160controls each unit of the projector100, and thus the projector100generates imaging light based on the content data363received from the information processing device300, and causes the generated imaging light to be displayed on the display surface10.

The projector100can cause a content such as video to be displayed on a three-dimensional projection target such as a building by so-called projection mapping. Further, the projector100can include a plurality of projectors100being arranged side by side, and also perform multiprojection that projects an image on the display surface10by the plurality of projectors100.

Next, a configuration of the HMD200will be described with reference toFIG. 3.

Note that the HMD200A and the HMD200B have the same configuration, and thus a configuration of the HMD200will be described herein.

As illustrated inFIG. 1, the image display unit210is a member having a shape of glasses. The image display unit210includes a main body including a right holding part21, a left holding part23, and a front frame27. The main body further includes a right display unit22, a left display unit24, a right light-guiding plate26, and a left light-guiding plate28. Further, the image display unit210is installed with, as a sensor, a camera211, a geomagnetic sensor213, and a six-axis sensor215.

The right holding part21and the left holding part23extend rearward from corresponding ends of the front frame27, to hold the image display unit210on the head of the user U.

The front frame27has a shape formed by coupling an end of the right light-guiding plate26and an end of the left light-guiding plate28to each other, and this coupling position corresponds to a position between eyebrows of the user U in a state where the user U wears the image display unit210.

Each of the right display unit22and the left display unit24is a module obtained by unitizing an optical unit and a peripheral circuit. The right display unit22causes an image to be displayed by the right light-guiding plate26, and the left display unit24causes an image to be displayed by the left light-guiding plate28. The right display unit22is attached to the right holding part21, and the left display unit24is attached to the left holding part23.

The right display unit22includes, as a configuration that causes a right eye to visually recognize an image, a right OLED unit that emits imaging light, and a right optical system that guides the imaging light emitted from the right OLED unit to the right light-guiding plate26. OLED is an abbreviation for Organic Light Emitting Diode. The right OLED unit and the right optical system are not illustrated.

The left display unit24includes, as a configuration that causes a left eye to visually recognize an image, a left OLED unit that emits imaging light, and a left optical system that guides the imaging light emitted from the left OLED unit to the left light-guiding plate28. The left OLED unit and the left optical system are not illustrated.

The camera211is provided on the front frame27and located such that the camera211does not block the outside light passing through the right light-guiding plate26and the left light-guiding plate28. The camera211is a digital camera including an imaging element such as a CCD and a CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera. An angle of view of the camera211includes at least a part of a range of an external scene visually recognized by the user U wearing the image display unit210through the right light-guiding plate26and the left light-guiding plate28.

The geomagnetic sensor213is a three-axis sensor that detects a magnetic field in an X-axis direction, a Y-axis direction, and a Z-axis direction. An X-axis, a Y-axis, and a Z-axis are respective three axis directions orthogonal to each other as illustrated inFIG. 1. The Z-axis direction corresponds to a vertical direction. The X-axis direction corresponds to a left-right direction of the head of the user U. The Y-axis direction corresponds to a front-back direction of the head of the user U.

The six-axis sensor215is a motion sensor including a three-axis acceleration sensor and a three-axis gyro sensor. The six-axis sensor215may adopt an IMU in which the sensors described above are provided as modules.

The six-axis sensor215and the geomagnetic sensor213perform sampling for each predetermined period such as, for example, 50 msec, and outputs an analog voltage value indicating a detection result. The output voltage value is converted into a digital signal by an A/D converter and is output to an HMD control unit250.

Next, the control device230will be described.

The control device230includes an operation unit231, a communication unit233, a beacon reception unit235, and the HMD control unit250.

The operation unit231is a reception unit that includes a button, a switch, and the like, and receives an operation of the user U. The operation unit231outputs an operation signal corresponding to the received operation to the control device230.

The communication unit233corresponds to a wireless communication protocol such as Bluetooth and Wi-Fi, and performs wireless communication with the information processing device300. Note that Bluetooth and Wi-Fi are a registered trademark.

The beacon reception unit235receives a beacon signal.

A plurality of beacon devices that transmit a beacon signal are installed in a field. The field is a region in which the projector100, the information processing device300, and the display surface10are installed, and the user U can move, and may be outdoor or indoor. The beacon reception unit235receives a beacon signal transmitted from the beacon devices.

The HMD control unit250includes an HMD storage unit260and a processor270.

The HMD storage unit260includes a ROM and a RAM. Further, the HMD storage unit260may be configured to include a non-volatile memory such as a flash memory and the like in addition to the ROM and the RAM. The HMD storage unit260corresponds to a second storage unit of the present disclosure.

The HMD storage unit260stores a program executed by the processor270and various kinds of data processed by the processor270in a non-volatile manner. For example, the HMD storage unit260stores an operating system as a basic control program executed by the processor270, an application program operating on the operating system, and the like. Further, the processor270stores data processed during execution of the application program and data about a processing result.

Further, the HMD storage unit260stores identification information265identifying the HMD200. Furthermore, the HMD storage unit260stores positional information indicating an installation position of a beacon device that transmits a beacon signal in the field, and identification information identifying each beacon device.

The processor270is configured with a CPU, a microcomputer, a DSP, and the like, and is configured to execute a program to control each unit of the control device230. The processor270may be a SoC that integrates the processor270and the HMD storage unit260. DSP is an abbreviation for Digital Signal Processor. SoC is an abbreviation for System-on-a-chip.

The HMD control unit250achieves various functional configurations by the processor270executing a program stored in the HMD storage unit260. The HMD control unit250includes, as functional configurations, a communication control unit271, a position detection unit273, a head orientation detection unit275, a sound processing unit277, a display control unit279, and a detection control unit281.

The communication control unit271controls the communication unit233, and performs data communication with the information processing device300.

The position detection unit273detects a moving amount and a moving direction of the HMD200, based on sensor data of a three-axis acceleration sensor and a three-axis gyro sensor included in the six-axis sensor215. The position detection unit273detects a position of the HMD200in the field, based on the moving amount and the moving direction being detected.

Further, the position detection unit273detects a position of the HMD200in the field, based on a beacon signal received by the beacon reception unit235. The position detection unit273receives a beacon signal from at least three beacon devices, and estimates a distance between the beacon device serving as a transmission source of the beacon signal and the HMD200, based on signal intensity of the received beacon signal. Then, the position detection unit273estimates a current position of the HMD200by three-point positioning or four-point positioning, based on the estimated distance between the at least three beacon devices and the HMD200. In order for the position detection unit273to estimate a current position based on a beacon signal, the beacon signal needs to be received from at least three beacon devices while the HMD200is stationary. Thus, the beacon device needs to be disposed such that reaching ranges of beacon signals transmitted from a plurality of beacon devices partially overlap each other, and transmission power of the beacon device needs to be set. Further, when the plurality of beacon devices are disposed in close proximity, a transmission period is adjusted such that transmission timing of the beacon signal does not match. The reason is to prevent an increase in probability of failure in receiving some beacon signals and failure in acquiring information superimposed on a beacon signal when the HMD200simultaneously receives beacon signals transmitted from the plurality of beacon signals.

Further, the position detection unit273may be configured to correct a position of the HMD200detected based on the signal intensity of the beacon signal by the sensor data of the three-axis acceleration sensor and the three-axis gyro sensor included in the six-axis sensor215.

The head orientation detection unit275detects a position and a movement of the head of the user U, based on sensor data of the six-axis sensor215and the geomagnetic sensor21. Further, the head orientation detection unit275may detect a movement of the upper half of the body including the head of the user U.

The head orientation detection unit275detects an azimuth angle indicating a direction in which the head of the user U faces, based on the sensor data of the geomagnetic sensor213. The azimuth angle corresponds to an orientation of the head of the present disclosure. Further, the head orientation detection unit275detects a pitch angle that is a rotation angle about the X-axis of the head, based on the sensor data of the gyro sensor included in the six-axis sensor215. The detected pitch angle is a slope with respect to a horizontal plane that is a plane including the X-axis set in the left-right direction of the head and the Y-axis set in the front-rear direction of the head. The slope is an angle of elevation or an angle of depression. The angle of elevation or the angle of depression corresponds to an angle of the head with respect to the vertical direction of the present disclosure.

The HMD control unit250transmits, to the information processing device300for each predetermined time, notification information including positional information indicating a position of the HMD200, azimuth angle information indicating an orientation of the head, elevation angle information or depression angle information, and identification information of the HMD200.

When sound data is included in the content data363received from the information processing device300, the sound processing unit277processes the sound included in the content and outputs the sound from an earphone220.

The display control unit279executes various kinds of processing for causing an image to be displayed on the image display unit210, based on video data included in the content data363received from the information processing device300. The display control unit279executes various kinds of processing such as cutting out of a frame from video data, resolution conversion, scaling, intermediate frame generation, and frame rate conversion, and outputs the video data after execution of various kinds of processing to the left display unit24and the right display unit22.

The detection control unit281analyzes a captured image of the camera211, and detects an operation performed by the user U. The camera211and the detection control unit281correspond to a reception unit of the present disclosure.

When the operation unit231receives a preset operation in a state where the image display unit210displays the object image500, the detection control unit281detects an image of a hand of the user from a captured image of the camera211, and receives an operation that specifies a moving amount and a moving direction of the object image500. The detection control unit281detects a moving direction that is a direction in which the hand of the user U moves and a moving amount in which the hand of the user U moves, based on the image of the hand detected from the captured image of the camera211being continuously captured. In addition, the user U may input, by operating the operation unit231, a moving direction and a moving amount in which a position in the field associated with the object image500is moved.

When the detection control unit281detects a moving amount and a moving direction of the hand, the detection control unit281transmits information indicating the moving amount and the moving direction being detected to the information processing device300. When the information processing device300receives the information indicating the moving amount and the moving direction from the HMD200, the information processing device300rewrites a position of the field associated with the object image500being recorded in three-dimensional map data361described later.

Next, a configuration of the information processing device300will be described with reference toFIG. 4.FIG. 4is a block configuration diagram illustrating the configuration of the information processing device300.

The information processing device300is a so-called computer device, and a notebook PC desktop PC is used. PC is an abbreviation for Personal Computer. The information processing device300includes a first communication unit310, a second communication unit320, an input/output I/F330, a display unit331, an input device333, and a processing control unit350. The first communication unit310, the second communication unit320, the input/output I/F330, and the processing control unit350are coupled to each other via a bus305. I/F is an abbreviation for Interface.

The first communication unit310includes a conductive connector and an interface circuit conforming to a prescribed communication standard, and transmits and receives data to and from the projector100coupled thereto via a cable.

The second communication unit320corresponds to a wireless communication protocol such as Bluetooth and Wi-Fi, and performs wireless communication with the HMD200A and the HMD200B.

The input/output I/F330is coupled to the display unit331and the input device333, outputs data input via the bus305to the display unit331, and outputs data input from the input device333to the processing control unit350.

The display unit331includes, for example, a display panel such as a liquid crystal panel, and causes a display image generated by the processing control unit350to be displayed. The liquid crystal panel is not illustrated.

The input device333includes an input device such as a mouse and a keyboard, and receives an operation of a user. The input device333outputs an operation signal corresponding to the received operation to the processing control unit350via the input/output I/F330. Note that the user is different from the user U wearing the image display unit210of the HMD200on the head.

The processing control unit350includes a processing device storage unit360and a processor370. The processing device storage unit360includes a memory such as a ROM, a RAM, and a flash memory, and a storage device such as an HDD and an SSD. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. The processing device storage unit360corresponds to a first storage unit of the present disclosure.

The processing device storage unit360stores an operating system as a basic control program and an application program operating on the operating system. Further, the processing device storage unit360stores background image data that is data about a background image displayed on the display surface10by the projector100.

Furthermore, the processing device storage unit360stores the three-dimensional map data361, the content data363, and an identification information registration table365. The three-dimensional map data361corresponds to map data of the present disclosure.

In the three-dimensional map data361, information indicating a position in the field associated with the object image500and the display surface10is registered. The field corresponds to the real space of the present disclosure. The information indicating a position in the field may be coordinate information of a three-dimensional Cartesian coordinate system set in the field, or may be a latitude or a longitude indicating a position in the field. In the present exemplary embodiment, a case in which the information indicating a position in the field is coordinate information of the three-dimensional Cartesian coordinate system will be described. In the following, coordinate axes of the three-dimensional Cartesian coordinate system are denoted as an x-axis, a y-axis, and a z-axis. The z-axis is an axis extending in the vertical direction of the field. The x-axis is an axis perpendicular to the z-axis. The y-axis is an axis perpendicular to the z-axis and the x-axis. Further, in the following, a position in the field associated with the object image500and the display surface10is referred to as a corresponding position.

The coordinate information indicating the corresponding position of the object image500being registered in the three-dimensional map data361can be changed by an operation of the input device333. When the input device333receives the operation of changing the corresponding position of the object image500, the processing control unit350rewrites the coordinate information indicating the corresponding position of the object image500being registered in the three-dimensional map data361according to the received operation.

The content data363is image data of the object image500. The content data363may be image data of a still image or image data of a moving image. The content data363may also include sound data.

The content data363includes data for displaying the object image500by AR. AR is an abbreviation for Augmented Reality. An image of an AR displayed virtual object is an image visually recognized by being superimposed on an external scene visually recognized by the user U through the image display unit210, i.e., a scene and an object in the real space. The visibility of the external scene when the image of the AR displayed virtual object is determined by a ratio between the amount of imaging light emitted from each of the right display unit22and the left display unit24and the amount of external light incident on the right eye and the left eye of the user U from the outside of the image display unit210.

Further, the content data363includes data for three-dimensionally displaying the object image500. The data for three-dimensionally displaying the object image500may be a moving image or a still image. The data for three-dimensionally displaying the object image500includes image data for the right eye displayed by the right display unit22and image data for the left eye displayed by the left display unit24. The parallax between the image data for the right eye and the image data for the left eye may be preset. Further, the parallax may be adjustable by the HMD200, and may be configured to be able to be specified by an operation of the user U.

The identification information registration table365is a table in which identification information of the HMD200and identification information of the content data363stored in the processing device storage unit360are registered in association with each other. The content data363stored in the processing device storage unit360includes a content permitted to be visually recognized or test-listened only by a preset specific user U. In the identification information registration table365, identification information of a content permitted to be visually recognized or test-listened only by the specific user U and the identification information of the HMD200worn by the user U who is permitted to visually recognize or test-listen to the content are registered in association with each other.

The processor370is configured with a CPU, a microcomputer, a DSP, and the like, and is configured to execute a program to control each unit of the information processing device300. The processor370may be a SoC that integrates the processor370and the processing device storage unit360.

The processing control unit350achieves various functional configurations by the processor370executing an operating system and an application program stored in the processing device storage unit360.

Next, an operation of the processing control unit350will be described.

The second communication unit320of the information processing device300receives the notification information transmitted from the HMD200. The second communication unit320receives the notification information transmitted from the HMD200for each predetermined time. The notification information received by the second communication unit320is input to the processing control unit350. The processing control unit350starts processing by receiving an input of the notification information.

First, the processing control unit350acquires the positional information, the azimuth angle information, and the elevation angle information or the depression angle information of the HMD200included in the notification information. The processing control unit350estimates a visual field range being visually recognizable by the user U. The processing control unit350sets a visual field range405of the user U, based on the positional information, the azimuth angle information, and the elevation angle information or the depression angle information being acquired. A method of setting the visual field range405will be described with reference toFIG. 5.FIG. 5is a diagram illustrating the visual field range405of the user U.

First, the processing control unit350sets coordinates indicated by the acquired positional information to a user position. The user position is a position of the user U wearing the HMD200. Next, the processing control unit350sets a viewpoint position401that is a position of the eye of the user U wearing the HMD200, based on the positional information. The position of the user U in the field uses the positional information included in the notification information, and a height is set to a predetermined height being preset. The height can be more accurate by setting different values for men and women. Further, information indicating a height of the user U or a height to the eye may be acquired in advance.

Next, the processing control unit350sets the visual field range405of the user U, based on the azimuth angle information and the elevation angle information or the depression angle information included in the notification information. The processing control unit350sets a line-of-sight direction403, based on the azimuth angle information and the elevation angle information or the depression angle information with reference to the set viewpoint position401.FIG. 5illustrates the azimuth angle information with an azimuth angle as an angle α, and the line-of-sight direction403with an angle of elevation indicated by the elevation angle information as an angle β. Next, the processing control unit350sets, to the visual field range405, a range at a predetermined distance from the line-of-sight direction403in the up-down and left-right directions with the line-of-sight direction403as the center. At this time, a distance in a horizontal direction between the viewpoint position401and the visual field range405is set to a preset distance. For example, the distance may be a distance from a position of the user U to an end portion of the field in the line-of-sight direction403.

When the visual field range405is set, the processing control unit350detects the object image500having a position of the associated field included within the range acquired by connecting the viewpoint position401and four vertexes of the visual field range405. The range acquired by connecting the viewpoint position401and the four vertexes of the visual field range405is referred to as a detection range.

For example, when the coordinate information indicating a position in the field associated with the object image500includes three-dimensional information of the x-axis, the y-axis, and the z-axis, the processing control unit350detects the object image500having a position associated with the detection range.

Further, when the coordinate information indicating a position in the field associated with the object image500includes two-dimensional information of the x-axis and the y-axis, the processing control unit350converts the detection range into a range in a xy plane constituted by the x-axis and the y-axis. The processing control unit350detects the object image500having a position associated within the range in the xy plane of the detection range.

When the processing control unit350detects the object image500, the processing control unit350determines whether the identification information is set for the detected object image500. When the identification information is not set for the detected object image500, the processing control unit350reads the content data363corresponding to the detected object image500from the processing device storage unit360. The processing control unit350transmits the read content data363to the HMD200that has received the notification information.

Further, when the detected object image500is the object image500for which the identification information is set, the processing control unit350refers to the identification information registration table365. The processing control unit350refers to the identification information registration table365, and determines whether the identification information of the object image500is associated with the identification information of the HMD200that has received the notification information.

When the identification information of the object image500is associated with the identification information of the HMD200that has received the notification information, the processing control unit350reads the content data363corresponding to the detected object image500from the processing device storage unit360. The processing control unit350transmits the read content data363to the HMD200.

Further, when the identification information of the object image500is not associated with the identification information of the HMD200that has received the notification information, the processing control unit350does not transmit the content data363to the HMD200. In this way, only the user U who is permitted to test-listen to or visually recognize the content data363can test-listen to or visually recognize the content data363.

Next, a case in which the object image500is included in the set visual field range405, but a distance between a position of the user U and a corresponding position of the object image500is longer than a reference distance will be described.

The processing control unit350selects whether to display the object image500on the projector100or to display the object image500on the HMD200, based on the positional relationship between a position of the HMD200and the corresponding position of the object image500. In the present exemplary embodiment, with a distance between the HMD200and the display surface10as the reference distance, the processing control unit350causes the object image500having a distance to the HMD200shorter than the reference distance to be displayed on the HMD200. Further, the processing control unit350causes the object image500having a distance to the HMD200longer than the reference distance to be displayed on the display surface10by the projector100.

The processing control unit350calculates a distance from a user position indicated by the positional information to the display surface10. For example, the processing control unit350may calculate a distance from the user position to the center of the display surface10. Further, the processing control unit350may detect, as a distance to the display surface10, a position of the display surface10closest to the user position or a position of the display surface10farthest from the user position. The distance calculated herein is referred to as the reference distance.

When the processing control unit350calculates the reference distance, the processing control unit350determines whether there is the object image500having a distance from the user position longer than the reference distance among the detected object images500. The processing control unit350calculates a distance to the object image500, based on coordinates of the user position and coordinates of the corresponding position of the object image500, and compares the calculated distance with the reference distance.

When the processing control unit350detects the object image500having a distance from the user position longer than the reference distance, the processing control unit350determines that the object image500is an image to be displayed on the display surface10.

When the processing control unit350detects the object image500having a corresponding position included in the visual field range405and having a distance from the user position longer than the reference position, the processing control unit350determines that the object image500is an image to be displayed on the display surface10.

Next, the processing control unit350determines a position of the display surface10on which the object image500is displayed. The processing will be described with reference toFIGS. 6 and 7.FIGS. 6 and 7are diagrams illustrating a corresponding position of the object image500on the display surface10.

First, a case in which the number of users U in a position where the object image500is visually recognizable is one will described with reference toFIG. 6.

As illustrated inFIG. 6, the processing control unit350sets a straight line531connecting the viewpoint position401of the user U and the center of the object image500, and determines a display position on the display surface10such that the center of the object image500is located in a position where the set straight line531intersects the display surface10.

Next, a case in which the number of users U in a position where the object image500is visually recognizable is plural will be described with reference toFIG. 7. Here, a case in which two users of a user U1and a user U2are present as the user U in the position where the object image500is visually recognizable will be described. The user U1is a user wearing the HMD200A, and the user U2is a user wearing the HMD200B.

As illustrated inFIG. 7, the processing control unit350sets a first straight line533connecting the viewpoint position401of the user U1and the center of the object image500, and sets a second straight line535connecting the viewpoint position401of the user U2and the center of the object image500. Further, the processing control unit350determines an intersection point of the first straight line533with the display surface10as a first display position541, and determines an intersection point of the second straight line535with the display surface10as a second display position543.

The processing control unit350sets a display position of the object image500within a range sandwiched between the set first straight line533and the set second straight line535. For example, the processing control unit350sets a third straight line545connecting the first display position541and the second display position543. Furthermore, the processing control unit350sets a fourth straight line537. The fourth straight line537is set such that the fourth straight line537intersects the third straight line545between the first display position541and the second display position543, and an angle θ formed with the first straight line533and an angle θ formed with the second straight line535are the same angle. The processing control unit350determines the display position such that the center of the object image500is located in an intersection position where the fourth straight line537intersects the display surface10.

When the processing control unit350determines the display position of the object image500, the processing control unit350outputs the content data363and positional information indicating the determined display position to the projector100. The projector100causes the content data363input from the information processing device300to be superimposed on a display position of a background image indicated by setting information and to be displayed as the object image500on the display surface10.

In this way, as illustrated inFIG. 8, the object image500located at a distance shorter than the reference distance that is the distance from the user position to the display surface10is displayed in AR display or three-dimensional display by the HMD200, and is thus recognized as a virtual object by the user U. Further, the object image500located at a distance longer than the reference distance is displayed as a two-dimensional image on the display surface10by the projector100.

AlthoughFIG. 8illustrates a case in which only one display surface10is installed, a plurality of display surfaces10may be installed in the field. For example, two display surfaces10may be bonded together, and an image may be displayed on each of the two display surfaces10by two installed projectors100. In this case, both sides of the bonded display surfaces10are fields, and a display position of the object image500is set on the both sides of the bonded display surfaces10.

FIG. 9is a flowchart illustrating an operation of the information processing device300that performs an image display method. The operation of the information processing device300will be described with reference to the flowchart illustrated inFIG. 9.

When the processing control unit350receives an operation preset by the input device333, the processing control unit350first reads background image data from the processing device storage unit360. The processing control unit350transmits the read background image data to the projector100. When a plurality of pieces of background image data are stored in the processing device storage unit360, the processing control unit350selects background image data selected by the input device333and background image data corresponding to a theme selected by the input device333.

When the projector100receives the background image data from the information processing device300, the projector100causes a background image that is an image based on the received background image data to be displayed on the display surface10.

Next, the processing control unit350determines whether notification information including identification information, positional information, azimuth angle information, and elevation angle information or depression angle information is received from the HMD200(step S2). When the notification information is not received from the HMD200(step S2/NO), the processing control unit350proceeds to a determination in step S14. Step S2corresponds to a reception step of the present disclosure.

When the notification information is received from the HMD200(step S2/YES), the processing control unit350determines the object image500to be displayed on the display surface10by the projector100. First, the processing control unit350calculates a reference distance that is a distance from a position of the user U indicated by the received positional information to the display surface10. For example, the processing control unit350calculates, as the reference distance, a distance from the position of the user U to the center of the display surface10.

Next, the processing control unit350determines whether there is the object image500having a distance from the position of the user U to a corresponding position registered in the three-dimensional map data361longer than the calculated reference distance (step S3). When there is no object image500having a distance from the current position of the user U longer than the reference distance (step S3/NO), the processing control unit350proceeds to processing in step S8. Step S3corresponds to a selection step of the present disclosure.

Further, when there is the object image500having a distance from the position of the user U to the corresponding position longer than the reference position, the processing control unit350determines whether notification information is received from the other HMD200(step S4). In other words, the processing control unit350determines whether there is another user U who uses the display system1. When the notification information is not received from the other HMD200(step S4/NO), the processing control unit350determines a display position on the display surface10in which the object image500is displayed, based on the positional information included in the notification information received in step S2(step S6).

Further, when the notification information is received from the other HMD200(step S4/YES), the processing control unit350determines a display position on the display surface10in which the object image500is displayed, based on the positional information of the plurality of users U (step S5). The processing control unit350transmits the determined display position and the content data363about the object image500to the projector100(step S7). Step S7corresponds to a transmission step of the present disclosure.

Next, the processing control unit350selects the object image500to be displayed on the image display unit210of the HMD200that has received the notification information. First, the processing control unit350sets the visual field range405of the user U (step S8). The processing control unit350sets the visual field range405visually recognizable by the user U, based on the positional information, the azimuth angle information, and the elevation angle information or the depression angle information included in the received notification information. When the processing control unit350sets the visual field range405, the processing control unit350detects the object image500having the corresponding position included within a range acquired by connecting the viewpoint position401and four vertexes of the visual field range405(step S9).

Next, the processing control unit350determines whether the identification information is set for the detected object image500(step S10). When the identification information is set for the detected object image500(step S10/YES), the processing control unit350determines whether the identification information of the HMD200is associated with the identification information of the object image500(step S11).

When the identification information of the HMD200is not associated with the identification information of the object image500(step S11/NO), the processing control unit350proceeds to the determination in step S14. Further, when the identification information of the HMD200is associated with the identification information of the object image500(step S11/YES), the processing control unit350acquires the content data363about the detected object image500from the processing device storage unit360(step S12). The processing control unit350transmits the acquired content data363to the corresponding HMD200(step S13).

Further, in step S10, when the identification information is not set for the detected object image500(step S10/NO), the processing control unit350acquires the content data363about the detected object image500from the processing device storage unit360(step S12). The processing control unit350transmits the acquired content data363to the corresponding HMD200(step S13).

Next, the processing control unit350determines whether an end operation is received (step S14). When the end operation is not received (step S14/NO), the processing control unit350proceeds to the determination in step S2. Further, when the end operation is received (step S14/YES), the processing control unit350causes the processing flow to end.

As described above, the display system1according to the present exemplary embodiment includes the projector100configured to cause the object image500to be displayed on the display surface10, the HMD200mounted on the head of the user U and configured to display the object image500so as to be visually recognizable together with an external scene, and the information processing device300communicably coupled to the projector100and the HMD200.

The information processing device300refers to the three-dimensional map data361in which a position in the real space associated with the object image500and a position of the display surface10in the real space are recorded, and acquires the position in the real space associated with the object image500and the position of the display surface10in the real space.

Further, the information processing device300causes the object image500to be displayed on any one of the projector100and the HMD200, based on a position of the HMD200notified from the HMD200, the acquired position in the real space associated with the object image500, and the acquired position of the display surface10in the real space.

The information processing device300causes the object image500to be displayed on any one of the projector100and the HMD200, based on a positional relationship between a position of the HMD200and a position in the real space associated with the object image500.

For example, when the position of the HMD200and the position in the real space associated with the object image500are closer than the positions of the HMD200and the display surface10, the object image500can be displayed on the HMD200. Further, when the position of the HMD200and the position in the real space associated with the object image500are farther than the positions of the HMD200and the display surface10, the object image500can be displayed on the display surface10.

Therefore, a processing load on the HMD200can be reduced further than that when all the object image500is displayed on the HMD200. Further, since a part of the object image500is displayed on the display surface10, a processing load on the device of the HMD200can be reduced while suppressing a reduction in realism provided to the user U.

The HMD200includes the geomagnetic sensor213, the six-axis sensor215, the beacon reception unit235, and the position detection unit273configured to detect a position of the HMD200, based on data output from the geomagnetic sensor213, the six-axis sensor215, and the beacon reception unit235. The HMD200transmits, as notification information, the position of the HMD200detected by the position detection unit273to the information processing device300.

Further, the information processing device300includes the processing device control unit350and the processing device storage unit360configured to store the three-dimensional map data361.

The processing device control unit350transmits the content data363about the object image500to the projector100when a distance between the position of the HMD200notified by the notification information and the position in the real space associated with the object image500is longer than the distance between the HMD200and the display surface10.

The projector100causes the object image500based on the content data363received from the information processing device300to be displayed on the display surface10.

For example, in a case in which the HMD200three-dimensionally displays the object image500, when the position in the real space associated with the object image500is too far from the user U, the user U may not be able to recognize the object image500as a stereoscopic object. Thus, the projector100displays, on the display surface10, the object image500that cannot be recognized as a stereoscopic object by the user U, and thus a processing load on the HMD200can be reduced while suppressing a reduction in realism.

The processing device control unit350transmits the content data363about the object image500to the HMD200when the distance between the position of the HMD200and the position in the real space associated with the object image500is shorter than the distance between the HMD200and the display surface10.

The HMD200displays an image based on the content data363about the object image500received from the information processing device300such that an external scene is visually recognizable.

Therefore, the HMD200displays the object image500that can be three-dimensionally displayed by the HMD200, and thus a reduction in realism provided to the user U can be suppressed.

The HMD200includes the head orientation detection unit275configured to detect an orientation of the head of the user U and an angle of the head with respect to the vertical direction. The HMD200transmits, as the notification information, the orientation and the angle of the head detected by the head orientation detection unit275to the information processing device300.

The processing device control unit350estimates the visual field range405of the user U, based on the orientation and the angle of the head notified by the notification information, and detects, as a candidate, the object image500being visually recognizable in the estimated visual field range405and associated with the position in the real space. Further, the processing device control unit350selects whether to display the object image500detected as the candidate on the projector100or to display the object image500on the HMD200.

Therefore, the object image500visually recognizable by the user U of the HMD200can be selected as the candidate and displayed by the HMD200or the projector100.

The HMD200includes the HMD storage unit260configured to store identification information identifying the HMD200. The HMD200transmits, as the notification information, the position of the HMD200detected by the position detection unit273and the identification information of the HMD200to the information processing device300.

The processing device storage unit360stores the identification information registration table365in which identification information identifying the object image500and the identification information identifying the HMD200are registered in association with each other. The processing device control unit350transmits the content data363about the object image500to the HMD200when the distance between the position of the HMD200and the position in the real space associated with the object image500is shorter than the distance between the HMD200and the display surface10, and the identification information of the object image500is associated with the identification information of the HMD200.

Therefore, the object image500can be displayed only on the HMD200associated with the identification information of the object image500.

The processing device control unit350determines a position of the display surface10on which the object image500is displayed, based on the position of the HMD200and the position in the real space associated with the object image500, when the distance between the position of the HMD200and the position in the real space associated with the object image500is longer than the distance between the HMD200and the display surface10.

Further, the processing control unit350transmits positional information indicating the determined position of the display surface10together with the content data363about the object image500to the projector100.

Therefore, the position of the display surface10on which the object image500is displayed can be determined based on the position of the HMD200and the position in the real space associated with the object image500.

The processing device control unit350determines the position of the display surface10on which the object image500is displayed such that the position of the HMD200, the position in the real space associated with the object image500, and the position of the display surface10on which the object image500is displayed are aligned on a straight line.

Therefore, the object image500can be displayed in the position of the display surface10corresponding to the direction of the position in the real space associated with the object image500.

The display system1includes the HMD200A and the HMD200B as the HMD200. The processing device control unit350determines a position of the display surface10on which the object image500is displayed, based on a position of the HMD200A, a position of the HMD200B, and the position in the real space associated with the object image500, when a distance between the position of the HMD200A and the position in the real space associated with the object image500is longer than a distance between the HMD200A and the display surface10, and a distance between the position of the HMD200B and the position in the real space associated with the object image500is longer than a distance between the HMD200B and the display surface10.

The processing device control unit350transmits positional information indicating the determined position of the display surface10together with the content data363about the object image500to the projector100.

Therefore, the position of the display surface10on which the object image500is displayed can be determined based on the position of the HMD200A, the position of the HMD200B, and the position in the real space associated with the object image500.

The processing device control unit350selects the first display position541of the object image500on the display surface10so that the first display position541is located on a straight line connecting the position of the HMD200A and the position in the real space associated with the object image500. Further, the processing device control unit350selects the second display position543of the object image500on the display surface10so that the second display position543is located on a straight line connecting the position of the HMD200B and the position in the real space associated with the object image500.

The processing device control unit350determines a position of the display surface10on which the object image500is displayed between the first display position541and the second display position543, and transmits positional information indicating the determined position of the display surface10together with the content data363about the object image500to the projector100.

Therefore, the position of the display surface10on which the object image500is displayed can be set so as to reduce a difference between when viewed in the direction of the position in the real space associated with the object image500from a position of the user U of the HMD200A and when viewed in the direction of the position in the real space associated with the object image500from a position of the user U of the HMD200B.

The HMD200includes the operation unit231, or the camera211and the detection control unit281as a reception unit configured to receive an operation of the user U.

The HMD200transmits, when the reception unit receives an operation of changing the position in the real space associated with the object image500and an operation of specifying a moving amount and a moving direction of the position in the real space, information indicating the received moving amount and the received moving direction to the information processing device300.

The processing device control unit350changes the position in the real space associated with the object image500, based on the information indicating the moving amount and the moving direction received from the HMD200.

Therefore, the position in the real space associated with the object image500can be changed by an operation of the user U of the HMD200.

Next, an application example of the display system1will be described with specific examples.

For example, when the user U conducts work for operating equipment, the display system1can be used for the purpose of supporting work by a supporter providing information related to the operation of the equipment from a remote place, such as describing and guiding the operation of the equipment. Specifically, the display system1is applicable to a case in which a setup and an operation of an application program are performed with a personal computer as a work target object. Further, for example, the display system1is applicable to a case in which an operation and maintenance of a machine are performed with the machine such as a machine tool, an industrial machine, and various household electronic apparatuses as a work target object. In other words, when the user U who performs maintenance of various machines visits an installation place of the machine as a work target in response to a request by a user, and performs maintenance, the display system1can be used for an application in supporting the user U with a low skill level by a supporter.

In this example, a machine such as a machine tool, an industrial machine, and various household electronic apparatuses, a personal computer, and the like as a work target object of a work target can be installed on a work stand. Further, a position where the equipment, the personal computer, and the like are installed can also be regarded as a work stand. In this case, the user U wearing the HMD200can perform hands-free work at the installation place of the machine such as a machine tool, an industrial machine, and various household electronic apparatuses, the personal computer, and the like as a work target.

In such a case, the user U at a supported site at a remote place and a supporter at a supporting site such as a service center can share information, and the supporter at the supporting site can provide an instruction and change display by displaying an AR image on the HMD200. Then, support can be provided by the HMD200showing, as a virtual image, an operation by a hand and fingers of the supporter at the supporting site. In this case, it is also effective to give description while the supporter at the support site places the same equipment as a work target object at the supported site in front and actually operates the equipment. The supporter can provide an instruction and teach by performing a position instruction operation on a specific portion of an actual object at a site, and can efficiently provide support with a simple operation.

Further, the information processing device300may display a process table indicating an overall process of work and an enlarged image of an AR image displayed on the HMD200.

The present disclosure is not limited to the configurations described in the exemplary embodiments described above, and the present disclosure can be implemented in various aspects without departing from the gist of the disclosure.

For example, a current position of the HMD200may be detected based on a captured image of the camera211. In this case, the HMD storage unit260previously stores map data in which an arrangement of objects present in the field is described. The HMD control unit250may detect an object captured in a captured image of the camera211, and detect a current position of the HMD200, based on a distance from the detected object.

Further, a movement and a position of the head of the user U may be detected by a position tracking sensor. For example, an infrared ray sensor is installed in a fixed position in the field. By detecting an infrared ray emitter or an infrared ray reflection marker provided on a surface of the HMD200by using the infrared ray sensor, an orientation of the head of the user U wearing the HMD200and a position of the HMD200may be detected.

At least some of the functional blocks illustrated inFIGS. 2 to 4may be achieved in the form of hardware or may be achieved by a cooperation of hardware and software, and, is not limited to a configuration in which independent hardware resources are arranged as illustrated in the drawings. Further, the program executed by the processor370of the information processing device300may be configured to acquire and execute a program stored in an external device via the first communication unit310or the second communication unit320.