Head-mounted display apparatus, information processing device, system, and method for controlling use of captured images from head-mounted display apparatus

An HMD mounted on a head of a user includes a storage unit configured to store process flow data that defines a process flow including a plurality of work blocks in which an order of execution is predetermined, a camera configured to capture an external scene, and a control unit configured to execute the process flow according to the process flow data. At least any one of the plurality of work blocks in the process flow data includes a setting that determines whether use of captured image data of the camera is permitted. The control unit controls, when executing the process flow, propriety of using the captured image data in each of the work blocks according to the setting of the process flow data about the work block.

BACKGROUND

1. Technical Field

The present invention relates to a head-mounted display apparatus, an information processing device, an information processing system, and a method for controlling a head-mounted display apparatus.

2. Related Art

A head-mounted display apparatus that is mounted on a head of a user and displays an image visually recognizable by the user has been known (for example, see JP-A-2016-212345). This head-mounted display apparatus is equipped with various functions, and the functions have been improving. Thus, when the head-mounted display apparatus is used at a work site, measures against a leak of information are needed.

JP-A-2016-212345 discloses a head-mounted display apparatus including a function of limiting a capturing function.

When the head-mounted display apparatus is used for work support, captured image data may be desired to be stored by causing an imaging unit of the head-mounted display apparatus to perform capturing depending on a work scene. However, when capturing is limited by a function of a basic control program (OS) installed in the head-mounted display apparatus, only whether the capturing function is uniformly limited can be set. This causes a decrease in convenience.

SUMMARY

An advantage of some aspects of the invention is to achieve securing of security and improvement in user convenience by controlling propriety of using captured image data for each operation step.

To solve the above-described problems, the invention is a head-mounted display apparatus mounted on a head of a user. The head-mounted display apparatus includes a storage unit configured to store control data that defines an operation flow including a plurality of operation steps in which an order of execution is predetermined, an imaging unit configured to capture an external scene, and a control unit configured to execute the operation flow according to the control data. At least any one of the plurality of operation steps in the control data includes a setting that determines whether use of captured image data of the imaging unit is permitted. The control unit controls, when executing the operation flow, propriety of using the captured image data in each of the operation steps according to the setting of the control data about the operation step.

According to this configuration, when executing the control data, the control unit can control the propriety of using the captured image data for each of the operation steps. Therefore, securing of security and improvement in user convenience can be achieved by controlling the propriety of using the captured image data for each of the operation steps.

In the invention, the control unit switches, when executing the operation step, the imaging unit between a state of being operated and a state of being stopped according to the setting of the control data about the operation step.

According to this configuration, switching the operating state of the imaging unit can be performed by causing the control unit to execute the control data.

The invention includes an image storage unit configured to store captured image data of the imaging unit. The control unit controls, when executing the operation step, whether to use captured image data of the imaging unit stored in the image storage unit according to the setting of the control data about the operation step.

According to this configuration, the propriety of using the captured image data stored in the image storage unit can be controlled by causing the control unit to execute the control data.

In the invention, when executing the operation step including processing of capturing an image including code information, the control unit controls use of captured image data of the imaging unit, extracts the code information from the captured image data of the imaging unit, and causes the storage unit to store the code information.

According to this configuration, when the processing of capturing an image including code information is performed, the code information can be extracted from the captured image data.

In the invention, when executing the operation step including the processing of capturing an image including code information, the control unit determines a number of usable times of the captured image data based on the code information extracted from the captured image data, and allows the captured image data to be used in another operation step included in the operation flow with the number of usable times as an upper limit.

According to this configuration, an upper limit of the number of usable times of the captured image data can be set.

In the invention, when executing the operation step in which the captured image data is set to be usable, the control unit allows use of the captured image data corresponding to capturing by the imaging unit for a plurality of times.

According to this configuration, capturing for a plurality of times can be performed in the operation step in which the captured image data is set to be usable.

The invention includes a notification unit. The control unit makes notification by the notification unit when the captured image data of the imaging unit stored in the image storage unit is stored in the storage unit.

According to this configuration, the notification that the captured image data is stored in the storage unit can be made.

To solve the above-described problems, the invention is an editing device allowed to edit control data that defines an operation flow including a plurality of operation steps in which an order of execution is determined. The editing device includes an input unit configured to accept an input, a display unit configured to cause a display face to perform displaying, and a control unit configured to cause the display face to display a work screen including a work region, arrange objects each indicating the operation step and including an image indicating a content of the operation step in the work region in accordance with an input accepted by the input unit, create the operation flow based on arrangement of the objects in the work region, and generate the control data that defines the operation flow created. The control data includes a setting that determines whether use of captured image data of an imaging unit is permitted for at least any one of the operation steps.

According to this configuration, the operation flow set with whether use of the captured image data of the imaging unit is permitted for each of the operation steps in the operation flow can be created.

The invention includes a connection unit attached to a head-mounted display apparatus. The control unit outputs the control data that defines the operation flow created to be associated with the head-mounted display apparatus to the head-mounted display apparatus connected to the connection unit.

According to this configuration, the control data created to be associated with the head-mounted display apparatus is output to the head-mounted display apparatus, and the control data can be executed by the head-mounted display apparatus.

In the invention, a plurality of the head-mounted display apparatuses are allowed to be connected to the connection unit, and the control unit outputs the control data that defines the operation flow created to be associated with each of the plurality of head-mounted display apparatuses to the head-mounted display apparatus connected to the connection unit.

According to this configuration, the control data created to be associated with each of the plurality of head-mounted display apparatuses is output to the corresponding head-mounted display apparatus, and the control data can be executed by the corresponding head-mounted display apparatus.

To solve the above-described problems, the invention is an information processing system that includes a head-mounted display apparatus mounted on a head of a user and an information processing device. The head-mounted display apparatus includes a first communication unit configured to perform communication, a storage unit configured to store control data that defines an operation flow including a plurality of operation steps in which an order of execution is predetermined, an imaging unit configured to capture an external scene, and a first control unit configured to execute the operation flow according to the control data. At least any one of the operation steps in the control data includes a setting that determines whether use of captured image data of the imaging unit is permitted. The first control unit controls, when executing the operation flow, propriety of using the captured image data in each of the operation steps according to the setting of the control data about the operation step, and sends, when use of the captured image data is permitted, the captured image data captured by the imaging unit from the first communication unit to the information processing device.

According to this configuration, the propriety of using the captured image data in each of the operation steps can be controlled. When the use of the captured image data is permitted, the captured image data being captured can be sent to the information processing device. Therefore, when the use of the captured image data is permitted, sending the captured image data being captured to the information processing device can prevent a leak of the captured image data to the outside and secure security.

In the invention, the first control unit adds limited information limiting use of the captured image data to the captured image data and sends the captured image data to the information processing device.

According to this configuration, a leak of the captured image data to the outside can be prevented more efficiently, and security can be secured.

In the invention, the first control unit adds first identification information identifying the head-mounted display apparatus to the captured image data and sends the captured image data to the information processing device.

According to this configuration, a leak of the captured image data to the outside can be prevented more efficiently, and security can be secured.

In the invention, the information processing device includes a second communication unit configured to perform communication, a data storage unit configured to associate the captured image data received from the head-mounted display apparatus with the first identification information, and store the data, an identification information storage unit configured to associate second identification information identifying an external device in which the captured image data is usable with the first identification information of the head-mounted display apparatus from which the external device corresponding to the second identification information is allowed to acquire captured image data, and store the information, and a second control unit configured to determine, when receiving the second identification information from the second communication unit, whether the second identification information coinciding with the second identification information received is stored in the identification information storage unit, and send, when the second identification information that is identical is stored in the identification information storage unit, the captured image data in the first identification information associated with the second identification information that is identical to the external device that has received the second identification information.

According to this configuration, a leak of the captured image data to the outside can be prevented more efficiently, and security can be secured.

To solve the above-described problems, the invention is a method for controlling a head-mounted display apparatus mounted on a head of a user. The method for controlling a head-mounted display apparatus includes executing control data that defines an operation flow including a plurality of operation steps in which an order of execution is predetermined, and controlling the imaging unit according to a setting that determines whether capturing by an imaging unit is permitted, the setting being included in each of the plurality of operation steps to be executed.

According to this configuration, when executing the control data, the control unit can control the propriety of using the captured image data in each of the operation steps. Therefore, securing of security and improvement in user convenience can be achieved by controlling the propriety of using the captured image data for each of the operation steps.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1is a system block diagram of an information processing system1.

The information processing system1includes, as a system configuration, a configuration in which a plurality of head-mounted display apparatuses (hereinafter expressed as head mounted displays (HMDs))100are connected to a server device5through a communication network7.FIG. 1illustrates two HMDs100being an HMD100A and an HMD100B, but the number of the HMDs100is not limited to two and is arbitrary. Further, when the HMD100A and the HMD100B do not need to be distinguished from each other hereinafter, the HMD100A and the HMD100B are expressed as the HMD100.

The communication network7is achieved by various communication lines, such as wireless communication lines including a public network, dedicated lines, and cellular phone lines, and communication lines of a backbone of these lines, or by a combination of the various communication lines. A specific configuration of the communication network7is not limited. The communication network7may be a wide-area communication network that can connect remote places, or a local area network (LAN) laid in specific facilities and buildings. Further, the communication network7may include a network appliance such as a server device, a gateway device, and a router device connecting the various communication lines described above.

The HMD100performs data communication with the other HMD100and the server device5through a wireless access point9. The wireless access point9is a communication device such as an access point and a router, and relays data communication between the HMD100A and the HMD100B and relays data communication between the HMDs100A and100B and the communication network7.

The server device5includes a communication unit51, a storage unit53, and a control unit55. The server device5corresponds to an “information processing device” of the invention. The communication unit51corresponds to a “second communication unit” of the invention. The storage unit53corresponds to a “data storage unit” and an “identification information storage unit” of the invention. The control unit55corresponds to a “second control unit” of the invention.

The communication unit51is connected to the communication network7, and performs data communication with the HMD100A, the HMD100B, and another device connected to the communication network7.

The storage unit53is formed of, for example, an auxiliary storage device such as a hard disk device and a semiconductor storage device. The storage unit53stores a control program executed by the control unit55and captured image data received from the HMD100. Further, the storage unit53stores an information management table57.

FIG. 2is a diagram illustrating a configuration of the information management table57.

The information management table57is a table in which HMD identification information, user identification information such as a user ID and a password, and image identification information are associated with terminal identification information and registered.

The HMD identification information is identification information about the HMD100. The HMD100A and the HMD100B are provided with different HMD identification information, and can be distinguished from each other based on the HMD identification information. For example, a media access control (MAC) address and an Internet protocol (IP) address can be used as the HMD identification information. Further, the HMD identification information may be set independently by a system manager of the information processing system1.

The user ID and the password included in the user identification information are information identifying each user using the information processing system1. When a plurality of users use one HMD100, each of the users using the HMD100needs to be identified. Thus, the user identification information is associated with the HMD identification information and registered in the information management table57.

The image identification information is identification information about captured image data received from the HMD100. The image identification information may be provided by the HMD100before captured image data is sent to the server device5, or by the server device5. The image identification information is associated with the HMD identification information and the user identification information and registered in the information management table57to identify a user who sends captured image data and the HMD100that receives the captured image data.

The terminal identification information is identification information identifying a terminal device (external device) permitted to download captured image data from the server device5. The terminal identification information corresponds to “second identification information” of the invention. A media access control (MAC) address and an Internet protocol (IP) address may be used as the terminal identification information, and a system manager of the information processing system1may independently set the terminal identification information.

In the exemplary embodiment, a case where a terminal device permitted to download captured image data is set for each HMD100is described. Thus, the terminal identification information is associated with the HMD identification information and registered in the information management table57. The terminal identification information is, for example, identification information about a terminal device used by a manager who manages a worker.

Further, identification information about the manager (manager identification information) may be associated with user information about a user who is a subordinate of the manager and registered in the information management table57. An ID and a password of the manager are used as the manager identification information. For this configuration, the manager inputs the ID and the password to log in to the server device5, and can download, from the server device5, captured image data uploaded to the server device5by a worker managed by the manager.

The control unit55includes a CPU, a ROM, a RAM, and the like (none of which are illustrated) as hardware. The RAM is used as a work area of the CPU. The CPU decompresses a control program read from the ROM and a storage unit (not illustrated) in the RAM, and controls each of the units of the server device5by executing the decompressed control program. The control unit55may include one CPU (or semiconductor chip) or include a plurality of CPUs (or semiconductor chips).

FIG. 3is a diagram illustrating an external configuration of the HMD100. Since the HMD100A and the HMD100B have almost the same configuration, a configuration of the HMD100will be described below. The HMD100is a display device including a control device10and an image display unit20.

The control device10includes an operation unit that accepts an operation by the user and functions as a controller with which the user operates the HMD100. The control device10accepts the operation by the user and controls the image display unit20in accordance with the accepted operation. The image display unit20is mounted on a head of a user and causes the user to visually recognize a virtual image. The user represents a person who wears the image display unit20on the head.

The control device10includes a flat box-shaped case10A (also may be referred to as a housing or a main body) as illustrated inFIG. 3. The case10A includes respective units of an operation button11, an LED indicator12, a track pad14, up and down keys15, a switching switch16, and a power switch18. The operation button11, the up and down keys15, the switching switch16, and the power switch18are collectively referred to as an operator13(FIG. 6). The user can operate the HMD100by operating the operator13and the track pad14.

The operation button11includes a menu key, a home key, a return key, and the like, and particularly, the keys and switches that are displaced by a pressing operation. The LED indicator12lights up or blinks in accordance with an operating state of the HMD100.

The track pad14includes an operation surface that detects a contact operation, and outputs an operation signal in response to an operation on the operation surface. Various detection methods such as an electrostatic method, a pressure detecting method, and an optical method can be adopted as a method for detecting an operation on the operation surface of the track pad14.

The up and down keys15are used to input an instruction to turn up or down the volume output from a right earphone32and a left earphone34and input an instruction to increase or decrease the brightness of display of the image display unit20. The switching switch16is a switch for switching an input corresponding to an operation of the up and down keys15. The power switch18is a switch for switching between ON and OFF of a power supply of the HMD100and is formed of, for example, a slide switch.

The image display unit20is a mounting body mounted on the head of the user, and has an eyeglasses-like shape in the exemplary embodiment. The image display unit20includes a right display unit22, a left display unit24, a right light-guiding plate26, and a left light-guiding plate28in a main body including a right holding portion21, a left holding portion23, and a front frame27.

The right holding portion21and the left holding portion23extend from both end portions of the front frame27and hold the image display unit20on the head of the user like temples (sidepieces) of eyeglasses. Herein, it is assumed that an end portion of both end portions of the front frame27located on the right side of the user wearing the image display unit20is an end portion ER, and an end portion located on the left side of the user is an end portion EL. The right holding portion21extends from the end portion ER of the frame27to a position corresponding to a right side of the head of the user wearing the image display unit20. The left holding portion23extends from the end portion EL to a position corresponding to a left side of the head of the user wearing the image display unit20.

The right light-guiding plate26and the left light-guiding plate28are provided in the front frame27. The right light-guiding plate26is located in front of a right eye of the user wearing the image display unit20, and causes the right eye to visually recognize an image. The left light-guiding plate28is located in front of a left eye of the user wearing the image display unit20, and causes the left eye to visually recognize an image.

The front frame27has such a shape that one end of the right light-guiding plate26and one end of the left light-guiding plate28are coupled to each other. This coupled position corresponds to a place between eyebrows of the user wearing the image display unit20. The front frame27may be provided with a nose pad portion contacting the nose of the user wearing the image display unit20in the coupled position of the right light-guiding plate26and the left light-guiding plate28. In this case, the nose pad portion, the right holding portion21, and the left holding portion23can hold the image display unit20on the head of the user. A belt (not illustrated) contacting the back of the head of the user wearing the image display unit20may be coupled to the right holding portion21and the left holding portion23. In this case, the belt can hold the image display unit20on the head of the user.

The right display unit22achieves display of an image by the right light-guiding plate26. The right display unit22is provided on the right holding portion21and located close to the right side of the head of the user wearing the image display unit20. The left display unit24achieves display of an image by the left light-guiding plate28. The left display unit24is provided on the left holding portion23and located close to the left side of the head of the user wearing the image display unit20.

The right light-guiding plate26and the left light-guiding plate28are optical portions formed by light transmissive resin and the like and are formed of, for example, a prism. The right light-guiding plate26and the left light-guiding plate28respectively guide image light output from the right display unit22and the left display unit24to the eyes of the user.

A light control plate (not illustrated) may be provided on the surfaces of the right light-guiding plate26and the left light-guiding plate28. The light control plate is a thin-plate optical element varying in transmittance depending on a wavelength range of light, and functions as a so-called a wavelength filter. The light control plate is, for example, disposed so as to cover a front side of the front frame27on a side opposite to a side of the eyes of the user. A light transmittance in an arbitrary wavelength range of visible light, infrared light, ultraviolet light, and the like can be adjusted by appropriately selecting an optical property of this light control plate. An amount of external light being incident on the right light-guiding plate26and the left light-guiding plate28from the outside and passing through the right light-guiding plate26and the left light-guiding plate28can be adjusted.

The image display unit20is a transmissive display device that causes an external scene to be visually recognized by allowing external light to pass through. The image display unit20guides image light generated by the right display unit22and the left display unit24to the right light-guiding plate26and the left light-guiding plate28, respectively. The image light guided by the right light-guiding plate26and the image light guided by the left light-guiding plate28are incident on the right eye and the left eye of the user and cause the user to visually recognize a virtual image. In this way, the image display unit20displays an image. A region in which the image display unit20can display an image in a visual field range FV of the user who wears the image display unit20on the head is referred to as a display region.

When external light passes through the right light-guiding plate26and the left light-guiding plate28from the front of the user and is incident on the eyes of the user, image light forming a virtual image and the external light are incident on the eyes of the user and intensity of the external light thus affects visibility of the virtual image. For this reason, ease of visual recognition of the virtual image can be adjusted by, for example, mounting the light control plate on the front frame27and appropriately selecting or adjusting an optical property of the light control plate. For a typical example, the light control plate having light transmissivity to the extent that the user wearing the HMD100can visually recognize at least an external scene can be used. Further, effects of protecting the right light-guiding plate26and the left light-guiding plate28and suppressing damage, adhesion of dirt, and the like to the right light-guiding plate26and the left light-guiding plate28can be expected from using the light control plate. The light control plate may be detachable from the front frame27or each of the right light-guiding plate26and the left light-guiding plate28, and different types of light control plates may be exchangeable and attachable. The image display unit20may not include the light control plate.

A camera60is disposed on the front frame27of the image display unit20. The camera60corresponds to an “imaging unit” of the invention. A configuration and arrangement of the camera60are determined such that the camera60captures an external scene visually recognized by the user wearing the image display unit20. The external scene represents a scene outside in a line-of-sight direction of the user wearing the image display unit20on the head. For example, the camera60is provided in a position of a front surface of the front frame27so as not to block external light passing through the right light-guiding plate26and the left light-guiding plate28. In the example illustrated inFIG. 3, the camera60is disposed on the end portion ER side of the front frame27, and functions as a detection unit that detects an object attached as an indicator to a part of the body or the body of the user.

The camera60is a digital camera including an imaging element such as a CCD and a CMOS and an imaging lens. In the exemplary embodiment, the camera60is a monocular camera, but may be a stereo camera. The camera60captures at least a part of an external scene in a direction of the front side of the HMD100, that is, in a view direction of the user wearing the HMD100. In other words, the camera60captures an image in a range or a direction overlapping a view of the user and captures an image in a direction in which the user fixates. A direction and a size of an angle of view of the camera60can be appropriately set. In the exemplary embodiment, as described later, the angle of view of the camera60includes the outside world visually recognized by the user through the right light-guiding plate26and the left light-guiding plate28. The angle of view of the camera60is more preferably set such that the camera60can capture the entire view of the user visually recognizable through the right light-guiding plate26and the left light-guiding plate28.

The camera60performs capturing in accordance with control by a capturing control unit153(FIG. 7) included in a control unit150. When the power switch18is turned on and a main processor140is activated by receiving the power supplied from a power supply unit130, the main processor140causes the power supply unit130to start supplying the power to the camera60and turns on the camera60. When the camera60turns on, the capturing control unit153causes the camera60to perform capturing at regular time intervals and generate captured image data. The captured image data is written to a memory121(seeFIG. 6) and temporarily stored.

The HMD100may include a distance sensor (not illustrated) that detects a distance between a measurement target located in a preset measurement direction and the HMD100. For example, the distance sensor may be disposed on a portion of the front frame27where the right light-guiding plate26and the left light-guiding plate28are coupled to each other. In this case, while the image display unit20is mounted, the distance sensor is located almost in the middle of both eyes of the user in a horizontal direction and above both eyes of the user in a perpendicular direction. The measurement direction of the distance sensor may be, for example, a direction of the front side of the front frame27, that is, a direction overlapping the capturing direction of the camera60. The distance sensor may include, for example, a light source such as an LED and a laser diode and a light receiving unit that receives reflection light obtained by light emitted from the light source being reflected by a measurement target. The distance sensor may perform triangular distance measuring processing or distance measuring processing based on a time difference in accordance with control by the control unit150. The distance sensor may include a sound source that generates ultrasonic waves and a detection unit that receives ultrasonic waves reflected by a measurement target. In this case, the distance sensor may perform distance measuring processing based on a time difference until the time when ultrasonic waves are reflected in accordance with control by the control unit150.

FIG. 4is a plan view of a principal part illustrating a configuration of an optical system provided in the image display unit20.FIG. 4illustrates a left eye LE and a right eye RE of the user for description.

As illustrated inFIG. 4, the right display unit22and the left display unit24are symmetrically formed. The right display unit22includes, as a configuration that causes the right eye RE of the user to visually recognize an image, an organic light emitting diode (OLED) unit221that emits image light L and a right optical system251including a lens group and the like that guide image light L emitted from the OLED unit221. The image light L is guided to the right light-guiding plate26by the right optical system251.

The OLED unit221includes an OLED panel223and an OLED driving circuit225that drives the OLED panel223. The OLED panel223is a spontaneous light-emitting display panel including light emitting elements arranged in a matrix. The light emitting elements are caused to emit respective color light in red (R), green (G), and blue (B) by organic electro luminescence. The OLED panel223includes a plurality of pixels on the assumption that a unit including each one of the R, G, and B elements is one pixel, and generates an image with the pixels arranged in a matrix. The OLED driving circuit225selects the light emitting element provided in the OLED panel223and energizes the light emitting element in accordance with control by the control unit150(FIG. 7), and causes the light emitting element of the OLED panel223to emit light. The OLED driving circuit225is fixed to a back surface of the OLED panel223, that is, the back side of a light-emitting surface by bonding and the like. The OLED driving circuit225may be formed of, for example, a semiconductor device that drives the OLED panel223and installed on a substrate (not illustrated) fixed to the back surface of the OLED panel223. A temperature sensor217is installed on this substrate.

Note that, the OLED panel223may include light emitting elements that emit light in white being arranged in a matrix and include color filters corresponding to respective colors of R, G, and B being disposed to overlap each other. The OLED panel223having a WRGB configuration including light emitting elements that emit white (W) light in addition to the light emitting elements that each radiate color light of R, G, and B may be used.

The right optical system251includes a collimating lens that collimates the image light L emitted from the OLED panel223. The image light L collimated by the collimating lens is incident on the right light-guiding plate26. A plurality of reflection surfaces that reflect the image light L are formed in an optical path for guiding light inside the right light-guiding plate26. The image light L is reflected a plurality of times inside the right light-guiding plate26and guided to the right eye RE side. A half mirror261(reflection surface) located in front of the right eye RE is formed in the right light-guiding plate26. The image light L is reflected by the half mirror261and emitted from the right light-guiding plate26toward the right eye RE. The image light L forms an image on a retina of the right eye RE and causes the user to visually recognize the image.

Further, the left display unit24includes, as a configuration that causes the left eye LE of the user to visually recognize an image, an OLED unit241that emits image light L and a left optical system252including a lens group and the like that guide image light L emitted from the OLED unit241. The image light L is guided to the left light-guiding plate28by the left optical system252.

The OLED unit241includes an OLED panel243and an OLED driving circuit245that drives the OLED panel243. The OLED panel243is a spontaneous light-emitting display panel formed similarly to the OLED panel223. The OLED driving circuit245selects the light emitting element provided in the OLED panel243and energizes the light emitting element in accordance with control by the control unit150(FIG. 7), and causes the light emitting element of the OLED panel243to emit light. The OLED driving circuit245is fixed to a back surface of the OLED panel243, that is, the back side of a light-emitting surface by bonding and the like. The OLED driving circuit245may be formed of, for example, a semiconductor device that drives the OLED panel243and installed on a substrate (not illustrated) fixed to the back surface of the OLED panel243. A temperature sensor239is installed on this substrate.

The left optical system252includes a collimating lens that collimates the image light L emitted from the OLED panel243. The image light L collimated by the collimating lens is incident on the left light-guiding plate28. The left light-guiding plate28is an optical element in which a plurality of reflection surfaces that reflect the image light L are formed, and is a prism, for example. The image light L is reflected a plurality of times inside the left light-guiding plate28and guided to the left eye LE side. A half mirror281(reflection surface) located in front of the left eye LE is formed in the left light-guiding plate28. The image light L is reflected by the half mirror281and emitted from the left light-guiding plate28toward the left eye LE. The image light L forms an image on a retina of the left eye LE and causes the user to visually recognize the image.

According to this configuration, the HMD100functions as a see-through display device. In other words, the image light L reflected by the half mirror261and external light OL passing through the right light-guiding plate26are incident on the right eye RE of the user. Also, the image light L reflected by the half mirror281and external light OL passing through the left light-guiding plate28are incident on the left eye LE of the user. In this way, the HMD100causes the image light L of the image processed internally and the external light OL to overlap each other and be incident on the eyes of a user. The user can see an external scene through the right light-guiding plate26and the left light-guiding plate28. The image by the image light L overlapping the external scene or around the external scene is displayed to be visually recognizable. The half mirrors261and281are image extracting units that reflect image light output from the right display unit22and the left display unit24, respectively, and extract an image, and may be referred to as display units.

Note that, the left optical system252and the left light-guiding plate28are collectively referred to as a “left light-guiding unit”, and the right optical system251and the right light-guiding plate26are collectively referred to as a “right light-guiding unit”. The configurations of the right light-guiding unit and the left light-guiding unit are not limited to the above-described example. An arbitrary method can be used when image light is used to form a virtual image in front of the eyes of the user. For example, a diffraction grating or a semitransparent reflective film may be used.

Referring back toFIG. 3, the control device10and the image display unit20are connected to each other with a connection cable40. The connection cable40is removably connected to a connector (not illustrated) provided in a lower portion of the case10A, and connects a tip of the left holding portion23and various circuits provided inside the image display unit20. The connection cable40includes a metal cable or an optical fiber cable that transmits digital data, and may include a metal cable that transmits an analog signal. A connector46is provided at a mid-point of the connection cable40. The connector46is a jack to which a stereo mini-plug is connected. The connector46and the control device10are connected to each other with, for example, a line that transmits an analog audio signal. In the configuration example illustrated inFIG. 3, a headset30including the right earphone32and the left earphone34constituting a stereo headphone and a microphone63is connected to the connector46.

The control device10and the image display unit20may be connected to each other in a wireless manner. For example, the control device10and the image display unit20may exchange a control signal and data through wireless communication in conformity with standards such as Bluetooth (trade name) and a wireless LAN (including Wi-Fi (trade name)).

As illustrated inFIG. 3, for example, the microphone63is disposed such that a sound collecting unit of the microphone63is directed in the line-of-sight direction of the user, collects sound, and outputs an audio signal to an audio interface180(FIG. 6). The microphone63may be, for example, a monaural microphone or a stereo microphone, may be a directional microphone, and may be a non-directional microphone.

FIG. 5is a perspective view illustrating a configuration of the image display unit20, and illustrates a principal configuration of the image display unit20seen from a head of the user side.FIG. 5illustrates a side contacting the head of the user of the image display unit20, that is, a side seen from the right eye RE and the left eye LE of the user. In other words, the back side of the right light-guiding plate26and the left light-guiding plate28is seen.

InFIG. 5, the half mirror261that irradiates the right eye RE of the user with the image light and the half mirror281that irradiates the left eye LE with the image light are each seen as a substantially rectangular region. The entire right light-guiding plate26and the entire left light-guiding plate28respectively including the half mirrors261and281allow the external light to pass through as described above. Thus, the user visually recognizes an external scene through the entire right light-guiding plate26and the entire left light-guiding plate28, and visually recognizes a rectangular display image in a position of each of the half mirrors261and281.

The camera60is disposed at the end portion on the right side of the image display unit20, and captures an image in a direction in which both eyes of the user are directed, that is, in the front for the user. An optical axis of the camera60is assumed to be in a direction including the line-of-sight direction of the right eye RE and the left eye LE. An external scene visually recognizable by the user wearing the HMD100is not necessarily infinitely distant. For example, when the user fixates on a target located in front of the user with both eyes, a distance between the user and the target is often 30 cm to 10 m approximately and more often 1 m to 4 m approximately. Thus, estimates of an upper limit and a lower limit of the distance between the user and the target in normal use of the HMD100may be determined. The estimates may be obtained from an investigation or an experiment, or may be set by the user. The optical axis and the angle of view of the camera60are preferably set such that a target is included in the angle of view when the distance between the user and the target in normal use corresponds to set estimates of an upper limit and a lower limit.

In general, a visual field angle of a person is approximately 200 degrees in the horizontal direction and approximately 125 degrees in the vertical direction. An effective visual field at the visual field angle having an excellent information receiving capacity is approximately 30 degrees in the horizontal direction and 20 degrees in the vertical direction. Furthermore, it is assumed that a stable field of fixation in which a stable fixation point fixated on by a person is seen quickly is approximately from 60 to 90 degrees in the horizontal direction and from 45 to 70 degrees in the vertical direction. When a fixation point is on a target located in front of the user, the effective visual field in the visual field of the user is approximately 30 degrees in the horizontal direction and 20 degrees in the vertical direction with the line-of-sight direction of each of the right eye RE and the left eye LE at the center. The stable field of fixation is approximately from 60 to 90 degrees in the horizontal direction and from 45 to 70 degrees in the vertical direction. The visual field angle is approximately 200 degrees in the horizontal direction and 125 degrees in the vertical direction. Furthermore, the actual visual field visually recognized by the user through the right light-guiding plate26and the left light-guiding plate28may be referred to as an actual field of view (FOV). In the configuration of the exemplary embodiment illustrated inFIGS. 3 and 4, the actual field of view corresponds to the actual visual field visually recognized by the user through the right light-guiding plate26and the left light-guiding plate28. The actual field of view is narrower than the visual field angle and the stable field of fixation, but is wider than the effective visual field.

The camera60captures an image in a range including an external scene visually recognizable together with an image displayed by the image display unit20. The angle of view of the camera60is preferably set such that the camera60can capture an image in a range wider than a visual field of the user. Specifically, the angle of view is preferably set to be wider than at least an effective visual field of the user. The angle of view is also preferably set to be wider than an actual field of view of a user. The angle of view is more preferably set to be wider than a stable field of fixation of the user. It is the most preferable that the angle of view be wider than a visual field angle of both eyes of the user.

The camera60may include a so-called wide-angle lens as an imaging lens and be able to capture an image at a wide angle of view. The wide-angle lens may include a lens called a super-wide-angle lens or a semi-wide-angle lens, and may be a single-focus lens or a zoom lens. The camera60may include a lens group including a plurality of lenses.

FIG. 6is a block diagram illustrating a configuration of each of the units constituting the HMD100.

The control device10includes the main processor140that executes a program to control the HMD100. The memory121and a nonvolatile storage unit123are connected to the main processor140. A six-axis sensor111and a magnetic sensor113as types of sensors are connected to the main processor140. A GPS receiving unit115, a communication unit117, an audio codec182, an external connector184, an external memory interface186, a USB connector188, a sensor hub192, and an FPGA194are connected to the main processor140. The components function as interfaces to the outside. An LED indicator12, an LED display unit17, a vibrator19, an operation unit110, and the power supply unit130are connected to the main processor140.

The main processor140is installed in a controller substrate120built in the control device10. The memory121, the nonvolatile storage unit123, the six-axis sensor111, the magnetic sensor113, the GPS receiving unit115, the communication unit117, the audio codec182, and the like in addition to the main processor140are further installed in the controller substrate120. In the exemplary embodiment, the external connector184, the external memory interface186, the USB connector188, the sensor hub192, the FPGA194, and an interface196are installed in the controller substrate120.

When the main processor140executes a control program, the memory121constitutes a work area in which the control program to be executed and data to be processed are temporarily stored. The memory121corresponds to an “image storage unit” of the invention. The volatile storage unit123is formed of a flash memory and an embedded multi media card (eMMC). The volatile storage unit123stores a program executed by the main processor140and various data processed by the main processor140executing a program.

FIG. 6illustrates the configuration that achieves the functions of the control device10by one main processor140, but a plurality of processors or semiconductor chips may achieve the functions of the control device10. For example, a co-processor such as a system-on-a-chip (SoC), a micro control unit (MCU), and a field-programmable gate array (FPGA) may be further installed in the controller substrate120. The control device10may cause both of the main processor140and the co-processor to work in cooperation, or may perform various control by selectively using one of the main processor140and the co-processor.

The six-axis sensor111is a motion sensor (an inertial sensor) including a three-axis acceleration sensor and a three-axis gyro (angular velocity) sensor. An inertial measurement unit (IMU) in which the above-described sensors are modularized may be adopted as the six-axis sensor111. The magnetic sensor113is, for example, a three-axis geomagnetic sensor.

The six-axis sensor111and the magnetic sensor113each output a detected value to the main processor140according to predesignated sampling cycles. The six-axis sensor111and the magnetic sensor113each output a detected value to the main processor140at timings designated by the main processor140according to a request from the main processor140.

The GPS receiving unit115includes a GPS antenna, which is not illustrated, and receives a GPS signal sent from a GPS satellite. The GPS receiving unit115outputs the received GPS signal to the main processor140. The GPS receiving unit115measures a signal strength of the received GPS signal and outputs the signal strength to the main processor140. For example, information about received signal strength indication (RSSI), electric field strength, magnetic field strength, and a signal-to-noise ratio (SNR) may be used for the signal intensity.

The communication unit117performs wireless communication between an external apparatus and the communication unit117. The communication unit117includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or is formed of a device that combines an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like. The communication unit117performs wireless communication in conformity with standards such as Bluetooth and a wireless LAN (including Wi-Fi). The communication unit117corresponds to a “first communication unit” of the invention.

The audio interface180is an interface that inputs and outputs an audio signal. In the exemplary embodiment, the audio interface180includes the connector46(FIG. 3) provided on the connection cable40. The connector46is connected to the headset30. An audio signal output from the audio interface180is input to the right earphone32and the left earphone34, and thus the right earphone32and the left earphone34output sound. The microphone63provided in the headset30collects sound and outputs an audio signal to the audio interface180. The audio signal input from the microphone63to the audio interface180is input to the external connector184.

The audio codec182is connected to the audio interface180, and encodes and decodes an audio signal input and output through the audio interface180. The audio codec182may include an A/D converter that converts an analog audio signal to digital audio data and a D/A converter that converts digital audio data to an analog audio signal in reverse. For example, the HMD100in the exemplary embodiment outputs sound to the right earphone32and the left earphone34and collects sound with the microphone63. The audio codec182converts digital audio data output from the main processor140to an analog audio signal, and outputs the analog audio signal through the audio interface180. The audio codec182converts an analog audio signal input to the audio interface180to digital audio data, and outputs the digital audio data to the main processor140.

The external connector184is a connector that connects the main processor140and an external device communicating with the main processor140. The external connector184is an interface that connects an external device to the main processor140when, for example, the external device is connected to debug a program executed by the main processor140and collect logs of operations of the HMD100.

The external memory interface186is an interface to which a portable memory device can be connected, and includes, for example, a memory card slot capable of reading data with a card-type recording medium attached and an interface circuit. A size, a shape, and a specification of the card-type recording medium in this case are not restricted and may be appropriately changed.

The universal serial bus (USB) connector188includes a connector and an interface circuit in conformity with USB standards. The USB connector188can connect a USB memory device, a smartphone, a computer, and the like. A size, a shape, a version of conformable USB standards of the USB connector188may be appropriately selected and changed.

The sensor hub192and the FPGA194are connected to the image display unit20through the interface (I/F)196. The sensor hub192acquires a detected value of various sensors provided in the image display unit20and outputs the detected value to the main processor140. The FPGA194performs processing on data transmitted and received between the main processor140and each of the units of the image display unit20and transmission through the interface196.

The LED indicator12lights up or blinks in accordance with an operating state of the HMD100. The LED display unit17controls turning on and turning off of the LED indicator12in accordance with control by the main processor140. The LED display unit17may include an LED (not illustrated) disposed directly below the track pad14and a driving circuit that turns on this LED. In this case, the LED display unit17causes the LED to turn on, blink, and turn off in accordance with control by the main processor140.

When captured image data of the camera60is stored in a storage unit160and uploaded to the server device5, the LED indicator12lights up or blinks, and notifies that the captured image data is stored.

The vibrator19includes a motor and an eccentric rotor (none of which are illustrated), and may include another necessary configuration. The vibrator19rotates the motor in accordance with control by the main processor140to generate vibrations. For example, when an operation on the operation unit110is detected, when captured image data of the camera60is stored in the storage unit160, and when captured image data is uploaded the server device5, the HMD100causes the vibrator19to generate vibrations in predetermined vibration patterns.

The operation unit110includes the operator13and the track pad14. The operator13includes the operation button11, the up and down keys15, the switching switch16, and the power switch18. When the operator13and the track pad14accept an operation, the operation unit110outputs, to the control unit150, an operation signal including identification information about the operator13and the track pad14that have accepted the operation and information indicating a content of the accepted operation.

In the exemplary embodiment, any of the keys included in the operator13is associated with an operation of a shutter button of the camera60. When a key of the operator13is pressed, the capturing control unit153described later causes the camera60to perform capturing. In the exemplary embodiment, any of the keys included in the operator13is associated with the operation of the shutter button, but an operation of the track pad14may be associated with the operation of the shutter button.

The control device10includes the power supply unit130and operates with power supplied from the power supply unit130. The power supply unit130includes a chargeable battery132and a power supply control circuit134that controls detection of remaining capacity of the battery132and charging of the battery132. The power supply control circuit134is connected to the main processor140, and outputs a detected value of remaining capacity of the battery132or a detected value of voltage to the main processor140. Power may be supplied from the control device10to the image display unit20based on power supplied by the power supply unit130. The main processor140may be able to control a state of the power supplied from the power supply unit130to each of the units of the control device10and the image display unit20.

The right display unit22and the left display unit24of the image display unit20are each connected to the control device10. As illustrated inFIG. 3, in the HMD100, the connection cable40is connected to the left holding portion23, wiring connected to this connection cable40is laid inside the image display unit20, and the right display unit22and the left display unit24are each connected to the control device10.

The right display unit22includes a display unit substrate210. An interface (I/F)211connected to the interface196, a reception unit (Rx)213that receives data input from the control device10through the interface211, and an EEPROM215are installed in the display unit substrate210.

The interface211connects the reception unit213, the EEPROM215, the temperature sensor217, the camera60, an illumination sensor65, and an LED indicator67to the control device10.

The electrically erasable programmable read-only memory (EEPROM)215stores various data readable by the main processor140. The EEPROM215stores, for example, data about a light-emitting property and a display property of the OLED units221and241provided in the image display unit20, data about a property of a sensor provided in the right display unit22or the left display unit24, and the like. Specifically, the EEPROM215stores a parameter according to gamma correction of the OLED units221and241, data compensating for a detected value of the temperature sensors217and239, and the like. These data are generated when the HDM100is inspected at the time of shipment from a factory and written into the EEPROM215. After the shipment, the main processor140uses the data in the EEPROM215to perform processing.

The camera60performs capturing in response to a signal input through the interface211, and outputs captured image data or a signal indicating a capturing result to the control device10.

As illustrated inFIG. 3, the illumination sensor65is provided at the end portion ER of the front frame27and disposed so as to receive external light from the front of the user wearing the image display unit20. The illumination sensor65outputs a detected value corresponding to an amount of received light (strength of received light).

As illustrated inFIG. 3, the LED indicator67is disposed at the end portion ER of the front frame27close to the camera60. The LED indicator67lights up while the camera60performs capturing, and notifies that capturing is being performed. When captured image data of the camera60is stored in the storage unit160and uploaded to the server device5, the LED indicator67lights up or blinks, and notifies that the captured image data is stored.

The temperature sensor217detects a temperature and outputs, as a detected value, a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor217is installed on the back surface side of the OLED panel223(FIG. 4). The temperature sensor217may be installed in the same substrate as the OLED driving circuit225, for example. With this configuration, the temperature sensor217detects a temperature of mainly the OLED panel223.

The reception unit213receives data sent from the main processor140through the interface211. When receiving image data about an image displayed on the OLED unit221, the reception unit213outputs the received image data to the OLED driving circuit225(FIG. 4).

The left display unit24includes a display unit substrate230. An interface (I/F)231connected to the interface196and a reception unit (Rx)233that receives data input from the control device10through the interface231are installed in the display unit substrate230. A six-axis sensor235and a magnetic sensor237are installed in the display unit substrate230. The interface231connects the reception unit233, the six-axis sensor235, the magnetic sensor237, and the temperature sensor239to the control device10.

The six-axis sensor235is a motion sensor (an inertial sensor) including a three-axis acceleration sensor and a three-axis gyro (angular velocity) sensor. An IMU in which the above-described sensors are modularized may be adopted as the six-axis sensor235. The magnetic sensor237is, for example, a three-axis geomagnetic sensor.

The temperature sensor239detects a temperature and outputs, as a detected value, a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor239is installed on the back surface side of the OLED panel243(FIG. 4). The temperature sensor239may be installed in the same substrate as the OLED driving circuit245, for example. With this configuration, the temperature sensor239detects a temperature of mainly the OLED panel243. The temperature sensor239may be built in the OLED panel243or the OLED driving circuit245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel243serving as a Si-OLED is installed as an integrated circuit on an integrated semiconductor chip together with the OLED driving circuit245and the like, the temperature sensor239may be installed in this semiconductor chip.

The camera60, the illumination sensor65, and temperature sensor217provided in the right display unit22and the six-axis sensor235, the magnetic sensor237, and the temperature sensor239provided in the left display unit24are connected to the sensor hub192.

The sensor hub192sets and initializes a sampling cycle of each of the sensors in accordance with control by the main processor140. The sensor hub192energizes each of the sensors, sends control data, acquires a detected value, and the like in accordance with a sampling cycle of each of the sensors. The sensor hub192also outputs a detected value of each of the sensors provided in the right display unit22and the left display unit24to the main processor140at preset timing. The sensor hub192may include a function of temporarily holding a detected value of each of the sensors in accordance with timing of output to the main processor140. The sensor hub192may include a function of converting an output value of each of the sensors to data in unified data format to handle a difference in signal format or data format of the output value and outputting the data to the main processor140.

The sensor hub192starts and stops a current to pass through the LED indicator67in accordance with control by the main processor140, and causes the LED indicator67to light up or blink in accordance with timing at which the camera60starts and ends capturing.

FIG. 7is a function block diagram of the storage unit160and the control unit150constituting a control system of the control device10. The storage unit160illustrated inFIG. 7is a logical storage unit formed of the nonvolatile storage unit123(FIG. 6) and the EEPROM215.

The storage unit160stores an application program161, process flow data162, content data163, HMD identification information164, and user identification information165.

The application program161is an application program including a specific function executed on an OS151. The application program161is a program configured to allow the HMD100to execute a process flow created with a PC300(seeFIG. 12) described later. The application program161may be a special application program configured to execute a process flow, or may be a general-purpose program, such as a web browser. A specific configuration of the application program161can be selected as desired in conformity with a specification of the process flow data162.

The process flow data162corresponds to “control data” of the invention and is data about a process flow created with the PC300. The control unit150acquires output data327(seeFIG. 12) created with the PC200and causes the storage unit160to store the output data327as the process flow data162. The process flow data162may be general-purpose data executable by various apparatuses (for example, a device without the camera60) including the HMD100and an HMD100with a different type or specification.

Herein, a process flow and the process flow data162will be described.

The process flow is a so-called work flow representing a business operation including a plurality of work operations. The process flow is a specific example of an “operation flow” of the invention. The operation flow includes an operation step or a plurality of operation steps, and represents a sequence in which an order of execution of the operation steps is defined. For example, the operation flow represents processing of outputting information about display, inputting information, making a determination, and the like. The operation step is regarded as one unit when creating and editing an operation flow, but otherwise is not particularly restricted. The operation step may include a plurality of operations or processing.

The process flow includes a plurality of work operations performed by an operator to achieve a set object, and includes a process configured to output information to the operator who is engaged with the plurality of work operations. One unit of a work operation performed by an operator and one output of information to the operator each corresponds to a work block in an operation flow. The work block corresponds to an “operation step” of the invention.

Further, the process flow data162is data indicating a work process of work operations performed by an operator. The process flow data162includes work blocks included in a process flow, an order of execution of the work blocks, and setting contents set to each of the work blocks, and may include other information.

The content data163is data about contents including a display image and a video displayed by the image display unit20under control of the control unit150, and includes image data or video data. The content data163may include music and audio data.

The HMD identification information164is identification information identifying the HMD100. For example, a MAC address and an IP address of the HMD100can be used as the HMD identification information164.

The user identification information165is a user ID and a password that identify a user of the HMD100and are set by the user. When a plurality of users share one HMD100, each of the users needs to be identified. Thus, when using the HMD100, the user inputs a user ID and a password from the operation unit110. The control unit150performs user authentication with the user ID and the password received from the operation unit110and performs log-in processing.

The control unit150performs various processing by using data stored in the storage unit160and controls the HMD100. The control unit150corresponds to a “control unit” and a “first control unit” of the invention. The control unit150includes, as functional blocks, the operating system (OS)151, an image processing unit152, the capturing control unit153, an input detection unit154, a communication control unit155, a display control unit156, an information acquisition unit157, and a process flow execution unit158. The functional blocks represent, as blocks, functions achieved by the main processor140executing a control program for the sake of convenience, and do not represent a specific application program and hardware.

A function of the OS151is a function of a control program stored in the storage unit160. The other units are functions of an application program executed on the OS151.

For example, the image processing unit152reads the content data163from the storage unit160and separates a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the read content data163. The image processing unit152generates a clock signal PCLK by using a phase locked loop (PLL) circuit and the like (not illustrated) in accordance with cycles of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit152may perform various image processing such as processing of converting a resolution, adjusting brightness and saturation, and performing 2D/3D conversion on image data included in the content data163as necessary.

The image processing unit152decompresses every one frame, which is a display unit of an image, of image data subjected to the image processing in a DRAM in the storage unit160. Hereinafter, a region of the DRAM in which one frame of image data is decompressed is referred to as a frame region. The image processing unit152reads image data from the frame region and causes the image display unit20to display the read image data.

The image processing unit152may be formed of the main processor140and other hardware (such as a digital signal processor (DSP)) other than the configuration achieved by the main processor140executing a program.

The capturing control unit153controls the camera60. While the power of the HMD100is on, the capturing control unit153causes the camera60to perform capturing at regular time intervals to generate captured image data. The capturing control unit153also causes the camera60to perform capturing to generate captured image data when the key of the operator13associated with the operation of the shutter button is operated. The captured image data captured by the camera60is written to the memory121and temporarily stored. Since the memory121is a volatile memory, the captured image data written to the memory121is deleted when the power of the HMD100is turned off. The capturing control unit153overwrites the memory121with captured image data generated at regular time intervals and stores the captured image data. In other words, a region of the memory121in which the captured image data has been written is overwritten with newly generated captured image data.

When a predetermined condition is satisfied, the capturing control unit153causes the storage unit160to store the captured image data written to the memory121such that the captured image data is not deleted. The predetermined condition represents a case where the control unit150executes the process flow data162and a case where use of captured image data is permitted in a work block included in the process flow data162. In the exemplary embodiment, “use” means that captured image data is permitted to be stored and processing by using the captured image data is permitted during or after execution of the process flow data162. The capturing control unit153causes the storage unit160to store captured image data captured during execution of a work block permitted to use the captured image data. The capturing control unit153reads captured image data captured during execution of a work block permitted to use the captured image data from the memory121and uploads the captured image data to the server device5.

When executing a work block permitted to use captured image data during execution of the process flow data162, the capturing control unit153may bring the camera60into a state of operating by supplying the power to the camera60. In this case, the capturing control unit153causes the camera60to perform capturing at regular time intervals and causes the storage unit160to store captured image data. The capturing control unit153may also cause the camera60to perform capturing when the key of the operator13associated with the operation of the shutter button is operated.

When the execution of a work block permitted to use captured image data is completed, the capturing control unit153brings the camera60into a state of stopping operation by stopping to supply the power to the camera60.

The input detection unit154detects an input from an operator based on operation data input from the operation unit110.

The communication control unit155performs data communication with the server device5and the other HMD100by controlling the communication unit117.

The display control unit156controls and causes the image display unit20to display various screens in a display region. For example, when executing the application program161, the display control unit156causes various screens corresponding to work blocks to be displayed.

The information acquisition unit157reads data from an external device to which the information acquisition unit157is connected through the external connector184, the external memory interface186, and the USB connector188. For example, when a storage device connected to the external memory interface186is detected, the information acquisition unit157reads data from the storage device, and causes the storage unit160to store the data. The HMD100can thus read and use the process flow data162stored in the storage device.

The process flow execution unit158executes the application program161stored in the storage unit160, and executes a process flow in accordance with the process flow data162. The process flow data162includes at least work blocks included in a process flow, an order of execution of the work blocks, and setting contents set to each of the work blocks.

The setting contents set to each of the work blocks include processing performed by the control unit150. The processing performed by the control unit150includes processing of causing a display screen including an image and text to be displayed as information for an operator and reading of a two-dimensional code (2D code) such as a QR code (registered trademark). Further, the processing performed by the control unit150includes processing of accepting an input of text and accepting an input of whether a work result is positive or negative. Further, the processing performed by the control unit150includes, as work blocks, accepting a selection input using a check box, accepting a selection input using a radio button, and the like.

For example, the work blocks include at least a procedure display block, a read block, an input block, a determination block, a selection input block, and a termination block.

The procedure display block is a block that causes the image display unit20to display a screen and text indicating a work procedure in a display region. An operator conducts work while referring to the screen and the text displayed in the display region by the image display unit20.

The read block is a block that causes the camera60to perform capturing and reads a QR code (registered trademark) and a two-dimensional code such as a bar code from captured image data of the camera60.

For example, the control unit150causes a read frame of a two-dimensional code and a guide display that guides an operator to place the two-dimensional code in the read frame to be displayed. When an image of the two-dimensional code is detected from the read frame, the control unit150causes the storage unit160to store the detected image of the two-dimensional code and analyzes code information included in the two-dimensional code.

The input block is a block that accepts an input from an operator from the operation unit110and the like. A method for accepting an input from an operator is not limited to the method using the operation unit110. For example, text recognition processing using captured image data of the camera60and sound recognition processing for sound collected with the microphone63may be used to detect various inputs including text. A bar code reader connected with the external connector184may be used to read a bar code and to acquire read data. The acquired data may then be accepted as an input from an operator. A motion of an operator acquired from a detected value of the six-axis sensors235and111may be accepted as an input from an operator.

The determination block is a block that accepts a result of executing work from an operator. For example, the operator conducts work according to a work procedure displayed in the procedure display block, and inputs a work result indicating “positive” or “negative” as a work result from the operation unit110and the like. When the control unit150accepts an input of “positive” from the operation unit110, the control unit150makes an affirmative determination. In this case, the control unit150executes a work block corresponding to the affirmative determination. For example, the control unit150executes a procedure display block and causes a screen displaying a next work content to be displayed. When the control unit150accepts an input of “negative” from the operation unit110, the control unit150makes a negative determination. In this case, the control unit150executes a work block corresponding to the negative determination. For example, the control unit150causes the same work content to be displayed again and the same work to be executed again, causes the camera60to capture a work result, and the like.

The selection input block is a block that accepts a selection input using a check box and a radio button. When the selection input block is executed, the control unit150causes a check box and a radio button to be displayed to be visually recognizable by an operator and accepts an operation of selecting the check box or the radio button. When accepting the selection operation, the control unit150executes a work block corresponding to the accepted selection.

The termination block is a block that indicates an end of a process flow.

Each of the work blocks including the procedure display block, the read block, the input block, and the determination block includes permission information set with whether use of captured image data is permitted as a setting content. When the key corresponding to the shutter button of the operation unit110is pressed during execution of a work block including the permission information in which permission is set, the control unit150causes the storage unit160to store captured image data captured by the camera60. When the key corresponding to the shutter button of the operation unit110is pressed during execution of a work block including the permission information in which permission is set, the control unit150uploads captured image data captured by the camera60to the server device5.

Switching between storing of captured image data in the storage unit160and uploading of the captured image data to the server device5may be set as a setting content for each of the work blocks, or may be set by each operator. In other words, setting information in which whether uploading of captured image to the server device5or storing of the captured image data in the storage unit160is set may be associated with user information and stored in the storage unit160.

When a work block during execution is a block that does not include the permission information, or when rejection is set in the permission information, the control unit150does not cause the storage unit160to store captured image data captured by the camera60. Similarly, when a work block during execution does not include the permission information, or when rejection is set in the permission information, the control unit150does not upload captured image data captured by the camera60to the server device5.

FIG. 8is a flowchart illustrating a first operation of the control unit150. The first operation is an operation of the control unit150to cause the storage unit160to store captured image data when a work block including permission information in which permission is set is executed.

The control unit150reads the process flow data162from the storage unit160, and executes a work block included in the process flow data162. Specifically, the control unit150selects a work block executed according to an order of execution included in the process flow data162(Step S1). Next, the control unit150determines whether the selected work block is a termination block (Step S2). When the selected work block is the termination block (YES in Step S2), the control unit150ends this processing flow. When the selected work block is not the termination block (NO in Step S2), the control unit150determines whether permission information is included in the selected work block (Step S3).

When the permission information is not included in the selected work block (NO in Step S3), the control unit150performs processing according to a setting content of the work block (Step S4). Examples of the processing performed by the control unit150include processing of causing the image display unit20to display a screen and text indicating a work content and a work procedure to be performed by an operator, processing of causing the camera60to perform capturing and reading a two-dimensional code, and the like.

The control unit150performs the processing according to the setting content and then determines whether the processing set to the setting content of the selected work block is completed (Step S5). For example, when the processing set to the setting content of the selected work block is processing of reading a two-dimensional code, the control unit150determines whether an image of the two-dimensional code has been acquired from captured image data of the camera60. When the image of the two-dimensional code has been able to be acquired, the control unit150determines that the processing set to the setting content of the work block is completed.

When the processing set to the setting content of the work block is not completed (NO in Step S5), the processing returns to Step S4and the control unit150continues the processing according to the setting content. When the processing set to the setting content of the work block is completed (YES in Step S5), the processing returns to Step S1and the control unit150selects a next work block (Step S1).

When the control unit150determines that the permission information is included in Step S3(YES in Step S3), the control unit150determines whether “permission” is set in the permission information (Step S6). When “permission” is not set in the permission information, that is, when “rejection” is set (NO in Step S6), the processing proceeds to Step S4and the control unit150performs the processing according to the setting content of the work block. Subsequently, the control unit150determines whether the processing set in the work block is completed in Step S5.

When “permission” is set in the permission information (YES in Step S6), the control unit150performs the processing according to the setting content set in the work block (Step S7). The control unit150determines whether an operation on the key corresponding to the shutter button has been accepted during execution of the processing (Step S8). When the operation on the key corresponding to the shutter button has not been accepted (NO in Step S8), the control unit150determines whether the processing set to the setting content of the work block is completed (Step S12). When the processing set to the setting content of the work block is not completed (NO in Step S12), the processing returns to Step S8, and the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted during execution of the processing (Step S8). When the processing set to the setting content of the work block is completed (YES in Step S12), the processing returns to Step S1and the control unit150selects a next work block (Step S1).

When the operation on the key corresponding to the shutter button has been accepted (YES in Step S8), the control unit150causes the camera60to perform capturing (Step S9) and causes the storage unit160to store captured image data being captured (Step S10). The control unit150makes notification when causing the storage unit160to store the captured image data (Step S11). In the exemplary embodiment, the control unit150causes at least one of the LED indicators12and67to light up or blink as the notification. In addition, the control unit150may make the notification by causing the earphones32and34to output preset sound, or by causing the vibrator19to vibrate in predetermined vibration patterns. The LED indicators12and67, the earphones32and34, and the vibrator19correspond to “notification units” of the invention.

The control unit150makes the notification and causes the storage unit160to store the captured image data, and then determines whether the processing set to the setting content of the work block is completed (Step S12). When the processing set to the setting content of the work block is not completed (NO in Step S12), the processing returns to Step S8, and the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted during execution of the processing (Step S8). When the processing set to the setting content of the work block is completed (YES in Step S12), the processing returns to Step S1and the control unit150selects a next work block (Step S1).

FIG. 9is a flowchart illustrating a second operation of the control unit150. The second operation is an operation to upload captured image data to the server device5when a work block including permission information in which permission is set is executed. Note that, inFIG. 9, operations in Steps S21to S27are the same as the first operation described with reference toFIG. 8, and thus description will be omitted.

When “permission” is set in the permission information (YES in Step S26), the control unit150performs the processing according to the setting content set in the work block (Step S27). The control unit150determines whether an operation on the key corresponding to the shutter button has been accepted during execution of the processing (Step S28). When the operation on the key corresponding to the shutter button has not been accepted (NO in Step S28), the processing proceeds to Step S32, and the control unit150determines whether the processing set to the setting content of the work block is completed (Step S32). When the processing is completed (YES in Step S32), the processing proceeds to Step S21and the control unit150selects a work block.

When the operation on the key corresponding to the shutter button has been accepted (YES in Step S28), the control unit150causes the camera60to perform capturing (Step S29), and writes captured image data being captured to the memory12and causes the memory12to temporarily store the captured image data. The control unit150generates image identification information identifying captured image data, and associates the generated image identification information with the captured image data. For example, a serial number and the like associated with each captured image data uploaded to the server device5by the HMD100is used as the image identification information.

Next, the control unit150acquires operator information about an operator in a logged-in state and HMD identification information from the storage unit160(Step S30). The operator information acquired from the storage unit160may be at least one of a user ID and a password. Next, the control unit150adds the image identification information, the operator information, and the HMD identification information to the captured image data temporarily stored in the memory121and uploads the captured image data to the server device5(Step S31). Operator information, a user ID, and a password correspond to “limited information” of the invention. Further, HMD identification information corresponds to “first identification information” of the invention.

Timing at which captured image data of the camera60is uploaded to the server device5is not limited to only timing at which the camera60performs capturing to generate captured image data. For example, captured image data may be uploaded to the server device5when processing set to a setting content of a work block is completed and before a next work block to be executed is selected in Step S21, or after execution of the process flow data162. Further, captured image data may also be uploaded to the server device5in a shutdown sequence after the power of the HMD100is turned off.

The control unit150uploads the captured image data to the server device5, and then determines whether the processing set to the setting content of the work block is completed (Step S32). When the processing set to the setting content of the work block is not completed (NO in Step S32), the processing returns to Step S28, and the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted during execution of the processing (Step S28). When the processing set to the setting content of the work block is completed (YES in Step S32), the processing returns to Step S21and the control unit150selects a next work block (Step S21).

FIG. 10is a flowchart illustrating an operation of the server device5.

When the communication unit51receives data through the communication network7(YES in Step S41), the control unit55identifies the received data. First, the control unit55determines whether captured image data sent from the HMD100is included in the received data (Step S42).

When the captured image data is included in the received data (YES in Step S42), the control unit55extracts image identification information, operator information, and HMD identification information from the received data. The control unit55registers the extracted information in a cell in which the corresponding HMD identification information of the information management table57is registered. The control unit55also associates the received captured image data with the image identification information and stores the data in the storage unit53(Step S43).

When the data received through the communication network7is not captured image data (NO in Step S42), the control unit55determines whether the received data is a request for acquiring the captured image data (Step S44). When the received data is not a request for acquiring the captured image data (NO in Step S44), the control unit55performs processing corresponding to the received data (Step S45). When the received data is a request for acquiring the captured image data (YES in Step S44), the control unit55acquires terminal identification information from the terminal device that has received the data in Step S41or from the data received in Step S41(Step S46). The control unit55determines whether the acquired terminal identification information is registered in the information management table57(Step S47).

When the acquired terminal identification information is not registered in the information management table57(NO in Step S47), the control unit55ends the processing flow without downloading the captured image data to the terminal device that has received the data.

When the acquired terminal identification information is identification information registered in the information management table57(YES in Step S47), the control unit55causes the terminal device to display a list of downloadable captured image data. For example, the control unit55generates a thumbnail image and causes the terminal device to display the thumbnail image. The captured image data is captured image data about image identification information associated with the acquired terminal identification information and registered in the information management table57. When the control unit55accepts an input of selecting captured image data from the terminal device, the control unit55downloads the selected captured image data to the terminal device (Step S48).

Next, a third operation of the control unit150will be described with reference to a flowchart illustrated inFIG. 11. The third operation is an operation to acquire captured image data in which a bar code is captured from captured image data of the camera60and permit storing of the captured image data in the storage unit160for a number of times set in the bar code. Operations in Steps S51and S52illustrated inFIG. 11are the same as those in Steps S1and S2illustrated inFIG. 8, and thus description will be omitted.

The control unit150determines whether a selected work block is a read block (Step S53). When the selected work block is not the read block (NO in Step S53), the control unit150performs processing according to the selected work block (Step S54). Operations in Steps S54and S55are the same as those in Steps S4and S5illustrated inFIG. 8, and thus description will be omitted.

When the selection block is the read block (YES in Step S53), the control unit150acquires captured image data from the memory121(Step S56). The control unit150causes the camera60to perform capturing at regular time intervals, and writes the generated captured image data to the memory121and causes the memory121to temporarily store the captured image data.

The control unit150analyzes the acquired captured image data, and determines whether an image of a bar code is included in the captured image data (Step S57). When the image of the bar code is not included (NO in Step S57), the processing returns to Step S56, and the control unit150acquires next captured image data from the memory121and determines whether an image of a bar code is included.

When the image of the bar code is included (YES in Step S57), the control unit150extracts code information from the image of the bar code (Step S58), and causes the storage unit160or the memory121to store the extracted code information. The control unit150analyzes the extracted code information and determines a number of usable times (Step S59). The number of usable times stipulates an upper limit of the number of times captured image data is permitted to be stored in the storage unit160. The control unit150permits use of captured image data in a work block during execution with the number of usable times as an upper limit. In a case where the camera60is caused to perform capturing only when an operation of the key corresponding to the shutter button is accepted, the number of usable times stipulates an upper limit of the number of times capturing by the camera60is permitted. The number of usable times can be set as code information about a bar code and can be set previously for each user identification information and each work block.

When the number of usable times cannot be determined with acquired code information, that is, when an image of a bar code is an image of a bar code unidentifiable by the HMD100, the control unit150may cause the image display unit20to display an error. After the control unit150causes the image display unit20to display an error, the processing then proceeds to Step S56, and the control unit150acquires captured image data from the memory121and extracts code information.

The control unit150determines the number of usable times, and then executes an input block that accepts an input from an operator from the operation unit110and the like according to the process flow data162(Step S60). When the input block is executed, the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted (Step S61). When the operation on the key corresponding to the shutter button has not been accepted (NO in Step S61), the control unit150determines whether the processing is completed (Step S62). The determination in Step S62may be determination of whether an operation indicating completion of the processing has been accepted from the operation unit110, or determination of whether captured image data corresponding to the number of usable times is stored in the storage unit160. When the control unit150determines that the processing is not completed (NO in Step S62), the processing returns to Step S61, and the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted. When the control unit150determines that the processing is completed (YES in Step S62), the processing returns to Step S51and the control unit150selects a work block.

When the operation on the key corresponding to the shutter button has been accepted (YES in Step S61), the control unit150causes the camera60to perform capturing (Step S63) and causes the storage unit160to store captured image data being captured (Step S64). The control unit150makes notification when causing the storage unit160to store the captured image data (Step S65).

Next, the control unit150determines whether the number of times captured image data stored in the storage unit160reaches the number of usable times (Step S66). When the number of usable times is not reached (NO in Step S66), the processing returns to Step S61, and the control unit150determines whether the operation on the key corresponding to the shutter button has been accepted. When the number of usable times is reached (YES in Step S66), the control unit150changes a setting to not store captured image data in the storage unit160(Step S67), the processing returns to Step S51, and the control unit150then selects a work block. At this time, the control unit150may cause the image display unit20to display that the number of usable times is reached.

As described above, the HMD100in the exemplary embodiment includes the storage unit160, the camera60, and the control unit150.

The storage unit160stores the process flow data162that defines a process flow including a plurality of work blocks in which an order of execution is predetermined. The camera60captures an external scene. The control unit150executes a process flow according to the process flow data162. At least any one of the work blocks included in the process flow data162includes permission information set with whether use of captured image data of the camera60is permitted. When executing a process flow, the control unit150controls propriety of using the captured image data in each of the work blocks according to a setting of the process flow data162about the work block.

When the capturing function of the camera60is controlled by the OS, whether the camera60is operated can only be set uniformly, and a program of the OS cannot be rewritten. As a result, this causes convenience to decrease.

In the exemplary embodiment, capturing of the camera60is controlled by the process flow data162being an application program including a specific function executed on the OS151. When the process flow data162is executed, propriety of using captured image data is controlled for each work block included in a process flow. Therefore, the propriety of using captured image data can be controlled for each work block, and securing of security is compatible with improvement in user convenience.

When executing a work block, the control unit150controls the camera60between a state of being operated and a state of being stopped according to the setting of the process flow data162about the work block.

Therefore, the camera60can be switched between the state of being operated and the state of being stopped for each work block by causing the control unit150to execute the process flow data162.

The HMD100further includes the memory121that stores captured image data of the camera60. When executing a work block, the control unit150controls whether to use captured image data of the camera60stored in the memory121according to the setting of the process flow data162about the work block.

Therefore, propriety of using the captured image data stored in the memory121can be controlled according to the setting of the process flow data162.

When executing a work block including processing of capturing an image including code information, the control unit150controls use of captured image data of the camera60. When controlling the use of the captured image data of the camera60, the control unit150extracts the code information from the captured image data of the camera60and causes the storage unit160to store the code information.

Therefore, when executing a work block that captures an image including code information, the code information can be extracted from captured image data and processing corresponding to the extracted code information can be performed.

When executing the work block including the processing of capturing an image including code information, the control unit150determines the number of usable times of the captured image data based on the code information extracted from the captured image data. The control unit150allows the captured image data to be used in another work block included in a process flow with the number of usable times as an upper limit.

According to this configuration, an upper limit of the number of usable times of captured image data can be set.

When executing a work block in which captured image data is set to be usable, the control unit150allows use of captured image data corresponding to capturing by the camera60for a plurality of times.

Therefore, capturing for a plurality of times can be performed in the work block in which the captured image data is set to be usable.

The HMD100includes the LED indicators12and67as notification units.

The control unit150makes notification when causing the storage unit160to store the captured image data of the camera60stored in the memory121.

Therefore, the notification that the captured image data is stored in the storage unit160can be made.

When use of the captured image data is permitted, the control150of the HMD100sends the captured image data captured by the camera60from the communication unit117to the server device5.

Therefore, when the use of the captured image data is permitted, the captured image data is sent to the server device5. Therefore, a leak of the captured image data to the outside can be prevented.

The control unit150adds a password as limited information limiting use of the captured image data to the captured image data and sends the captured image data to the server device5.

Therefore, the captured image data can be prevented from being visually recognized by a third party, and a leak of information can be more efficiently prevented.

The control unit150also adds HMD identification information identifying the HMD100to the captured image data and sends the captured image data to the server device5.

Therefore, the HMD100that generates the captured image data can be identified in the server device5, and the captured image data can be prevented from being sent to an external device.

The server device5includes the communication unit51, the storage unit53, and the control unit55.

The storage unit53associates the captured image data received from the HMD100with the HMD identification information, and stores the data. The storage unit53also associates terminal identification information identifying a terminal device in which the captured image data is usable with HMD identification information of the HMD100from which the terminal device corresponding to the terminal identification information is allowed to acquire captured image data, and stores the information.

When receiving the terminal identification information from the communication unit51, the control unit55determines whether the terminal identification information coinciding with the received terminal identification information is stored in the storage unit53.

When the identical terminal identification information is stored in the storage unit53, the control unit55sends the captured image data in the HMD identification information associated with the identical terminal identification information to the terminal device associated with the terminal identification information.

Therefore, the captured image data can be prevented from being sent to an external device.

Next, an editing device that creates and edits the process flow data162will be described.

FIG. 12is a function block diagram of a personal computer (PC)300operating as an editing device. The PC300includes a display332and an input device334. As long as the PC300is a computer operable by a user, a specific configuration is not limited. For example, the PC300may be a desktop computer, and may be a portable computer, such as a laptop computer and a tablet computer. As long as the PC300includes functions illustrated inFIG. 12, the PC300may be a small device, such as a smartphone.

The PC300includes a control unit310, a storage unit320, a display unit331, an input unit333, a communication unit336, and an interface (I/F) unit337, and these are connected to each other via a bus339. The interface unit337corresponds to a “connection unit” of the invention.

The control unit310includes a processor, such as a CPU, and is configured to allow the processor to execute a program to control the PC300, and to achieve various functions of the PC300. The control unit310may include a RAM configured to prepare a work area for the processor. The control unit310may include a ROM configured to store in a non-volatile manner a basic control program executed by the processor.

The storage unit320includes a magnetic storage medium, an optical storage medium, a semiconductor storage device, or the like, and has a storage region used to store programs and data. The storage unit320is configured to store in a non-volatile manner programs to be executed by the control unit310and data to be processed by the control unit310. Programs and data stored in the storage unit320will be described later.

The display unit331is connected to the display332, and is configured to cause the display332to display various screens with text and/or images in accordance with control by the controller310. The display332corresponds to a “display face” of the invention.

The display332includes a liquid crystal display device, an organic electro luminescence (EL) display device, or another display device, and is driven by the display unit331.

The input unit333is connected to the input device334, and is configured to detect an operation of the input device334, and to accept an input through the operation of the input device334. The input unit333outputs, to the control unit310, data indicating a content of the input through the input device334.

The input device334may be a text input device, such as a keyboard, or a pointing device, such as a mouse, a digitizer, and a pen tablet. The input device334may be a configuration integrated with the display332, such as a touch panel. The input device334may be a software keyboard or a graphical user interface (GUI) incorporated into a screen displayed on the display332.

The interface (I/F) unit337is an interface used to connect an external device to the PC300, and includes a universal serial bus (USB) interface, for example. The interface337includes a plurality of connectors and interface circuits (none of which are illustrated) that connect cables, and can connect a plurality of HMDs100at the same time. The interface unit337exchanges data with the external device connected to the interface unit337in accordance with control by the control unit310.

The communication unit336is configured to execute wired or wireless communications with the external device attached to the PC300in accordance with control by the control unit310. The communication unit336executes communications in accordance with various protocols, such as an Ethernet (registered trademark) protocol, a wireless LAN (including WiFi (registered trademark)), and a Bluetooth (registered trademark).

As examples of programs and data to be stored in the storage unit320,FIG. 12illustrates an operating system (OS)321, a process flow definition tool322, display-related data323, condition definition information324, process flow data325, output setting data326, and output data327. The OS321is a control program used by the control unit310to control the PC300, and configures a platform allowing the control unit310to operate an application program. When the control unit310executes the OS321, a basic function of the PC300is provided as an application program interface (API) for the application program to be executed by the control unit310. The basic function of the PC300includes display processing to be executed by the display unit331, input detection processing to be executed by the input unit333, data input and output processing to be executed by the interface unit337, communication processing to be executed by the communication unit336, and other processing, for example.

The process flow definition tool322is an application program used to create and edit a process flow with the PC300. When the control unit310executes the process flow definition tool322, an operation flow to be executed by the HMD100can be created and edited. A process flow created by the process flow definition tool322is a specific example of an operation flow.

The display-related data323includes data about images and/or text to be displayed by the display unit331when the control unit310executes the process flow definition tool322. When the control unit310executes the process flow definition tool322, the control unit310refers to the display-related data323, and causes the display332, via the display unit331, to display an image and/or text based on the display-related data323.

The condition definition information324is information defining conditions with respect to work blocks included in a process flow, and includes information about some or all of work blocks to be created or edited by executing the process flow definition tool322. A condition with respect to a work block represents a condition to be set with respect to a content of the work block. For example, a work block corresponding to an input of information needs to be set with a format of information to be input, an input method, and an action when no input is made, for example. The condition definition information324is set with how to define, as a condition for a work block corresponding to an input of information, a format of information to be input, an input method, and an action when no input is made.

The process flow data325is data about a process flow generated when the control unit310has executed the process flow definition tool322. The process flow data325includes work blocks included in the process flow, an order of execution of the work blocks, and setting contents including inputs with respect to the work blocks, and may include other information. The process flow data325has a data format that can be interpreted when the control unit310executes the process flow definition tool322. When the control unit310executes the process flow definition tool322, the control unit310can read the process flow data325from the storage unit320, and can edit a process flow represented by the read process flow data325. In this case, based on the edited process flow, the control unit310can update the process flow data325stored in the storage unit320.

The output setting data326is data relating to a setting when the process flow data325is converted to the output data327. The output data327is, in the HMD100or a device other than the HMD100, data used to execute a process flow by following the process flow data325, and differs in data format and the like from the process flow data325.

The output data327is data interpretable and executable by the HMD100, and is described in a general-purpose data format, for example. A general-purpose data format represents a data format that can be processed by the HMD100via a web browser, for example. Specifically, general-purpose data is data described in Extensible Markup Language (XML), Hypertext Markup Language (HTML), and the like.

The output data327may be general-purpose data used to execute a process flow in various devices including general-purpose devices, such as PCs. The output data327may be data corresponding to any of a type, a configuration, and a specification of a device used to execute a process flow. For example, the output data327corresponding to a device equipped with a camera may be the output data327that specifies performing text recognition using captured image data of the camera, as a method for inputting information. When the output data327is executed by a device equipped with a camera, an operator uses the camera to easily make an input with fewer burdens. The output data327corresponding to a device equipped with a bar-code reader may be data that specifies acquiring data read by the bar-code reader, as a method for inputting information. In the output data327, a background color of a screen to be displayed, resolution of the screen, sizes of images and/or text to be displayed on the screen, and the like may be adjusted so as to be compatible with the HMD100. A burden on visual performance of an operator wearing the HMD100can be reduced by adjusting these pieces of information.

The control unit310executes the process flow definition tool322to create and edit the process flow data325. The control unit310executes the process flow definition tool322to generate (convert) the output data327from the process flow data325.

FIG. 13is a flowchart illustrating an operation of the PC300.

The operation inFIG. 13is executed while the controller310is executing the OS321. In response to an operation by a user, for example, the control unit310reads the process flow definition tool322from the storage unit320and launches the process flow definition tool322(step S101). The control unit310reads the display-related data323, and causes the display unit331to display an editing screen arranged with an image and the like included in the display-related data323(step S102).

The control unit310displays an editing screen401(step S102), and executes process flow editing processing of creating or editing the process flow data325in accordance with an operation by a user (step S103). The process flow data325created or edited by the process flow editing processing may be the process flow data325for mutually causing a plurality of HMDs100to operate, or may be the process flow data325created for each HMD100. When the process flow data325is created for each HMD100, the flows in Steps S101to S104are executed for the number of times corresponding to the number of HMDs100for which the process flow data325is created. When the process flow data325corresponding to any of the plurality of HMDs100has already been created, the control unit310can edit the process flow data325that has already been created to create the process flow data325to be executed by another HMD100.

The control unit310stores the process flow data325representing the process flow created or edited in the process flow editing processing in the storage unit320(step S104). When an identical process flow is already stored in the process flow data325, the process flow data325is overwritten and updated.

The control unit310determines whether the process flow data325is to be output to an external device attached to the HMD100and the like (step S105). When outputting of the process flow data325is instructed with an operation by the input device334, and the like (YES in Step S105), the control unit310refers to the output setting data326, and acquires settings corresponding to a type and a function of an output-destination device (step S106). The output setting data326may include a setting of one general-purpose type. In this case, the control unit310may acquire a setting of one type, which is included in the output setting data326.

The control unit310follows the setting acquired in step S106to generate and output the output data327based on the process flow data325(step S107). In Step S107, the control unit310may store the output data327in the storage unit320, may output the output data327to a device connected to the interface unit337, or may send the output data327to the communication unit336. When the output data327is the output data327generated based on the process flow data325to be executed by the HMD100A, the output data327is output to the interface337connected to the HMD100A. Similarly, when the output data327is the output data327generated based on the process flow data325to be executed by the HMD100B, the output data327is output to the interface337connected to the HMD100B. When the process flow data325is not to be output (NO in Step S105), the control unit310ends the processing.

FIG. 14is a diagram illustrating a display example of the PC300, and illustrates a configuration example of the editing screen401.

The editing screen401is a screen displayed through a function of the process flow definition tool322for creating and editing a process flow. The editing screen401is roughly separated into a candidate region410and an editing region420. At an upper section of the editing screen401, an instruction section401ais arranged. The instruction section401aincludes an icon instructing end of creating and editing of a process flow, as well as including an icon instructing storing the process flow. The editing screen401corresponds to a “work screen” of the invention. The editing region420corresponds to a “work region” of the invention.

The candidate region410displays a list of work blocks that can be incorporated into a process flow to be created or edited. The candidate region410is arranged with, in regions separated per work block, text describing each of the work blocks, and icons that are images representing symbols indicating the work blocks.

In the editing screen401inFIG. 14, the candidate region410is arranged with, in line with seven work blocks, work block display sections411,412,413,414,415,416, and417.

The work block display section411includes a work block description display section411aas text describing that the section corresponds to a procedure display block, and an icon411bindicating the procedure display block. The work block display section412includes a work block description display section412aas text describing that the section corresponds to a read block, and an icon412bindicating the read block. The work block display section413includes a work block description display section413aas text describing that the section corresponds to an input block, and an icon413bindicating the input block. The work block display section414includes a work block description display section414aas text describing that the section corresponds to a determination block, and an icon414bindicating the determination block. The work block display section415includes a work block description display section415aas text describing that the section corresponds to a selection input block with a check box, and an icon415bindicating the selection input block. The work block display section416includes a work block description display section416aas text describing that the section corresponds to a selection input block with a radio button, and an icon416bindicating the selection input block. The work block display section417includes a work block description display section417aas text describing that the section corresponds to a termination block, and an icon417bindicating the termination block.

The work block display sections411to417arranged in the candidate region410are candidates for work blocks to be incorporated into a process flow to be created and edited. A user can operate the input device334to select any of the work block display sections411to417and drag and drop the work block display sections selected into the editing region420to incorporate desired work blocks into a process flow.

The editing region420is a display region used to display a process flow. The editing region420is arranged with work blocks selected by the user from among the work block display sections411to417arranged in the candidate region410. A process flow is created with one work block or a plurality of work blocks arranged in the editing region420. The editing region420is arranged with objects represented by images corresponding to work blocks.

A user selects (drags) an icon of any one of the work blocks from the work block display section411and drops the selected icon into the editing region420. When the icon is dropped into the editing region420, an object421is displayed in the editing region420. The object421represents the work block displayed in the work block display section411. The object421includes an icon421awith an image identical to the image of the icon411b. With the icon421a, the object421representing the work block in the work block display section411is easily and visually identified.

The editing region420displays a guide430indicating a position onto which an object can be added. The guide430is located below the object421, indicating that a work block to be executed after the object421can be arranged. A user selects (drags) an icon of any one of the work blocks from the work block display section411and drops the selected icon into the guide430. With this operation, the editing region420displays an object executed after the object421and the guide430indicating a position onto which an object can be added.

FIG. 15is a diagram illustrating a display example of the PC300, and illustrates a configuration example of a details setting screen451.FIG. 15illustrates an example of the details setting screen451being switched and displayed, with a plurality of objects arranged in the editing region420of the editing screen401.

The details setting screen451includes a process flow display region460, a view editing region470, and a details setting region480. At an upper section of the details setting screen451, an instruction section451ais arranged. The instruction section451aincludes an icon instructing storing of details settings, as well as includes an icon instructing ending of details setting processing.

The process flow display region460is a region in which a process flow461is displayed, and objects indicating work blocks that configure the process flow461are arranged in an order of execution. In the example inFIG. 15, the process flow461including an object421and an object422is displayed. The process flow461does not include an object indicating termination since the flow is still under creation. Thus, a guide430is displayed at a bottom of the process flow461.

The view editing region470is a region used to display a configuration of a screen displayed by the HMD100when the HMD100executes the process flow461created in the processing. The view editing region470displays the configuration of the screen in accordance with any of the objects configuring the process flow461displayed in the process flow display region460.

FIG. 15illustrates a screen (referred to as a user view)471corresponding to the object421being a first object in the process flow461in the view editing region470. The user view471includes a process display section472displaying an order of a plurality of work blocks included in the process flow461. The process display section472is a display section notifying, to an operator, a position of a work block being executed in a whole process.

The details setting region480is a region used to input setting contents defined in the condition definition information324for the object421selected in the process flow461. In the details setting region480inFIG. 15, a title setting section481and a work information setting section482are arranged for accepting inputs in accordance with setting contents for the object421. The title setting section481includes input boxes used to input a work name and a work ID. The work information setting section482includes input boxes used to input contents to be displayed to an operator such as a work procedure and a hint about work.

A check box483indicating whether to permit use of captured image data captured by the camera60is displayed in the details setting region480. When a check mark is input to the check box483, permission information permitting use of captured image data is set in a work block corresponding to the object421. When the check mark is removed from the check box483, permission information rejecting the use of captured image data is set in a work block corresponding to the object421.

As described above, the PC300in the exemplary embodiment is an editing device allowed to edit control data that defines a process flow including a plurality of work blocks in which an order of execution is determined.

The PC300includes the input unit333, the display unit331configured to cause the display332to perform displaying, and the control unit310.

The control unit310causes the display332to display the editing screen401including the editing region420. The control unit310arranges objects each indicating the operation step and including an image indicating a content of the operation step in the editing region420in accordance with an input accepted by the input unit333. The control unit310creates a process flow based on arrangement of the objects in the editing region420and generates the process flow data162that defines the created process flow.

The process flow data162includes a setting that determines whether use of captured image data of the camera60is permitted for at least any one of the operation steps.

Therefore, a process flow set with whether use of captured image data of the camera60is permitted for each work block can be created.

The PC300includes the interface unit337as a connection unit attached to the HMD100. The control unit310outputs the output data327that defines a process flow created to be associated with the HMD100to the HMD100connected to the interface unit337.

Therefore, the output data327executable by the HMD100can be output to the HMD100and executed.

A plurality of HMDs100are allowed to be connected to the interface unit337.

The control unit310outputs a process flow created to be associated with each of the plurality of HMDs100to the corresponding HMD100.

Therefore, the process flow created to be associated with each of the plurality of HMDs100can be executed by the corresponding HMD100.

The above-described exemplary embodiment is an exemplary embodiment suitable for the invention. However, the invention is not limited to this, and various modifications can be applied to the exemplary embodiment without departing from the purpose of the invention.

For example, a transmissive head-mounted display apparatus is described as an example in the above-described exemplary embodiment. However, the head-mounted display apparatus may be non-transmissive and not allow the outside world to be visually recognized with the display device being mounted. The HMD100is described as an example in the exemplary embodiment, but may be a tablet PC. A projector is also applicable.

The HMD100may include an interface (not illustrated) that connects various external devices being supply sources of contents. For example, the interface may be an interface compatible with wired connection, such as a USB interface, a micro USB interface, a memory card interface, or may be a wireless communication interface. The external devices in this case are image supply devices that supply an image to the HMD100. A personal computer (PC), a cellular phone terminal, a portable game machine, and the like are used. In this case, the HMD100can output an image and sound based on the content data163input from these external devices.

The above-described exemplary embodiment exemplifies the configuration in which the control device10and the image display unit20are connected to each other in the wired manner. However, the invention is not limited to this, and the image display unit20may be connected to the control device10in a wireless manner. The method exemplified as a communication method compatible with the communication unit117may be adopted as a wireless communication method in this case, or another communication method may be used.

A part of the functions of the control device10may be provided in the image display unit20. The control device10may be achieved by a plurality of devices. In other words, the control device10is not limited to a configuration including the box-shaped case10A. For example, a device wearable by a user on a body of the user or with clothes on may be used instead of the control device10. For example, a device wearable by a user may be a device such as a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, and a pen-type device.

In the above-described exemplary embodiment, the configuration in which the image display unit20and the control device10are separated from each other and connected through the connection cable40is described as an example. The invention is not limited to this, and a configuration mounted on a head of the user in which the control device10and the image display unit20are integrally formed may also be used.

A laptop computer, a tablet computer, or a desktop computer may be used as the control device10. Portable electronic devices including a game machine, a portable telephone, a smartphone, and a portable media player, or other dedicated devices may be used as the control device10.

For example, an image display unit in another mode such as an image display unit mounted like a hat may be adopted instead of the image display unit20. The image display unit may include a display unit configured to display an image corresponding to a left eye LE of a user and a display unit configured to display an image corresponding to a right eye RE of the user. A head-up display used instead of the image display unit20can be installed in vehicles such as automobiles and planes. For example, when the head-up display is installed in a vehicle, an operation surface corresponding to the operation surface of the track pad14is provided on a handle of the vehicle and the like.

For example, the head-up display may be formed as a head-mounted display apparatus built in a protective device such as a helmet. In this case, a section in which a position is determined with respect to a body of the user and a section in which a position is determined with respect to the section can serve as mounted sections.

The configuration in which a virtual image is formed by the half mirrors261and281in the part of the right light-guiding plate26and the left light-guiding plate28is exemplified as an optical system that guides image light to eyes of a user. The invention is not limited to this, and an image may be displayed in a display region having an area accounting for the entire surface or most of the right light-guiding plate26and the left light-guiding plate28. In this case, processing of reducing an image may be included in an operation to change a display position of the image.

Furthermore, the optical element of the invention is not limited to the right light-guiding plate26and the left light-guiding plate28respectively including the half mirrors261and281. The optical element may be an optical component that causes image light to be incident on eyes of a user. Specifically, a diffraction grating, a prism, and a holography display unit may be used.

Such a configuration may be adopted that at least some of the function blocks illustrated inFIGS. 6, 7, and the like are achieved with hardware, or achieved together with hardware and software. The invention is not limited to the configuration in which independent hardware resources are arranged as illustrated in the drawings. A program executed by the control unit150may be stored in the nonvolatile storage unit123or another storage device (not illustrated) in the control device10. A program stored in an external device may be acquired and executed through the communication unit117and the external connector184. The operation unit110in the configuration formed in the control device10may be formed as a user interface (UI).

A processing unit in the flowcharts illustrated inFIGS. 8, 9, and 11is obtained by dividing processing in accordance with a main processing content to facilitate the understanding of the processing of the control unit150in the HMD100. Thus, the invention is not limited by a method for dividing processing into processing units and a name. The processing of the control unit150can be divided into more processing units in accordance with a processing content, and can be divided such that one processing unit includes more processing. The order of processing in the above-described flowcharts is also not limited to the illustrated examples.

The same applies to the flowchart illustrating the operation of the server device5illustrated inFIG. 10and the flowchart illustrating the operation of the PC300illustrated inFIG. 13.

The entire disclosure of Japanese Patent Application No.: 2017-236422, filed Dec. 8, 2017 is expressly incorporated by reference herein.