Patent Description:
A technique relating to mixed reality (hereinafter referred to as "MR") is known of displaying virtual information using computer graphics or the like with respect to a real object arranged in a real space (see, for example, Patent Document <NUM>). Patent Document <NUM> discloses a technique of synthesizing display of a virtual video and a real video obtained by capturing an image of an architectural model that is a real object. Patent Document <NUM> discloses a solution to perform during-surgery support of surgery with high accuracy and at low cost. Patent Document <NUM> discloses a solution for placing and rendering of virtual objects in three dimensional environments. Patent Document <NUM> discloses a method for generating an augmented reality image, wherein a real image of a real environment is recorded by means of a camera. Patent Document <NUM> discloses a system for providing visual cues to a welder via an augmented reality display for a sequence of operations. Patent Document <NUM> discloses a welding jig for welding a target plate to a die steel. Patent Document <NUM> discloses a head mounted display for a welding guided by augmented reality. Non-Patent Document <NUM> discloses an algorithm for improving manual welding by passing the same information to the human operator as used in the robotic cell.

As a method of using an MR technique, associating a virtual video for supporting a user's work through a head mount display (HMD) (hereinafter referred to as an "MR device") or the like with a real object and providing it to the user can be considered. For example, when welding steel members, work such as reading and understanding of drawings, ruling on steel members, and confirmation of ruling is required. Even skilled engineers take time to understand complicated drawings, and reworking due to simple mistakes is needed when there are many portions to be welded. For this reason, conventional welding of steel members has required skilled techniques. Consequently, superimposition display of a virtual object indicating a mounting member on a real steel member using the MR device can be considered.

However, the difficulty of mounting work may not be improved by only superimposition display of a virtual object indicating a mounting member on a structure such as a steel member. For example, in a state in which a mounting member having relatively large dimensions or weight is inclined with respect to a structure, mounting work such as welding may be performed between the mounting member and the structure. In such a case, it is difficult to perform the mounting work while maintaining a state in which the mounting member is arranged in accordance with display of the virtual object.

The present disclosure was invented in view of the above problems, and an object thereof is to provide a video display system that makes it possible to easily mount a member on a structure.

In order to achieve the above object, according to the present disclosure, there is provided a video display system used in mounting a member on a structure according to claim <NUM>. Additional features for advantageous embodiments of the present invention are provided in the dependent claims.

In the video display system according to the present disclosure, information on the arrangement of a jig for positioning a member on a structure is stored, and a display process indicating the arrangement of the jig is performed on the structure arranged in a real space. A user can easily arrange the jig in accordance with the display process indicating the arrangement of the jig. As a result, the member is positioned by the jig on the structure, and thus it is possible to easily mount the member on the structure.

According to the present disclosure, it is possible to provide a video display system that makes it possible to easily mount a member on a structure.

Hereinafter, an embodiment of a video display system according to the present disclosure will be described in detail with reference to the accompanying drawings. Meanwhile, in the description of the drawings, the same components are denoted by the same reference numerals and signs, and thus description thereof will not be repeated.

First, a video display system according to the present embodiment will be described. <FIG> shows a block diagram of a video display system. A video display system <NUM> displays a virtual object based on computer graphics (CG) associated with a real space (a world coordinate system). The video display system <NUM> is realized by, for example, an information processing terminal or the like including an HMD which is worn by a user. The video display system <NUM> may be a tablet terminal, projection mapping, or the like. Hereinafter, a case in which the video display system <NUM> is an HMD used in an MR technique will be described as an example. The video display system <NUM> employs an optical see-through scheme for allowing a user to visually recognize a real object in a real space and a virtual object using a half mirror. The video display system <NUM> may employ a video see-through scheme for merging a virtual object with video in a real space and allowing a user to visually recognize the result. In the present embodiment, the video display system <NUM> uses a video see-through scheme.

The video display system <NUM> performs image capturing in the same image capturing direction as a user's eye line, or the like. The video display system <NUM> superimposes a virtual object on a captured image to display the result. By viewing this display, a user can view a virtual object that does not exist in reality according to the arrangement of a real object arranged in a real space. The "arrangement" referred to here means the state of six degrees of freedom defined by coordinates on predetermined three-dimensional coordinate axes and rotation around each of the three-dimensional coordinate axes (orientation in three-dimensional coordinates).

The video display system <NUM> displays a virtual object using a virtual space (a screen coordinate system) which is three-dimensional coordinates. The video display system <NUM> arranges a virtual object at preset position in a virtual space, and calculates a correspondence relation between the virtual space and the real space. The arrangement of a virtual object in a virtual space is referred to as "display arrangement. " In the following description, a coordinate axis indicating the position of a virtual object in a virtual space is referred to as a "virtual coordinate axis α. " In the video display system <NUM>, an image viewed from a position and direction in a virtual space corresponding to an image capturing position and image capturing direction in a real space is defined as display of a virtual object. That is, an image of a virtual object in a virtual space viewed from a virtual user's visual line is displayed. The display of a virtual object can be carried out using an MR technique of the related art. The video display system <NUM> detects a user's movement, and changes a display process in accordance with the movement so as to match a change in the user's visual line according to the user's movement.

The video display system <NUM> is used in mounting a member on a structure <NUM> arranged in a real space (hereinafter referred to as a "mounting member"). <FIG> shows display that the video display system <NUM> allows a user to visually recognize. <FIG> shows a state in which mounting members are mounted using the video display system <NUM>. The video display system <NUM> displays a virtual object indicating the arrangement of a jig <NUM> on a display to correspond to the arrangement of the structure <NUM> which is a real object arranged in a real space. In other words, the video display system <NUM> performs a display process indicating the arrangement of the jig <NUM> on the structure <NUM> arranged in a real space. In the present embodiment, the structure <NUM> is an H steel member having an elongated shape.

The jig <NUM> is a member for positioning a mounting member on the structure <NUM>. For example, in a state in which a mounting member is positioned on the structure <NUM> by the jig <NUM>, a user performs mounting work such as welding between the mounting member and the structure. In the present embodiment, the above-described virtual object contains information indicating the structure <NUM> in a real space, information indicating a mounting member scheduled to be mounted on the structure <NUM>, and information indicating a distance from a reference position to a position at which the jig <NUM> is arranged. The reference position is an end 110a of the structure <NUM>.

In the present embodiment, as shown in <FIG>, the video display system <NUM> displays virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> indicating mounting members scheduled to be mounted on the structure <NUM> and information <NUM> and <NUM> indicating distances from the reference position to the position at which the jig <NUM> is arranged. The video display system <NUM> performs display allowing a virtual object indicating the arrangement of the jig <NUM> to be visually recognized by a user in a state in which it overlaps the structure <NUM> arranged in a real space, and indicates a position at which the jig <NUM> is to be mounted on the structure <NUM>. The virtual mounting member <NUM> shown in <FIG> corresponds to a mounting member <NUM>. An arrangement in which the virtual mounting member <NUM> is displayed is an arrangement in which the mounting member <NUM> is scheduled to be mounted. The virtual mounting member <NUM> shown in <FIG> corresponds to a mounting member <NUM>. An arrangement in which the virtual mounting member <NUM> is displayed is an arrangement in which the mounting member <NUM> is scheduled to be mounted.

In the present embodiment, the video display system <NUM> displays a shortest distance from the end 110a of the structure <NUM> in its extending direction to a mounting member and a longest distance from the end 110a of the structure <NUM> in its extending direction to the mounting member as distances from the reference position to the position at which the jig <NUM> is arranged. As shown in <FIG> and <FIG>, through the video display system <NUM>, it can be understood that the shortest distance from the end 110a of the structure <NUM> in its extending direction to the mounting member is <NUM>, and that the longest distance from the end 110a of the structure <NUM> in its extending direction to the mounting member is <NUM>.

The video display system <NUM> detects the reference position on the basis of a reference marker <NUM> arranged on the structure <NUM> in a real space. The reference marker <NUM> is arranged in a predetermined arrangement relation with the structure <NUM> before work is performed. In the present embodiment, the reference marker <NUM> is provided along the end 110a of the structure <NUM> on a face plate <NUM> of the structure <NUM>. The reference marker <NUM> has a feature point that makes it possible for the video display system <NUM> to recognize the three-dimensional arrangement of the reference marker <NUM> in a real space through image recognition. For example, the reference marker <NUM> may contain a diagram having such a feature point (for example, a two-dimensional code) or the like. Insofar as the video display system <NUM> can recognize the three-dimensional arrangement of the reference marker <NUM> in a real space, the reference marker <NUM> is not limited to this form. For example, the feature point of the reference marker <NUM> may be shown by the color or blinking of the reference marker <NUM>. The three-dimensional position of the reference marker <NUM> may be identified through a method of identifying a three-dimensional position using a magnetic signal, a sound wave signal, or the like.

<FIG> shows an example a state in which a mounting member is mounted on the structure <NUM>. <FIG> shows a three-sided view of a state in which the mounting member is mounted to be inclined with respect to the structure <NUM>. As shown in <FIG>, the structure <NUM> has a configuration in which three face plates <NUM>, <NUM>, and <NUM> extending in a longitudinal direction are integrally connected to each other. The face plates <NUM>, <NUM>, and <NUM> all exhibit a rectangular shape.

The face plate <NUM> has main surfaces 111a and 111b that face each other. The face plate <NUM> has main surfaces 112a and 112b that face each other. The face plate <NUM> has main surfaces 113a and 113b that face each other. The main surfaces 111a, 111b, 112a, 112b, 113a, and 113b of the face plates <NUM>, <NUM>, and <NUM> extend in a longitudinal direction and are in contact with the mounting member. The face plate <NUM> is connected to the face plates <NUM> and <NUM> so as to be perpendicular thereto. The central portion of the main surface 112a of the face plate <NUM> and the central portion of the main surface 113a of the face plate <NUM> are connected to a pair of long sides of the face plate <NUM>.

<FIG> shows relationships among an outward appearance when viewed from a direction orthogonal to the main surface 112b of the face plate <NUM>, an outward appearance when viewed from a direction orthogonal to the main surface 111a of the face plate <NUM>, and an outward appearance when viewed from the extending direction of the structure <NUM>, with respect to the structure <NUM> on which the mounting member <NUM> is mounted. As shown in <FIG>, the mounting member <NUM> is trapezoidal when seen in a plan view, and has four corner portions 131a, 131b, 131c, and 131d. <FIG> shows the same corner portions connected by a dashed-dotted line.

As shown in <FIG>, the mounting member <NUM> is also inclined with respect to the main surface 112a of the face plate <NUM> and the main surface 113a of the face plate <NUM> while being inclined with respect to the main surface 111a of the face plate <NUM>. A side connecting the corner portion 131b and the corner portion 131c is in contact with the main surface 111a of the structure <NUM>. A side connecting the corner portion 131a and the corner portion 131b is in contact with the main surface 112a of the structure <NUM>. A side connecting the corner portion 131c and the corner portion 131d is in contact with the main surface 113a of the structure <NUM>. In this case, in the extending direction of the structure <NUM>, the position of the corner portion 131a is a position of the shortest distance from the end 110a of the structure <NUM>. In the extending direction of the structure <NUM>, the position of the corner portion 131c is a position of the longest distance from the end 110a of the structure <NUM>.

<FIG> is a partial enlarged view illustrating an arrangement relation among the structure <NUM>, the jig <NUM>, and display of the video display system <NUM>. Here, (a) of <FIG> shows an outward appearance when viewed from a direction orthogonal to the main surface of the face plate <NUM>. In addition, (b) of <FIG> shows an outward appearance when viewed from a direction orthogonal to the main surface 111a of the face plate <NUM>. The virtual mounting member <NUM> has the same shape as the mounting member <NUM>, and has the four corner portions 131a, 131b, 131c, and 131d.

As shown in <FIG> and <FIG>, the jig <NUM> includes a first jig <NUM> and a second jig <NUM>. The first jig <NUM> has a flat surface 121a. The second jig <NUM> has a flat surface 122a. In the present embodiment, the flat surface 121a of the first jig <NUM> and the flat surface 122a of the second jig <NUM> are arranged perpendicularly to the main surface 111a and the main surface 113a. Each of the first jig <NUM> and the second jig <NUM> has a rectangular parallelepiped shape. Each of the first jig <NUM> and the second jig <NUM> is arranged to be in contact with at least the main surface 111a and the main surface 113a of the structure <NUM>.

On the basis of the display of the video display system <NUM>, the flat surface 121a of the first jig <NUM> is arranged at the position of the shortest distance from the end 110a in the extending direction of the structure <NUM> to the mounting member <NUM>. On the basis of the display of the video display system <NUM>, the flat surface 122a of the second jig <NUM> is arranged at the position of the longest distance from the end 110a in the extending direction of the structure <NUM> to the mounting member.

The video display system <NUM> also displays the virtual mounting member <NUM> in addition to display of the information <NUM> and <NUM> indicating distances from the reference position to the position at which the jig <NUM> is arranged. The first jig <NUM> is arranged to be in contact with a portion of the virtual mounting member <NUM> which is located at the shortest distance from the end 110a in the extending direction of the structure <NUM>. The second jig <NUM> is arranged to be in contact with a portion of the virtual mounting member <NUM> which is located at the longest distance from the end 110a in the extending direction of the structure <NUM>. In the present embodiment, the first jig <NUM> is arranged so that the flat surface 121a is in contact with the corner portion 131a of the virtual mounting member <NUM>, and the second jig <NUM> is arranged so that the flat surface 122a is in contact with the corner portion 131d of the virtual mounting member <NUM>. In this case, the flat surface 121a of the first jig <NUM> and the flat surface 122a of the second jig <NUM> face each other. Thereby, the jig <NUM> is arranged to restrict the position of the mounting member <NUM> in the extending direction of the structure <NUM>; the angle of inclination of the mounting member <NUM> with respect to the main surface 111a of the structure <NUM>; and the angle of rotation of the mounting member <NUM> with respect to an axis orthogonal to the main surface 111a. For example, the mounting member <NUM> is arranged to abut the main surface 111a of the structure <NUM>; the flat surface 121a of the first jig <NUM>; and the flat surface 122a of the second jig <NUM>, so that the mounting member <NUM> is positioned with a desired arrangement.

<FIG> is a diagram illustrating an arrangement of the jig in mounting work according to a modification example of the present embodiment. In the present modification example, a mounting member <NUM> having a length in a direction intersecting the main surface 111a of the face plate <NUM> which is larger than the mounting member <NUM> is mounted on the structure <NUM>. The mounting member <NUM> also has the four corner portions 131a, 131b, 131c, and 131d. In the direction intersecting the main surface 111a, the length of the mounting member <NUM> is larger than the length of the jig <NUM>. In this case, the flat surface 122a is in contact with the corner portion 131d, but the flat surface 121a cannot be in contact with the corner portion 131a. In this case, the first jig <NUM> is arranged so that the flat surface 121a is in contact with the side connecting the corner portion 131a and the corner portion 131b. The video display system <NUM> shows a portion of the side connecting the corner portion 131a and the corner portion 131b with which the flat surface 121a of the first jig <NUM> is in contact. In other words, the video display system <NUM> displays the position of the shortest distance from the end 110a to the mounting member <NUM> within a region interposed between the first jig <NUM> and the second jig <NUM> in the extending direction of the structure <NUM> as display of information indicating the arrangement of the flat surface 121a of the first jig <NUM>. The video display system <NUM> displays the position of the longest distance from the end 110a to the mounting member within the region interposed between the first jig <NUM> and the second jig <NUM> in the extending direction of the structure <NUM> as display of information indicating the arrangement of the flat surface 122a of the second jig <NUM>.

As shown in <FIG>, the video display system <NUM> includes an image capturing unit <NUM>, an inertial measurement unit <NUM>, a display unit <NUM>, an environmental information detection unit <NUM>, a storage unit <NUM>, an operation detection unit <NUM>, and a control unit <NUM>. The image capturing unit <NUM>, the inertial measurement unit <NUM>, the display unit <NUM>, the environmental information detection unit <NUM>, the storage unit <NUM>, the operation detection unit <NUM>, and the control unit <NUM> may be housed in one housing, or may be separately housed in a plurality of housings. In the case of being housed in a plurality of housings, the above units may be connected to each other in a wired manner, or may be wirelessly connected to each other.

At least one of the image capturing unit <NUM> and the display unit <NUM> in the video display system <NUM> is portable. The video display system <NUM> may not include the image capturing unit <NUM>. When the display unit <NUM> is carried without carrying the image capturing unit <NUM>, the display unit <NUM> and the inertial measurement unit <NUM> are housed in one housing. For example, the video display system <NUM> can be used not only when an HMD is used, but also when a user carries a tablet or the like in which the image capturing unit <NUM> does not operate and when the user remotely operates the movement of a terminal having the image capturing unit <NUM> while making confirmation with the display unit <NUM> at hand. In the present embodiment, a case in which the image capturing unit <NUM>, the inertial measurement unit <NUM>, and the display unit <NUM> are housed in one housing and an HMD having the housing fixed to a user's head is used will be mainly described.

The image capturing unit <NUM> captures an image of a real space, and sequentially transmits the captured image to the environmental information detection unit <NUM>. In the present embodiment, the image capturing unit <NUM> is fixed to a user's head, and the position and posture of the image capturing unit <NUM> fluctuate in accordance with the movement of the user's head. The position of the image capturing unit <NUM> is a coordinate position of the image capturing unit <NUM>. The posture of the image capturing unit <NUM> is rotation around each axis of three-dimensional coordinates (orientation in three-dimensional coordinates). The image capturing unit <NUM> is constituted by a plurality of cameras such as a depth camera and an RGB camera. The image capturing unit <NUM> includes not only a front camera that captures an image of a user's visual line direction, but also an environment camera that captures an image of the user's side or the like.

The inertial measurement unit <NUM> measures external force acting on the image capturing unit <NUM> or the display unit <NUM>, and sequentially transmits measurement results to the environmental information detection unit <NUM>. In the present embodiment, the inertial measurement unit <NUM> is constituted by an acceleration sensor, a gyro sensor, an orientation sensor, and the like which are fixed to a user's head.

The display unit <NUM> includes a display that performs display. The display unit <NUM> displays a virtual video including a virtual object in accordance with the position and posture of the image capturing unit <NUM> or the display unit <NUM>. In the present embodiment, the display unit <NUM> is fixed to a user's head in a state in which the user can visually recognize a virtual object appropriately. The display unit <NUM> is configured such that the position and posture of the display unit <NUM> fluctuate in accordance with the movement of the user's head. The position of the display unit <NUM> is a coordinate position of the display unit <NUM>. The posture of the display unit <NUM> is rotation around each axis of three-dimensional coordinates (orientation in three-dimensional coordinates). In the present embodiment, the display unit <NUM> performs display indicating the structure <NUM> in a real space, display indicating a mounting member scheduled to be mounted on the structure <NUM>, and display of information indicating a distance from the reference position to the position at which the jig <NUM> is arranged, in accordance with instructions from the control unit <NUM> to be described later.

The environmental information detection unit <NUM> detects various types of information on the basis of at least one of an image captured by the image capturing unit <NUM> and a measurement result of the inertial measurement unit <NUM>. The video display system <NUM> may not include the inertial measurement unit <NUM>. In this case, the video display system <NUM> performs various processes in the environmental information detection unit <NUM> on the basis of the image captured by the image capturing unit <NUM>. In a case where the video display system <NUM> does not include the image capturing unit <NUM>, the video display system <NUM> performs various processes in the environmental information detection unit <NUM> on the basis of the inertial measurement unit <NUM>. The environmental information detection unit <NUM> includes a reference position detection unit <NUM>, a structure arrangement detection unit <NUM>, a user position detection unit <NUM>, a visual line direction detection unit <NUM>, a display arrangement detection unit <NUM>, and a jig arrangement detection unit <NUM>.

The reference position detection unit <NUM> detects a reference position for arranging the origin N of a virtual coordinate axis α. In the present embodiment, the reference position detection unit <NUM> detects a feature point that makes it possible to recognize the three-dimensional arrangement of the reference marker <NUM> in a real space from the image captured by the image capturing unit <NUM>. The reference position detection unit <NUM> detects a reference position in a real space from the detected feature point. For example, in a state in which the reference marker <NUM> is arranged along the end 110a of the structure <NUM>, the reference position detection unit <NUM> detects the center of the end 110a in the face plate <NUM> from the feature point of the reference marker <NUM> as the reference position. The result of detection of the reference position performed by the reference position detection unit <NUM> is, for example, a relative position of the reference position from the image capturing unit <NUM> or the display.

The reference position detection unit <NUM> may detect the reference position not from the reference marker <NUM> but from the arrangement of the edge of the structure <NUM> in the image captured by the image capturing unit <NUM>, the surrounding environment of the structure <NUM> in the image captured by the image capturing unit <NUM>, or the like. The reference position detection unit <NUM> may separately acquire a signal indicating the reference position in a real space in a wireless or wired manner. The result of detection of the reference position performed by the reference position detection unit <NUM> may be a position on the image captured by the image capturing unit <NUM>.

The structure arrangement detection unit <NUM> detects the arrangement of the structure <NUM> arranged in a real space. In the present embodiment, the structure arrangement detection unit <NUM> detects the arrangement of the structure <NUM> on the basis of the reference position detected by the reference position detection unit <NUM> and another feature point for specifying the direction of the structure <NUM>. The other feature point for specifying the direction of the structure <NUM> may be the direction of the reference marker <NUM>, the arrangement of a marker provided in the structure <NUM> separately from the reference marker <NUM>, the arrangement of the edge of the structure <NUM>, or the surrounding environment of the structure <NUM>. For example, the structure arrangement detection unit <NUM> detects the arrangement of the edge of the structure <NUM> in a real space from the image captured by the image capturing unit <NUM>.

The result of detection of the arrangement of the structure <NUM> performed by the structure arrangement detection unit <NUM> is, for example, a relative arrangement of the structure <NUM> with respect to the image capturing unit <NUM> or the display. The structure arrangement detection unit <NUM> may separately acquire a signal indicating the arrangement of the structure <NUM> in a real space in a wireless or wired manner. The result of detection of the arrangement of the structure <NUM> performed by the structure arrangement detection unit <NUM> may be an arrangement on the image captured by the image capturing unit <NUM>.

The user position detection unit <NUM> detects the position of a user who performs mounting work. In the present embodiment, the user position detection unit <NUM> detects a relative position of the image capturing unit <NUM> with respect to the reference position as the user's position. For example, the user position detection unit <NUM> detects a feature point of the surrounding environment from the image captured by the image capturing unit <NUM>. The user position detection unit <NUM> detects the position of the image capturing unit <NUM> in a real space from the detected feature point. The user position detection unit <NUM> may detect the position of the image capturing unit <NUM> on the basis of the image captured by the image capturing unit <NUM>, the result of detection of the reference position, and the measurement result of the inertial measurement unit <NUM>.

The user position detection unit <NUM> may detect a relative position of the display of the display unit <NUM> with respect to the reference position as the user's position. The user position detection unit <NUM> may detect the relative position of the image capturing unit <NUM> with respect to the reference position as the position of the display. The user position detection unit <NUM> may detect the position of the display on the basis of the image captured by the image capturing unit <NUM>, the result of detection of the reference position, and the measurement result of the inertial measurement unit <NUM>. The user position detection unit <NUM> may detect the position of the display without using the image captured by the image capturing unit <NUM>. The user position detection unit <NUM> may separately acquire a signal indicating the position of the image capturing unit <NUM> or the display in a real space from, for example, an external image capturing unit or the like.

The user position detection unit <NUM> may detect the position of the user with respect to any position of the structure <NUM> rather than the position of the user with respect to the reference position. The user position detection unit <NUM> may detect the position of the user with respect to the jig <NUM>.

The visual line direction detection unit <NUM> detects the visual line direction of the user who performs mounting work. Meanwhile, the "visual line direction" may be a direction in which a front camera is directed, may be the center of a captured image, may be a direction in which the display is directed, or may be a direction calculated from the direction of the user's eyeball. In the present embodiment, the visual line direction detection unit <NUM> detects the posture of the image capturing unit <NUM> with respect to the reference position as the visual line direction. For example, the visual line direction detection unit <NUM> detects a feature point of the surrounding environment from the image captured by the image capturing unit <NUM>. The visual line direction detection unit <NUM> detects the posture of the image capturing unit <NUM> in a real space from the detected feature point. The visual line direction detection unit <NUM> may detect the posture of the image capturing unit <NUM> on the basis of the image captured by the image capturing unit <NUM>, the result of detection of the reference position, and the measurement result of the inertial measurement unit <NUM>.

The visual line direction detection unit <NUM> may detect the posture of the display of the display unit <NUM> with respect to the reference position as the visual line direction. The visual line direction detection unit <NUM> may detect the posture of the image capturing unit <NUM> with respect to the reference position as the posture of the display. The visual line direction detection unit <NUM> may detect the posture of the display on the basis of the image captured by the image capturing unit <NUM>, the result of detection of the reference position, and the measurement result of the inertial measurement unit <NUM>. The visual line direction detection unit <NUM> may detect the posture of the display without using the image captured by the image capturing unit <NUM>. The visual line direction detection unit <NUM> may separately acquire a signal indicating the posture of the image capturing unit <NUM> or the display in a real space from, for example, an external image capturing unit or the like.

The display arrangement detection unit <NUM> detects the arrangement of the display. In the present embodiment, the display arrangement detection unit <NUM> detects the arrangement of the display with respect to the reference position from the detection result obtained by the user position detection unit <NUM> and the detection result obtained by the visual line direction detection unit <NUM>.

The jig arrangement detection unit <NUM> detects the arrangement of the jig <NUM> arranged in a real space. In the present embodiment, the jig arrangement detection unit <NUM> detects a feature point of each of the first jig <NUM> and the second jig <NUM> from the image captured by the image capturing unit <NUM>. As shown in <FIG>, the feature point of each of the first jig <NUM> and the second jig <NUM> may be, for example, a marker <NUM> provided in each of the first jig <NUM> and the second jig <NUM>. In the present embodiment, a QR code (registered trademark) is attached to the marker <NUM>. The jig arrangement detection unit <NUM> recognizes, for example, the QR code of the marker <NUM> from the image captured by the image capturing unit <NUM>, and calculates the arrangement of the first jig <NUM> and the second jig <NUM>. The marker <NUM> may have characteristic patterns other than the QR code. The jig arrangement detection unit <NUM> may detect color, blinking, electromagnetic information, or the like from the first jig <NUM> and the second jig <NUM>, and detect the arrangement of each of the first jig <NUM> and second jig.

As shown in <FIG>, the jig arrangement detection unit <NUM> may detect the arrangement of each of the first jig <NUM> and second jig on the basis of information acquired from laser distance measurement devices <NUM> and <NUM> arranged on both ends of the structure <NUM>. For example, the laser distance measurement device <NUM> transmits a distance from one end of the structure <NUM> to the end of the first jig <NUM> to the jig arrangement detection unit <NUM>. The jig arrangement detection unit <NUM> detects the arrangement of the first jig <NUM> from the distance from one end of the structure <NUM> to the end of the first jig <NUM>. The laser distance measurement device <NUM> transmits a distance from the other end of the structure <NUM> to the end of the second jig <NUM> to the jig arrangement detection unit <NUM>. The jig arrangement detection unit <NUM> detects the arrangement of the second jig <NUM> based on the distance from the other end of the structure <NUM> to the end of the second jig <NUM>. The jig arrangement detection unit <NUM> may detect the arrangement of each of the first jig <NUM> and second jig using the entire length of the structure <NUM> and the thicknesses of the first jig <NUM> and the second jig <NUM> which are acquired in advance. The video display system <NUM> may include the laser distance measurement devices <NUM> and <NUM>.

The storage unit <NUM> previously stores various types of information (CG data) required for display of a virtual object. The storage unit <NUM> stores information on the arrangement of the jig <NUM> on the structure <NUM>. In the present invention, the information on the arrangement of the jig <NUM> includes information on the arrangement of the first jig <NUM> and the second jig <NUM> of which the flat surface 121a and the flat surface 122a are in contact with the mounting member. The information on the arrangement of the first jig <NUM> and the second jig <NUM> includes information on the arrangement of the jig that restricts the position of the mounting member in the extending direction of the structure <NUM>; the angle of inclination of the mounting member with respect to the main surface 111a of the structure <NUM>; and the angle of rotation of the mounting member with respect to the axis orthogonal to the main surface 111a.

For example, the storage unit <NUM> previously stores the display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> associated in advance with the virtual coordinate axis α in a virtual space and the information <NUM> and <NUM> indicating the distance from the reference position to the position at which the jig <NUM> is arranged. The storage unit <NUM> stores the display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> and the coordinates on the virtual coordinate axis α of the position at which the jig <NUM> is arranged. The display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> corresponding to the position and posture of the image capturing unit <NUM> in a real space and the position at which the jig <NUM> is arranged are determined by associating the virtual coordinate axis α with the real space.

The storage unit <NUM> also previously stores a predetermined arrangement relation between the reference marker <NUM> arranged in a real space and the virtual coordinate axis α. In the present embodiment, the position of a reference point M in a real space of the reference marker <NUM> and the reference position at which the origin N of the virtual coordinate axis α is arranged are associated with each other. That is, the arrangement of the reference marker <NUM> in a real space and the origin N of the virtual coordinate axis α are associated with each other.

The operation detection unit <NUM> detects a user's operation. For example, in the touch panel display of the video display system <NUM>, a position touched by a user is detected. The operation detection unit <NUM> sequentially transmits detected information to the control unit <NUM>. For example, the operation detection unit <NUM> transmits signals indicating selected virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> to the control unit <NUM> in accordance with a user's operation. The user's operation which is detected by the operation detection unit <NUM> may be a hand gesture, a blink, or a button input.

The control unit <NUM> performs various types of control on the basis of the detection result obtained by the environmental information detection unit <NUM>, the detection result obtained by the operation detection unit <NUM>, and information stored in the storage unit <NUM>. The control unit <NUM> includes a reference arrangement setting unit <NUM>; a display control unit <NUM>; and a member determination unit <NUM>.

The reference arrangement setting unit <NUM> sets a relationship between the virtual coordinate axis α for specifying the display arrangement of a virtual object and the position and posture of the image capturing unit <NUM> or the display detected by the user position detection unit <NUM> and the visual line direction detection unit <NUM>. In the present embodiment, the reference arrangement setting unit <NUM> associates the virtual coordinate axis α with a real space in accordance with the arrangement of the structure <NUM> detected by the structure arrangement detection unit <NUM>. In other words, the reference arrangement setting unit <NUM> arranges the origin N of the virtual coordinate axis α at the reference position on the basis of the detection result detected by the structure arrangement detection unit <NUM>, and sets the direction of the virtual coordinate axis α in a real space.

The display control unit <NUM> performs a display process indicating the arrangement of the jig <NUM> on the structure <NUM> arranged in a real space on the basis of the information on the arrangement of the jig <NUM> stored in the storage unit <NUM> and the detection result obtained by the structure arrangement detection unit <NUM>. For example, the display control unit <NUM> instructs the display unit <NUM> to display information indicating the arrangement of a plurality of members, and causes the display unit <NUM> to display the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> on the structure <NUM> arranged in a real space as shown in <FIG>. In the present embodiment, the display control unit <NUM> acquires the information on the arrangement of the jig <NUM> from the storage unit <NUM>. The display control unit <NUM> associates the display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> and the arrangement of the information <NUM> and <NUM> indicating the distance from the reference position to the position at which the jig <NUM> is arranged with the virtual coordinate axis α associated with the real space in the reference arrangement setting unit <NUM>, in accordance with the acquired information on the arrangement of the jig <NUM>. As a result, the display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> and the arrangement of the information <NUM> and <NUM> indicating the distance from the reference position to the position at which the jig <NUM> is arranged are associated with the reference position. In the present embodiment, the display control unit <NUM> causes the display unit <NUM> to display the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> and the information <NUM> and <NUM> indicating the distance to the position at which the jig <NUM> is arranged which are associated with the reference position, in accordance with the user's position and visual line direction.

The display control unit <NUM> may change the display arrangement of information indicating the arrangement of the jig <NUM> in accordance with the arrangement of the display detected by the display arrangement detection unit <NUM>. For example, when a user has changed the arrangement of the display, the display control unit <NUM> may change the display arrangement of the structure <NUM>, the display arrangement of the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM>, and the display arrangement of the information <NUM> and <NUM> indicating the distance to the position at which the jig <NUM> is arranged, on the display, in accordance with the change.

The display control unit <NUM> displays information indicating a position shift of the jig <NUM> on the basis of the arrangement of the jig <NUM> detected by the jig arrangement detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. For example, when the arrangement of the jig <NUM> detected by the jig arrangement detection unit <NUM> shifts from the information on the arrangement of the jig <NUM> stored in the storage unit <NUM> by an allowable error or more, the display control unit <NUM> performs a display process indicating the occurrence of the shift on the display unit <NUM>. For example, when it is determined that the shift has occurred, the display control unit <NUM> causes the display unit <NUM> to display information <NUM> indicating that the shift has occurred, as shown in <FIG>. When it is determined that the shift has occurred, the display control unit <NUM> may change the color of the virtual mounting member.

The member determination unit <NUM> determines a mounting member on which mounting work is performed by a user from a plurality of mounting members corresponding to a plurality of virtual mounting members stored in the storage unit <NUM>. For example, the member determination unit <NUM> determines at least one mounting member out of the plurality of mounting members as the mounting member on which mounting work is performed by the user on the basis of, for example, at least one of the detection result obtained by the operation detection unit <NUM>, the position of the user with respect to the structure <NUM>, and the user's visual line direction. The display control unit <NUM> determines information indicating the arrangement of the jig <NUM> corresponding to the mounting member determined by the member determination unit <NUM> as information used in the arrangement of the jig <NUM> by the user, and performs a display process on information indicating the arrangement of the jig <NUM> corresponding to the other mounting members. For example, when the member determination unit <NUM> has determined the mounting member <NUM> corresponding to the virtual mounting member <NUM> as the mounting member on which mounting work is performed by the user, the display control unit <NUM> instructs the display unit <NUM> not to display information indicating the arrangement of the jig <NUM> corresponding to the remaining mounting members. A display process for the remaining jig <NUM> is not limited to non-display, but includes changing display such as semi-transparent display, blinking display, contour display, color change, and display density change. For the remaining jig <NUM>, only information indicating the distance of the jig <NUM> may be changed in display.

The display control unit <NUM> may change display of information indicating the arrangement of the jig <NUM> corresponding to the mounting member determined by the member determination unit <NUM>. <FIG> shows a state in which information relating to the virtual mounting members <NUM>, <NUM>, and <NUM> is not displayed. The display control unit <NUM> may cause the display unit <NUM> to display information indicating the arrangement of the jig <NUM> corresponding to the mounting member determined by the member determination unit <NUM> from the state of non-display.

For example, the member determination unit <NUM> determines a mounting member corresponding to the user's operation among a plurality of mounting members on the basis of the detection result obtained by the operation detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. The display control unit <NUM> performs a display process of information indicating the arrangement of the jig <NUM> corresponding to the determined mounting member. For example, when the user touches the virtual mounting member <NUM>, the member determination unit <NUM> determines the mounting member <NUM> corresponding to the virtual mounting member <NUM> operated by the user as the mounting member on which mounting work is performed by the user. The display control unit <NUM> may leave display of the virtual mounting member <NUM> operated by the user and display of a distance corresponding thereto, as shown in <FIG>, from a state in which the virtual mounting members <NUM>, <NUM>, <NUM>, and <NUM> are displayed on the structure <NUM>, and do not display the virtual mounting members <NUM>, <NUM>, and <NUM>. When the information <NUM> and <NUM> indicating the distance of the jig <NUM> are manipulated instead of the virtual mounting member in the operation detection unit <NUM>, the member determination unit <NUM> may determine a mounting member corresponding to the manipulated information <NUM> and <NUM> as the mounting member on which mounting work is performed by the user.

For example, the member determination unit <NUM> determines a mounting member corresponding to the detection result obtained by the visual line direction detection unit <NUM> among a plurality of mounting members on the basis of the detection result obtained by the visual line direction detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. The display control unit <NUM> performs a display process of information indicating the arrangement of the jig <NUM> relating to the determined mounting member. For example, as shown in <FIG>, the display control unit <NUM> may leave display of the virtual mounting member <NUM> corresponding to the visual line direction detected by the visual line direction detection unit <NUM> and display of the information <NUM> and <NUM> corresponding thereto, and do not display the virtual mounting members <NUM>, <NUM>, and <NUM>. The virtual mounting member corresponding to the visual line direction may be a front-most virtual mounting member located on a half line in the visual line direction. The display control unit <NUM> may change and display the color of the virtual mounting member located on an extended line in the visual line direction, and definitely determine display used in the arrangement of the jig <NUM> in accordance with the detection result for the user's operation performed by the operation detection unit <NUM>.

For example, the member determination unit <NUM> determines a mounting member corresponding to the detection result obtained by the user position detection unit <NUM> among a plurality of mounting members on the basis of the detection result obtained by the user position detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. The display control unit <NUM> performs a display process of information indicating the arrangement of the jig <NUM> relating to the determined mounting member. For example, the member determination unit <NUM> determines where on the left, right, top and bottom the user is located relative to the structure <NUM> on the basis of the detection result obtained by the user position detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. For example, as shown in <FIG>, a plane orthogonal to the extending direction of the structure <NUM> is divided into four regions R1, R2, R3, and R4. In a case where the user's position detected by the user position detection unit <NUM> is the region R1, the member determination unit <NUM> outputs the virtual mounting member <NUM> corresponding to the region R1 as a determination result. The display control unit <NUM> may leave display of information indicating the virtual mounting member <NUM> and a distance corresponding thereto, and do not display the remaining information. In a case where the user's position is located near between different regions, the member determination unit <NUM> outputs a virtual mounting member corresponding to both the regions as a determination result.

The member determination unit <NUM> may calculate a distance between the user and the structure <NUM> from the detection result obtained by the user position detection unit <NUM>, and determine a mounting member in accordance with the distance. For example, the member determination unit <NUM> may output a virtual mounting member located within two meters of the user as a determination result. In this case, the display control unit <NUM> may, for example, display a virtual mounting member located within two meters of the user in blue, and may not display a virtual mounting member located two meters or more away from the user or display it semi-transparently. Even in the case of a virtual mounting member located within two meters of the user, the display control unit <NUM> may not display the virtual mounting member located on the opposite side of the user with the structure <NUM> interposed therebetween. For example, in <FIG>, even in a case where the virtual mounting member <NUM> is located within two meters of the user, the virtual mounting member <NUM> is located on the opposite side of the user with the structure <NUM> interposed therebetween, and thus the display control unit <NUM> does not display the virtual mounting member <NUM>. The member determination unit <NUM> may acquire an arrangement relation between the user and the jig <NUM> from the detection result obtained by the user position detection unit <NUM>, and determine a mounting member in accordance with the arrangement relation.

Next, an example of processing which is performed in the video display system <NUM> will be described with reference to <FIG> is a flow chart illustrating an example of processing which is performed in the video display system <NUM>.

In step S101, the reference position detection unit <NUM> detects a reference position for arranging the origin N of the virtual coordinate axis α. Subsequently, the process proceeds to step S102.

In step S102, the structure arrangement detection unit <NUM> detects the arrangement of the structure <NUM> arranged in a real space. Subsequently, the process proceeds to step S103.

In step S103, the display arrangement detection unit <NUM> detects the arrangement of the display of the display unit <NUM>. Subsequently, the process proceeds to step S104.

In step S104, the jig arrangement detection unit <NUM> detects the arrangement of the jig <NUM> arranged in a real space. Subsequently, the process proceeds to step S105.

In step S105, the control unit <NUM> performs a display process on the basis of information detected in steps S102 to S104. The display unit <NUM> displays various types of information in accordance with instructions from the control unit <NUM>.

Next, a display process which is performed by the control unit <NUM> will be described with reference to <FIG> is a flow chart illustrating a display process which is performed by the control unit <NUM>.

In step S111, the control unit <NUM> determines whether there is information on the selection of a member. The information on the selection of a member is, for example, a determination result determined by the member determination unit <NUM> on the basis of at least one of the detection result obtained by the operation detection unit <NUM>, the position of the user with respect to the structure <NUM>, and the user's visual line direction. In a case where it is determined that there is no information on the selection of a member (NO in step S111), the control unit <NUM> advances the process to step S112. In a case where it is determined that there is the information on the selection of a member (YES in step S111), the control unit <NUM> advances the process to step S113.

In step S112, the control unit <NUM> causes the display control unit <NUM> to display all the virtual mounting members stored in the storage unit <NUM> on the display unit <NUM>, and advances the process to step S <NUM>.

In step S113, the control unit <NUM> determines whether the jig arrangement has been detected in the jig arrangement detection unit <NUM>. In a case where it is determined that the jig arrangement is not detected (NO in step S113), the control unit <NUM> advances the process to step S114. In a case where it is determined that the jig arrangement is detected (YES in step S113), the control unit <NUM> advances the process to step S115.

In step S114, the control unit <NUM> causes the display control unit <NUM> to display a virtual mounting member corresponding to a mounting member determined by the member determination unit <NUM> on the display unit <NUM>, and advances the process to step S117.

In step S115, the control unit <NUM> determines whether the jig arrangement is appropriate on the basis of the detection result obtained by the jig arrangement detection unit <NUM>. In a case where it is determined that the jig arrangement is appropriate (YES in step S115), the control unit <NUM> advances the process to step S114. In a case where it is determined that the jig arrangement is not appropriate (NO in step S115), the control unit <NUM> advances the process to step S116.

In step S116, the control unit <NUM> causes the display control unit <NUM> to display a virtual mounting member corresponding to a mounting member determined by the member determination unit <NUM> and an error on the display unit <NUM>, and the control unit <NUM> advances the process to step S117. In this case, the display control unit <NUM> displays, for example, the information <NUM> indicating that a shift has occurred on the display unit <NUM>.

In step S117, the control unit <NUM> determines whether to end the display process. In a case where it is determined that the display process is not ended (NO in step S117), the control unit <NUM> advances the process to step S111. In a case where it is determined that the display process is ended (YES in step S117), the control unit <NUM> ends the display process.

Next, main operations and effects of the video display system <NUM> according to the present embodiment will be described.

In the video display system <NUM>, the information on the arrangement of the jig <NUM> for positioning a mounting member on the structure <NUM> is stored, and the display process indicating the arrangement of the jig <NUM> is performed on the structure <NUM> arranged in a real space. A user can easily arrange the jig <NUM> in accordance with the display process indicating the arrangement of the jig <NUM>. As a result, the mounting member is positioned by the jig <NUM> on the structure <NUM>, and thus it is possible to easily mount the mounting member on the structure <NUM>.

For example, the mounting member <NUM> is mounted on the structure <NUM> in a state in which it is also inclined with respect to the main surface 112a of the face plate <NUM> and the main surface 113a of the face plate <NUM> while being inclined with respect to the main surface 111a of the face plate <NUM>. It is very difficult to perform mounting work in such a state. The jig <NUM> includes the first jig <NUM> having the flat surface 121a and the second jig <NUM> having the flat surface 122a facing the flat surface 121a. The information on the arrangement of the jig <NUM> includes the information on the arrangement of the jig <NUM> of which the flat surface 121a and the flat surface 122a abut the mounting member. Therefore, according to the video display system <NUM>, it is possible to easily and appropriately arrange the jig <NUM> and to easily position the mounting member <NUM> on the structure <NUM>.

The structure <NUM> has the main surface 111a which extends in a predetermined direction and with which the mounting member is in contact. The information on the arrangement of the jig <NUM> includes the information on the arrangement of the jig <NUM> for restricting the position of the mounting member in a predetermined direction, the angle of inclination of the mounting member with respect to the main surface 111a, and the angle of rotation of the member with respect to the axis orthogonal to the main surface 111a. Therefore, according to the video display system <NUM>, it is possible to easily and appropriately arrange the jig <NUM> and to easily position the mounting member <NUM> on the structure <NUM>.

The video display system <NUM> includes the display arrangement detection unit <NUM> that detects the arrangement of the display that performs display based on the display process. The control unit <NUM> changes display of the structure and the display arrangement of information indicating the arrangement of the jig <NUM> on the display in accordance with the arrangement of the display detected by the display arrangement detection unit <NUM>. Therefore, the user can confirm the arrangement of the jig <NUM> on the structure at any position by moving the display.

The video display system <NUM> includes the jig arrangement detection unit <NUM> that detects the arrangement of the jig <NUM> arranged in a real space. The control unit <NUM> displays the information <NUM> indicating a position shift of the jig <NUM> on the basis of the arrangement of the jig <NUM> detected by the jig arrangement detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. Therefore, the user can easily confirm whether the arrangement of the jig <NUM> is appropriate.

The video display system <NUM> includes the operation detection unit <NUM> that detects a user's operation. The control unit <NUM> performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's operation among a plurality of mounting members on the basis of the detection result obtained by the operation detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. Therefore, the user can select information indicating the arrangement of the jig to be displayed.

The storage unit <NUM> stores information on the arrangement of a plurality of mounting members on the structure <NUM>. The control unit <NUM> gives an instruction for display of information indicating the arrangement of a plurality of mounting members on the basis of the information on the arrangement of the plurality of members stored in the storage unit <NUM>. The control unit <NUM> determines a mounting member corresponding to the detection result obtained by the operation detection unit <NUM> among the plurality of mounting members, and performs a display process indicating the arrangement of the jig <NUM> relating to the determined mounting member. Therefore, the user can select information indicating the arrangement of the jig by operating the displayed mounting member.

The video display system <NUM> includes the user position detection unit <NUM> that detects a user's position. The control unit <NUM> performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's position among a plurality of mounting members on the basis of the detection result obtained by the user position detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. Therefore, information indicating appropriate arrangement of the jig can be displayed in accordance with the user's position.

The video display system <NUM> includes the visual line direction detection unit <NUM> that detects a user's visual line direction. The control unit <NUM> performs a display process of information indicating the arrangement of the jig relating to a mounting member corresponding to the user's visual line direction among a plurality of mounting members on the basis of the detection result obtained by the visual line direction detection unit <NUM> and the information on the arrangement of the jig <NUM> stored in the storage unit <NUM>. Therefore, information indicating appropriate arrangement of the jig can be displayed in accordance with the user's visual line direction.

Meanwhile, the block diagram used in the description of the above embodiment represents blocks in units of functions. These functional blocks (constituent elements) are realized by any combination of at least one of hardware and software. In addition, a method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device which is physically or logically coupled, or may be realized using two or more devices which are physically or logically separated from each other by connecting the plurality of devices directly and/or indirectly (for example, using a wired or wireless manner or the like). The functional block may be realized by combining software with the one device or the plurality of devices.

Examples of the functions include determining, deciding, judging, calculating, computing, process, deriving, investigating, search, ascertaining, receiving, transmitting, output, access, resolving, selecting, choosing, establishing, comparing, assuming, expecting, considering, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (or mapping), assigning, and the like, but there is no limitation thereto. For example, a functional block (constituent element) for causing transmitting to function is referred to as a transmitting unit or a transmitter. As described above, any realization methods are not particularly limited.

For example, the video display system in an embodiment of the present disclosure may function as a computer that performs processing in the present embodiment. <FIG> is a diagram illustrating an example of a hardware configuration of the video display system <NUM> according to an embodiment of the present disclosure. The video display system <NUM> described above may be physically configured as a computer device including a processor <NUM>, a memory <NUM>, a storage <NUM>, a communication device <NUM>, an input device <NUM>, an output device <NUM>, a bus <NUM>, and the like. In addition to the above components, the video display system <NUM> also includes hardware such as the image capturing unit <NUM>, a sensor used for the inertial measurement unit <NUM>, and a display.

Meanwhile, in the following description, the word "device" can be replaced with a circuit, a unit, or the like. The hardware configuration of the video display system <NUM> may be configured to include one or a plurality of devices shown in the drawing, or may be configured without including some devices.

The processor <NUM> performs an arithmetic operation by reading predetermined software (a program) onto hardware such as the processor <NUM> or the memory <NUM>, and thus each function of the video display system <NUM> is realized by controlling communication in the communication device <NUM> or controlling at least one of reading-out and writing of data in the memory <NUM> and the storage <NUM>.

The processor <NUM> controls the whole computer, for example, by operating an operating system. The processor <NUM> may be constituted by a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic operation device, a register, and the like. For example, the environmental information detection unit <NUM>, the control unit <NUM>, and the like may be realized by the processor <NUM>.

In addition, the processor <NUM> reads out a program (a program code), a software module, data, or the like from at least one of the storage <NUM> and the communication device <NUM> into the memory <NUM>, and executes various types of processes in accordance therewith. An example of the program which is used includes a program causing a computer to execute at least some of the operations described in the foregoing embodiment. For example, the display unit <NUM>, the environmental information detection unit <NUM>, the operation detection unit <NUM>, and the control unit <NUM> are stored in the memory <NUM>, and may be realized by a control program which is operated in the processor <NUM>. Similarly, other functional blocks may be realized. The execution of various types of processes described above by one processor <NUM> has been described, but these processes may be simultaneously or sequentially executed by two or more processors <NUM>. One or more chips may be mounted in the processor <NUM>. Meanwhile, the program may be transmitted from a network through an electrical communication line.

The memory <NUM> is a computer readable recording medium, and may be constituted by at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), and the like. The memory <NUM> may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory <NUM> can store a program (a program code), a software module, or the like that can be executed in order to carry out a wireless communication method according to an embodiment of the present disclosure.

The storage <NUM> is a computer readable recording medium, and may be constituted by at least one of, for example, an optical disc such as a compact disc ROM (CD-ROM), a hard disk drive, a flexible disk, a magnetooptic disc (for example, a compact disc, a digital versatile disc, or a Blu-ray (registered trademark) disc), a smart card, a flash memory (for example, a card, a stick, or a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. The storage <NUM> may be referred to as an auxiliary storage device. The foregoing storage medium may be, for example, a database including at least one of the memory <NUM> and the storage <NUM>, a server, or another suitable medium.

The communication device <NUM> is hardware (a transmitting and receiving device) for performing communication between computers through at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device <NUM> is an input device (such as, for example, a keyboard, a mouse, a microphone, a switch, a button, or a sensor) that receives an input from the outside. The output device <NUM> is an output device (such as, for example, a display, a speaker, or an LED lamp) that executes an output to the outside. Meanwhile, the input device <NUM> and the output device <NUM> may be an integrated component (for example, a touch panel). In addition, respective devices such as the processor <NUM> and the memory <NUM> are connected to each other through the bus <NUM> for communicating information. The bus <NUM> may be configured using a single bus, or may be configured using a different bus between devices.

In addition, the video display system <NUM> may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), or some or all of the respective functional blocks may be realized by the hardware. For example, the processor <NUM> may be mounted using at least one of the hardware.

The aspects/embodiments described in the present disclosure may be used alone, may be used in combination, or may be switched during implementation thereof. In addition, notification of predetermined information (for example, notification of "X") is not limited to explicit transmission, and may be performed by implicit transmission (for example, the notification of the predetermined information is not performed).

Hereinbefore, the present invention has been described in detail, but it is apparent to those skilled in the art that the present invention should not be limited to the embodiments described in the present disclosure. The present invention can be implemented with modified and changed aspects without departing from the scope of the present invention, which is determined by the claims. Therefore, the description of the present disclosure is intended for illustrative explanation only, and does not impose any limited interpretation on the present invention,.

The aspects/embodiments described in the present disclosure may be applied to at least one of a system employing long term evolution (LTE), LTE-Advanced (LTE-A), SUPER <NUM>, IMT-Advanced, a 4th generation mobile communication system (<NUM>), a 5th generation mobile communication system (<NUM>), Future Radio Access (FRA), new Radio (NR), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, Ultra Mobile Broadband (UMB), IEEE <NUM> (Wi-Fi (registered trademark)), IEEE <NUM> (WiMAX (registered trademark)), IEEE <NUM>, Ultra-WideBand (UWB), Bluetooth (registered trademark), or other appropriate systems and a next-generation system extended on the basis thereof. In addition, a plurality of systems may be combined (for example, <NUM> and at least one of LTE and LTE-A are combined or the like) and be applied.

The order of the processing sequences, the sequences, the flow charts, and the like of the aspects/embodiments described above in the present disclosure may be changed as long as they are compatible with each other. For example, in the methods described in the present disclosure, various steps as elements are presented using an exemplary order but the methods are not limited to the presented order.

The input or output information or the like may be stored in a specific place (for example, a memory) or may be managed using a management table. The input or output information or the like may be overwritten, updated, or added. The output information or the like may be deleted. The input information or the like may be transmitted to another device.

Determination may be performed using a value (<NUM> or <NUM>) which is expressed by one bit, may be performed using a Boolean value (true or false), or may be performed by comparison of numerical values (for example, comparison thereof with a predetermined value).

Information, a signal or the like described in the present disclosure may be expressed using any of various different techniques. For example, data, an instruction, a command, information, a signal, a bit, a symbol, and a chip which can be mentioned in the overall description may be expressed by a voltage, a current, an electromagnetic wave, a magnetic field or magnetic particles, an optical field or photons, or any combination thereof.

The term "determining" which is used in the present disclosure may include various types of operations. The term "determining" may include regarding operations such as, for example, judging, calculating, computing, processing, deriving, investigating, looking up/search/inquiry (for example, looking up in a table, a database or a separate data structure), or ascertaining as an operation such as "determining. " In addition, the term "determining" may include regarding operations such as receiving (for example, receiving information), transmitting (for example, transmitting information), input, output, or accessing (for example, accessing data in a memory) as an operation such as "determining. " In addition, the term "determining" may include regarding operations such as resolving, selecting, choosing, establishing, or comparing as an operation such as "determining. " That is, the term "determining" may include regarding some kind of operation as an operation such as "determining. " In addition, the term "determining" may be replaced with the term "assuming," "expecting," "considering," or the like.

An expression "on the basis of ~" which is used in the present disclosure does not refer to only "on the basis of only ~," unless otherwise described. In other words, the expression "on the basis of ~" refers to both "on the basis of only ~" and "on the basis of at least ~.

Any reference to elements having names such as "first" and "second" which are used in the present disclosure does not generally limit amounts or an order of the elements. The terms can be conveniently used to distinguish two or more elements in the present disclosure. Accordingly, reference to first and second elements does not mean that only two elements are employed or that the first element has to precede the second element in any form.

In the present disclosure, when the terms "include" and "including" and modifications thereof are used, these terms are intended to have a comprehensive meaning similarly to the term "comprising. " Further, the term "or" which is used in the present disclosure is intended not to mean an exclusive logical sum.

In the present disclosure, when articles are added by translation like, for example, "a," "an" and "the" in English, the present disclosure may include that nouns that follow these articles are plural forms.

In the present disclosure, an expression "A and B are different" may mean that "A and B are different from each other. " Meanwhile, the expression may mean that "A and B are different from C. " The terms "separated," "coupled," and the like may also be construed similarly to "different.

Claim 1:
A video display system (<NUM>) used in mounting a member (<NUM>, <NUM>) on a structure (<NUM>), comprising:
a storage unit (<NUM>) configured to store information on arrangement of a jig (<NUM>) for positioning the member (<NUM>, <NUM>) on the structure (<NUM>);
a structure arrangement detection unit (<NUM>) configured to detect an arrangement of the structure (<NUM>) arranged in a real space; and
a control unit (<NUM>) configured to perform a display process indicating the arrangement of the jig (<NUM>) on the structure (<NUM>) arranged in the real space on the basis of the information (<NUM>, <NUM>) on the arrangement of the jig (<NUM>) stored in the storage unit (<NUM>) and a detection result obtained by the structure arrangement detection unit (<NUM>);
characterised in that:
the structure (<NUM>) extends in a predetermined direction with a main surface (111a) with which the member (<NUM>) is in contact, and the information (<NUM>, <NUM>) on the arrangement of the jig (<NUM>) includes information (<NUM>, <NUM>) on the arrangement of the jig (<NUM>) for restricting a position of the member (<NUM>, <NUM>) in the predetermined direction; an angle of inclination of the member (<NUM>, <NUM>) with respect to the main surface (111a); and an angle of rotation of the member (<NUM>, <NUM>) with respect to an axis orthogonal to the main surface (111a); and
wherein the jig (<NUM>) includes a first jig (<NUM>) having a first surface (121a) and a second jig (<NUM>) having a second surface (122a) facing the first surface (121a), and the information (<NUM>, <NUM>) on the arrangement of the jig (<NUM>) includes information (<NUM>, <NUM>) on arrangement of the first and second jigs (<NUM>, <NUM>) of which the first surface (121a) and the second surface (122a) abut the member (<NUM>).