Image generation device and image generation method

An image generation device includes an image capturing unit installed on a water moving object, a target information acquisition unit which acquires target information about a target, an augmented image generation unit which generates an augmented image in which the target information acquired by the target information acquisition unit is displayed in a captured image captured by the image capturing unit, a marker display unit which displays a marker for selecting the target by a user such that the marker moves according to a display place of the target in the augmented image, and a marker moving unit moves the display place of at least one of a plurality of the markers so that determination regions which determining that each marker has been selected do not overlap each other.

FIELD

Embodiments described herein relate generally to an image generation device which generates an image including a target.

BACKGROUND

Generally, an image generation device which georeferences an image stream from a camera is disclosed. For example, there is an image generation device which receives stream image data captured by a camera of a ship, acquires the position of a target (another ship or the like) around the ship, and generates an image to be displayed on a display (see US 2015/0350552 A).

However, in order for the ship to navigate, it is necessary to acquire various kinds of information about a target (for example, a water moving object) around the ship. Therefore, when all pieces of information necessary for navigation are simply superimposed on the image captured by the camera of the ship and displayed on the display, the displayed pieces of information are difficult to understand. Therefore, such an image is less convenient.

BRIEF SUMMARY

An object of embodiments described herein is to provide an image generation device which improves convenience by displaying information on an image captured from a ship.

In accordance with an aspect of the embodiments, there is provided an image generation device. the image generation device includes an image capturing unit installed on a water moving object, a target information acquisition unit which acquires target information about a target, an augmented image generation unit which generates an augmented image in which the target information acquired by the target information acquisition unit is displayed in a captured image captured by the image capturing unit, a marker display unit which displays a marker for selecting the target by a user such that the marker moves according to a display place of the target in the augmented image, and a marker moving unit moves the display place of at least one of a plurality of the markers so that determination regions which determining that each marker has been selected do not overlap each other.

DETAILED DESCRIPTION

Embodiment

FIG.1is a block diagram showing the configuration of an image generation device1according to an embodiment.FIG.2is a side view showing a side surface of a ship4according to the present embodiment. The same portions in the drawings are denoted by the same reference numbers, and explanations of them which are considered redundant are appropriately omitted.

The image generation device1is installed on the ship4which is a water moving object moving on the water. The ship4may be any water moving object as long as it sails on the water of the sea, lake, river or the like. The image generation device1generates an augmented reality (AR) image which expands and expresses the situation around the ship4by displaying detailed information in an image captured by a camera (imaging capturing device)3. Although the generated image is hereinafter explained as an AR image, the generated image may be any image as long as it is an image in which information is added to the captured image.

A display2displays the AR image generated by the image generation device1. For example, the display2is provided as a steering support device referred to by an operator who steers the ship4. Note that the display2may be a display of a portable computer carried by a steering assistant who monitors a surrounding situation from the ship4, a display for passengers to watch in a cabin of the ship4, a display of a wearable terminal worn by a person on board, or the like.

Next, various devices connected to the image generation device1will be explained.FIG.2is a side view showing a side surface provide with various devices of the ship4according to the present embodiment.

The camera3is a wide-angle video camera which captures the image of the situation around the ship4. The camera3has a live output function, and generates moving image data as the image capturing result (image data, stream data or the like) in real time and outputs it to the image generation device1. The camera3is installed on the ship4such that the image capturing direction is horizontally forward with respect to the hull. The camera3includes a rotation mechanism which performs a rotation operation such as panning or tilting. The camera3changes the image capturing direction within a predetermined angle range with reference to the hull of the ship4based on a signal indicating the rotation operation input from the image generation device1. The height and attitude of the ship4vary depending on the natural environment such as waves. Along with this, the height and attitude (image capturing direction) of the camera3also three-dimensionally change.

The image generation device1is connected to a global navigation satellite system (GNSS) compass5, an angular velocity sensor6, a GNSS receiver7, an acceleration sensor8, an automatic identification system (AIS) receiver9, an electronic chart display and information system (ECDIS)10, a plotter11, a radar12and a sonar13as ship devices in addition to the camera3. The ship devices are information sources of detailed information. The angular velocity sensor6, the GNSS receiver7and the acceleration sensor8are incorporated in the GNSS compass5. All or part of the angular velocity sensor6, the GNSS receiver7and the acceleration sensor8may be provided independently of the GNSS compass5. In addition, the ship devices are not limited to those explained here but may be any devices.

The GNSS compass5has the functions of a direction sensor and an attitude sensor. The GNSS compass5includes a plurality of GNSS antennas (positioning antennas) fixed to the ship4. The GNSS compass5calculates the positional relationship of the respective GNSS antennas based on radio waves received from a positioning satellite. In particular, the GNSS compass5acquires the positional relationship of the respective GNSS antennas based on a phase difference of the carrier phases of the radio waves received by the respective GNSS antennas. A method of acquiring the positional relationship of the GNSS antennas can employ a known processing method. Accordingly, the GNSS compass5can acquire the bow direction of the ship4.

The GNSS compass5three-dimensionally acquires the attitude of the ship4. More specifically, the GNSS compass5detects not only the bow direction (that is, the yaw angle of the ship4) but also the roll angle and pitch angle of the ship4. The attitude information about the ship4acquired by the GNSS compass5is output to an attitude acquisition unit25and other ship devices.

The angular velocity sensor6is composed of, for example, a vibration gyro sensor. The angular velocity sensor6detects the yaw angular velocity, roll angular velocity and pitch angular velocity of the ship4in shorter cycles than detection intervals (for example, 1 second) at which the GNSS compass5detects the attitude information. By using the angle detected by the GNSS compass5and the integral of the angular velocity detected by the angular velocity sensor6together, the attitude of the ship4can be acquired at shorter time intervals than when the GNSS compass5is used alone. In addition, when the radio wave from a positioning satellite of the GNSS is blocked by, for example, an obstacle such as a bridge and the attitude detection by the GNSS compass5cannot be performed, the angular velocity sensor6functions as an alternative means of acquiring the attitude information.

The GNSS receiver7acquires the position of the ship4based on the radio waves received from the positioning satellite by the GNSS antennas. For example, the position of the ship4is the latitude, longitude and height of the GNSS antennas. The GNSS receiver7outputs the acquired position information to a position acquisition unit24and other ship devices.

The acceleration sensor8is, for example, a capacitance detection type sensor. The GNSS receiver7detects an acceleration on the yaw axis, roll axis and pitch axis of the ship4in shorter cycles than detection intervals (for example, 1 second) at which the GNSS receiver7detects the position information. By using the position information detected by the GNSS receiver7and the double integral of the acceleration detected by the acceleration sensor8together, the position of the ship4can be acquired at shorter time intervals than when the GNSS receiver7is used alone. In addition, when the radio wave from a positioning satellite of the GNSS is blocked and the position detection by the GNSS receiver7cannot be performed, the acceleration sensor8functions as an alternative means of acquiring the position information.

The AIS receiver9is a device for receiving AIS information transmitted from other ships, a land station or the like. The AIS information includes various kinds of information such as information about other ships navigating around the ship4, positions of landmarks, and identification information. The information about another ship includes, for example, a position (latitude/longitude), a hull length, a hull width, a ship type, identification information, a ship speed, a course, a destination and the like.

The ECDIS10acquires the position information about the ship4from the GNSS receiver7and outputs the information about the situation around the ship4to the image generation device1based on electronic chart information stored in advance.

The plotter11generates information about the navigation track of the ship4by continuously acquiring the position of the ship4from the GNSS receiver7. In addition, the plotter11generates, by letting the user set a plurality of waypoints (points where the ship4is scheduled to pass), a planned route by sequentially connecting these waypoints.

The radar12detects a target such as another ship which is present around the ship4. In addition, the radar12has a target tracking function (TT) capable of capturing and tracking a target. The radar12acquires the position and velocity vector (TT information) about a target by this TT. The radar12outputs the acquired TT information to the image generation device1.

The sonar13detects a target by transmitting an ultrasonic wave to the water and receiving the reflected wave of the ultrasonic wave reflected by a target such as a school of fish. The sonar13outputs the detected detection information to the image generation device1.

An input device14for the user to input information is connected to the image generation device1. The input device14is a keyboard, a mouse or the like. Note that the input device14may be a touch panel for inputting information by touching the display2, a joystick, or any other device as long as it can input information.

The user gives various instructions about the AR image by the input device14. For example, the user gives instructions about an operation of changing the attitude of the camera3, setting of whether or not to display various kinds of information, setting of the viewpoint of the AR image and the like.

Next, the configuration of the image generation device1will be explained.

The image generation device1includes an AR image generation unit20, a captured image input unit21, an image capturing position setting unit22, an antenna position setting unit23, a position acquisition unit24, an attitude acquisition unit25, a detailed information acquisition unit26, a storage unit27, a viewpoint setting unit28, and a display setting unit29.

The image generation device1is mainly composed of a computer. For example, the computer includes a processing unit and a storage unit. The processing unit is a central processing unit (CPU), a graphics processing unit (GPU) for performing three-dimensional image processing at high speed or the like. The storage unit is various memories, a hard disk drive (HDD) or the like. Note that the computer may be configured in any way. The image generation device1operates by executing software (program or the like) for realizing various functions.

The image data output from the camera3is input to the captured image input unit21, for example, at a rate of 30 frames per second. The captured image input unit21outputs the input image data to the AR image generation unit20.

The image capturing position setting unit22sets the position (image capturing position) of the camera3on the ship4. For example, the image capturing position is a position in the length direction of the hull, a position in the width direction of the hull and a position in the vertical direction (height). The height of the camera3is, for example, a height from the waterline normally assumed on the ship4but may be a height from the bottom of the ship or may be based on another reference. In addition, the image capturing position may be set to the image capturing position setting unit22by actually measuring the position of the camera3by the user and inputting the measurement by the input device14. The image capturing position setting unit22outputs the set image capturing position to the AR image generation unit20.

The antenna position setting unit23sets the position of the GNSS antennas (antenna position) on the ship4. The antenna position represents a position in the length direction, width direction and vertical direction of the hull with reference to, for example, a reference point4aof the ship4shown inFIG.2. The reference point4ais a place to be a control reference. In the present embodiment, the reference point4ais a position at the center of the hull and at the same height as the normally assumed waterline. However, the reference point4amay be determined in any way. For example, as is the case of the image capturing position described above, the antenna position may be set by inputting an actual measurement by the user. The antenna position setting unit23outputs the set antenna position to the AR image generation unit20.

The position acquisition unit24acquires the current position of the ship4in real time based on the detection results of the GNSS receiver7and the acceleration sensor8. The position acquisition unit24outputs the acquired current position information about the ship4to the AR image generation unit20.

The attitude acquisition unit25acquires the current attitude of the ship4in real time based on the detection results of the GNSS compass5and the angular velocity sensor6. The attitude acquisition unit25outputs the acquired current attitude information about the ship4to the AR image generation unit20.

The detailed information acquisition unit26acquires the detailed information to be added to the image captured by the camera3based on the information acquired from the AIS receiver9, the ECDIS10, the plotter11, the radar12, the sonar13and the like. The detailed information is input in real time from these ship devices to the image generation device1. The detailed information acquisition unit26outputs the acquired detailed information to the AR image generation unit20. The detailed information acquisition unit26may add, to each detailed information, identification information (for example, an identification number) for uniquely identifying and managing the detailed information.

The storage unit27is configured as a memory which stores various kinds of information. The storage unit27may store, for example, a three-dimensional shape of a virtual reality object expressing various kinds of detailed information as a template. The template of the three-dimensional shape stored in the storage unit27is, for example, a small ship, a large ship, a buoy, a lighthouse or the like but is not limited to this. The storage unit27outputs the stored information to the AR image generation unit20.

The viewpoint setting unit28makes a setting for the viewpoint of the AR image generated by the AR image generation unit20by using the input device14by the user. The viewpoint setting unit28outputs setting information about the viewpoint of the AR image to the AR image generation unit20.

The display setting unit29makes a setting for the display of the detailed information in the AR image generated by the AR image generation unit20by using the input device14by the user. The display setting unit29outputs setting information about the display of the detailed information to the AR image generation unit20.

The AR image generation unit20generates the AR image expressing augmented reality by synthesizing three-dimensional computer graphics with the captured image of the camera3input to the captured image input unit21.

The AR image generation unit20includes a detailed information generation unit31, a three-dimensional scene generation unit32, a screen information generation unit33and a data synthesis unit34.

The detailed information generation unit31generates detailed information to be displayed in the AR image based on the detailed information input from the detailed information acquisition unit26. The detailed information includes character information which is information represented by characters, figure information which is information represented by figures, and the like. The detailed information generation unit31outputs the generated detailed information to the three-dimensional scene generation unit32or the screen information generation unit33. The detailed information may be any information as long as it is needed by the user. For example, the detailed information may be determined based on the object or function of the image generation device1or may be information required for the navigation of the ship4.

For example, detailed information about a target such as a water moving object includes a name (ship name), a position (ship position), a direction (bow direction), a course, a distance, a speed, a turning angular velocity, a destination, a nationality (ship nationality), a type (ship type), a size (length, width, height and the like), a state (navigation state), an identification code, a distance to a closest point of approach (DCPA), a time to a closest point of approach (TCPA), a bow crossing time (BCT), a bow crossing range (BCR), and the like. Here, the DCPA is the distance when the distance to another ship is minimized. The TCPA is the time until the distance to another ship is minimized. The BCT is the time until another ship crosses the bow of the own ship. The BCR is the distance when another ship crosses the bow of the own ship.

In addition, information about the position of a buoy, a virtual buoy or the like may be used as the detailed information based on the AIS information. The virtual buoy is a virtual (insubstantial) buoy which is not actually installed on the sea for a certain reason such as difficulty in installation or the like, but the virtual buoy is displayed as a sign on the screen of a navigation device. As the information included in the electronic chart by the ECDIS10, a dangerous water, a navigation prohibition area, a lighthouse, a buoy or the like may be used as detailed information. As the information based on the plotter11, the recoded track of the ship4, a planned route, a waypoint, an arrival area, a stop-off area or the like may be used as detailed information. As the information based on the radar12, the position, speed or the like of a detected target may be used as detailed information. As the information based on the sonar13, the position or the like of a detected school of fish may be used as detailed information.

FIG.3is a conceptual diagram showing an example of the situation around the ship4according to the present embodiment.

A plurality of waypoints41and a polygonal route line42indicating a planned route to a destination are shown on the sea surface (on the water surface). A polygonal (rectangular) stop-off area43is shown close to the route line42. The waypoints41, the route line42and the stop-off area43are set by operating the plotter11by the user.

At a slightly distant point in front of the ship4, another ship44is sailing toward the right side of the ship4. A virtual buoy45is present close to the diagonally left front of the ship4. This information is detected by the AIS information.

The detailed information includes information indicating a position (latitude and longitude) on the sea surface (water surface) at which the object of the detailed information is arranged. For example, the detailed information about the route line42includes the position information about the positions of two changing points (inflection points of the polygonal line). Here, the positions of the two changing points match the positions of the two waypoints41. The detailed information about the stop-off area43includes information about the positions of the points which are the vertexes of the polygonal shape.

FIG.4is a conceptual diagram showing a three-dimensional virtual space40according to the present embodiment.FIG.4shows the three-dimensional virtual space40corresponding to the situation around the ship4shown inFIG.3. In addition, the waypoints41, the route line42, the stop-off area43, another ship44and the virtual buoy45shown inFIG.3correspond to virtual reality objects41v,42v,43v,44vand45vshown inFIG.4, respectively.

The three-dimensional scene generation unit32generates three-dimensional scene data (three-dimensional display data)48of the virtual reality in the three-dimensional virtual space40as shown inFIG.4. The three-dimensional scene generation unit32updates the three-dimensional scene data48such that the detailed information generated by the detailed information generation unit31is displayed in the generated three-dimensional scene based on the setting information set to the display setting unit29.

If the detailed information is figure information, the three-dimensional scene generation unit32generates a virtual reality object corresponding to the detailed information and adds it to the three-dimensional scene data48. At this time, the template of the virtual reality object stored in the storage unit27may be used.

The figure information generating the virtual reality object is, for example, a target not shown on the image captured by the camera3such as a virtual buoy, a dangerous water, a navigation prohibition area, the track of the ship4, a planned route, a waypoint, an arrival area or a stop-off area. In addition, the three-dimensional scene generation unit32may generate a virtual reality object indicating a visible target such as another ship as the figure information. In this case, the generated virtual reality object may be displayed superimposed on the actual target shown on the image captured by the camera3.

The screen information generation unit33generates necessary screen information other than the captured image of the camera3and the three-dimensional scene data48. For example, the screen information is information necessary for various settings or operations of the image generation device1, information for improving operability or visibility, information for displaying a distance or a direction, or the like on the screen of the AR image displayed on the display2. In addition, in order for the user to grasp situations of the other ships or the like, the screen information generation unit33may generate a top view screen (for example, a radar image) representing the surrounding situation centering on the ship4as the screen information. Accordingly, the user can grasp the surrounding situation of the ship4even outside the range of the AR image (beside or behind the ship4).

The data synthesis unit34draws the three-dimensional scene data48generated by the three-dimensional scene generation unit32in the captured image of the camera3input from the captured image input unit21, adds the screen information generated by the screen information generation unit33, and thereby generates the AR image. The data synthesis unit34outputs the generated AR image to the display2. Accordingly, the display2displays the AR image.

Next, a method of constructing the three-dimensional virtual space40will be explained.

The three-dimensional virtual space40in which the virtual reality objects41vto45vare arranged is composed of a Cartesian coordinate system using a reference position (for example, the above-described reference point4a) of the ship4as the origin and is set such that the xz plane which is a horizontal plane simulates the sea surface (water surface) as shown inFIG.4. In the example ofFIG.4, the coordinate axes are defined such that the +z direction always matches the bow direction, the +x direction is the rightward direction, and the +y direction is the upward direction. Each point (coordinates) in the three-dimensional virtual space40is set such that it corresponds to an actual position around the ship4.

The virtual reality objects41vto45vare arranged in contact with the xz plane such that the relative positions to the ship4are reflected with reference to the bow direction. In order to determine the arrangement positions of the virtual reality objects41vto45v, calculation is performed using the position of the GNSS antennas set by the antenna position setting unit23.

For example, the virtual reality objects41vto45vare generated as follows.

The virtual reality object44vindicating another ship44is expressed using a template having the shape of a ship modeled on a large ship. The orientation of the virtual reality object44vis arranged such that it indicates the orientation of another ship44acquired by the AIS information. The virtual reality object45vindicating the virtual buoy45is expressed using a template having a shape modeled on a buoy.

The virtual reality object41vof the waypoint41is expressed by a thin disk-like three-dimensional shape. The virtual reality object42vof the route line42is expressed by a three-dimensional shape that a long thin plate having constant thickness and width is bent into a polygonal line. The virtual reality object43vof the stop-off area43is expressed by such a three-dimensional shape as a plate having a constant thickness and having the contour of the stop-off area43. These virtual reality objects41vto45vmay be created each time without using a template.

InFIG.4, the virtual reality objects41vto45vare arranged with reference to the direction using the position of the ship4as the origin. Therefore, if the position of the ship4changes from the state ofFIG.3in the east-west direction or the north-south direction or if the bow direction of the ship4changes by turning or the like, the three-dimensional scene data48is updated to a new three-dimensional scene in which the virtual reality objects41vto45vare rearranged. In addition, if the content of the detailed information such as the movement of another ship44is changed from the state ofFIG.3, the three-dimensional scene data48is updated so as to reflect the latest detailed information.

The data synthesis unit34arranges a projection screen51, which defines a position and a range where the image captured by the camera3is projected, in the three-dimensional virtual space40. The position and orientation of a viewpoint camera55are set such that both the projection screen51and the virtual reality objects41vto45vare included in the viewfield.

The data synthesis unit34simulates the position and orientation of the camera3installed on the ship4in the three-dimensional virtual space40and arranges the projection screen51in such a way as to directly face the camera3. In the simulation of the position of the camera3, the position of the camera3with reference to the hull is determined based on the image capturing position set to the image capturing position setting unit22.

In the simulation of the position and orientation of the camera3, a change in the orientation by an operation such as panning or tilting of the camera3is taken into consideration. In addition, this simulation is performed such that a change in the position and orientation of the camera3by a change in the attitude and height of the ship4is reflected based on the position information acquired by the position acquisition unit24and the attitude information acquired by the attitude acquisition unit25. The data synthesis unit34changes the position and orientation of the projection screen51arranged in the three-dimensional virtual space40such that they correspond to the change in the position and orientation of the camera3.

The data synthesis unit34generates a two-dimensional image by performing rendering processing to the three-dimensional scene data48and the projection screen51. More specifically, the data synthesis unit34arranges the viewpoint camera55as a virtual camera in the three-dimensional virtual space40, and sets a view frustum56which defines a range of rendering processing. The view frustum56is set such that the viewpoint camera55is the apex and the viewline direction from the viewpoint camera55is the central axis.

Next, among the polygons constituting the virtual reality objects41vto45vand the projection screen51, the data synthesis unit34converts the vertex coordinates of the polygons located inside the view frustum56into the coordinates of a two-dimensional virtual screen by perspective projection. This virtual screen corresponds to a display area in which the AR image is displayed on the display2. The data synthesis unit34generates a two-dimensional image by performing processing such as generation and processing of pixels at a predetermined resolution based on the vertex coordinates arranged on the virtual screen.

The generated two-dimensional image includes figures acquired by the drawing of the three-dimensional scene data48(that is, figures as the rendering results of the virtual reality objects41vto45v). In the process of generating the two-dimensional image, the image captured by the camera3is arranged such that it is attached to a position corresponding to the projection screen51. Accordingly, the image synthesis by the data synthesis unit34is realized. The projection screen51is formed in a curved shape along the spherical shell centered on the camera3, and prevents distortion of the captured image by perspective projection.

The viewpoint camera55defines the viewpoint of the AR image. Normally, the position and orientation of the viewpoint camera55is determined by the setting of the viewpoint setting unit28. By making a special setting to the viewpoint setting unit28, the data synthesis unit34is set to a mode in which the position and orientation of the viewpoint camera55automatically changes so to as to always match the position and orientation of the camera3as the mode during AR image generation (viewpoint following mode). In the viewpoint following mode, the entire viewfield of the viewpoint camera55is always covered with the projection screen51(that is, the image captured by the camera3). Accordingly, an AR image with a sense of reality can be realized.

The data synthesis unit34may include a mode in which the position and orientation of the viewpoint camera55follow the viewpoint set to the viewpoint setting unit28by the operation of the input device14regardless of the position and orientation of the camera3(independent viewpoint mode). In the independent viewpoint mode, the user can check detailed information at a position outside the viewfield of the captured image of the camera3by freely moving the viewpoint.

The relationship between the image captured by the camera3and the AR image will be explained with reference toFIGS.5and6.FIG.5is an image diagram showing an example of the image captured by the camera3.FIG.6is an image diagram showing the AR image output from the data synthesis unit34.

FIG.5shows the image captured by the camera3of the ship4in the situation shown inFIG.3. The captured image shows another ship44rfloating on the sea surface. In addition, a bow part of the ship4is shown at the lower center of the captured image.

Since the virtual buoy45is virtual, it does not appear in the captured image. Since the waypoints41, the route line42and the stop-off area43are created by the plotter11, they do not appear in the captured image, either.

The AR image shown inFIG.6is the image in which the two-dimensional image acquired by rendering the three-dimensional scene data48ofFIG.4is synthesized with the captured image shown inFIG.5. In the AR image ofFIG.6,FIGS.41f,42f,43f,44fand45fexpressing detailed information are arranged overlapping the captured image shown inFIG.5. Here, inFIG.6, a captured image part of the camera3is shown by a broken line so as to be differentiated from the other part (the same applies toFIGS.8to10). TheFIGS.41fto45fcorrespond to the virtual reality objects41vto45v, respectively. TheFIG.44frepresenting another ship is arranged substantially overlapping the position of another ship44rin the captured image.

TheFIGS.41fto45fare generated as a result of drawing the three-dimensional shapes of the virtual reality objects41vto45vconstituting the three-dimensional scene data48shown inFIG.4from the viewpoint of the same position and orientation as the camera3. Therefore, even if theFIGS.41fto45fare superimposed on the realistic image captured by the camera3, a sense of discomfort in appearance hardly occurs.

As shown inFIG.6, theFIGS.41fto45fexpressing detailed information in a virtual reality are arranged in the AR image such that they are placed on the water surface of the captured image. In other words, theFIGS.41fto45fexpressing detailed information in a virtual reality are arranged along the water surface of the captured image.

This arrangement is realized as follows. The virtual reality objects41vto45vshown inFIG.4are arranged such that they are in contact with the xz plane located below with respect to the camera3by a distance calculated based on the height set by the image capturing position setting unit22. In addition, the position of the projection screen51is correctly arranged in consideration of the position and orientation of the camera3. Accordingly, the arrangement of theFIGS.41fto45fon the water surface is realized.

For targets indicating information about the navigation of the ship4(own navigation information), information about the target (name, position or the like) may be each displayed at the end of a line drawn above from the target, and for the other targets, information about the target may be each displayed at the end of a line drawn below from the target. By displaying in this way, the own navigation information and the other information can be easily differentiated from each other. For example, the own navigation information includes a waypoint, a route line, a stop-off area and the like.

Next, a change in the AR image by the swaying of the ship4will be explained.FIG.7is a conceptual diagram showing a case where the ship4sways in the pitch direction and the roll direction from the state ofFIG.4.FIG.8is an image diagram showing the AR image in the state ofFIG.7.

Since the camera3is mounted on the ship4, the position and orientation change as the attitude of the ship4is tilted by a wave or the like or the ship4rides a wave. If swaying (pitching, rolling and heaving) occurs in the ship4, the data synthesis unit34changes the position and orientation of the camera3in the three-dimensional virtual space40so as to simulate a change in the attitude of the ship4acquired by the attitude acquisition unit25and a change in the vertical direction of the position of the ship4acquired by the position acquisition unit24. Along with this change, the position of the projection screen51is changed.

In the example ofFIG.7, the ship4tilts forward and leftward, and the position and orientation of the camera3change so as to reflect this tilt. In conjunction with this, the projection screen51moves so as to directly face the camera3whose position and orientation are changed.

In this example, by the viewpoint following mode, the position and orientation of the viewpoint camera55are changed so as to follow the camera3whose position and orientation are changed. Even if the position and orientation of the camera3are changed by the swaying of the ship4, the position and orientation of the projection screen51are changed in conjunction with this, and the position and orientation of the viewpoint camera55which renders the three-dimensional scene are changed. Accordingly, the AR image generation unit20continuously generates an AR image without a sense of discomfort as shown inFIG.8.

In the viewpoint following mode, every time the pitch angle or roll angle is changed by greater than or equal to a predetermined value by the swaying of the ship4, the drawing of the three-dimensional scene data48by the data synthesis unit34is updated, and theFIGS.41fto45fbased on the latest viewpoint are generated. By updating the drawing, the display of theFIGS.41fto45fchanges so as to remain placed on the sea surface with respect to the image captured by the camera3in which the inclination of the sea surface changes by the swaying of the ship4.

Accordingly, the figures expressed in a virtual reality appear to be floating on the sea surface, and the AR image becomes a natural and highly realistic image. In addition, since the user comprehensively see theFIGS.41fto45frepresenting the virtual reality objects41vto45vby looking at the sea surface of the AR image displayed on the display2, the user can acquire necessary information without omission.

With reference toFIGS.9and10, a configuration for displaying scale information91indicating a direction in the AR image generated by the AR image generation unit20will be explained. Note that the scale information91may be selectively displayed or the scale information91may not be displayed.

In order to display the scale information91, the detailed information generation unit31collects necessary information from ship devices and the like. The detailed information generation unit31outputs the collected information to the screen information generation unit33.

The screen information generation unit33generates an image or the like to be displayed as the scale information91in the AR image based on the information received from the detailed information generation unit31. The screen information generation unit33outputs the generated scale information91to the data synthesis unit34.

While synthesizing the three-dimensional scene data48generated by the three-dimensional scene generation unit32with the image captured by the camera3, the data synthesis unit34also synthesizes the image to be the scale information91generated by the screen information generation unit33.

The scale information91is displayed, for example, at a predetermined position such as an upper part, a lower part or the like of the AR image as shown inFIG.9. The position of the scale information91may be automatically moved or changed so as not to overlap the detailed information such as theFIGS.41fto45fas shown inFIG.10. In addition, the scale information91may be displayed tilted so as to always remain parallel to the horizon according to the tilt of the hull of the ship4as shown inFIG.10. By displaying in this way, the scale information91can always indicate an accurate direction.

A method of displaying the detailed information about the target in the AR image will be explained with reference toFIGS.11and12.FIG.11is an image diagram showing an AR image in which pieces of detailed information D1, D2and D20are displayed.FIG.12is a flow diagram showing an outline of a procedure for displaying the pieces of detailed information D1, D2and D20in the AR image.

Here, the explanation is based on the assumption that there are three water moving objects S1, S2and S3in front of the ship4in the AR image. In addition, although the explanation mainly uses the water moving objects S1to S3as the target, detailed information may be displayed similarly for any target (including an intangible object) other than the water moving objects S1to S3.

The detailed information held by the image generation device1is divided into three kinds, that is, main information (corresponding to first detailed information), sub-information (corresponding to second detailed information) and non-display information. The main information is displayed in a first space SP1. The sub-information is displayed in a second space SP2. The non-display information is not displayed in the AR image. The pieces of detailed information (items) corresponding respectively to the main information, the sub-information and the non-display information are arbitrarily set by the input device14by the user. For example, a list of items is displayed in the AR image, and items corresponding respectively to the main information, the sub-information and the non-display information are selected in a drop-down manner.

When the target water moving objects S1to S3are detected by the ship device such as the AIS receiver9or the radar12(step ST101), markers M1, M2and M3for making a selection and corresponding to the water moving objects S1to S3in the AR image are displayed (step ST102). The markers M1to M3are connected to correspondence lines L1, L2and L3indicating a correspondence relationship and extending downward from the corresponding water moving objects S1to S3, respectively.

The shape of the markers M1to M3may vary depending on the type of the ship device which detects. For example, the shape of the marker is a rhomboid shape when the target is detected by the AIS receiver9, and the shape of the marker is a circular shape when the target is detected by the radar12. In addition, the target may be detected by the ship device such as the ECDIS10or the sonar13. Furthermore, when one target is detected by more than one ship device, markers M1to M3detected from one target and corresponding to the respective ship devices may be displayed, only arbitrarily selected markers may be displayed, or markers combined into one may be displayed.

When the user displays more detailed detailed information about the water moving objects S1to S3, the user selects (for example, clicks) the markers M1to M3connected to the water moving objects S1to S3of the detailed information to be displayed (step ST103). When the markers M1to M3are selected, the display may be changed such that the color is inverted or the like.

When the markers M1to M3are selected, the pieces of main information about the targets corresponding to the selected markers M1to M3are displayed at a predetermined place of the AR image (step ST104).FIG.11shows a state where the markers M1and M2of two water moving objects S1and S2are selected, and the pieces of main information D1and D2of the water moving objects S1and S2are displayed in the first space SP1in the lower part of the AR image as the predetermined place. For example, the pieces of main information D1and D2each include the distance to the own ship (the ship4), the speed, the DCPA and the TCPA. In order to display the correspondence relationships of the water moving objects S1and S2to the pieces of main information D1and D2in an easy-to-understand manner, the markers M1and M2and the pieces of main information D1and D2are connected by correspondence lines L11and L12that the correspondence lines L1and L2are extended, respectively.

If the sub-information is displayed in addition to the main information D1for the water moving object S1, the user selects a display field in which the main information D1is displayed (step ST105). When the display field of the main information D1is selected, as shown inFIG.11, sub-information D20corresponding to the selected main information D1is displayed in the second space SP2below the first space SP1(step ST106). The sub-information D20is connected by a correspondence line L20indicating that it corresponds to the main information D1in the first space SP1.

The user can select whether to display or hide the sub-information D20. When the sub-information is hidden, the second space SP2disappears, and the first space SP1displaying the pieces of main information D1and D2moves to the lower side of the AR image. When the sub-information D20is not required, the captured image part of the camera3in the AR image is expanded by eliminating the second space SP2. Note that the second space SP2may be always reserved or may appear when either of the display fields of the pieces of the main information D1and D2is selected.

Although a case where a piece of sub-information D20is displayed is explained here, two or more pieces of sub-information may be displayed or the sub-information may be eliminated.

A basic method of displaying the makers M1and M2in the AR image will be explained with reference toFIG.13.

A case where the second (another) water moving object S2passes by the side of the first water moving object S1in the state of being oriented sideways and substantially stopped from the right to the left of the screen in front of the ship4will be explained here.

One size larger quadrangles shown by broken lines surrounding the outsides of the markers M1and M2represent touch determination regions R1and R2, respectively. The touch determination regions R1and R2are regions for determining whether or not the markers M1and M2are touched, respectively.

For example, when the input device14is a touch panel, if a finger or the like touches the insides of the touch determination regions R1and R2, it is determined that the markers M1and M2are touched (that is, selected). When the input device14is a mouse, if a predetermined operation such as clicking is performed in a state where a cursor is in the touch determination regions R1and R2, it is determined that the markers M1and M2are touched. By making the touch determination regions R1and R2one size larger than the markers M1and M2, even if touching is performed slightly away from the markers M1and M2, it is still determined that the markers M1and M2are touched. Accordingly, the operability of the user is improved.

The second water moving object S2at an initial position is sufficiently away from the first water moving object S1. At this time, the correspondence lines L1and L2aconnected to two water moving objects S1and S2have an initially set and smallest length. It is assumed that the length of the correspondence line L1of the first water moving object S1thereafter remains unchanged. The initially set length may not be the smallest.

The second water moving object S2after t seconds from the initial position is close to the first water moving object S1. In this state, if the correspondence lines L1and L2bof two water moving objects S1and S2have the initially set length, the touch determination regions R1and R2of two water moving objects S1and S2partly overlap. Therefore, the correspondence line L2bof the second water moving object S2is displayed such that it gradually extends downward until the touch determination region R2does not overlap the touch determination region R1of the first water moving object S1. By gradually changing the length of the correspondence line L2bas described above, the marker M2is moved visually continuously. This makes it easier for the user to visually follow the moving marker M2. While the length of the correspondence line L2bof the second water moving object S2is changing, two touch determination regions R1and R2may temporarily overlap.

The second water moving object S2after another t seconds from the state of being close to the first water moving object S1is sufficiently away from the first water moving object S1again, and two touch determination regions R1and R2do not overlap regardless of the length of the correspondence line L2c. Therefore, the correspondence line L2cof the second water moving object S2shrinks until it reaches the initially set state of having the smallest length.

As described above, the lengths of the correspondence lines L1and L2ato L2cof the respective water moving objects S1and S2change increasing and decreasing so that two touch determination regions R1and R2do not overlap.

Although a case where the lengths of the correspondence lines L2ato L2cof the moving second water moving object S2change is described here, the length of the correspondence line L1of the stopped first water moving object S1may change.

An example of the change of the length of the correspondence line L1connecting the first water moving object S1and the marker M1will be explained with reference toFIG.14. InFIG.14, the horizontal axis represents the elapsed time and the vertical axis represents the length of the correspondence line L1changing in the vertical direction in the AR image.

At a time t0, a length a1of the correspondence line L1of the first water moving object S1is an initially set and smallest length.

At a time t1, when the touch determination region R1of the first water moving object S1comes into contact with the touch determination region of another water moving object (second water moving object), the screen information generation unit33calculates a length a2of the correspondence line L1with which two touch determination regions do not overlap. Two touch determination regions may be spaced apart as much as needed.

The screen information generation unit33does not necessarily start calculation after two touch determination regions come into contact with each other, and may start calculation before two touch determination regions come into contact with each other. In addition, whether or not two touch determination regions come into contact with each other is not necessarily determined when the positions of two touch determination regions are directly detected but may be determined based on the position of the water moving objects, the markers, the correspondence lines or the like.

The screen information generation unit33changes (draws) the correspondence line L1so that the length a1of the correspondence line L1gradually becomes the calculated length a2. Accordingly, the correspondence line L1of the first water moving object S1reaches the intended length a2.

For example, the screen information generation unit33changes the length of the correspondence line L1as shown by a curve Cr shown inFIG.14. The curve Cr indicates that the rate of change gradually decreases as the length of the correspondence line L1approaches the intended length a2. This makes it easier for the user to visually follow the marker M1. A method of drawing such that the length a1of the correspondence line L1gradually changes may employ an existing method used in graphics, animation or the like.

At a time t2, while the correspondence line L1of the first water moving object S1remains extended to the length a2, the touch determination region R1of the first water moving object S1comes into contact with the touch determination region of another water moving object (third water moving object). The screen information generation unit33calculates a length a3with which the touch determination region R1of the first water moving object S1does not overlap the touch determination region of the newly contacting third water moving object. As is the case of the time t1, the screen information generation unit33changes the correspondence line L1such that it gradually extends from the current length a2to the calculated length a3. Accordingly, the correspondence line L1of the first water moving object S1reaches the intended length a3.

At a time t3, when the second water moving object and the third water moving object disappear from close to the first water moving object S1, the screen information generation unit33changes the length of the correspondence line L1such that the length a3of the correspondence line L1gradually decreases (returns) to the initially set length a4. When the length of the correspondence line L1decreases, as is the case of increasing, by changing the correspondence line L1in the curve Cr, the length of the correspondence line L1gradually changes. Accordingly, the correspondence line L1of the first water moving object S1returns to the initially set length a4.

At a time t4, when the touch determination region R1of the first water moving object S1comes into contact with the touch determination region of another water moving object (fourth water moving object), as is the case of the times t1and t2, the screen information generation unit33calculates the intended length a5of the correspondence line L1.

Accordingly, the correspondence line L1of the first water moving object S1reaches the intended length a5.

At a time t5, when the fourth water moving object disappears from close to the first water moving object S1, as is the case of the time t3, the screen information generation unit33starts returning the correspondence line L1to the initially set length.

At a time t6, before the correspondence line L1reaches the initially set length, the touch determination region R1of the first water moving object S1comes into contact with the touch determination region of another water moving object (fifth water moving object). The screen information generation unit33calculates an intended length a6of the correspondence line L1, and changes the correspondence line L1such that it gradually extends from the state of decreasing to the initially set length to the intended length a6. Accordingly, the correspondence line L1of the first water moving object S1reaches the intended length a6.

At a time t7, when the fifth water moving object disappears from close to the first water moving object S1, as is the case of the times t3and t5, the screen information generation unit33starts returning the correspondence line L1to the initially set length. Accordingly, the correspondence line L1of the first water moving object S1reaches the initially set length a7.

By changing the length of the correspondence line L1such that the touch determination region R1of the first water moving object S1does not overlap the touch determination region of another water moving object as described above, the user can easily select the markers of the respective water moving objects.

Here, for the sake of convenience of explanation, attention is paid to the first water moving object S1and the correspondence line L1is changed. In reality, however, in the respective water moving objects, as is the case of the first water moving object S1, the lengths of the correspondence lines connected to the markers are changed such that the touch determination regions do not overlap.

If the touch determination regions of a plurality of water moving objects overlap, the correspondence line whose length is to be changed is determined in order of priority determined based on the respective water moving objects, the respective markers or the like. The order of priority may be preassigned to the respective water moving objects or the respective markers. The order of priority may be the display order of the markers (in the order from new to old or from old to new) or may be determined based on the positions (top/bottom or left/right of the screen) of the respective water moving objects or the respective markers. For example, as the marker is located at a lower position on the screen, the length of the correspondence line may be changed more preferentially. The length to be increased when the correspondence line of the marker at a lower position is extended is less than when the correspondence line of the marker at an upper position is extended. In addition, the correspondence line to be changed may be determined based on the current lengths of the respective correspondence lines or the like.

A method of displaying the pieces of main information D1and D2will be explained with reference toFIGS.11and15.FIG.15is a flow diagram showing a procedure of displaying the pieces of main information D1and D2on the AR image.

In the first space SP1ofFIG.11, display fields Dh for displaying main information which are indicated by dashed lines and in which nothing is actually displayed are reserved in addition to parts in which the pieces of main information D1and D2are displayed. The display fields Dh are determined as follows.

The user selects items to be displayed as the main information D1and D2by the input device14(step ST201).FIG.11shows an example of the AR image in which the items such as the distance to the own ship, the speed, the DCPA and the TCPA are displayed as the main information D1and D2.

The screen information generation unit33determines the size of the display fields Dh based on the items selected as the main information D1and D2(step ST202). The height of the display fields is determined based on the number of selected items. The width of the display fields is determined based on the largest number of characters of the numbers of characters reversed respectively for the selected items. The number of characters reserved for an item expressed only by a number and a unit such as a position, a distance, a speed or a time is basically small. The number of characters reserved for an item which can displays a proper noun (such as a ship name or a place name) or a sentence is basically large.

The screen information generation unit33determines the number of display fields Dh based on the determined size of the display fields Dh (step ST203). For example, if the width of the first space SP1is divided by the width of the display fields Dh, the number of display fields Dh which can be arranged side by side can be found. The correspondence relationship between the size and number of the display fields Dh may be stored in advice as table data.FIG.11shows an example where the display fields Dh are displayed in one row at one level. However, when one row is assumed to be one level, the display fields Dh may be displayed at two or more levels stacked one above the other. In this case, as is the case of the number of display fields Dh arranged side by side, the number of levels at which the display fields Dh are stacked one above the other may be determined based on the height of the first space SP1and the height of the display fields Dh.

The screen information generation unit33determines the arrangement of the display fields Dh in the first space SP1based on the determined number of display fields Dh (step ST204).FIG.11shows the AR image when the number of display fields Dh is six. When the number of display fields Dh is four, the AR image is displayed as shown inFIG.16. In the AR image shown inFIG.16, since the ship name is selected as the item of the main information D1and D2, the width of one display field Dh is large. Therefore, the number of display fields Dh arranged side by side is reduced to four.

Steps ST202to ST204are repeatedly executed every time the items of the main information D1and D2are changed in step ST201.

When the user selects the markers M1and M2of the water moving objects S1and S2, the screen information generation unit33determines the display fields Dh for displaying the pieces of main information D1and D2of the water moving objects S1and S2(step ST205).

More specifically, the display fields Dh located closest to where lines drawn directly below in the vertical direction from the middles in the width direction of the water moving objects S1and S2come into contact with the first space SP1are set as the display fields Dh which displays the pieces of main information D1and D2of the water moving objects S1and S2, respectively. That is, on the screen of the AR image, when the horizontal direction is assumed to be the x-axis, the display fields Dh at the x-coordinates closest to the x-coordinates of the water moving objects S1and S2are set as the display fields Dh of the water moving objects S1and S2, respectively. In addition, when the main information about another water moving object is already displayed in the display field Dh selected as described above, the next most suitable display field Dh (for example, the display field Dh of the next closest x-coordinate) is selected.

Furthermore, when pieces of main information are displayed in all display fields Dh, one of the already displayed pieces of main information is hidden, and a display field Dh in which a new piece of main information can be displayed is created. For example, the main information to be hidden may be main information displayed earliest (that is, oldest) or main information about a target having the lowest risk, or may be determined in other ways. In addition, in order to prevent specific main information from being automatically hidden, the user may perform an operation such as pinning to fix the display.

For example, the degree of risk is determined based on any combination of DCPA, TCPA, BCT, or BCR. If each element of a predetermined combination of the elements is higher than a preset threshold value, it is determined that the degree of risk is high. The combination of elements is the combination of DCPA and TCPA, the combination of BCT and the BCR, or the like.

The screen information generation unit33displays the pieces of main information D1and D2of the water moving objects S1and S2in the determined display fields Dh (step ST206).

When there is the water moving objects S1and S2having a high risk, the water moving objects S1and S2, the markers M1and M2, the display fields Dh of the pieces of main information D1and D2or the like may be visually distinguished using a prominent color or a different display mode such as blinking so that the user can easily find the high-risk targets.

FIG.17shows the AR image after a predetermined time from the state of the AR image shown inFIG.11.

The first water moving object S1shown inFIG.17is moving toward the lower left of the screen of the AR image from the position of the first water moving object S1shown inFIG.11. The second water moving object S2shown inFIG.17is moving toward the right of the screen of the AR image from the position of the second water moving object S2shown inFIG.11.

The main information D1of the first water moving object S1has moved from the second display field Dh from the left of the screen to the leftmost display field Dh since the first water moving object S1has moved to the lower left of the screen. The main information D2of the second water moving object S2is in the process of moving from the fourth display field Dh from the left of the screen to the fifth display field Dh since the second water moving object S2has moved to the right of the screen.

When the display field Dh of the main information D1and D2switches, the display of the main information D1and D2gradually moves between two display fields Dh. In this case, as is the case of the main information D2ofFIG.17, while the display field Dh is switching, the main information D2is displayed not only at the position of the display field Dh arranged in advance but also between the two display fields Dh.

By gradually moving the displays of the main information D1and D2as described above, the pieces of main information D1and D2are moved visually continuously. Accordingly, the user can confirm the positions of the display fields Dh after movement while visually following the pieces of main information D1and D2. Therefore, it is possible to prevent the user from losing sight of the display positions of the main information D1, D2or misidentify as the pieces of main information D1, D2of other water moving objects S1and S2during the switching of the display fields Dh. The method of drawing such that the displays of the main information D1, D2gradually move may be similar to the method of drawing such that the length of the correspondence line L1gradually changes.

Note that the present embodiment may be modified as follows.

The data synthesis unit34may not render the three-dimensional scene data48and the projection screen51simultaneously. That is, the data synthesis unit34may create a two-dimensional image which is a rendering result of the three-dimensional scene data48(images such as theFIGS.41fto45f) and a two-dimensional image which is a rendering result of the projection screen51(a captured image of the camera3attached to the projection screen51) separately. After that, the AR image is generated by synthesizing these two-dimensional images created separately. In this case, the rendering processing of the three-dimensional scene data48may be performed as needed according to the movement of the ship4or the like, and the rendering processing of the projection screen51may be performed at short time intervals according to the frame rate of the image by the camera3.

The camera3may not include the function of performing the rotation operation such as panning or tilting, and the image capturing direction may be fixed. In this case, the image capturing direction may be fixed in any of the forward, rearward or other directions. In addition, the camera3may be configured to simultaneously capture an image in all directions 360 degrees around the ship4. Furthermore, when the user performs an operation of changing the orientation of the viewpoint camera55, the rotation operation of the camera3may be automatically performed so as to follow this.

The generation of the three-dimensional scene data48by the three-dimensional scene generation unit32is explained based on the configuration that the virtual reality objects41vto45vare arranged with reference to the bow using the position of the ship4as the origin as shown inFIG.4. However, it is not limited to this. The virtual reality objects41vto45vmay be arranged not with reference to the bow but with reference to the true north where the +z direction is the true north. In this case, when the bow direction is changed by the turning of the ship4or the like, instead of rearranging the virtual reality objects41vto45v, the orientation of the ship4in the three-dimensional virtual space40is changed to the yaw direction. The change in the position and orientation of the camera3at this time is simulated in the three-dimensional virtual space40, and the position and orientation of the viewpoint camera are changed in conjunction to this, and rendering is performed. Accordingly, a rendering result can be acquired as is the case of arranging with reference to the bow.

The coordinate system of the three-dimensional virtual space40may use an arbitrarily selected fixed point on the earth as the origin instead of using the position of the ship as the origin, and may fix the relationship of the coordinate axes to the directions such that the +z direction is the true north and the +x direction is the true east. In this case, the coordinate system of the three-dimensional virtual space40is fixed on the earth, and the position and orientation where the ship4is arranged change based on the position information and the attitude information. Along with the change, the change in the position and orientation of the camera3is simulated in the three-dimensional virtual space40.

The image generation device1may perform processing of reducing the swaying of the AR image caused by the swaying of the ship4. As the processing, for example, the three-dimensional scene generation unit32may suppress a fluctuation in the position and orientation of the viewpoint camera55even when the ship4sways.

According to the present embodiment, the following effects can be acquired.

By changing the length of the correspondence line connecting the target and the marker so that the touch determination regions of two or more markers do not overlap, the operability for the user to select the marker can be improved.

By displaying the marker for displaying the detailed information in the vertical direction (upward direction or downward direction) of the water moving object (target), the marker in the AR image can be easily seen. For example, if the marker is displayed in the horizontal direction (rightward direction or leftward direction) of the water moving object, in the AR image, the water moving objects tend to be densely packed in the horizontal direction. Therefore, the markers overlap each other or the marker and the display of another water moving object overlap, and the markers tend to be difficult to see.

By changing the display mode of the marker (for example, the shape) according to the ship device (such as the AIS receiver9or the radar12) which recognizes the target, the user can know the information source of the main information simply by looking at the marker.

By making the area of the touch determination region for determining whether or not the marker is touched larger than that of the marker including the marker, the operability of the user when selecting the marker can be improved.

By reserving a place to display the main information about each target at a predetermined place (for example, the bottom of the screen) of the AR image, the main information can be arranged in an easy-to-see manner in the AR image. In addition, by moving the display place of the corresponding main information according to the movement of the position of the target in the AR image, the correspondence relationship between the target and the main information can be easily understood.

When it is determined that a certain target has a high risk, the display of the main information about the target is distinguished so that the high-risk target or the main information of it can be visually notified to the user.

The display method of the marker and the display method of the main information and the like may be employed separately and independently, and even when either of them is employed, effects by the employed display method can be still received. In addition, when both of them are employed, further effects can be acquired by the combination.

The present invention is not limited to the above-described embodiment, and constituent elements may be deleted, added, modified and the like. In addition, the constituent elements in more than one embodiment may be combined, replaced or the like to constitute a new embodiment. Even if such an embodiment is directly different from the above-described embodiment, the embodiment having the same spirit as the present invention is considered as being explained as the embodiment of the present invention, and explanation is omitted.