Patent Description:
Tidal currents have been known to influence a course of the marine vessel on water. Vessel navigating personnel, for example, a captain, crew, or other navigating personnel on-board the marine vessel may be unaware of when and how each tidal current may influence the course of the marine vessel. For example, in waters where tidal currents may, or stated differently - are likely to, be encountered by the marine vessel, a position, direction and/or speed of the tidal current may be unknown by the vessel navigating personnel and consequently, the vessel navigating personnel may be forced to choose or elect a sub-optimal course of travel for the marine vessel in such waters. Depending on a severity of, or by simply encountering over a prolonged period of time, these tidal currents may contribute in negatively impacting a structural integrity and/or an operational performance of the marine vessel. "<NPL>) provides an augmented reality system with an interface for representing a wide spectrum of marine data. <CIT> discloses an image generating device that includes processing circuitry configured to acquire positional information indicative of a position of a water-surface movable body where an imaging device is to be installed, acquire posture information indicative of a posture of the water-surface movable body, acquire additional display information including information indicative of positions of a plurality of locations, generate three-dimensional display data for displaying a graphic indicative of the additional display information by superimposedly placing the graphic on a water surface portion of an image outputted from the imaging device based on the positional information, the posture information, and the positions of the plurality of locations included in the additional display information, and output the graphic rendering the three-dimensional display data. <CIT> discloses an apparatus that is configured to at least perform; detecting user selection of a part of an image wherein the image is displayed on a display; obtaining context information; and determining information to be provided to the user based on the user selection, the displayed image and the obtained context information. <CIT> discloses a Heads-Up-Display ("HUD") system for projecting safety/mission critical data onto a display pair of light weight projection glasses or monocular creating a virtual <NUM> degree. The HUD system includes a see-through display surface, a workstation, application software, and inputs containing the safety/mission critical information (Current User Position, Total Collision Avoidance System-TCAS, Global Positioning System-GPS, Magnetic Resonance Imaging-MRI Images, CAT scan images, Weather data, Military troop data, real-time space type markings etc.). The workstation software processes the incoming safety/mission critical data and converts it into a three-dimensional stereographic space for the user to view. Selecting any of the images may display available information about the selected item or may enhance the image. Predicted position vectors may be displayed as well as three-dimensional terrain.

Some systems have been developed in the past to assist in the navigation of marine vessels. Usually, a geographical chart showing a chart of a predetermined area and a tidal current at each location within an area is provided from an external base through communication equipment. The information indicating the tidal current generally includes the measuring position and the direction and speed of the tidal current at the measuring position. A ship navigating in the above area can obtain information on the tidal current by using a communication means such as satellite communication. The obtained information is displayed on the navigation monitor of a vessel to help determine the vessel route. However, many of these systems have mostly relied on on-board equipment for sensing various parameters associated with the surroundings of the marine vessel. Nevertheless, it is complex, challenging, and sometimes even impossible to detect or measure aspects such as tidal currents using such on-board equipment. In fact, such on-board equipment may not be simple, yet reliable, for use in the detection or measurement of aspects of the surroundings such as tidal currents.

Conventional Augmented Reality (AR) based navigation systems can display image information captured by an image sensor (camera) and information about surrounding ships and land acquired based on information captured by a sensor such as a radar, however, it is difficult to display the tidal current information by the conventional AR navigation system because the tidal current information is not obtained by the sensor of the own vessel except around the own vessel, but obtained from an external base through communication equipment. There does not exist an AR based navigation system that is able to display tidal current information provided from other than one's own vessel on an AR display screen.

For the aforementioned reasons, there is a need for providing a system and method that takes into account tidal currents and their potential impacts on the marine vessel for assisting in the navigation of the marine vessel.

The present invention is defined by the claimed subject-matter. Embodiments not covered by the claims do not form part of the invention.

The problem of not being able to display tidal current information provided from other than one's own vessel on an AR display screen is solved by visually capturing the state of the tidal current on the sea surface by a camera, and aligning the position and direction of the tidal current with the surface azimuth and position of the sea surface of the camera image.

The illustrated embodiments of the subject matter will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of devices, systems, and processes that are consistent with the subject matter as claimed herein:.

Example apparatus are described herein. Other example embodiments or features may further be utilized, and other changes may be made, without departing from the claimed subject matter presented herein. In the following detailed description, reference is made to the accompanying drawings, which form a part thereof.

<FIG> is a block diagram illustrating an entire configuration of an augmented reality (AR) based tidal current display device <NUM> in which an image sensor <NUM> is attached to a movable body <NUM> according to one embodiment of the present disclosure. <FIG> illustrates a chart <NUM> of a region <NUM> including the movable body <NUM>, and a sub-region <NUM> in the chart <NUM> that has a reference axis with respect to a heading direction of the movable body <NUM>. <FIG> illustrates the superimposing of tidal current information on an image captured by the image sensor <NUM>.

Next, mainly referring to <FIG>, the image sensor <NUM> (hereinafter also referred to as camera <NUM>) may be attached to the movable body <NUM> (hereinafter also referred to as ship <NUM>, and electrically connected to the display device <NUM>.

The tidal current display device <NUM> of this embodiment may be located on-board the ship <NUM> and provided in electrical connection to the camera <NUM> as the ship instrument for purposes as will be explained in detail later herein.

The camera <NUM> may be configured as, for example, a limited-viewing angle or a wide-angle video camera which images the water surface W in the vicinity, or around at least a portion of the perimeter of the ship <NUM>. This camera <NUM> may have a live output function, capable of generating video data (image data) as the imaged result on real time. , and outputting it to a display screen <NUM>. As illustrated in <FIG>, the camera <NUM> may be installed in the ship <NUM> so that an imaging direction generally faces onto the water surface W forward of the hull.

The camera <NUM> may be attached to the ship <NUM> through a rotating mechanism (not illustrated) and, therefore, the imaging direction can be changed in a given angle range on the basis of the hull of the ship <NUM> by inputting a signal for instructing pan/tilt from the display device <NUM>. Moreover, as the height and the posture of the ship <NUM> may change due to waves etc., the height h of the camera <NUM> may change with respect to the water surface W.

The AR based tidal current display device <NUM> may be communicatively coupled to the display screen <NUM> for generating an image expressing the situation around the movable body <NUM> using AR based on, among other things, a position and azimuthal orientation of the camera <NUM>.

The display screen <NUM> may be configured as, for example, a display screen that forms part of a navigation assisting device to which a ship operator who operates the ship <NUM> refers. However, the display screen <NUM> is not limited to the above configuration, and, for example, it may be a display screen for a portable computer which is carried by a ship operator's assistant who monitors the surrounding situation from the ship <NUM>, a display screen for a passenger to watch in the cabin of the ship <NUM>, or a display part for a head mounted display, such as a wearable glass, worn by a passenger. In an embodiment of the present disclosure, the camera <NUM> and display screen <NUM> are external to the display device <NUM>. In another embodiment of the present disclosure, the camera <NUM>, and the display screen <NUM> are an integral part of the display device <NUM>. Additionally, or optionally, the image sensor <NUM>, the display screen <NUM> and the AR based tidal current display device <NUM>, together forms an AR based navigation apparatus for facilitating a user to navigate the ship <NUM> across the sea.

The AR based tidal current display device <NUM> may also be connected to variety of peripheral devices including, but not limited to, a keyboard and a mouse which the user may operate for performing various functions consistent with the present disclosure. For example, the user can provide various kinds of instructions to the AR based tidal current display device <NUM> and the camera <NUM> about generation of an image by operating the keyboard and/or the mouse. The instructions may include the pan/tilt operation of the camera <NUM>, setting of displaying or not-displaying of various types of information, and a setup of a viewpoint from which the image is captured.

Next, a configuration of the AR based tidal current display device <NUM> is described in detail mainly referring to <FIG>. The AR based tidal current display device <NUM> includes an image sensor information terminal <NUM>, a chart information terminal <NUM>, a tidal current information terminal <NUM>, a processing circuitry <NUM>, and an image information terminal <NUM>. The image information terminal <NUM> is configured to receive an image captured by the image sensor <NUM>, and output the image data to the display screen <NUM>.

The image sensor information terminal <NUM> is configured to receive and store image sensor information including a position and an azimuthal orientation of the image sensor <NUM> with respect to a reference axis of a global geographical chart. As shown in <FIG>, the image sensor information may further include a height (h) of the image sensor <NUM> from the water surface (W), a depression angle (α) of the image sensor <NUM> and a viewing angle (β) of the image sensor <NUM>. The image sensor <NUM> is fixedly attached to the ship <NUM>, and therefore, the position of the image sensor <NUM> on the chart <NUM> is assumed to be similar to a position of the ship <NUM> in the chart <NUM>, and a heading direction of the ship <NUM> is used to determine the azimuthal orientation of the image sensor <NUM>, and vice versa.

Referring back to <FIG>, the chart information terminal <NUM> is configured to receive and store the geographical chart <NUM> of the region <NUM> including the ship <NUM> based on electronic nautical chart information stored beforehand.

The tidal current information terminal <NUM> is configured to receive and store tidal current information including positions of tidal currents on the water surface (W) from an external communication equipment, for example, a land station, a Global Navigation Satellite System (GNSS) receiver, an Electronic Chart Display and Information System (ECDIS), an Automated Identification System (AIS) receiver, a radar device etc. The tidal current information may further include directions and speeds of the tidal currents. The tidal current information terminal <NUM> is generally configured to store tidal current information of tidal currents of the region <NUM> including the ship <NUM>. The tidal current information terminal <NUM> is configured to store the tidal current information with respect to a reference axis of the chart <NUM>, which means that the tidal current information terminal <NUM> is configured to store chart/global co-ordinates of the tidal currents in the region <NUM>.

Referring to <FIG>, <FIG> and <FIG>, the processing circuitry <NUM> includes a tidal current selection unit <NUM> that selects one or more tidal currents <NUM> and <NUM> located in a field of view of the image sensor <NUM> based on the tidal current information, a position calculation unit <NUM> that calculates a corresponding display position of each selected tidal current <NUM> and <NUM> on the display screen <NUM> based on the tidal current information and the image sensor information, and a tidal current information superimposition unit <NUM> that generates symbols 204a and 205a for each selected tidal current <NUM> and <NUM> to indicate position of each tidal current <NUM> and <NUM> on an image <NUM> displayed on the display screen <NUM>. As shown, each of the symbols 204a and 205a is configured in the shape of a circular marker for indicating position of corresponding tidal currents <NUM> and <NUM> on the image <NUM>. Although, the symbols 204a and 205a are shown to be circular markers, it would be apparent to one of ordinary skill in the art, that the symbols 204a and 205a may include other configurations such as elliptical or square markers to indicate the positions of respective tidal currents on the image <NUM> displayed on the display screen <NUM>.

The tidal current superimposition unit <NUM> outputs the symbols 204a and 205a to the display screen <NUM> to superimpose the symbols 204a and 205a on the image <NUM> captured by the image sensor <NUM> and display the superimposed image to provide visual information about positions of the tidal currents <NUM> and <NUM> with respect to sea surface, and in a manner which is easy for a user to comprehend.

In the context of the present disclosure, the processing circuitry <NUM> includes a processor, computer, microcontroller, or other circuitry that controls the operations of various components such as the operation panel, and the memory. The processing circuitry <NUM> may execute software, firmware, and/or other instructions, for example, that are stored on a volatile or non-volatile memory, or otherwise provided to the processing circuitry <NUM>.

Referring to <FIG>, <FIG> and <FIG> together, the tidal current selection unit <NUM> receives the image sensor information, chart information and tidal current information from the image sensor information terminal <NUM>, chart information terminal <NUM>, and the tidal current information terminal <NUM> respectively. The tidal current selection unit <NUM> operably selects one or more tidal currents <NUM> and <NUM> located in a field of view <NUM>, or stated differently, a viewing angle (β) of the image sensor <NUM>. More specifically, the tidal current selection unit <NUM> receives the chart <NUM> of the region <NUM> surrounding the movable body <NUM>, determines the sub-region <NUM> based on the position and azimuthal orientation of the image sensor <NUM>, and selects the first and second tidal currents <NUM> and <NUM> in the sub-region <NUM> based on the tidal current information.

Although two selected tidal currents are shown herein, it would be apparent to one of ordinary skill in the art that the tidal current selection unit <NUM> may select more than or less than two tidal currents.

Referring to <FIG>,<FIG>, <FIG>, <FIG>, and <FIG>, the position calculation unit <NUM> is configured to calculate a position of each selected tidal current <NUM> and <NUM> to be displayed on the display screen <NUM> based on the image sensor information and the tidal current information. More specifically, the position calculation unit <NUM> is configured to calculate display coordinates of the first and second tidal currents <NUM> and <NUM> for displaying on the display screen <NUM>.

A configuration of the position calculation unit <NUM> is described in detail mainly referring to <FIG>. The position calculation unit <NUM> includes a local coordinate transformation unit <NUM> that receives and converts chart coordinates of each selected tidal current <NUM> and <NUM> to corresponding local coordinates based on the position and the azimuthal orientation of the image sensor <NUM>. In the context of the present disclosure, a chart coordinate is a position on the chart <NUM>, and a local coordinate is a position in the sub-region <NUM> with respect to the reference axis <NUM> in relation to a heading direction of the movable body <NUM>.

The position calculation unit <NUM> further includes a display coordinate transformation unit <NUM> that receives and converts local coordinates of each selected tidal current <NUM> and <NUM> to corresponding display coordinates based on the height (h) of the image sensor <NUM> from the water surface W, and a depression angle (α) and a viewing angle (β) of the image sensor <NUM>. In the context of the present disclosure, a display coordinate corresponds to a display position on the display screen <NUM>. Thus, the position calculation unit <NUM> receives the chart coordinates of the tidal currents <NUM> and <NUM> in the chart <NUM>, and generates corresponding display coordinates for displaying on the display screen <NUM>.

Additionally, or optionally, the local coordinate transformation unit <NUM> receives a direction and a velocity of each selected tidal current <NUM> and <NUM> from the tidal current selection unit <NUM> , and determines a direction and a velocity associated with each tidal current <NUM> and <NUM> in the sub-region <NUM> based on the position and the azimuthal orientation of the image sensor <NUM>. As illustrated with reference to <FIG>, the direction of the tidal currents <NUM> and <NUM> in the chart <NUM> is with respect to a global reference axis, and the direction of the tidal currents <NUM> and <NUM> in the sub-region <NUM> is with respect to the reference axis <NUM> in relation to the heading direction of the movable body <NUM>.

Based on the direction and velocity associated with each selected tidal current <NUM> and <NUM> in the sub-region <NUM>, the display coordinate transformation unit <NUM> calculates, for example, using mathematical and/or graphical transformations, the direction and velocity of each selected tidal current <NUM> and <NUM> on the display screen <NUM> based on the height h of the image sensor <NUM> from the water surface W, a depression angle α and a viewing angle β of the image sensor <NUM>. Thus, the position calculation unit <NUM> receives the direction of the tidal currents <NUM> and <NUM> with respect to the chart <NUM>, and generate direction information of the tidal currents <NUM> and <NUM> for displaying on the display screen <NUM>.

Referring to <FIG>, <FIG>, the tidal current information superimposition unit <NUM> generates symbols 505b and 507b to indicate position and direction of respective tidal currents on an image <NUM> displayed on the display screen <NUM>, based on information generated by the position calculation unit <NUM>. The tidal current information superimposition unit <NUM> further superimposes the symbols 505b and 507b on the image <NUM> to generate a superimposed image <NUM> to indicate positions and directions of respective tidal currents in the superimposed image <NUM>. The symbols 505b and 507b may be configured, for example, in the shape of an arrow having an arrowhead to indicate a direction of the corresponding tidal current.

Additionally, or optionally, a length of the arrow, a thickness of the arrow, and a color of the arrow in the superimposed image <NUM> may be varied based on a current speed of the corresponding tidal current. For instance, to indicate a weak (slow-speed) tidal current on the image, a short or thin sized arrow may be used. Further, such arrow may be rendered with the use of a green or yellow color for superimposing on the image <NUM>. Alternatively, to indicate a strong (high-speed) tidal current on the image, a long or thick sized arrow may be superimposed on the image <NUM>. Additionally, or optionally, the long and/or thick sized arrow may be further rendered graphically on the image <NUM> with the use of, for example, a red color. Additionally, the tidal current superimposition unit <NUM> may generate the symbols 505b and 507b based on, and to indicate, depths of respective tidal currents.

With respect to the foregoing explanation of the symbols, the characteristics of the symbols 505b and 507b are not limited to any specific shape, size, or color. These characteristics, that is, the shapes, sizes, and colors may be varied to suit one or more maritime code requirements and/or other application specific requirements.

Further, when displaying the symbols 505b and 507b superimposedly on the image <NUM>, the limited display area of the display screen <NUM> can be effectively utilized by simultaneously displaying additional information <NUM> including tabulation of directions and speeds. Optionally, the positions of the symbols 505b and 507b may be changed, as required, or even removed from the superimposed image <NUM> so that the graphics pertaining to the image <NUM> are not obscured, or hidden, by the symbols 505b and 507b as much as possible.

Additionally, the tidal current information superimposition unit <NUM> generates the symbols 505b and 507b parallel to the water surface W, indicating the direction and speed of corresponding tidal current based on the depression angle α of the image sensor <NUM>. Referring to <FIG>, the symbols 505b and 507b may be generated parallel to the water surface W based on transformation of two dimensional symbols 505c and 507c into corresponding three dimensional symbols 505b and 507b respectively.

Moreover, the symbols 505b and 507b may be displayed so as to incline according to the declination or depression angle β of the camera <NUM>. By displaying in such a way, graphics pertaining to the water surface W can be obtained from image sensor <NUM> and such graphics is always available for visual observation by the user and for the superimposition of the symbols 505b and 507b when needed even when, for example, the hull and the image sensor <NUM> incline with respect to the water surface W.

<FIG> illustrates a display screen <NUM> that displays a superimposed image <NUM> for indicating positions of one or more tidal currents, and a sub-display <NUM> for indicating directions of tidal currents whose position is shown in the superimposed image <NUM>. The sub-display <NUM> has a reference axis in relation to the heading direction of the movable body <NUM>.

The superimposed image <NUM> includes symbols 514a and 516a (similar to the symbols 204a and 205a) for indicating positions of one or more tidal currents on an image captured by the image sensor <NUM>. The sub-display <NUM> displays two-dimensional symbols 514c and 516c (similar to the symbols 505c and 507c) for indicating directions of tidal currents corresponding to symbols 514a and 516a of the superimposed image <NUM>. The symbols 514a, 516a, and 514c and 516c are generated by the tidal current superimposition unit <NUM>. The superimposed image <NUM> and the sub-display <NUM> are synchronous in time with respect to each other. A change in the superimposed image <NUM> is reflected in the sub-display <NUM> as well.

<FIG> illustrates the display screen <NUM> that displays the superimposed image <NUM> for indicating positions of one or more tidal currents, and an external display screen <NUM> for indicating directions of tidal currents whose position is shown in the superimposed image <NUM>. The external display screen <NUM> has a reference axis in relation to the heading direction of the movable body <NUM>.

The superimposed image <NUM> includes symbols 514a and 516a (similar to the symbols 204a and 205a) for indicating positions of one or more tidal currents on an image captured by the image sensor <NUM>. The external screen <NUM> displays two-dimensional symbols 514c and 516c (similar to the symbols 204c and 205c) for indicating directions of tidal currents corresponding to symbols 514a and 516a of the superimposed image <NUM>. The symbols 514a, 516a, 514c and 516c are generated by the tidal current superimposition unit <NUM>. The contents displayed on the display screen <NUM> and the external display screen <NUM> are synchronous in time with respect to each other.

Additionally, or optionally, the position calculation unit <NUM> may also predict future positions of one or more tidal currents displayed on an image captured by the image sensor. Accordingly, the tidal current information superimposition unit <NUM> may also generate the symbols to correspond with the predicted future positions and superimpose the generated symbols on the image for displaying on the display screen <NUM>.

The ship instruments (information source of the additional display information) connected to the display device <NUM> is not limited to what is described in <FIG>, and other ship instruments may be included.

Further, the present disclosure is applicable not only to the ship which travels on the sea, but may also be applicable to arbitrary water-surface movable bodies which can travel, for example, the sea, a lake, or a river.

<FIG> is a flowchart illustrating a method for superimposing tidal current information on an image displayed on the display screen <NUM>, in accordance with an embodiment of the present disclosure.

At step <NUM>, image data based on the image captured by an image sensor <NUM> is received and stored by the image information terminal <NUM>.

At step <NUM>, image sensor information including a position and an azimuthal orientation of the image sensor <NUM> attached to the movable body <NUM> on a water surface (W) is received and stored in the image sensor information terminal <NUM>.

At step <NUM>, chart information indicating a chart <NUM> of the region <NUM> including the movable body <NUM> is received and stored in the chart information terminal <NUM>.

At step <NUM>, tidal current information pertaining to one or more locations in the chart <NUM> is received and stored in the tidal current information terminal <NUM>.

At step <NUM>, the tidal current selection unit <NUM> determines a sub-region <NUM> in the region <NUM> of the chart <NUM> based on the image sensor information.

At step <NUM>, the tidal current selection unit <NUM> selects one or more tidal currents <NUM> and <NUM> located in the sub-region <NUM> based on the tidal current information.

At step <NUM>, the position calculation unit <NUM> calculates a display position of each selected tidal current on the display screen <NUM> based on the tidal current information and the image sensor information.

At step <NUM>, the tidal current superimposition unit <NUM> generates a symbol for each selected tidal current corresponding to each display position.

Claim 1:
An augmented reality, AR, based tidal current display device (<NUM>) for superimposing tidal current information on an image displayed on a display screen (<NUM>), comprising:
an image information terminal (<NUM>) configured to receive an image captured by an image sensor (<NUM>) and output image data for displaying on the display screen (<NUM>);
an image sensor information terminal (<NUM>) configured to receive and store image sensor information including a position and an azimuthal orientation of the image sensor (<NUM>) attached to a movable body (<NUM>) on a water surface (W);
a chart information terminal (<NUM>) configured to receive and store chart information indicating a chart (<NUM>) of the region (<NUM>) including the movable body (<NUM>);
a tidal current information terminal (<NUM>) configured to receive and store tidal current information including positions of tidal currents on the water surface (W) pertaining to one or more locations in the chart from an external communication equipment;
a tidal current selection unit (<NUM>) configured to:
determine a sub-region (<NUM>) in the region (<NUM>) of the chart (<NUM>) based on the image sensor information; and
select one or more tidal currents (<NUM>, <NUM>) located in the sub-region (<NUM>);
a position calculation unit (<NUM>) configured to, for each selected tidal current, calculate a corresponding display position on the display screen (<NUM>) based on the tidal current information and the image sensor information;
a tidal current superimposition unit (<NUM>) configured to generate a symbol for each selected tidal current to indicate each display position on the display screen (<NUM>), and output each symbol to the display screen (<NUM>), wherein the symbol is configured in the shape of a marker indicating a display position of corresponding tidal current on the display screen (<NUM>); and
wherein the tidal current superimposition unit (<NUM>) is further configured to superimpose each symbol at the respective display position on an image displayed on the display screen based on the image data.