Patent Description:
The present disclosure relates to a system and method for multi-image-based vessel proximity situation recognition support.

An unmanned ship is a ship that automatically navigates a set route without a crew member, and refers to a vessel whose navigation and engine parts (e.g. engine, rudder) can be controlled from a remote control center, if necessary. To this end, a remote control center is needed on shore to control the unmanned ship remotely, and a qualified person must directly conduct command and control at the remote control center in order to resolve technical and legal issues.

Meanwhile, due to the recent development of GPS and a variety of sensors, it is not difficult for a vehicle to autonomously drive on the shortest route, but it is not so in the case of ship navigation. This is because, unlike land-based vehicles, such as cars and motorcycles, ships have very large inertial forces due to the nature of being operated on water, and thus it is very difficult for ships to instantly adjust their speed or direction. In addition, the reality is that it is very difficult to navigate on a scheduled route at sea because the sea does not have a fixed road like on land and is greatly affected by multiple variables such as weather. Moreover, crew members should always look ahead to prevent unexpected collisions with other ships and reefs.

Particularly, in case of heavy rain or snow, high concentration of fog, smog, yellow dust or fine dust during ship operation, visibility is rapidly reduced, making it difficult for the crew members to detect objects with the naked eye even if they look ahead. Consequently, the ship and its crew members may be put in very dangerous situations where they may collide with other ships and reefs unexpectedly, which is problematic. In other words, when it comes to ship navigation, collision avoidance is essential, and in order to adjust the speed or direction of a ship, it is necessary to predict the ship's navigation path in advance and operate a steering wheel or shift lever in advance.

(Patent Document <NUM>) <CIT>). <CIT> is another piece of relevant prior art, which discloses determining vessel collision avoidance maneuvers on the basis of historical navigational data which corresponds to the real-time data determined when the collision risk is above the predetermined threshold. The collision avoidance maneuvers are provided to an operator of the vessel. <CIT> and <CIT> disclose displaying the surroundings objects of a vessel with additional situational data.

Accordingly, the present disclosure has been made keeping in mind the above problems occurring in the related art, and the present disclosure is intended to provide a system and method for multi-image-based vessel proximity situation recognition support that detects obstacles around an unmanned surface vehicle (USV) using multiple cameras and various navigation sensors mounted on the USV, and provides remote situational awareness information regarding the risk of collision with the detected obstacles, according to respectively the subject-matter of claims <NUM> and <NUM>.

In order to achieve the above objective, according to an embodiment of the present disclosure, there is provided a system for multi-image-based vessel proximity situation recognition support.

The system for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure includes: an unmanned surface vehicle (USV) configured to detect and track surrounding objects by monitoring surroundings using surrounding images and navigation sensors; and a remote navigation control device configured to support proximity situation recognition of the unmanned surface vehicle according to detection of the surrounding objects, wherein the unmanned surface vehicle may include: an image acquisition unit configured to acquire multiple images showing the surroundings of the unmanned surface vehicle to detect objects around the unmanned surface vehicle through image analysis; a navigation sensor unit configured to acquire current navigation information of the unmanned surface vehicle and information on obstacles around the unmanned surface vehicle in real time; and a detection unit configured to monitor the surroundings of the unmanned surface vehicle by using the multiple images, current navigation information, and information on obstacles and, when an object close to the unmanned surface vehicle within a preset distance is detected as a result of monitoring, track the object detected.

The image acquisition unit may include: a thermal imaging camera, a panoramic camera, and a <NUM>-degree camera installed to photograph the surroundings of the unmanned surface vehicle, and acquires the multiple images including thermal images, panoramic images; and <NUM>-degree images of the surroundings of the unmanned surface vehicle in a form of Around View (AV).

The navigation sensor unit may include: a global positioning system (GPS), a gyro sensor, an automatic identification system (AIS), and RADAR, wherein the GPS and gyro sensor may obtain the current navigation information including location, speed, direction, and posture of the unmanned surface vehicle, while the AIS and RADAR may obtain the information on obstacles around the unmanned surface vehicle.

The remote navigation control device may include: an estimation unit configured to estimate a collision risk between the unmanned surface vehicle and the object detected by using the multiple images, current navigation information, and information on obstacles; and a situation recognition unit configured to display the collision risk estimated, along with the object detected, on an electronic navigational chart for each detected object and to output the collision risk in order to support proximity situation recognition.

The estimation unit may determine whether the object detected is located on an expected path of the unmanned surface vehicle by using the current navigation information, and when it is determined that the object detected is located on the expected path, may calculate the collision risk between the unmanned surface vehicle and the object detected, by using fuzzy inference.

The situation recognition unit may display the collision risk for the object detected in an augmented reality (AR) or virtual reality (VR) screen based on the multiple images and may output the screen in order to support the proximity situation recognition.

According to another embodiment of the present disclosure, there is provided a method for multi-image-based vessel proximity situation recognition support performed in a system including an unmanned surface vehicle and a remote navigation control device.

The method for multi-image-based vessel proximity situation recognition support according to another embodiment of the present disclosure may include: acquiring, by the unmanned surface vehicle, multiple images of surroundings to detect surrounding objects through image analysis; obtaining, by the unmanned surface vehicle, current navigation information and information on surrounding obstacles in real time; monitoring, by the unmanned surface vehicle, the surroundings using the multiple images, current navigation information, and information on surrounding obstacles and, when an object close to a surrounding area within a preset distance is detected as a result of monitoring, tracking the object detected; and supporting, by the remote navigation control device, proximity situation recognition of the unmanned surface vehicle according to detection of the object.

As described above, a system and method for multi-image-based vessel proximity situation recognition support according to an embodiment of the present disclosure has an effect that collision accidents with ships and obstacles in proximity can be prevented in advance during remote operation of an unmanned surface vehicle (USV) by detecting obstacles around the unmanned surface vehicle using multiple cameras and various navigation sensors mounted on the unmanned surface vehicle, and providing remote situational awareness information regarding the risk of collision with the detected obstacles.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise", "include", or "have" should not be construed as necessarily including all of the various components or steps described in the specification, but should be construed that some of the components or steps may not be included, or may further include additional components or steps. In addition, terms such as ". part", and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination thereof.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<FIG> is a view schematically illustrating an environment in which a system and method for multi-image-based vessel proximity situation recognition support according to an embodiment of the present disclosure may be implemented, <FIG> and <FIG> are views schematically illustrating the configuration of the system and method for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure, and <FIG> and <FIG> are views illustrating the system and method for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure. Hereinafter, with reference to <FIG>, the system for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure will be described.

Referring to <FIG>, the system for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure may be implemented in an unmanned surface vehicle (USV) and an onshore control center <NUM>.

For example, communication between the unmanned surface vehicle <NUM> and the onshore control center <NUM> may be performed through a marine communication network such as LTE-Maritime or a communication satellite.

Referring to <FIG>, the system for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure may be configured to include a remote navigation control device <NUM> installed in the unmanned surface vehicle <NUM> and the onshore control center <NUM>.

The unmanned surface vehicle <NUM> detects and tracks surrounding objects by monitoring the surroundings using surrounding images and navigation sensors.

The remote navigation control device <NUM> supports the recognition of surrounding situations by the unmanned surface vehicle in line with the detection of surrounding objects.

As shown in <FIG>, the unmanned surface vehicle <NUM> may include a communication unit <NUM>, a ship operation control unit <NUM>, an image acquisition unit <NUM>, a navigation sensor unit <NUM> and a detection unit <NUM>.

The communication unit <NUM> communicates with other ships or land-based communicable devices. For example, the communication unit <NUM> may include various communication media such as CDMA, satellite communication, LTE, and RF communication.

In particular, the communication unit <NUM> may communicate with the remote navigation control device <NUM> installed in the onshore control center <NUM> that remotely supports navigation of ships.

The ship operation control unit <NUM> controls the operation of the unmanned surface vehicle <NUM> according to control commands and control information received from the remote navigation control device <NUM> in the onshore control center <NUM>.

That is, the remote navigation control device <NUM> may receive control commands and control information from a user and transmit the received control commands and control information to the unmanned surface vehicle <NUM>.

Meanwhile, the ship operation control unit <NUM> may have an autonomous navigation function.

For example, the ship operation control unit <NUM> may receive destination information, generate an optimal navigation route to the destination, and control the unmanned surface vehicle <NUM> to navigate according to the generated optimal navigation route. The ship operation control unit <NUM> may be implemented as software, hardware, or a combination thereof, and may include an electronic navigational chart database, a route algorithm for calculating an optimal navigation route, and the like.

In addition, the ship operation control unit <NUM> may control the speed and direction of the unmanned surface vehicle <NUM> by controlling the engine and steering gear of the unmanned surface vehicle <NUM>. At this time, the ship operation control unit <NUM> may control the speed and direction of the unmanned surface vehicle <NUM> so that the unmanned surface vehicle <NUM> avoids surrounding ships or obstacles by utilizing the image acquisition unit <NUM> and the navigation sensor unit <NUM> to be described later to monitor the surrounding situations.

The image acquisition unit <NUM> acquires multiple images showing the surroundings of the unmanned surface vehicle <NUM> in order to detect objects around the unmanned surface vehicle <NUM> through image analysis.

That is, the image acquisition unit <NUM> may include a thermal imaging camera, a panoramic camera, and a <NUM>-degree camera installed to photograph the surroundings of the unmanned surface vehicle <NUM>. As such, the image acquisition unit <NUM> may acquire thermal images, panoramic images, and <NUM>-degree images showing the surroundings of the unmanned surface vehicle <NUM> in the form of Around View (AV).

The navigation sensor unit <NUM> acquires current navigation information of the unmanned surface vehicle <NUM> and information about obstacles around the unmanned surface vehicle <NUM> in real time.

That is, the navigation sensor unit <NUM> may include a global positioning system (GPS), a gyro sensor, an automatic identification system (AIS), RADAR, LiDAR, and the like.

The GPS and gyro sensor may obtain current navigation information including the location, speed, direction, and posture of the unmanned surface vehicle <NUM>.

The AIS, RADAR, and LiDAR may obtain information about obstacles around the unmanned surface vehicle <NUM>. The obstacle information may include information on an unidentified object (obstacle) located around the unmanned surface vehicle <NUM> or a ship operating around the unmanned surface vehicle <NUM>.

That is, the AIS receives and collects AIS data, which is the track data of ships around the unmanned surface vehicle <NUM>. At this time, the AIS data consists of static date and dynamic data. The static date includes information on ship name, specifications, and destination while the dynamic data includes navigation information such as the current location, course, and speed of a ship.

The RADAR and LiDAR detect objects existing around the unmanned surface vehicle <NUM>, and obtain location information, movement speed, movement direction, and shape information of the detected objects. At this time, the objects may include a ship sailing around the unmanned surface vehicle <NUM>, an iceberg, a reef, a floating object, and the like.

By using the multiple images showing the surroundings of the unmanned surface vehicle <NUM>, as well as obstacle information and current navigation information acquired by the image acquisition unit <NUM> and the navigation sensor unit <NUM>, the detection unit <NUM> monitors the surroundings of the unmanned surface vehicle <NUM> and, when an object close to the unmanned surface vehicle <NUM> within a preset distance is detected as a result of monitoring, tracks the detected object.

At this time, the detection unit <NUM> may transmit the multiple-image information, obstacle information, and current navigation information acquired by the image acquisition unit <NUM> and the navigation sensor unit <NUM> to the remote navigation control device <NUM>, and in case an object is detected, may also transmit information on the detected object to the remote navigation control device <NUM>.

Referring to FGI. <NUM>, the remote navigation control device <NUM> may be configured to include a communication unit <NUM>, an interface unit <NUM>, an estimation unit <NUM> and a situation recognition unit <NUM>.

The communication unit <NUM> communicates with various ships in operation. For example, the communication unit <NUM> may include various communication media such as CDMA, satellite communication, LTE, and RF communication.

In particular, the communication unit <NUM> may communicate with the unmanned surface vehicle <NUM> that a remote operator wants to control through the remote navigation control device <NUM>.

The interface unit <NUM> is a means for the remote operator in the onshore control center <NUM> to control the unmanned surface vehicle <NUM> located remotely.

For example, the interface unit <NUM> may include a display, a remote controller, and the like. The display may output the multiple images showing the surroundings of the unmanned surface vehicle <NUM>, as well as obstacle information and current navigation information acquired by the image acquisition unit <NUM> and the navigation sensor unit <NUM> and received from the unmanned surface vehicle <NUM>. In addition, the remote controller is a device for the remote operator to control the unmanned surface vehicle <NUM>, and may receive various control commands for the unmanned surface vehicle <NUM> and transmit the received control commands to the unmanned surface vehicle <NUM>.

The estimation unit <NUM> estimates a collision risk between the unmanned surface vehicle <NUM> and an object around the unmanned surface vehicle <NUM> detected by the detection unit <NUM> by using the multiple-image information, obstacle information, and current navigation information received from the unmanned surface vehicle <NUM>.

For example, the estimation unit <NUM> may calculate the collision risk by using fuzzy inference.

That is, the estimation unit <NUM> may determine whether the detected object is located on the expected path of the unmanned surface vehicle <NUM> by using the current navigation information including the location, speed, direction, and posture of the unmanned surface vehicle <NUM> and, when it is determined that the detected object is located on the expected path, may calculate the collision risk between the unmanned surface vehicle <NUM> and the detected object, by using fuzzy inference.

In addition, as a result of tracking by the detection unit <NUM>, when the detected object is identified as a moving object such as a ship, the estimation unit <NUM> may calculate the time when the detected object passes through the expected path of the unmanned surface vehicle <NUM>, and calculate the collision risk between the unmanned surface vehicle <NUM> and the detected object at the time of passing through the calculated predicted path by the detected object, by using fuzzy inference.

The situation recognition unit <NUM> displays the collision risk calculated by the estimation unit <NUM> through the display of the interface unit <NUM> along with the detected object on an electronic navigational chart for each detected object and outputs the displayed collision risk in order to support the remote operator's awareness of the surrounding situations of the unmanned surface vehicle <NUM>.

In addition, the situation recognition unit <NUM> may display the calculated collision risk for the detected object in an augmented reality (AR) or virtual reality (VR) screen based on the multiple images showing the surroundings of the unmanned surface vehicle <NUM> and output the screen in order to support the remote operator's awareness of the surrounding situations of the unmanned surface vehicle <NUM>.

For example, as shown in <FIG> and <FIG>, on the augmented reality or virtual reality-based surrounding situation recognition support screen that is output, detected objects are highlighted to increase their visibility, and among the detected objects, the one with a high risk of collision due to close proximity to the unmanned surface vehicle <NUM> may be displayed with a sign indicating the danger. <FIG> shows an augmented reality and virtual reality-based surrounding situation recognition support screen using <NUM>-degree images, and <FIG> shows an augmented reality-based proximity situation recognition support screen using a panoramic image.

<FIG> is a flowchart schematically illustrating an operating method of a system for multi-image-based vessel proximity situation recognition support according to the embodiment of the present disclosure.

In the step of S610, the unmanned surface vehicle <NUM> acquires multiple images of the surroundings by utilizing the image acquisition unit <NUM> and the navigation sensor unit <NUM>, obtains current navigation information of the unmanned surface vehicle <NUM> and information on obstacles around the unmanned surface vehicle <NUM> in real time, and transmits the acquired multi-image information, current navigation information, and obstacle information to the remote navigation control device <NUM>.

In the step of S620, the unmanned surface vehicle <NUM> monitors the surroundings using multiple images, current navigation information, and obstacle information, and as a result of monitoring, when an object close to the surrounding area within a preset distance is detected, the unmanned surface vehicle <NUM> tracks the detected object.

At this time, in case the object is detected, the unmanned surface vehicle <NUM> may also transmit information on the detected object to the remote navigation control device <NUM>.

In the step of S630, the remote navigation control device <NUM> estimates a collision risk between the unmanned surface vehicle <NUM> and the object around the unmanned surface vehicle <NUM> detected by the detection unit <NUM> by using the multiple-image information, obstacle information, and current navigation information received from the unmanned surface vehicle <NUM>.

In the step of S640, the remote navigation control device <NUM> displays the estimated collision risk along with the object on the electronic navigational chart for each detected object and outputs the displayed collision risk in order to support the remote operator's awareness of the surrounding situations of the unmanned surface vehicle <NUM>.

In addition, the remote navigation control device <NUM> may display and output the calculated collision risk for the detected object displayed on the augmented reality or virtual reality screen based on the multiple images showing the surroundings of the unmanned surface vehicle <NUM> in order to support the remote operator's awareness of the surrounding situations of the unmanned surface vehicle <NUM>.

Meanwhile, components of the above-described embodiment may be easily grasped from a process perspective. That is, each component may be identified as a separate process. In addition, the process of the above-described embodiment may be easily grasped from the point of view of components of the device.

Claim 1:
A system for multi-image-based vessel proximity situation recognition support, the system comprising:
an unmanned surface vehicle (USV) configured to detect and track surrounding objects by monitoring surroundings using surrounding images and navigation sensors; and
a remote navigation control device configured to support proximity situation recognition of the unmanned surface vehicle according to detection of the surrounding objects,
wherein the unmanned surface vehicle comprises:
an image acquisition unit configured to acquire multiple images showing the surroundings of the unmanned surface vehicle to detect objects around the unmanned surface vehicle through image analysis;
a navigation sensor unit configured to acquire current navigation information of the unmanned surface vehicle and information on obstacles around the unmanned surface vehicle in real time; and
a detection unit configured to monitor the surroundings of the unmanned surface vehicle by using the multiple images, current navigation information, and information on obstacles and, when an object close to the unmanned surface vehicle within a preset distance is detected as a result of monitoring, track the object detected,
wherein the remote navigation control device comprises:
an estimation unit configured to estimate a collision risk between the unmanned surface vehicle and the object detected by using the multiple images, current navigation information, and information on obstacles; and
a situation recognition unit configured to display the collision risk estimated, along with the object detected, on an electronic navigational chart for each detected object and to output the collision risk in order to support proximity situation recognition,
wherein the situation recognition unit displays the collision risk for the object detected in an augmented reality (AR) or virtual reality (VR) screen based on the multiple images and outputs the screen in order to support the proximity situation recognition,
wherein on the augmented reality or virtual reality-based surrounding situation recognition support screen that is output, detected objects are highlighted to increase their visibility.