TARGET MONITORING DEVICE, TARGET MONITORING METHOD, AND RECORDING MEDIUM

A target monitoring device includes: a data acquiring unit, acquiring image data including a ship observed by an imaging sensor; an image recognizing unit, detecting a region of the ship included in the image data; a distance acquiring unit, acquiring a distance to the ship from an observation position detected by a sensor that is different from the imaging sensor; a course acquiring unit, acquiring a course of the ship detected by the sensor that is different from the imaging sensor; and a ship body length estimating unit, estimating a ship body length of the ship based on a dimension of the region of the ship in a horizontal direction, the distance to the ship, and the course of the ship.

TECHNICAL FIELD

The disclosure relates to a target monitoring device, a target monitoring method, and a recording medium.

RELATED ART

Japanese Patent No. 5730565 discloses a technique that calculates a ship length by using a radar.

However, since radar has a low azimuth resolution, it is difficult to determine the size of a ship from radar echoes.

The disclosure provides a target monitoring device, a target monitoring method, and a program capable of increasing the accuracy in estimating a ship body length.

SUMMARY

A target monitoring device according to an aspect of the disclosure includes processing circuitry configured to: acquire image data including a ship observed by an imaging sensor; detect a region of the ship included in the image data; acquire a distance to the ship from an observation position detected by a sensor that is different from the imaging sensor; acquire a course of the ship detected by the sensor that is different from the imaging sensor; and estimate a ship body length of the ship based on a dimension of the region of the ship in a horizontal direction, the distance to the ship, and the course of the ship. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

In the aspect, it may also be that the processing circuitry is further configured to: estimate the ship body length of the ship further based on a particular ship body aspect ratio. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

In the aspect, it may also be that the imaging sensor is a camera, and the processing circuitry is further configured to: calculate an occupancy angle of the ship in a view angle of the camera in the horizontal direction based on a dimension of the image data in the horizontal direction, the dimension of the region of the ship in the horizontal direction, and the view angle, and estimate the ship body length of the ship based on the occupancy angle of the ship, the distance to the ship, and the course of the ship. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

In the aspect, it may also be that the processing circuitry is further configured to: calculate the occupancy angle of the ship in the view angle based on the dimension of the image data in the horizontal direction, the dimension of the region of the ship in the horizontal direction, the view angle of the camera in the horizontal direction, a focal point distance of the camera, and an optical center of the camera. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

In the aspect, it may also be that the sensor that is different is a radar, and the processing circuitry is further configured to: acquire the distance to the ship based on data detected by the radar. Accordingly, it is possible to estimate the ship body length by using the distance to the ship detected by the radar.

In the aspect, it may also be that the sensor that is different is a radar, and the processing circuitry is further configured to: acquire the course of the ship based on data detected by the radar. Accordingly, it is possible to estimate the ship body length by using the course of the ship detected by the radar.

In the above aspect, it may also be that the processing circuitry is further configured to: detect a boundary box surrounding the ship included in the image data as the region of the ship. Accordingly, the dimension of the region of the ship in the horizontal direction is easily acquired.

In addition, a target monitoring method according to another aspect of the disclosure includes: acquiring image data including a ship observed by an imaging sensor; detecting a region of the ship included in the image data; acquiring a distance to the ship from an observation position detected by a sensor that is different from the imaging sensor; acquiring a course of the ship detected by the sensor that is different from the imaging sensor; and estimating a ship body length of the ship based on a dimension of the region of the ship in a horizontal direction, the distance to the ship, and the course of the ship. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

In addition, a non-transient computer-readable recording medium records a program according to another aspect of the disclosure executed by a computer to: acquire image data comprising a ship observed by an imaging sensor; detect a region of the ship comprised in the image data; acquire a distance to the ship from an observation position detected by a sensor that is different from the imaging sensor; acquire a course of the ship detected by the sensor that is different from the imaging sensor; and estimate a ship body length of the ship based on a dimension of the region of the ship in a horizontal direction, the distance to the ship, and the course of the ship. Accordingly, it is possible to improve the accuracy in estimating the ship body length.

DESCRIPTION OF EMBODIMENTS

In the following, the embodiments of the disclosure are described with reference to the drawings.

FIG.1is a block diagram illustrating a configuration example of a target monitoring system100. The target monitoring system100is a system mounted in a ship. In the following description, the ship in which the target monitoring system100is mounted is referred to as “own ship”, and another ship is referred to as “other ship”.

The target monitoring system100includes a target monitoring device1, a display unit2, a radar3, an AIS4, a camera5, a GNSS receiver6, a gyro compass7, an ECDIS8, a wireless communicating unit9, and a ship steering control unit10. Such components are connected with a network N, such as LAN, and are able to communicate with each other through network communication.

The target monitoring device1a computer that includes a CPU, a RAM, a ROM, a non-volatile memory, and an input/output interface, etc. The CPU of the target monitoring device1executes an information process in accordance with a program loaded from the ROM or the non-volatile memory to the RAM.

The program may also be supplied via an information storage medium, such as an optical disc or a memory card, and may also be supplied via a communication network such as the Internet or a LAN.

The display unit2displays a display image generated by the target monitoring device1. The display unit2also displays a radar image, a camera image, or an electronic chart, etc.

The display unit2, for example, is a display device having a touch sensor, i.e., a so-called touch panel. The touch sensor detects an indicated position in an image indicated by the user's finger, etc. However, the disclosure is not limited thereto. The indicated position may also be input by using a trackball, etc.

The radar3emits radio waves around the own ship, receives the reflected waves thereof, and generates echo data based on the received signals. In addition, the radar3recognizes a target from the echo data, and generates target tracking (TT) data indicating the position and the velocity of the target.

The automatic identification system (AIS)4receives AIS data from other ships present around the own ship or from land control. A VHF data exchange system (VDES) may also be used, instead of being limited to AIS. The AIS data include recognition symbols, ship names, positions, courses, velocities, ship types, ship body lengths, and destinations, etc., of other ships.

The camera5is a digital camera that images the outside from the own ship to generate image data. The camera5is disposed at a bridge of the own ship and toward a bow orientation, for example. The camera5may be a camera having pan/tilt/zoom functions, i.e., a so-called PTZ camera.

In addition, the camera5may also include an image recognizing unit that estimates a position and a type of a target, such as an other ship, included in the imaged image by using an object detecting model. In addition to the camera5, the image recognizing unit may also be realized in other devices, such as the target monitoring device1.

The GNSS receiver6detects the position of the own ship based on radio waves received from the global navigation satellite system (GNSS). The gyro compass7detects a bow orientation of the own ship. A GPS compass may also be used, instead of being limited to the gyro compass.

The electronic chart display and information system (ECDIS)8acquires the position of the own ship from the GNSS receiver6and displays the position of the own ship on an electronic chart. In addition, the ECDIS8also displays a planned route of the own ship on the electronic chart. A GNSS plotter may also be used, instead of being limited to ECDIS.

The wireless communication unit9includes various wireless components for ultra short wave band, very short wave band, medium and short wave band, short wave band, etc., for realizing the communication with other ships or land control.

The ship steering control unit10is a control device for realizing automatic ship steering, and controls a steering device of the own ship. In addition, the ship steering control unit10may also control the engine of the own ship.

In the embodiment, the target monitoring device1is an independent device. However, the disclosure is not limited thereto, and may also be integrated with another device, such as the ECDIS8. That is, the functional units of the target monitoring device1may also be realized by other devices.

In the embodiment, the target monitoring device1is mounted in the own ship and used to monitor a target, such as an other ship, present around the own ship. However, the disclosure is not limited thereto. For example, the target monitoring device1may also be disposed in the land control and configured to monitor a ship present in a controlled sea area.

FIG.2is a diagram illustrating a configuration example of the target monitoring device1. The control unit20of the target monitoring device1includes data acquiring units11,12,13, an image processing unit14, a data integrating unit15, a display control unit16, and a ship steering determining unit17. These functional units are realized by executing information processing according to a program by using the control unit20.

The control unit20of the target monitoring device1further includes a radar management database (DB)21, an AIS management DB22, a camera management DB23, and an integral management DB24. The storage units thereof are provided in the memory of the control unit20.

The data acquiring unit11sequentially acquires, as target data, TT data generated by the radar3, and registers the target data in the radar management DB21.

The target data registered in the radar management DB21include the position, the ship velocity, and the course, etc., of the target, such as an other ship, detected by the radar3. The target data registered in the radar management DB21may further include the track of the target, the elapsed time since detection, the size of the echo image, and the signal strength of the reflected waves, etc.

The data acquiring unit12acquires, as target data, the AIS data received by the AIS4and registers the AIS data in the AIS management DB22.

The target data registered in the AIS management DB22includes the position, the ship velocity, and the course, etc., of an other ship, detected by the AIS4. The target data registered in the AIS management DB22may also include the type, the ship name, and the destination, etc., of the other ship.

The data acquiring unit13acquires an image including the target, such as an other ship, imaged by the camera5. The data acquiring unit13sequentially acquires time-series images from the camera5, and sequentially provides the time-series images to the image processing unit14. The time-series images are, for example, still images (frames) included in motion image data.

The image processing unit14performs a particular image process, such as image recognition, on the image acquired by the data acquiring unit13, generates target data of the target recognized from the image, and registers the target in the camera management database23. Details of the image processing unit14will be described afterwards.

The target data registered in the camera management DB23includes the position, the ship velocity, and the course, etc., of the target, such as an other ship calculated by the image processing unit14. The target data registered in the camera management DB may also include the size of the target, the type of the target, and the elapsed time from detection, etc.

The position of the target detected by the radar3and the position of the target recognized from the image imaged by the camera5are relative positions with respect to the own ship. Therefore, the positions can be converted into absolute positions by using the position of the own ship detected by the GNSS receiver6.

The target detected by the radar3and the target recognized from the image imaged by the camera5are mainly ships, but may also include others such as a buoy.

The data integrating unit15registers the target data registered in the radar management DB21, the AIS management DB22, and the camera management DB23in the integral management DB24for cross-management of these databases. The target data registered in the integral management DB24include the position, the ship velocity, and the course, etc., of the target, such as an other ship.

In the case where the position included in the target data registered in one of the radar management DB21, the AIS management DB22, and the camera management DB23and the position included in the target data registered in another one of the radar management DB21, the AIS management DB22, and the camera management DB23are the same or similar, the data integrating unit15registers these target data in the integral management DB24.

The display control unit16generates a display image including an object indicating the target based on the target data registered in the integral management DB24and outputs the display image to the display unit2. The display image is, for example, a radar image, an electronic chart, or an image formed by synthesizing a radar image and an electronic chart. The object indicating the target is disposed at a position in the image corresponding to the actual position of the target.

The ship steering determining unit17performs ship steering determination based on the target data registered in the integral management DB24, and, in a case of determining that it is necessary to avoid a target, causes the ship steering control unit10to perform an avoidance steering operation. Specifically, the ship steering control unit10calculates an avoidance route for avoiding the target by using an avoidance steering algorithm, and controls the steering device and an engine, etc., so that the own ship follows the avoidance route.

FIG.3is a diagram illustrating a configuration example of the image processing unit14. The image processing unit14includes an image recognizing unit31, a distance and course acquiring unit32, and a ship body length estimating unit33. The same figure illustrates a configuration relating to a function of estimating a ship body length of an other ship among the functions realized by the image estimating unit14.

In the embodiment, the image processing unit14acquires an image including the target, such as an other ship, imaged by the camera5from the data acquiring unit13. The camera5is an example of an imaging sensor. However, the disclosure is not limited thereto. The image data under observation may also be acquired by other imaging sensors, such as a light detection and ranging (LiDAR). It is assumed that the imaged data observed through LiDAR is also included in the image data.

The image recognizing unit31detects a region of a ship included in the image acquired by the data acquiring unit13. Specifically, the image recognizing unit31calculates the region of the ship included in the image, the type of the target, and the estimation reliability by using a learned model generated in advance through machine learning. The type of the target may be a ship type, such as a tanker or a fishing boat. However, the disclosure is not limited thereto. The image recognizing unit31may also recognize the region included in the image and the type of the target by using a rule base.

The learned model is an object detection model, such as a single shot multibox detector (SSD) or a you only look once (YOLO), and detects a boundary box surrounding the ship included in the image as the region of the ship. However, the disclosure is not limited thereto. The learned model may also be a region split model, such as semantic segmentation or instance segmentation.

FIG.4is a diagram illustrating a recognition example of an image P.FIG.5is a diagram in which a boundary box BB is enlarged. As shown in these figures, an other ship SH included in the image P is surrounded by the boundary box BB in a rectangular shape. A label CF in which the type of the target and the estimation reliability are recorded is added to the boundary box BB.

Lb inFIG.4indicates the dimension of the image P in the horizontal direction. Lw inFIG.5indicates the dimension of the region of the other ship SH in the horizontal direction, that is, the dimension of the boundary box BB in the horizontal direction. The horizontal direction of the image is a direction corresponding to the horizontal direction in the actual space, and is the left-right direction in the illustrated example. The dimension is represented by using the number of pixels, for example.

Since the dimension Lw of the region of the other ship SH in the horizontal direction as detected from the image changes in accordance with the distance or the orientation with the ship SH, it is difficult to obtain the ship body length of the other ship SH by using only the dimension Lw. Therefore, in the embodiment, by using the data of the other ship SH detected by using a target detecting unit other than the camera5, the accuracy in estimating the ship body length is increased.

The distance and course acquiring unit32acquires the distance from the own ship to the other ship and the course of the other ship from the TT data generated by the radar3. Specifically, the distance and course acquiring unit32reads the target data of the other ship serving as the target from the radar management DB21, and acquires the distance from the own ship to the other ship and the course of the other ship from the read target data. The radar3is an example of “a sensor that is different”.

The course of the other ship is, for example, a course over ground (COG) of the other ship. However, the disclosure is not limited thereto. A heading (HDG) of the other ship may also be used. Although HDG may be used to estimate the ship body length, in the embodiment, COG is used for its ease of being acquired.

As shown inFIG.6, according to an echo EC detected by the radar3, a distance d to a target MK and a course DR of the target MK can be specified. However, since the echo EC forms a shape that expands in an angular direction, that is, the azimuth resolution of the radar3is low, it is difficult to specify the size of the target MK from the magnitude of the echo EC. In particular, the difficulty increases as the distance d increases.

Therefore, in the embodiment, as shown inFIG.7, by combining an azimuth width (occupancy angle) θ of the other ship SH acquired by the camera5having a high azimuth resolution, and the distance d and the course DR of the other ship SH acquired by the radar3having a high distance resolution, the accuracy in estimating the ship body length of the other ship SH is increased.

The distance and course acquiring unit32may also acquire the distance from the own ship to the other ship and the course of the other ship from the AIS data received from the AIS4.

The ship body length estimating unit33estimates the ship body length of the other ship based on the dimension of the region of the other ship in the horizontal direction as detected by the image recognizing unit31as well as the distance and the course of the other ship acquired by the distance and course acquiring unit32. In addition, the ship body length estimating unit33estimates the ship body length of the other ship further based on a particular ship body aspect ratio. Before the ship body length is estimated, a pre-process, such as distortion calibration on the image, may also be applied.

Specifically, as shown inFIG.8, firstly, the ship body length estimating unit33calculates the occupancy angle θ of the other ship SH within a view angle Θ based on the dimension Lb of the image in the horizontal direction, the dimension Lw of the region of the other ship SH in the horizontal direction, and the view angle Θ of the camera5in the horizontal direction. The dimensions Lb, Lw are dimensions (e.g., numbers of pixels) in the image. The distance d represents a distance in the actual space detected by the radar3.

The view angle Θ is an angle representing a range included in the image when imaged by using the camera5, and is an angle determined by the lens of the camera5. The occupancy angle θ of the other ship SH is an angle representing the range occupied by the other ship SH, and is an angle between the left end and the right end of the target SH with the camera5as the center.

Since the ratio between the dimension Lb of the image in the horizontal direction and the dimension Lw of the region of the other ship SH in the horizontal direction and the ratio between the view angle Θ of the camera5in the horizontal direction and the occupancy angle θ of the other ship SH can be considered as the same, the occupancy angle θ of the other ship SH can be calculated accordingly.

More specifically, the ship body length estimating unit33may also calculate the occupancy angle θ of the target SH in the view angle Θ further based on the focal point distance and the optical center of the camera5, in addition to the dimension Lb of the image in the horizontal direction, the dimension Lw of the region of the target SH in the horizontal direction, and the view angle Q of the camera5in the horizontal direction. That is, considering camera-internal parameters of a perspective projection model, the occupancy angle θ of the target SH may also be calculated by using the focal point distance and the optical center of the camera5representing the camera-internal parameters.

In addition, as shown inFIG.9, the ship body length estimating unit33estimates the ship body length L of the other ship SH based on the occupancy angle θ of the other ship SH, the distance d from the own ship SP to the other ship SH, a relative course φ of the other ship SH with respect to the own ship, and a deviation angle φaspin accordance with the aspect ratio of the ship body.

A ship body length L of the other ship SH is represented by Equation 1 below.

Ldigis the length of the diagonal when the ship body of the other ship SH is assumed to be rectangular, and corresponds to the occupancy angle θ of the other ship SH (that is, corresponds to the portion appearing in the image). φaspis a deviation angle of Ldigwith respect to the course q of the other ship that is determined by the aspect ratio of the ship body, and is represented by tan−1(ship body width/ship body length).

Ldigis represented by Equation 2 in the following.

Regarding the derivation of Equation 2, the description is made by using the case where φ is in the range of 0° to 90° as an example, with reference toFIG.10.

In the same figure, a line segment from a position P of the own ship to the left end of the diagonal of the other ship SH is set as A, and a line segment to the right end is set as B. In addition, the length of the diagonal of the other ship SH is set as L′(=Ldig), and a combined angle of the course q of the other ship SH and the deviation angle φaspis set as φ′.

A triangle surrounded by the left end and the right end of the diagonal of the other ship SH and the position P of the own ship is represented by Equation 3 below.

When 2 cos θ is applied to two ends of Equation 3, Equation 4 as follows is yielded.

In addition, vectors A, B when the position P of the own ship is set as the origin are represented by Equation 5 as follows.

By taking the inner product of the vectors A and B, Equation 6 below is yielded.

By substituting Equation 6 6 into Equation 4, Equation 7 as follows is yielded.

By arranging Equation 7, Equation 8 for L′ as follows is yielded.

By solving Equation 8 for L′ using a solution formula, Equation 9 is yielded.

Here, since the distance is positive, Equation 10 as follows is yielded.

Here, L′=Ldig, and φ′=φ+φasp. Therefore, Equation 10 is the same as Equation 2.

The ship body length estimating unit33estimates the ship body length L of the other ship SH like the above. The ship body length L of the other ship SH estimated by the ship body length estimating unit33is included in the target data together with the position, the ship velocity, and the course, etc., of the other ship SH and registered in the camera management DB23.

In the embodiment, although the region of the other ship SH is detected from the image imaged by the camera5through image recognition, the disclosure is not limited thereto. The region of the other ship SH may also be detected through image recognition from image data including the other ship as observed by an other sensor, such as a light detection and ranging (LiDAR). The sensor may have an azimuth resolution higher than that of the radar3.

FIG.11is a diagram illustrating a procedural example of a target monitoring method realized in the target monitoring system100. The same figure mainly illustrates a process of estimating the ship body length of the other ship. The target monitoring device1executes the process shown in the same figure according to the program.

When acquiring the image imaged by the camera5, the target monitoring device1performs an image recognizing process, and calculates the region of the other ship included in the image, the type of the other ship, and the reliability that is estimated (S11, the process as the image recognizing unit31).

Then, the target monitoring device1acquires the dimension of the region of the other ship in the horizontal direction calculated through the image recognizing process (S12). The dimension of the region of the other ship in the horizontal direction is the dimension Lw of the boundary box BB in the horizontal direction (seeFIG.5).

Then, based on the target data detected by the radar3, the target monitoring device1obtains the distance from the own ship to the other ship and the course of the other ship (S13, the process as the distance and course acquiring unit32).

Then, based on the dimension of the region of the other ship in the horizontal direction, the distance from the own ship to the other ship, and the course of the other ship, the target monitoring device1estimates the ship body length of the other ship (S14(the process as the ship body estimating unit33).

Then, the target monitoring device1generates the target data including the ship body length of the other ship, together with the position, the ship velocity, and the course, etc., of the other ship in the camera management DB23(S15). Accordingly, a series of processes on the image have ended.

Although the embodiments of the disclosure have been described above, the disclosure is not limited thereto. It goes without saying that various modifications can be made by those skilled in the art.

Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C. The same holds true for the use of definite articles used to introduce embodiment recitations. In addition, even if a specific number of an introduced embodiment recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

It will be understood by those within the art that, in general, terms used herein, are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

As used herein, the terms “attached,” “connected,” “mated,” and other such relational terms should be construed, unless otherwise noted, to include removable, moveable, fixed, adjustable, and/or releasable connections or attachments. The connections/attachments can include direct connections and/or connections having intermediate structure between the two components discussed.

Numbers preceded by a term such as “approximately”, “about”, and “substantially” as used herein include the recited numbers, and also represent an amount close to the stated amount that still performs a desired function or achieves a desired result. For example, the terms “approximately”, “about”, and “substantially” may refer to an amount that is within less than 10% of the stated amount. Features of embodiments disclosed herein preceded by a term such as “approximately”, “about”, and “substantially” as used herein represent the feature with some variability that still performs a desired function or achieves a desired result for that feature.