# The System Research and Implementation for Autorecognition of the Ship Draft via the UAV

Authors: Wei Zhan; Shengbing Hong; Yong Sun; Chenguang Zhu
Journal: International Journal of Antennas and Propagation
 (2021)

---

## Abstract

The reading of the ship draft is an important step in the process of weighing and pricing. The traditional detection method is time-consuming and labor-consuming, and it is easy to lead to misdetection. In order to solve the above problems, this paper introduces the computer image processing technology based on deep learning, and the specific process is divided into three steps: first, the video sampling is carried out by the UAV to obtain a large number of pictures of the ship draft reading face, and the images are preprocessed; then, the deep learning target detection algorithm of improved YOLOv3 is used to process the images to predict the position of the waterline and identify the draft characters; finally, the prediction results are analyzed and processed to obtain the final reading results. The experimental results show that the ship draft reading method proposed in this paper has obvious effects. The method has a good detection effect on high-quality images, and the accuracy rate can reach 98%. The accuracy rate can also reach 73% for the images with poor quality caused by improper capture, character corrosion, bad weather, etc. This method is a kind of artificial intelligence method with safe measurement process, high measurement effect, and accuracy, providing a new idea for related research.


---

## Body

## 1. Introduction

The ship draft reading is an important method widely used in measuring the ship cargo load and identifying the weight of import and export commodities. According to the statistics of 2021 Express, the cargo throughput of China’s ports above designated size has reached 2.3 billion tons, with a year-on-year growth of 6.4%, the container throughput has increased by 2.27% to 8.3%, and the growth rate is significantly faster than that of the cargo throughout of 6.4%. Among them, the import volume of foreign trade coal has increased significantly, and the growth rate exceeds the expected value [1].Accurate results of draft weight measurement are of great significance to the protection of the interests of carriers, consignors, and consignees. The current mainstream method is artificial observation. However, with the increasingly close economic exchanges among countries, the efficiency of manual observation is difficult to support the rapid growth of ship freight demand. In recent years, most of the new draft weight measurement methods have higher requirements on the cost and maintenance of weight measurement equipment.There are a variety of existing draft detection methods [2], mainly including (1) artificial observation, which is vulnerable to interference from subjective factors and objective condition [3]. The observation results have a large deviation from the actual real value and need a large amount of manpower and time consumption. (2) The pressure sensor detection, using the pressure sensor installed in advance on the outside of the ship hull, obtains the change of water depth through the pressure before and after loading, with high accuracy. However, due to the high density and accuracy of the instrument, the installation is inconvenient, and the later maintenance cost is high, so it is not easy to popularize. (3) Sonar detection [4]: because of the slow attenuation speed of ultrasonic energy transfer in water and strong penetration, sonar detection uses this characteristic to measure the waterline of the ship. However, due to the accuracy of the instrument and since the installation position is usually underwater, there are also problems of difficult installation and high maintenance cost in the later stage. (4) Computer image detection: with the current rapid development of artificial intelligence technology, its application scope spreads to various fields, among which the image processing technology is particularly vigorous. In fields such as UAV [5] and medicine [6], a lot of automatic image recognition technologies based on the principle of image processing [7] have been produced. There has been some exploration based on the image detection technology in the field of ship draft reading. Table 1 is the comparison of the parameters for the above water gauge detection methods.Table 1
Parameters of existing draft reading methods.
Contrast parametersArtificial observationPressure sensor detectionSonar detectionTraditional image recognition technology detectionDeep learning model detectionDetection accuracyAverageHigherHigherLowHigherMaintenance costLowerHigherHigherAverageAverageDistance from the shipAverageUnreachableHigherHigherHigherHuman resourcesHigherLowLowLowLowAs can be seen from Table1, deep learning image detection technology based on computer vision has the best effect, and the popularity of high-precision and low-cost camera equipment in the market also provides the possibility of draft reading based on computer vision. At present, the existing deep learning method is to use semantic segmentation algorithm to segment the target area in the image [8], obtain the waterline position, and then obtain the final draft according to the traditional method. At the same time, target detection technology has become more and more mature in recent years, such as Faster R-CNN, SSD, YOLO, and other deep learning algorithms, which have achieved good results in the field of target detection. Therefore, this paper carries out research from the direction of computer vision and proposes a deep learning object detection algorithm based on improved YOLOv3 [9] to conduct research on the draft reading. By preprocessing the sampled data in the early stage and inputting the YOLOv3 network for training, the position of the waterline and the draft character can be predicted. Then, the accuracy of the results can be judged according to the loss of the network. Finally, the draft reading formula can be obtained according to the processing of some fitting functions.


## 2. Materials and Methods

### 2.1. Experimental Materials

#### 2.1.1. Experimental Equipment and Environment

This paper is to study the application of deep learning target detection algorithm in ship draft reading. Because it is an image processing technology based on deep learning, it has high requirements on equipment. The hardware device used in this paper is Intel Core i9 9900k, 32 GB RAM, and GPU is RTX 2080Ti; the software experimental environment is Python 3.6.8 and PyCharm 2020.3 professional edition, using NumPy, OpenCV, and Matplotlib module package to process videos and images, and using PyTorch to build the neural network.


#### 2.1.2. Image Acquisition of the Ship Draft Surface

The ship draft surface video is shot at a customs office in Suzhou by F450 UAV with a resolution of 1920 × 1080. There are 50 videos (in different weather conditions), each of which lasts for 5 minutes, with a total of 250 minutes. Figure1 shows the effect of video shooting.Figure 1
Video capture of the ship draft surface.


### 2.2. Basic Principle of Ship Draft Logarithmic Detection

The process of reading the draft mainly involves two parts: one is the character of the draft on the ship and the other is the position of the waterline. The position of the waterline can be detected by image background subtraction or edge detection. However, considering the fluctuation of the water surface and various factors affecting the color difference, the deep learning detection algorithm using YOLOv3 will have a better effect. For the draft character, it is actually the process of target detection, which can be realized by using Faster R-CNN, SSD, YOLOv3, and other detection methods. However, since the first two algorithms are behind YOLOv3 in both mAP and detection speed, both parts are implemented by using YOLOv3 detection algorithm in the end.


## 2.1. Experimental Materials

### 2.1.1. Experimental Equipment and Environment

This paper is to study the application of deep learning target detection algorithm in ship draft reading. Because it is an image processing technology based on deep learning, it has high requirements on equipment. The hardware device used in this paper is Intel Core i9 9900k, 32 GB RAM, and GPU is RTX 2080Ti; the software experimental environment is Python 3.6.8 and PyCharm 2020.3 professional edition, using NumPy, OpenCV, and Matplotlib module package to process videos and images, and using PyTorch to build the neural network.


### 2.1.2. Image Acquisition of the Ship Draft Surface

The ship draft surface video is shot at a customs office in Suzhou by F450 UAV with a resolution of 1920 × 1080. There are 50 videos (in different weather conditions), each of which lasts for 5 minutes, with a total of 250 minutes. Figure1 shows the effect of video shooting.Figure 1
Video capture of the ship draft surface.


## 2.1.1. Experimental Equipment and Environment

This paper is to study the application of deep learning target detection algorithm in ship draft reading. Because it is an image processing technology based on deep learning, it has high requirements on equipment. The hardware device used in this paper is Intel Core i9 9900k, 32 GB RAM, and GPU is RTX 2080Ti; the software experimental environment is Python 3.6.8 and PyCharm 2020.3 professional edition, using NumPy, OpenCV, and Matplotlib module package to process videos and images, and using PyTorch to build the neural network.


## 2.1.2. Image Acquisition of the Ship Draft Surface

The ship draft surface video is shot at a customs office in Suzhou by F450 UAV with a resolution of 1920 × 1080. There are 50 videos (in different weather conditions), each of which lasts for 5 minutes, with a total of 250 minutes. Figure1 shows the effect of video shooting.Figure 1
Video capture of the ship draft surface.


## 2.2. Basic Principle of Ship Draft Logarithmic Detection

The process of reading the draft mainly involves two parts: one is the character of the draft on the ship and the other is the position of the waterline. The position of the waterline can be detected by image background subtraction or edge detection. However, considering the fluctuation of the water surface and various factors affecting the color difference, the deep learning detection algorithm using YOLOv3 will have a better effect. For the draft character, it is actually the process of target detection, which can be realized by using Faster R-CNN, SSD, YOLOv3, and other detection methods. However, since the first two algorithms are behind YOLOv3 in both mAP and detection speed, both parts are implemented by using YOLOv3 detection algorithm in the end.


## 3. Experimental Principle and Process

In this paper, the improved YOLOv3 network is used to detect the waterline and draft character. Its network structure is shown in Figure2. Its backbone network is Darknet53, which contains 53 convolution layers and is a feature extraction network. Darknet contains a series of residual structures that are the residual modules. The convolutional layer in the network actually includes three parts: Conv2d layer, BN layer, and LeakyReLu layer. Considering that the waterline in the draft image is clear and the draft character is also relatively easy to detect the character, it is unnecessary to use such a deep Darknet53 network as the feature extraction network. This paper designs a light feature extraction network S-Net (small network) for this detection task. After the feature extraction network, the prediction output of three branches is followed. The first prediction output corresponds to the prediction of a large target; the second prediction output is the fusion of a medium-sized feature map, predicting a medium-sized target; the third prediction output is the fusion of a large-sized feature map and, finally, is the prediction of a small target. Please refer to Figure 2 for specific information.Figure 2
Network structure of improved YOLOv3.


### 3.1. Image Preprocessing

OpenCV library is used to read the captured ship video images. According to the change of the ship position in the video, we finally decided to take a frame every 3 seconds. A total of 250 minutes of the video can be sampled to 5000 pictures. In the shooting video, the characters and numbers on the hull are not on a vertical line due to the shooting angle and hull shaking, which will greatly affect the training effect. Therefore, this paper is looking for a better processing method and finally decided to use the principle of image affine transformation to process the image in order to get the image of vertical characters and numbers. Image affine transformation is a linear transformation from two-dimensional coordinates (x, y) to two-dimensional coordinates u,v, and the mathematical formula is as follows:(1)u=a1x+b1y+c1,v=a2x+b2y+c2.Affine transformation maintains the straightness and parallelism of the two-dimensional image. In other words, the original straight line still remains the same, as well as the original parallel line, but the position coordinates of each point have changed considerably. The inclined straight line can be turned into vertical so that the problem that the hull characters are not in the vertical line can be solved. The image after affine processing is shown in Figure3.Figure 3
(a) The image of ship draft before affine transformation. (b) The image after affine transformation.
(a)(b)


### 3.2. Waterline Detection

The YOLOv3 grid is used to detect the waterline. Although the whole waterline can contain more waterline information, the final recognition result obtained based on this information cannot fit the waterline, so it is necessary to restore the water wave in the actual shipping process. Therefore, the YOLOv3 network can be used to mark and optimize the waterline in the actual shipping process. Due to different styles of each picture, this paper uses the artificial tagging, which is mainly to frame the picture of the ship dataset manually, use multiple squares to mark the waterline [6], transfer the manually marked picture to the YOLOv3 network for training, and then forecast the new picture to obtain the predicted waterline position of the new picture. The effect is shown in Figure 4, which corresponds to the waterline marked manually and detected by YOLOv3, respectively.Figure 4
(a) Marked image by YOLOv3. (b) Detection image of waterline by YOLOv3.
(a)(b)


### 3.3. Draft Character Recognition

The recognition of the draft character in this paper is also detected using the target detection algorithm of YOLOv3. The numbers and characters are manually labeled and input into the network for training.As the main network of YOLOv3 uses the Darknet network, it is a kind of convolutional neural network as shown in Figure5. The three characteristics of the convolutional neural network [10] are that it has local receptive field, weight sharing, and downsampling, greatly reduces the complexity of the network model and the number of parameters, and has a better processing effect on the grid data than the traditional DNN. Therefore, it is widely used in image classification and target recognition field.Figure 5
Convolutional neural network.Assuming that layerL is a convolution layer and layer L+1 is a pooling layer, the calculation process of the j feature map of layer L is shown in formula (2), where k is the convolution kernel.(2)xjL=f∑i∈Mjixl−1∗kijL+bjL.The residual calculation formula of thej feature map in layer L is as follows:(3)δjl=βjl+1f′ujl⊙upδjl+1.YOLO algorithm adopts a single convolutional neural network model to achieve end-to-end target detection. The input image is adjusted to448×448 and then input into the CNN to process the network prediction results to obtain the detected target, as shown in Figure 6.Figure 6
The result of prediction via the CNN of YOLO.Specifically, the CNN of YOLO divides the input image intoS×S grids [11], and each grid will detect the target whose center point falls in the grid and predict bounding boxes and the corresponding confidence score for the target, which is defined as(4)confidence=Probject∗IOUpredtruth,where Probject is the possibility that the bounding box contains the target, which is 1 when containing the target and 0, otherwise. IOUpredtruth is the accuracy of the bounding box when it contains the target, which is expressed by the intersection and union ratio of the actual box and the prediction box.This paper attempts to mark the draft character in 5000 pictures in different situations and obtains the effect as shown in Figure7.Figure 7
(a) Prediction result 1. (b) Prediction result 2.
(a)(b)Note that the left reading of the downmost character is “M,” and the minimum centimeter reading isC. Due to the complexity of the real situation, the prediction value should be further cleaned according to the correlation of characters in the vertical direction to reduce the possibility of wrong prediction after obtaining the prediction results.


### 3.4. Acquisition of Reading Results

After the implementation of the first two parts, the prediction results of the waterline and the draft character are finally obtained. The next problem is how to obtain the final draft reading according to these prediction results. The specific method is shown in Figure8, which requires a series of processing to obtain the final reading formula.Figure 8
Flowchart of the draft reading.


#### 3.4.1. Fitting Function of the Draft Character

Combined with the reading rules of the ship draft, under ideal conditions shown in Figure9, the draft is perpendicular to the water surface, and the number is 2, 4, 6, and 8 m counting characters and M characters, whose height is 10 cm, and the spacing between two adjacent characters is 10 cm.Figure 9
Draft reading rules.Since the arrangement of the ship draft character will be curved because of the curvature of the ship, the coordinate system is established with the upper left corner of the picture as the origin. It is assumed that, in the recognition result of the draft character, the bottom ordinate and center abscissa of the character prediction box with the lowesty value (negative value) in the picture are X1B,Y1B, with its top ordinate and center abscissa X1T,Y1T. Because of the curvature of the hull, the pixel changes between the characters with more spacing are not consistent, and there is no predictive value. However, the pixel changes between the characters with shorter spacing are basically consistent, which can fit the curve. After the experiment, this paper takes three numbers with the actual total distance less than 1 m from bottom to top and records them in Table 2.Table 2
Variables of the character fitting function.
X1B,Y1BX1T,Y1TX2B,Y2BX2T,Y2TX3B,Y3BX3T,Y3TX4B,Y4BX4T,Y4TAccording to Table2, the curve function Wx passing through the middle axis of the bottom three characters of the image is obtained by fitting the water ruler character curve with the least square method. When reading the draft, there will be an intersection point between this function and the waterline fitting function, and the draft reading can be obtained by obtaining the value of the ordinate of the intersection.


#### 3.4.2. Fitting the Mapping Function of Pixels in Height

The relationship between the actual height and the pixel changes by the mapping functionY, and the result goes 0, which is recorded in Table 3.Table 3
Variables of the mapping function.
Y1B,70Y1T,60Y2B,50Y2T,40Y3B,30Y3T,20Y4B,10Y4T,0According to Table3, the mapping function Z1y of interval pixel changing at the actual height is deduced by using the least square method. The least square method is the basis of classification regression algorithm, which seeks the best function matching of data by minimizing the sum of squares of errors and is defined as(5)S∈2=min∑i=1nyi−fxi2.The optimal value isy which makes S∈2 the minimum.When the independent variable of the polynomial isxi, fxi is the predicted value and yi is the real value. When the sum of squares between the two differences is the smallest, it can be considered that the deviation degree of all predicted values from the real value is to the least extent, which is the optimal fitting curve. The higher the degree of the polynomial is, the more fitting the curve will be, but the slower the fitting speed will be. After testing, eight groups of data and a six-degree polynomial can meet the needs.The mapping functionZ1y fitted by the least square method has a good anti-interference ability when the hull appears radian, but when the character recognition result box deviates, it will lead to a certain deviation of the mapping result. In this case, the mapping function Z2y needs to be calculated. According to the proportional relationship between the height of the character box at the bottom of the hull recognition and 10 cm, formulas (6) and (7) are obtained:(6)Y1B−Y1T10=Yjoin−Y1Bu,(7)Z2y=10×Yjoin−Y1B÷Y1B−Y1T,where u is the centimeter from the bottom of the lowest character to the surface of water. Compared with Z1y, Z2y reduces the error caused by the deviation of the character recognition box, but it does not have the advantage of Z1y in hull radian detection. Therefore, after unifying the specifications of X and Y, let the manual label reading be the dependent variable H, the variation of X1B and X4T be n, Z1y−70 be m, and Z2y be u. The least square method of three variables [12] is used to further study the relationship between the above three and fit the mapping function Zy. The function reflects the relationship between the pixel height of the draft reading and the actual height, through which the actual draft reading can be obtained.


#### 3.4.3. Fitting the Wave Curve

After the experiment of waterline prediction, the number of prediction boxes in a picture is usually about 20, and the central coordinate of the prediction box is taken. Here, two methods are tried to determine the waterline curveSx, and the fitting effect of the waterline is relatively good. The details are as follows:(1)
Fitting the wave curve with the least square method:Taking four center points as a set of data, the cubic polynomial is established, and the fitting function is calculated by using the least square method. Because the distribution of the prediction box is roughly uniform throughout the horizontal axis of the picture, the first coordinate of the latter set of data can be taken as the fourth coordinate of the previous set of data, and the final result is a fitting wave curve.(2)
Fitting the wave curve by cubic spline interpolation:In the real situation, the detected prediction box will show the phenomenon of misidentification. Thus, in the above method (1), the least square curve fitting is carried out in segments to reduce the error interference. Method (2) also uses the cubic spline difference method [13–18] to fit the curve based on the idea of segmentation.Cubic spline interpolation method is to divide an intervala,b into n intervals of x0=a<x1,……,xn−1<b=xn and use the known n+1 points to simulate an unknown function Sx. The function Sx satisfies the following conditions:①
Sxi=fxi=yi,i=0,1,2,…,n.②
Sx is a cubic polynomial on each subinterval xi−xi+1.③
Sx is second-order continuously differentiable on a,b, and Sx satisfies(8)Si−1xi=Sixi,Si−1′xi=Si′xi,Si−1″xi=Si″xi.The specific fitting method to fit the wave curve needs to be selected according to the actual dataset. The least square fitting curve is a polynomial form to determine myopia by tracing points. However, due to the error of given data, the mathematical model of fitting curve cannot be selected at the beginning, and it often needs multiple calculations and analyses to obtain a good fitting effect. Cubic spline interpolation method is related to the number of difference points, accuracy, and selection of difference points. The more the difference points, the better the result fitting. The dataset in this paper is limited, so the cubic spline interpolation method is used to fit the waterline. The fitting effect is good, and the speed is fast. Finally, the fitting functionSx is obtained.


#### 3.4.4. Draft Reading Formula

According to the previous fitting steps, the calculation formula of draft reading can be obtained:(1)
LetWx=Sx; then, Yjoin is obtained(2)
LetHdistance=ZYjoin to obtain the actual height change(3)
Compare the height relationship betweenHdistance and the draft characters predicted by the network, and select the lowest draft meters M and centimeters C of the height Hdistance in the character coordinate set(4)
Predictreading=M−1+C∗0.1−0.01∗Hdistance (unit: m)


#### 3.4.5. Application of Reading Results

After the above series of processing, the final formula of the draft reading is obtained. Based on this formula, we can realize the system of this draft reading method in order to apply it in the actual measurement process. For instance, a set of ship draft reading system can be developed, which can be deployed on the intelligent UAV, intelligent HD camera, and server so as to realize the reading of the ship draft. On both sides of the channel, solar-powered cameras can be installed to collect the images of the ship’s six-side draft through high-definition cameras. The data can be transmitted to the server through digital signals, and the system deployed on the server can be used to read the ship’s draft.In complicated ports, the UAV can be controlled on the shore or on the deck, and the high mobility and high zoom of the UAV can be used to collect the ship draft images. In addition, for the UAV equipped with DJI Onboard SDK custom programming, the cruise mode can be set to make the UAV fly according to the set inspection mode and automatically collect the draft images, further reducing the waste of human resources. In addition, if conditions permit, the UAV and intelligent camera with high chip performance can be deployed in the embedded system so that the UAV or camera can read the draft of the image while taking the image, reducing the calculation of the server and improving the reading speed.


## 3.1. Image Preprocessing

OpenCV library is used to read the captured ship video images. According to the change of the ship position in the video, we finally decided to take a frame every 3 seconds. A total of 250 minutes of the video can be sampled to 5000 pictures. In the shooting video, the characters and numbers on the hull are not on a vertical line due to the shooting angle and hull shaking, which will greatly affect the training effect. Therefore, this paper is looking for a better processing method and finally decided to use the principle of image affine transformation to process the image in order to get the image of vertical characters and numbers. Image affine transformation is a linear transformation from two-dimensional coordinates (x, y) to two-dimensional coordinates u,v, and the mathematical formula is as follows:(1)u=a1x+b1y+c1,v=a2x+b2y+c2.Affine transformation maintains the straightness and parallelism of the two-dimensional image. In other words, the original straight line still remains the same, as well as the original parallel line, but the position coordinates of each point have changed considerably. The inclined straight line can be turned into vertical so that the problem that the hull characters are not in the vertical line can be solved. The image after affine processing is shown in Figure3.Figure 3
(a) The image of ship draft before affine transformation. (b) The image after affine transformation.
(a)(b)


## 3.2. Waterline Detection

The YOLOv3 grid is used to detect the waterline. Although the whole waterline can contain more waterline information, the final recognition result obtained based on this information cannot fit the waterline, so it is necessary to restore the water wave in the actual shipping process. Therefore, the YOLOv3 network can be used to mark and optimize the waterline in the actual shipping process. Due to different styles of each picture, this paper uses the artificial tagging, which is mainly to frame the picture of the ship dataset manually, use multiple squares to mark the waterline [6], transfer the manually marked picture to the YOLOv3 network for training, and then forecast the new picture to obtain the predicted waterline position of the new picture. The effect is shown in Figure 4, which corresponds to the waterline marked manually and detected by YOLOv3, respectively.Figure 4
(a) Marked image by YOLOv3. (b) Detection image of waterline by YOLOv3.
(a)(b)


## 3.3. Draft Character Recognition

The recognition of the draft character in this paper is also detected using the target detection algorithm of YOLOv3. The numbers and characters are manually labeled and input into the network for training.As the main network of YOLOv3 uses the Darknet network, it is a kind of convolutional neural network as shown in Figure5. The three characteristics of the convolutional neural network [10] are that it has local receptive field, weight sharing, and downsampling, greatly reduces the complexity of the network model and the number of parameters, and has a better processing effect on the grid data than the traditional DNN. Therefore, it is widely used in image classification and target recognition field.Figure 5
Convolutional neural network.Assuming that layerL is a convolution layer and layer L+1 is a pooling layer, the calculation process of the j feature map of layer L is shown in formula (2), where k is the convolution kernel.(2)xjL=f∑i∈Mjixl−1∗kijL+bjL.The residual calculation formula of thej feature map in layer L is as follows:(3)δjl=βjl+1f′ujl⊙upδjl+1.YOLO algorithm adopts a single convolutional neural network model to achieve end-to-end target detection. The input image is adjusted to448×448 and then input into the CNN to process the network prediction results to obtain the detected target, as shown in Figure 6.Figure 6
The result of prediction via the CNN of YOLO.Specifically, the CNN of YOLO divides the input image intoS×S grids [11], and each grid will detect the target whose center point falls in the grid and predict bounding boxes and the corresponding confidence score for the target, which is defined as(4)confidence=Probject∗IOUpredtruth,where Probject is the possibility that the bounding box contains the target, which is 1 when containing the target and 0, otherwise. IOUpredtruth is the accuracy of the bounding box when it contains the target, which is expressed by the intersection and union ratio of the actual box and the prediction box.This paper attempts to mark the draft character in 5000 pictures in different situations and obtains the effect as shown in Figure7.Figure 7
(a) Prediction result 1. (b) Prediction result 2.
(a)(b)Note that the left reading of the downmost character is “M,” and the minimum centimeter reading isC. Due to the complexity of the real situation, the prediction value should be further cleaned according to the correlation of characters in the vertical direction to reduce the possibility of wrong prediction after obtaining the prediction results.


## 3.4. Acquisition of Reading Results

After the implementation of the first two parts, the prediction results of the waterline and the draft character are finally obtained. The next problem is how to obtain the final draft reading according to these prediction results. The specific method is shown in Figure8, which requires a series of processing to obtain the final reading formula.Figure 8
Flowchart of the draft reading.


### 3.4.1. Fitting Function of the Draft Character

Combined with the reading rules of the ship draft, under ideal conditions shown in Figure9, the draft is perpendicular to the water surface, and the number is 2, 4, 6, and 8 m counting characters and M characters, whose height is 10 cm, and the spacing between two adjacent characters is 10 cm.Figure 9
Draft reading rules.Since the arrangement of the ship draft character will be curved because of the curvature of the ship, the coordinate system is established with the upper left corner of the picture as the origin. It is assumed that, in the recognition result of the draft character, the bottom ordinate and center abscissa of the character prediction box with the lowesty value (negative value) in the picture are X1B,Y1B, with its top ordinate and center abscissa X1T,Y1T. Because of the curvature of the hull, the pixel changes between the characters with more spacing are not consistent, and there is no predictive value. However, the pixel changes between the characters with shorter spacing are basically consistent, which can fit the curve. After the experiment, this paper takes three numbers with the actual total distance less than 1 m from bottom to top and records them in Table 2.Table 2
Variables of the character fitting function.
X1B,Y1BX1T,Y1TX2B,Y2BX2T,Y2TX3B,Y3BX3T,Y3TX4B,Y4BX4T,Y4TAccording to Table2, the curve function Wx passing through the middle axis of the bottom three characters of the image is obtained by fitting the water ruler character curve with the least square method. When reading the draft, there will be an intersection point between this function and the waterline fitting function, and the draft reading can be obtained by obtaining the value of the ordinate of the intersection.


### 3.4.2. Fitting the Mapping Function of Pixels in Height

The relationship between the actual height and the pixel changes by the mapping functionY, and the result goes 0, which is recorded in Table 3.Table 3
Variables of the mapping function.
Y1B,70Y1T,60Y2B,50Y2T,40Y3B,30Y3T,20Y4B,10Y4T,0According to Table3, the mapping function Z1y of interval pixel changing at the actual height is deduced by using the least square method. The least square method is the basis of classification regression algorithm, which seeks the best function matching of data by minimizing the sum of squares of errors and is defined as(5)S∈2=min∑i=1nyi−fxi2.The optimal value isy which makes S∈2 the minimum.When the independent variable of the polynomial isxi, fxi is the predicted value and yi is the real value. When the sum of squares between the two differences is the smallest, it can be considered that the deviation degree of all predicted values from the real value is to the least extent, which is the optimal fitting curve. The higher the degree of the polynomial is, the more fitting the curve will be, but the slower the fitting speed will be. After testing, eight groups of data and a six-degree polynomial can meet the needs.The mapping functionZ1y fitted by the least square method has a good anti-interference ability when the hull appears radian, but when the character recognition result box deviates, it will lead to a certain deviation of the mapping result. In this case, the mapping function Z2y needs to be calculated. According to the proportional relationship between the height of the character box at the bottom of the hull recognition and 10 cm, formulas (6) and (7) are obtained:(6)Y1B−Y1T10=Yjoin−Y1Bu,(7)Z2y=10×Yjoin−Y1B÷Y1B−Y1T,where u is the centimeter from the bottom of the lowest character to the surface of water. Compared with Z1y, Z2y reduces the error caused by the deviation of the character recognition box, but it does not have the advantage of Z1y in hull radian detection. Therefore, after unifying the specifications of X and Y, let the manual label reading be the dependent variable H, the variation of X1B and X4T be n, Z1y−70 be m, and Z2y be u. The least square method of three variables [12] is used to further study the relationship between the above three and fit the mapping function Zy. The function reflects the relationship between the pixel height of the draft reading and the actual height, through which the actual draft reading can be obtained.


### 3.4.3. Fitting the Wave Curve

After the experiment of waterline prediction, the number of prediction boxes in a picture is usually about 20, and the central coordinate of the prediction box is taken. Here, two methods are tried to determine the waterline curveSx, and the fitting effect of the waterline is relatively good. The details are as follows:(1)
Fitting the wave curve with the least square method:Taking four center points as a set of data, the cubic polynomial is established, and the fitting function is calculated by using the least square method. Because the distribution of the prediction box is roughly uniform throughout the horizontal axis of the picture, the first coordinate of the latter set of data can be taken as the fourth coordinate of the previous set of data, and the final result is a fitting wave curve.(2)
Fitting the wave curve by cubic spline interpolation:In the real situation, the detected prediction box will show the phenomenon of misidentification. Thus, in the above method (1), the least square curve fitting is carried out in segments to reduce the error interference. Method (2) also uses the cubic spline difference method [13–18] to fit the curve based on the idea of segmentation.Cubic spline interpolation method is to divide an intervala,b into n intervals of x0=a<x1,……,xn−1<b=xn and use the known n+1 points to simulate an unknown function Sx. The function Sx satisfies the following conditions:①
Sxi=fxi=yi,i=0,1,2,…,n.②
Sx is a cubic polynomial on each subinterval xi−xi+1.③
Sx is second-order continuously differentiable on a,b, and Sx satisfies(8)Si−1xi=Sixi,Si−1′xi=Si′xi,Si−1″xi=Si″xi.The specific fitting method to fit the wave curve needs to be selected according to the actual dataset. The least square fitting curve is a polynomial form to determine myopia by tracing points. However, due to the error of given data, the mathematical model of fitting curve cannot be selected at the beginning, and it often needs multiple calculations and analyses to obtain a good fitting effect. Cubic spline interpolation method is related to the number of difference points, accuracy, and selection of difference points. The more the difference points, the better the result fitting. The dataset in this paper is limited, so the cubic spline interpolation method is used to fit the waterline. The fitting effect is good, and the speed is fast. Finally, the fitting functionSx is obtained.


### 3.4.4. Draft Reading Formula

According to the previous fitting steps, the calculation formula of draft reading can be obtained:(1)
LetWx=Sx; then, Yjoin is obtained(2)
LetHdistance=ZYjoin to obtain the actual height change(3)
Compare the height relationship betweenHdistance and the draft characters predicted by the network, and select the lowest draft meters M and centimeters C of the height Hdistance in the character coordinate set(4)
Predictreading=M−1+C∗0.1−0.01∗Hdistance (unit: m)


### 3.4.5. Application of Reading Results

After the above series of processing, the final formula of the draft reading is obtained. Based on this formula, we can realize the system of this draft reading method in order to apply it in the actual measurement process. For instance, a set of ship draft reading system can be developed, which can be deployed on the intelligent UAV, intelligent HD camera, and server so as to realize the reading of the ship draft. On both sides of the channel, solar-powered cameras can be installed to collect the images of the ship’s six-side draft through high-definition cameras. The data can be transmitted to the server through digital signals, and the system deployed on the server can be used to read the ship’s draft.In complicated ports, the UAV can be controlled on the shore or on the deck, and the high mobility and high zoom of the UAV can be used to collect the ship draft images. In addition, for the UAV equipped with DJI Onboard SDK custom programming, the cruise mode can be set to make the UAV fly according to the set inspection mode and automatically collect the draft images, further reducing the waste of human resources. In addition, if conditions permit, the UAV and intelligent camera with high chip performance can be deployed in the embedded system so that the UAV or camera can read the draft of the image while taking the image, reducing the calculation of the server and improving the reading speed.


## 3.4.1. Fitting Function of the Draft Character

Combined with the reading rules of the ship draft, under ideal conditions shown in Figure9, the draft is perpendicular to the water surface, and the number is 2, 4, 6, and 8 m counting characters and M characters, whose height is 10 cm, and the spacing between two adjacent characters is 10 cm.Figure 9
Draft reading rules.Since the arrangement of the ship draft character will be curved because of the curvature of the ship, the coordinate system is established with the upper left corner of the picture as the origin. It is assumed that, in the recognition result of the draft character, the bottom ordinate and center abscissa of the character prediction box with the lowesty value (negative value) in the picture are X1B,Y1B, with its top ordinate and center abscissa X1T,Y1T. Because of the curvature of the hull, the pixel changes between the characters with more spacing are not consistent, and there is no predictive value. However, the pixel changes between the characters with shorter spacing are basically consistent, which can fit the curve. After the experiment, this paper takes three numbers with the actual total distance less than 1 m from bottom to top and records them in Table 2.Table 2
Variables of the character fitting function.
X1B,Y1BX1T,Y1TX2B,Y2BX2T,Y2TX3B,Y3BX3T,Y3TX4B,Y4BX4T,Y4TAccording to Table2, the curve function Wx passing through the middle axis of the bottom three characters of the image is obtained by fitting the water ruler character curve with the least square method. When reading the draft, there will be an intersection point between this function and the waterline fitting function, and the draft reading can be obtained by obtaining the value of the ordinate of the intersection.


## 3.4.2. Fitting the Mapping Function of Pixels in Height

The relationship between the actual height and the pixel changes by the mapping functionY, and the result goes 0, which is recorded in Table 3.Table 3
Variables of the mapping function.
Y1B,70Y1T,60Y2B,50Y2T,40Y3B,30Y3T,20Y4B,10Y4T,0According to Table3, the mapping function Z1y of interval pixel changing at the actual height is deduced by using the least square method. The least square method is the basis of classification regression algorithm, which seeks the best function matching of data by minimizing the sum of squares of errors and is defined as(5)S∈2=min∑i=1nyi−fxi2.The optimal value isy which makes S∈2 the minimum.When the independent variable of the polynomial isxi, fxi is the predicted value and yi is the real value. When the sum of squares between the two differences is the smallest, it can be considered that the deviation degree of all predicted values from the real value is to the least extent, which is the optimal fitting curve. The higher the degree of the polynomial is, the more fitting the curve will be, but the slower the fitting speed will be. After testing, eight groups of data and a six-degree polynomial can meet the needs.The mapping functionZ1y fitted by the least square method has a good anti-interference ability when the hull appears radian, but when the character recognition result box deviates, it will lead to a certain deviation of the mapping result. In this case, the mapping function Z2y needs to be calculated. According to the proportional relationship between the height of the character box at the bottom of the hull recognition and 10 cm, formulas (6) and (7) are obtained:(6)Y1B−Y1T10=Yjoin−Y1Bu,(7)Z2y=10×Yjoin−Y1B÷Y1B−Y1T,where u is the centimeter from the bottom of the lowest character to the surface of water. Compared with Z1y, Z2y reduces the error caused by the deviation of the character recognition box, but it does not have the advantage of Z1y in hull radian detection. Therefore, after unifying the specifications of X and Y, let the manual label reading be the dependent variable H, the variation of X1B and X4T be n, Z1y−70 be m, and Z2y be u. The least square method of three variables [12] is used to further study the relationship between the above three and fit the mapping function Zy. The function reflects the relationship between the pixel height of the draft reading and the actual height, through which the actual draft reading can be obtained.


## 3.4.3. Fitting the Wave Curve

After the experiment of waterline prediction, the number of prediction boxes in a picture is usually about 20, and the central coordinate of the prediction box is taken. Here, two methods are tried to determine the waterline curveSx, and the fitting effect of the waterline is relatively good. The details are as follows:(1)
Fitting the wave curve with the least square method:Taking four center points as a set of data, the cubic polynomial is established, and the fitting function is calculated by using the least square method. Because the distribution of the prediction box is roughly uniform throughout the horizontal axis of the picture, the first coordinate of the latter set of data can be taken as the fourth coordinate of the previous set of data, and the final result is a fitting wave curve.(2)
Fitting the wave curve by cubic spline interpolation:In the real situation, the detected prediction box will show the phenomenon of misidentification. Thus, in the above method (1), the least square curve fitting is carried out in segments to reduce the error interference. Method (2) also uses the cubic spline difference method [13–18] to fit the curve based on the idea of segmentation.Cubic spline interpolation method is to divide an intervala,b into n intervals of x0=a<x1,……,xn−1<b=xn and use the known n+1 points to simulate an unknown function Sx. The function Sx satisfies the following conditions:①
Sxi=fxi=yi,i=0,1,2,…,n.②
Sx is a cubic polynomial on each subinterval xi−xi+1.③
Sx is second-order continuously differentiable on a,b, and Sx satisfies(8)Si−1xi=Sixi,Si−1′xi=Si′xi,Si−1″xi=Si″xi.The specific fitting method to fit the wave curve needs to be selected according to the actual dataset. The least square fitting curve is a polynomial form to determine myopia by tracing points. However, due to the error of given data, the mathematical model of fitting curve cannot be selected at the beginning, and it often needs multiple calculations and analyses to obtain a good fitting effect. Cubic spline interpolation method is related to the number of difference points, accuracy, and selection of difference points. The more the difference points, the better the result fitting. The dataset in this paper is limited, so the cubic spline interpolation method is used to fit the waterline. The fitting effect is good, and the speed is fast. Finally, the fitting functionSx is obtained.


## 3.4.4. Draft Reading Formula

According to the previous fitting steps, the calculation formula of draft reading can be obtained:(1)
LetWx=Sx; then, Yjoin is obtained(2)
LetHdistance=ZYjoin to obtain the actual height change(3)
Compare the height relationship betweenHdistance and the draft characters predicted by the network, and select the lowest draft meters M and centimeters C of the height Hdistance in the character coordinate set(4)
Predictreading=M−1+C∗0.1−0.01∗Hdistance (unit: m)


## 3.4.5. Application of Reading Results

After the above series of processing, the final formula of the draft reading is obtained. Based on this formula, we can realize the system of this draft reading method in order to apply it in the actual measurement process. For instance, a set of ship draft reading system can be developed, which can be deployed on the intelligent UAV, intelligent HD camera, and server so as to realize the reading of the ship draft. On both sides of the channel, solar-powered cameras can be installed to collect the images of the ship’s six-side draft through high-definition cameras. The data can be transmitted to the server through digital signals, and the system deployed on the server can be used to read the ship’s draft.In complicated ports, the UAV can be controlled on the shore or on the deck, and the high mobility and high zoom of the UAV can be used to collect the ship draft images. In addition, for the UAV equipped with DJI Onboard SDK custom programming, the cruise mode can be set to make the UAV fly according to the set inspection mode and automatically collect the draft images, further reducing the waste of human resources. In addition, if conditions permit, the UAV and intelligent camera with high chip performance can be deployed in the embedded system so that the UAV or camera can read the draft of the image while taking the image, reducing the calculation of the server and improving the reading speed.


## 4. Experimental Results and Analysis

According to the above experimental process and experimental steps, to train the experience, the mAP curve and loss curve during the training process and the loss curve during the verification process are obtained, as shown in Figure10. Use the trained model to predict the new ship video or image, then fit the predicted result through the draft character fitting function and the waterline fitting function, and finally use the draft reading formula to read; one can obtain as shown in Figure 11. The results shown can clearly detect the position of the draft characters and the waterline, and the draft reading results are recorded in the upper left corner of the picture.Figure 10
The curve of mAP and the loss of the training and verification set during training. (a) mAP@0.5 curve in the process of the training. (b) mAP@0.5 : 0.95 curve in the process of the training. (c) Loss curve of the training set bounding box. (d) Loss curve of the training set class. (e) Loss curve of the verification set bounding box. (f) Loss curve of the verification set class.
(a)(b)(c)(d)(e)(f)Figure 11
Sample diagram of experimental results.Based on the factors of time and data collection, this paper collected 5000 aerial draft images in different scenes (different weathers, waves, light, and other scenes), annotated the draft characters and waterline in the images, and selected 4000 of them randomly for training, and two test sets were tested, with the number of test sets in each group being 500 sheets. The experimental results are shown in Table4.Table 4
Statistical table of effective values of the two groups.
Group numberError < 0.02 mError < 0.01 m14804602485465Compared with the manual observation results and the existing semantic segmentation methods, we can obtain the results in Table5. In order to ensure the reliability of the contrast data, the semantic segmentation method uses the Fine-MobileNet [8] network improved based on the MobileNet V2 network and uses the same dataset and hardware environment for the experiment. The manual observation results are obtained by a number of professional meteorologists, respectively, reading the results and taking the average value. It can be seen that, in most cases, the error between the reading result of this method and the result of manual observation is within 0.3 cm. The semantic segmentation method can also control the error, but the average value of the absolute error of the method in this paper is small. In the actual ship draft reading, the robustness is good, and the detection accuracy and efficiency all satisfy the actual demand.Table 5
Comparison of experimental results.
Serial numberResults of manual observation (m)Results of semantic segmentation (m)Results of this article (m)Error (segmentation method/method of this article) (m)a18.28018.28218.2780.002/−0.002b6.8106.8026.813−0.008/0.003c7.7517.7567.7270.005/−0.024d7.7037.7167.7050.013/0.002e6.7526.7746.7570.022/0.005f8.7978.7958.790−0.002/−0.007Mean absolute value of difference (m)0.0087/0.0072We also divided the images into two categories according to the quality of the photos, which are the image samples corresponding to clear hull characters in sunny weather and the image samples corresponding to fuzzy hull characters in bad weather. These two kinds of pictures were used to test the training model, and the corresponding results in Table6 were finally obtained.Table 6
Prediction results of different quality images.
Picture qualitymAP@0.5 : 0.95mAP@0.5Better-quality images0.740.98Poor-quality images0.620.73According to the above experimental results, it can be seen that the sample experimental effect chart is clear, and the position of the waterline and the recognized characters can be clearly given, the draft reading result is given, and the error can be basically stabilized within 0.03 m, generally within 0.01 m. Moreover, this article compares their prediction effects when the picture quality is relatively poor and relatively good. For pictures with better quality, such as those with clear hull characters and clear weather, their prediction accuracy is 98%. For poor quality pictures, such as pictures in which the hull characters had been blurred by corrosion or in bad weather, their test accuracy is only 73%. Comparing with manual estimation, the method in this article can basically control the error within the millimeter level. Therefore, the research in this article has achieved relatively good experimental results.


## 5. Conclusions

In this paper, based on computer vision technology, through the exploration and comparison of various image processing technologies, the results show that using deep learning technology to predict the draft reading, with artificial reading deviation within 0.01 meters, is valid. The prediction accuracy of this method is 98% in the case of good image quality and 73% in the case of poor image quality caused by bad weather, hull corrosion, and other reasons. In fact, this kind of error is normal; even if the manual reading is also biased, this method has greatly reduced the error. Moreover, the image processing method using deep learning has better effect in speed and accuracy both compared with the traditional manual reading method and the traditional image detection technology.However, it cannot be ignored that there are still some errors in the experimental results, which are inseparable from the complex and changeable real navigation situation. And there will be various interferences in the actual detection, which requires further research to collect a large number of hull pictures in different environments, build a larger dataset, train the network, and enhance the robustness of the model. Meanwhile, the data preprocessing is also worthy of further investigation.



---

*Source: 4617242-2021-08-17.xml*