# Environment Perception Technologies for Power Transmission Line Inspection Robots

Authors: Minghao Chen; Yunong Tian; Shiyu Xing; Zhishuo Li; En Li; Zize Liang; Rui Guo
Journal: Journal of Sensors
 (2021)

---

## Abstract

With the fast development of the power system, traditional manual inspection methods of a power transmission line (PTL) cannot supply the demand for high quality and dependability for power grid maintenance. Consequently, the automatic PTL inspection technology becomes one of the key research focuses. For the purpose of summarizing related studies on environment perception and control technologies of PTL inspection, technologies of three-dimensional (3D) reconstruction, object detection, and visual servo of PTL inspection are reviewed, respectively. Firstly, 3D reconstruction of PTL inspection is reviewed and analyzed, especially for the technology of LiDAR-based reconstruction of power lines. Secondly, the technology of typical object detection, including pylons, insulators, and power line accessories, is classified as traditional and deep learning-based methods. After that, their merits and demerits are considered. Thirdly, the progress and issues of visual servo control of inspection robots are also concisely addressed. For improving the automation degree of PTL robots, current problems of key techniques, such as multisensor fusion and the establishment of datasets, are discussed and the prospect of inspection robots is presented.


---

## Body

## 1. Introduction

Traditional PTL inspection methods include line crawling inspection, ground-based inspection, and manual inspection with telescopes, as shown in Figure1. Their defects are clearer with the progress of the power system. These methods are slow and dangerous and may not be conducted sometimes. The reasons are as follows: (a) intricate and diverse workspace. The arrangement of PTL corridors includes several scenarios, such as overhead ground wire and multibundled conductors. The slope of conductors is different. What is more, a great variety of obstacles are on the PTL, as shown in Figure 2 [1]. (b) Geographical conditions of PTL are various. Part of the PTL is located in some complex areas such as swamps, lakes, and mountains. Although the speedy and maneuverable helicopter inspection method can overcome this difficulty, its detection precision of small-scale objects is affected by the long working distance. Consequently, it is also not the best inspection method. Automatic PTL inspection technology is eager to be developed in these cases, which is also a challenging task. Apart from mechanical design, PTL environment perception and control technologies are the foundation of automatic PTL inspection. And they are research hotspots in PTL inspection.Figure 1
Traditional inspection methods: (a) line crawling inspection; (b) ground-based inspection; (c) manual inspection with a telescope.
(a)(b)(c)Figure 2
Suspension clamps, vibration dampers, and insulator strings [1].In the whole process of automatic PTL inspection, the main tasks completed by inspection robots now are as follows: (a) 3D reconstruction of PTL corridors [2, 3]. 3D reconstruction can be employed for the visualization of PTL, which is helpful for inspectors to analyze the excursion trend of PTL corridors, interference of surrounding environment, broken point detection, and snow loading. (b) Fine inspection of pylons [4] and components [5, 6]. Object detection is exploited to recognize and locate them and their defects, such as cracked nut, bolt looseness, and fitting corrosion, which are difficult to be found by manual inspection. Besides, object detection can be interfered with by rain, snow, and wind. Therefore, 3D reconstruction is also applied to a local map to acquire a more precise target position. (c) Obstacle crossing and fault maintenance [7, 8]. Obstacle crossing and fault maintenance can be realized by remote teleoperation or autonomous control. Remote teleoperation is easy to realize, which depends on the manual operation of a ground control unit. However, it is difficult to achieve autonomous operation. Due to much interference existing in the PTL environment, the autonomous operation of inspection robots requires a more robust control system. In contrast to sensor-based control, visual servo control is more robust and flexible. It is suitable for the unpredictable environment of PTL corridors. Aimed at these tasks, 3D reconstruction, object detection, and visual servo of PTL inspection are summarized and analyzed for providing a widely comprehensible review.The rest of this paper is organized as follows: Section2 describes the development of the transmission line corridor in 3D reconstruction. Section 3 introduces the progress of traditional and deep learning-based detection algorithms in the field of PTL inspection. Then, their merits and demerits are considered. Section 4 concerns the visual servo of different inspection robots. Section 5 analyzes key technical problems and presents future development directions of inspection robots. Finally, the conclusion is presented in Section 6.


## 2. 3D Reconstruction of PTL Inspection

3D reconstruction is a means to obtain 3D data of measured objects. It is mainly used to acquire object texture, structure, and scale. There are two approaches to 3D reconstruction: the contact and noncontact methods. In contrast to the former way, noncontact methods are extensively used. Based on whether or not to actively transmit measurement signals, they are divided into active and passive visual means. According to various principles, active visual methods are classified as laser methods [9–13], structured light methods [14], and interferometry [15–17]. On the basis of the number of vision sensors used, passive visual methods can be classified as monocular vision [18, 19], binocular vision [20, 21], and multieye vision [22, 23].In the field of PTL inspection, active visual methods are chiefly used to construct the 3D model of the PTL corridor. The characteristics of objects can be intuitively expressed by the 3D model. It is helpful for inspectors to discover existing or potential hazards in the transmission line corridor.


### 2.1. The 3D Reconstruction of Power Lines

The power line plays a key role in the entire PTL corridor as the main carrier of electricity supply. 3D reconstruction of power lines is the foundation for the analysis of wire sag, icing, wind deviation, and distance measurement. Therefore, many studies of it are presented.


#### 2.1.1. Active Visual Methods

Now, laser scanning is normally adopted to complete power line reconstruction. It contains three sections, which are point cloud extraction, point cloud segmentation, and model fitting. Point cloud extraction is divided into simple extraction and fine extraction. The raw data contains point clouds of various objects. The simple extraction is utilized to eliminate point clouds of ground. In addition, fine extraction is employed to further eliminate point clouds that do not belong to lines. Point cloud segmentation falls into determining point clouds of one span and a single conductor. Firstly, the data of different spans are divided according to the highest suspension point. And then, the data of a single power line in one span are confirmed. Finally, the model is selected for fitting.The extraction of point clouds is generally based on filtering methods. Firstly, DTM [24] is generated by filtering. Then, the points of the ground and objects are differentiated. After that, features such as elevation are employed to obtain the point clouds of power lines. Methods based on TIN such as PTD [25] are used commonly. Yu et al. [26] utilized PTD to eliminate ground points. Furthermore, an angle filter was adopted to acquire point clouds of power lines. In addition to PTD, slope filtering [27] and morphological filtering [28] are also used for eliminating ground points. Nevertheless, they are less applied in the extraction of power lines compared to PTD. The elevation is used in conjunction with DTM to gain the point clouds of power lines. Based on nDSM, Liu et al. [29] used the elevation histogram statistical method to get point clouds of power lines. Besides, there is also direct use of elevation to complete extraction. Shen et al. [30] proposed an elevation threshold segmentation algorithm based on the subspace feature and an elevation density segmentation algorithm. Some other methods for point cloud extraction are also employed. Mclaughlin [31] used a GMM and the EM algorithm for extraction. What is more, Jwa and Sohn [32] adopted Hough transform, eigenvalue analysis, point density analysis, and the one-outlier testing technique for extraction.In the process of segmentation, point clouds of lines in one span are decided by the position of pylons. The position of pylons can be obtained by the height [26, 33] or the density of point clouds [29]. Moreover, the segmentation of point clouds can be also completed by the minimum linkage hierarchical clustering [34] and the second derivative [35]. There are three types of techniques to determine point clouds of a single conductor. (a) Clustering. To distinguish each power line, point clouds that belong to the same power line in one span are clustered. Lai et al. [36] proposed a cluster analysis method based on spatial distance restriction. What is more, Lin et al. [35] presented the 3D connected component analysis based on fixed-radius near neighbors (FNN) and the k-means algorithm [37] of normalized projection. (b) Hough transform (HT). Yu et al. [26] and Wu et al. [33] employed HT for the segmentation of the single power line. Melzer et al. [34] used an iterative version of the HT, but this method was unstable. (c) Local growth. The means gradually merges or grows to form the entire power line model, such as the local affine model [31] and the piecewise model growing (PMG) [32].Four types of curve fitting models of power lines are typically employed: (a) the polynomial model [33, 38], (b) the model that combines a line segment and a parabola [36, 37], (c) the model that combines a line segment and the quadratic polynomial with two variables [29, 39], and (d) the model that combines a line segment and a catenary [26, 31, 34, 35]. The first model directly fits the entire power line. However, the other three models split the power line into a line segment and a curved part for fitting. The difference between the latter three models is the expressions of curve fitting. The second model is the approximate expression of the fourth model. And the performance of the second model is much better because of its simpler formula. Besides, Zhang et al. [40] improved the second model and the fourth model. Overall, the second model is better.In the above literature, the methods used in each step of the paper are shown in Table1, which completed the whole process of 3D reconstruction.Table 1
The methods used in each step of 3D reconstruction in different studies.
ReferenceSimple extractionFine extractionDetermination of point clouds in one spanDetermination of point clouds in a single power lineFitting modelsMelzer and Briese [34]Interpolation method for DTM generationA grid culling mechanismA minimum linkage hierarchical clustering approachIterative HTThe fourth modelMclaughlin [31]Using a Gaussian mixture model and the EM algorithmA local affine modelThe fourth modelYu et al. [26]PTDAn angle filerThe position of pylons (by height)HTThe fourth modelLiu et al. [29]The elevation histogram statistical methodUsing point cloud densityThe position of pylons (by density)A spatial domain segmentation algorithmThe third modelWu et al. [33]Echoes of LiDAR dataGrid height elevation and elevation thresholdThe position of pylons (by height)HTThe first modelNow, 3D reconstruction of power lines is focused on areas with simple structures. The background of PTL, areas of different structures, and various types of power lines are considered much less. It is difficult to identify and reconstruct sleeve nodes and spacer nodes. Moreover, the density of LiDAR point clouds influences firsthand 3D reconstruction.


#### 2.1.2. Passive Visual Methods

Passive visual methods are less employed in the 3D reconstruction of power lines, and their accuracy is lower than that of active visual methods. Zhang et al. [41] developed a multiangle imaging power line detection system to reconstruct 3D power lines, but the method’s accuracy was low. Ganovelli et al. [42] presented a means of using a handful of images to reconstruct power lines. This means employed SFM. Because the method was based on specific assumptions, it could not be used in inspection tasks actually. Maurer et al. [43] presented a means that used semantic segmentation based on FCN and multiview geometry. The flowchart is shown in Figure 3. The method was effective but had high requirements on hardware. It is hard to match different images with the lack of effective point features of power lines by passive visual methods. Therefore, active visual methods are mostly employed in the power line reconstruction.Figure 3
The overview of the method based on FCN and multiview geometry [43].


### 2.2. The 3D Reconstruction of Ground Surface and Pylons

In addition to the reconstruction of power lines, the PTL corridors also need to reconstruct the ground surface and pylons. Nonetheless, their 3D reconstruction is in the preliminary stage. With regard to the ground surface, Maurer et al. [43] completed the 3D reconstruction of power lines and the ground surface simultaneously. In light of the reconstruction of pylons, Chen et al. [44] raised a model-driven approach to reconstruct pylons. The pylon could be divided into head, body, and foot. The three segments were reconstructed by different strategies. And they were assembled to the whole pylon by utilizing the direction and position. The result showed that the method’s accuracy could reach the centimeter level. Guo et al. [45] presented a means based on model library and stochastic geometry for pylon reconstruction. This method built an energy model of the correlation between point clouds and pylons. It also used simulated annealing and an MCMC sampler to reconstruct pylons. But this means required plenty of calculations.Recently, oblique photography has been applied to 3D reconstruction of PTL corridors. Xi et al. [46] designed a UAV power inspection system based on oblique photography. The system used the UAV to obtain multiview images of ground objects and accelerated the 3D reconstruction of the PTL corridors through data processing and model refinement. Pei et al. [47] designed a data acquisition device for 3D reconstruction of PTL corridors, which could obtain the structural information of the pylons’ four facades. Oblique photography has some defects in the field of PTL inspection. Oblique photography requires 60%-80% of the air belt overlap. Hence, it is necessary to collect data by multiple flights and the collection efficiency is low. Oblique photography is better for the 3D reconstruction of larger objects but ordinary for small and hollow objects such as power lines and pylons. Consequently, it is imperative to combine oblique photography with other means to gain good reconstruction results. It will be one of the research directions for the 3D reconstruction of PTL corridors in the future.


## 2.1. The 3D Reconstruction of Power Lines

The power line plays a key role in the entire PTL corridor as the main carrier of electricity supply. 3D reconstruction of power lines is the foundation for the analysis of wire sag, icing, wind deviation, and distance measurement. Therefore, many studies of it are presented.


### 2.1.1. Active Visual Methods

Now, laser scanning is normally adopted to complete power line reconstruction. It contains three sections, which are point cloud extraction, point cloud segmentation, and model fitting. Point cloud extraction is divided into simple extraction and fine extraction. The raw data contains point clouds of various objects. The simple extraction is utilized to eliminate point clouds of ground. In addition, fine extraction is employed to further eliminate point clouds that do not belong to lines. Point cloud segmentation falls into determining point clouds of one span and a single conductor. Firstly, the data of different spans are divided according to the highest suspension point. And then, the data of a single power line in one span are confirmed. Finally, the model is selected for fitting.The extraction of point clouds is generally based on filtering methods. Firstly, DTM [24] is generated by filtering. Then, the points of the ground and objects are differentiated. After that, features such as elevation are employed to obtain the point clouds of power lines. Methods based on TIN such as PTD [25] are used commonly. Yu et al. [26] utilized PTD to eliminate ground points. Furthermore, an angle filter was adopted to acquire point clouds of power lines. In addition to PTD, slope filtering [27] and morphological filtering [28] are also used for eliminating ground points. Nevertheless, they are less applied in the extraction of power lines compared to PTD. The elevation is used in conjunction with DTM to gain the point clouds of power lines. Based on nDSM, Liu et al. [29] used the elevation histogram statistical method to get point clouds of power lines. Besides, there is also direct use of elevation to complete extraction. Shen et al. [30] proposed an elevation threshold segmentation algorithm based on the subspace feature and an elevation density segmentation algorithm. Some other methods for point cloud extraction are also employed. Mclaughlin [31] used a GMM and the EM algorithm for extraction. What is more, Jwa and Sohn [32] adopted Hough transform, eigenvalue analysis, point density analysis, and the one-outlier testing technique for extraction.In the process of segmentation, point clouds of lines in one span are decided by the position of pylons. The position of pylons can be obtained by the height [26, 33] or the density of point clouds [29]. Moreover, the segmentation of point clouds can be also completed by the minimum linkage hierarchical clustering [34] and the second derivative [35]. There are three types of techniques to determine point clouds of a single conductor. (a) Clustering. To distinguish each power line, point clouds that belong to the same power line in one span are clustered. Lai et al. [36] proposed a cluster analysis method based on spatial distance restriction. What is more, Lin et al. [35] presented the 3D connected component analysis based on fixed-radius near neighbors (FNN) and the k-means algorithm [37] of normalized projection. (b) Hough transform (HT). Yu et al. [26] and Wu et al. [33] employed HT for the segmentation of the single power line. Melzer et al. [34] used an iterative version of the HT, but this method was unstable. (c) Local growth. The means gradually merges or grows to form the entire power line model, such as the local affine model [31] and the piecewise model growing (PMG) [32].Four types of curve fitting models of power lines are typically employed: (a) the polynomial model [33, 38], (b) the model that combines a line segment and a parabola [36, 37], (c) the model that combines a line segment and the quadratic polynomial with two variables [29, 39], and (d) the model that combines a line segment and a catenary [26, 31, 34, 35]. The first model directly fits the entire power line. However, the other three models split the power line into a line segment and a curved part for fitting. The difference between the latter three models is the expressions of curve fitting. The second model is the approximate expression of the fourth model. And the performance of the second model is much better because of its simpler formula. Besides, Zhang et al. [40] improved the second model and the fourth model. Overall, the second model is better.In the above literature, the methods used in each step of the paper are shown in Table1, which completed the whole process of 3D reconstruction.Table 1
The methods used in each step of 3D reconstruction in different studies.
ReferenceSimple extractionFine extractionDetermination of point clouds in one spanDetermination of point clouds in a single power lineFitting modelsMelzer and Briese [34]Interpolation method for DTM generationA grid culling mechanismA minimum linkage hierarchical clustering approachIterative HTThe fourth modelMclaughlin [31]Using a Gaussian mixture model and the EM algorithmA local affine modelThe fourth modelYu et al. [26]PTDAn angle filerThe position of pylons (by height)HTThe fourth modelLiu et al. [29]The elevation histogram statistical methodUsing point cloud densityThe position of pylons (by density)A spatial domain segmentation algorithmThe third modelWu et al. [33]Echoes of LiDAR dataGrid height elevation and elevation thresholdThe position of pylons (by height)HTThe first modelNow, 3D reconstruction of power lines is focused on areas with simple structures. The background of PTL, areas of different structures, and various types of power lines are considered much less. It is difficult to identify and reconstruct sleeve nodes and spacer nodes. Moreover, the density of LiDAR point clouds influences firsthand 3D reconstruction.


### 2.1.2. Passive Visual Methods

Passive visual methods are less employed in the 3D reconstruction of power lines, and their accuracy is lower than that of active visual methods. Zhang et al. [41] developed a multiangle imaging power line detection system to reconstruct 3D power lines, but the method’s accuracy was low. Ganovelli et al. [42] presented a means of using a handful of images to reconstruct power lines. This means employed SFM. Because the method was based on specific assumptions, it could not be used in inspection tasks actually. Maurer et al. [43] presented a means that used semantic segmentation based on FCN and multiview geometry. The flowchart is shown in Figure 3. The method was effective but had high requirements on hardware. It is hard to match different images with the lack of effective point features of power lines by passive visual methods. Therefore, active visual methods are mostly employed in the power line reconstruction.Figure 3
The overview of the method based on FCN and multiview geometry [43].


## 2.1.1. Active Visual Methods

Now, laser scanning is normally adopted to complete power line reconstruction. It contains three sections, which are point cloud extraction, point cloud segmentation, and model fitting. Point cloud extraction is divided into simple extraction and fine extraction. The raw data contains point clouds of various objects. The simple extraction is utilized to eliminate point clouds of ground. In addition, fine extraction is employed to further eliminate point clouds that do not belong to lines. Point cloud segmentation falls into determining point clouds of one span and a single conductor. Firstly, the data of different spans are divided according to the highest suspension point. And then, the data of a single power line in one span are confirmed. Finally, the model is selected for fitting.The extraction of point clouds is generally based on filtering methods. Firstly, DTM [24] is generated by filtering. Then, the points of the ground and objects are differentiated. After that, features such as elevation are employed to obtain the point clouds of power lines. Methods based on TIN such as PTD [25] are used commonly. Yu et al. [26] utilized PTD to eliminate ground points. Furthermore, an angle filter was adopted to acquire point clouds of power lines. In addition to PTD, slope filtering [27] and morphological filtering [28] are also used for eliminating ground points. Nevertheless, they are less applied in the extraction of power lines compared to PTD. The elevation is used in conjunction with DTM to gain the point clouds of power lines. Based on nDSM, Liu et al. [29] used the elevation histogram statistical method to get point clouds of power lines. Besides, there is also direct use of elevation to complete extraction. Shen et al. [30] proposed an elevation threshold segmentation algorithm based on the subspace feature and an elevation density segmentation algorithm. Some other methods for point cloud extraction are also employed. Mclaughlin [31] used a GMM and the EM algorithm for extraction. What is more, Jwa and Sohn [32] adopted Hough transform, eigenvalue analysis, point density analysis, and the one-outlier testing technique for extraction.In the process of segmentation, point clouds of lines in one span are decided by the position of pylons. The position of pylons can be obtained by the height [26, 33] or the density of point clouds [29]. Moreover, the segmentation of point clouds can be also completed by the minimum linkage hierarchical clustering [34] and the second derivative [35]. There are three types of techniques to determine point clouds of a single conductor. (a) Clustering. To distinguish each power line, point clouds that belong to the same power line in one span are clustered. Lai et al. [36] proposed a cluster analysis method based on spatial distance restriction. What is more, Lin et al. [35] presented the 3D connected component analysis based on fixed-radius near neighbors (FNN) and the k-means algorithm [37] of normalized projection. (b) Hough transform (HT). Yu et al. [26] and Wu et al. [33] employed HT for the segmentation of the single power line. Melzer et al. [34] used an iterative version of the HT, but this method was unstable. (c) Local growth. The means gradually merges or grows to form the entire power line model, such as the local affine model [31] and the piecewise model growing (PMG) [32].Four types of curve fitting models of power lines are typically employed: (a) the polynomial model [33, 38], (b) the model that combines a line segment and a parabola [36, 37], (c) the model that combines a line segment and the quadratic polynomial with two variables [29, 39], and (d) the model that combines a line segment and a catenary [26, 31, 34, 35]. The first model directly fits the entire power line. However, the other three models split the power line into a line segment and a curved part for fitting. The difference between the latter three models is the expressions of curve fitting. The second model is the approximate expression of the fourth model. And the performance of the second model is much better because of its simpler formula. Besides, Zhang et al. [40] improved the second model and the fourth model. Overall, the second model is better.In the above literature, the methods used in each step of the paper are shown in Table1, which completed the whole process of 3D reconstruction.Table 1
The methods used in each step of 3D reconstruction in different studies.
ReferenceSimple extractionFine extractionDetermination of point clouds in one spanDetermination of point clouds in a single power lineFitting modelsMelzer and Briese [34]Interpolation method for DTM generationA grid culling mechanismA minimum linkage hierarchical clustering approachIterative HTThe fourth modelMclaughlin [31]Using a Gaussian mixture model and the EM algorithmA local affine modelThe fourth modelYu et al. [26]PTDAn angle filerThe position of pylons (by height)HTThe fourth modelLiu et al. [29]The elevation histogram statistical methodUsing point cloud densityThe position of pylons (by density)A spatial domain segmentation algorithmThe third modelWu et al. [33]Echoes of LiDAR dataGrid height elevation and elevation thresholdThe position of pylons (by height)HTThe first modelNow, 3D reconstruction of power lines is focused on areas with simple structures. The background of PTL, areas of different structures, and various types of power lines are considered much less. It is difficult to identify and reconstruct sleeve nodes and spacer nodes. Moreover, the density of LiDAR point clouds influences firsthand 3D reconstruction.


## 2.1.2. Passive Visual Methods

Passive visual methods are less employed in the 3D reconstruction of power lines, and their accuracy is lower than that of active visual methods. Zhang et al. [41] developed a multiangle imaging power line detection system to reconstruct 3D power lines, but the method’s accuracy was low. Ganovelli et al. [42] presented a means of using a handful of images to reconstruct power lines. This means employed SFM. Because the method was based on specific assumptions, it could not be used in inspection tasks actually. Maurer et al. [43] presented a means that used semantic segmentation based on FCN and multiview geometry. The flowchart is shown in Figure 3. The method was effective but had high requirements on hardware. It is hard to match different images with the lack of effective point features of power lines by passive visual methods. Therefore, active visual methods are mostly employed in the power line reconstruction.Figure 3
The overview of the method based on FCN and multiview geometry [43].


## 2.2. The 3D Reconstruction of Ground Surface and Pylons

In addition to the reconstruction of power lines, the PTL corridors also need to reconstruct the ground surface and pylons. Nonetheless, their 3D reconstruction is in the preliminary stage. With regard to the ground surface, Maurer et al. [43] completed the 3D reconstruction of power lines and the ground surface simultaneously. In light of the reconstruction of pylons, Chen et al. [44] raised a model-driven approach to reconstruct pylons. The pylon could be divided into head, body, and foot. The three segments were reconstructed by different strategies. And they were assembled to the whole pylon by utilizing the direction and position. The result showed that the method’s accuracy could reach the centimeter level. Guo et al. [45] presented a means based on model library and stochastic geometry for pylon reconstruction. This method built an energy model of the correlation between point clouds and pylons. It also used simulated annealing and an MCMC sampler to reconstruct pylons. But this means required plenty of calculations.Recently, oblique photography has been applied to 3D reconstruction of PTL corridors. Xi et al. [46] designed a UAV power inspection system based on oblique photography. The system used the UAV to obtain multiview images of ground objects and accelerated the 3D reconstruction of the PTL corridors through data processing and model refinement. Pei et al. [47] designed a data acquisition device for 3D reconstruction of PTL corridors, which could obtain the structural information of the pylons’ four facades. Oblique photography has some defects in the field of PTL inspection. Oblique photography requires 60%-80% of the air belt overlap. Hence, it is necessary to collect data by multiple flights and the collection efficiency is low. Oblique photography is better for the 3D reconstruction of larger objects but ordinary for small and hollow objects such as power lines and pylons. Consequently, it is imperative to combine oblique photography with other means to gain good reconstruction results. It will be one of the research directions for the 3D reconstruction of PTL corridors in the future.


## 3. Object Detection of PTL Inspection

Object detection refers to a method of identifying and positioning targets from the background. In nearly two decades, the development of object detection is separated into the traditional detection period and deep learning-based detection period [48]. In the early stage of research on PTL inspection robots, traditional detection algorithms were used to identify and locate power components. With the performance of handcrafted features that tended to be stable, the research on traditional detection algorithms was almost stagnant. The development of inspection robots’ object detection fell into a bottleneck. On account of the emergence of R-CNN [49], the progress of object detection algorithms had taken a quantum leap. Neural networks are now used in object detection. They are able to boost detection performance and acquire robust characteristics. As a result, the research on object detection for PTL inspection robots has gradually changed from traditional detection algorithms to deep learning-based detection algorithms.


### 3.1. Traditional Object Detection Algorithms

Traditional object detection algorithms used on inspection robots were mostly foreground modeling-based methods over the last decades. They are classified as region selection, feature extraction, and classification. Sliding windows are used to acquire many candidate bounding boxes of the entire picture. And then, features of candidate bounding boxes are extracted. Classifiers, such as SVM [50], AdaBoost [51], and multilayer perceptron (MLP), are used to determine objects of candidate bounding boxes as the target or background. At last, NMS [52] is used to complete the detection. In the field of PTL inspection, SIFT [53], SURF [54], LBP [55], and HOG [56] are often used in traditional detection algorithms. Relevant research can be divided into two aspects: (a) pylons and (b) insulators and fittings.


#### 3.1.1. Pylons

While conducting pylon detection, pylons and birds’ nests are mainly detected. For pylons, Sampedro et al. [57] advanced a power pylon detection method based on HOG and MLP. Two MLPs were trained in this method. They were used for background-foreground segmentation and the classification of four types of pylons. The system architecture is shown in Figure 4. Cerón et al. [58] advanced a pylon detection method based on a line detection method and a grid of two-dimensional feature descriptors. Pylons could be characterized well by the descriptors. Wang et al. [59] used HOG and SVM to complete the rapid detection of pylons, but this method’s dismissal alarm rate was higher than the false alarm rate. Wang et al. [60] presented a method based on HOG for detecting the pylon components from far to near. This method used HOG features of the pylon in different orientations to train the MLP. It was suitable for pylon detection in an open environment. Aiming at the bird’s nest, Zhang et al. [61] raised the means based on the coarse-fine search tactics. It used HOG and AdaBoost to detect pylons’ position and roughly determined the candidate bounding boxes of birds’ nests. Then, color features were used to subtly detect birds’ nests. Xu et al. [62] proposed a method based on HSV and texture features. This method could eliminate much interference caused by areas, which were similar to the color and texture of the bird’s nest under the complex background.Figure 4
The system architecture: (a) pylon detection stage; (b) pylon classification stage [57].
(a)(b)


#### 3.1.2. Insulators and Fittings

In terms of insulators and fittings, many of the detection algorithms are designed for a single object, while only a few algorithms are for multiobjects. For insulators, Zhao et al. [63] came up with an insulator detection means based on SIFT and RANSAC. This method used RANSAC to remove abnormal SIFT features of insulators and completed the insulator detection through affine transformation. After that, Zhao and Liu [64] presented an approach based on SURF and intuitionistic fuzzy set (IFS). The steps are shown in Figure 5. The insulator’s SURF features were divided by IFS. Then, the connected regions of all categories were calculated to obtain the smallest circumscribed rectangle of each region for positioning. Prasad et al. [65] used SVM and local binary pattern histogram Fourier (LBP-HF) to complete health status detection of insulators, with a precision of 93.33%.Figure 5
The steps of the identification and localization of insulators [64].For fittings, Zhang et al. [66] proposed a fitting means based on HOG. This method used PCA to reduce the dimension of HOG features. Then, these features were identified by SVM. In contrast to other traditional algorithms, this means is more accurate. Zhang et al. [67] raised a method that could detect vibration dampers under the complex background. This method used Relief-F to weight and merge aggregate channel features (ACF) and complex frequency domain features. Ultimately, AdaBoost and NMS were used to complete the detection of vibration dampers. Feng et al. [68] employed HOG and SVM to accomplish the detection of bolts, but the accuracy of the means was low. Fan et al. [69] presented a method based on an improved HT for bolt detection. The peak selection strategy of HT was ameliorated for the improvement of detection accuracy.On the grounds of size, the targets of PTL can be classified into the pylon level, the device level, the part level, and the component level [70]. At the moment, object detection research is focused on targets of pylon level and device level. There is less research on detecting targets of the part level and component level. Traditional object detection algorithms applied to pylon level and device level generally use the framework of “feature expression” + “classifier”. Consequently, features and classifiers are vital for the precision of detecting different objects. In traditional algorithms, basic features usually cannot perfectly characterize the detected target. Features need to be improved as the detected object is replaced. This phenomenon is more obvious in the field of PTL inspection. There are nearly two thousand kinds of fittings in different specifications used in the field of PTL inspection. The features acquired from different regions cannot be unified. For the purpose of ensuring precision, it is imperative to design features for specific targets. Although the features designed for specific objects are intuitive and accessible, it has drawbacks such as low efficiency and poor generalization ability. In addition, tasks of PTL inspection require the robustness and real-time performance of algorithms. Facing the changes of light and viewing angle in the environment, handcrafted features are less robust. Besides, detection accuracy will be immensely influenced in the face of harsh operating environments. Moreover, handcrafted features are highly complex. And they are poorly real-time, which cannot satisfy needs of PTL tasks.


### 3.2. Deep Learning-Based Object Detection Algorithms

Powered by deep learning, R-CNN [49] broke the deadlock of stagnation of object detection. Plenty of similar approaches emerged in the following years, and they are also applied in PTL. These algorithms are classified as the “two-stage detection” and the “one-stage detection”. In “two-stage detection” algorithms, since the speed of Faster R-CNN is relatively quick [71, 72], it is often used in PTL inspection. In “one-stage detection” algorithms, the algorithms of the YOLO series [73–76] are used much more. Object detection algorithms, which are based on deep learning, can also be classified as two aspects like traditional algorithms.


#### 3.2.1. Pylons

In terms of pylons, Guo et al. [77] presented a real-time pylon detection model based on YOLO. The method used k-means to improve the anchor parameters in YOLO. Its detection rate was about twenty frames per second, and mAP was 94.09%. Wang et al. [78] verified the performance of different networks for detecting pylons. For birds’ nests, Shi et al. [79] proposed a bird’s nest detection model by employing RetinaNet [80]. This model was accurate but not real-time. No other neural network models had been tried as the backbone in this model, so there were some limitations in the experiment. Aiming at the same task, Wang et al. [81] proposed an approach, which was used to detect birds’ nests in a multiscale of high-voltage pylons. For enhancing the detection ability for birds’ nests, multiscale convolution features and region proposals were employed in this model, which was based on Faster R-CNN. The model was evaluated on a test set composed of two thousand images. And the average detection precision reached 84.55%.


#### 3.2.2. Insulators and Fittings

For insulators, Liu et al. [82] employed Faster R-CNN to detect insulators. The precision reached 94%. Zhao et al. [83] proposed an approach based on improved Faster R-CNN. This approach achieved better detection of insulators by improving NMS and anchor generation in RPN. It also solved the problem of missed detection caused by insulator shielding. Han et al. [84] designed a new neural network to detect insulators. There was high average precision but low real-time performance in the model.In terms of fittings, the detection based on deep learning is no longer limited to a single target. In recent years, research on multiobject detection has emerged. The targets include a variety of fittings and insulators. Lin et al. [85] advanced a means based on improved Faster R-CNN. The means could keep a good detection performance in different resolutions, angles, and positions. Dong [86] came up with a real-time multidevice detection method based on YOLOv3. Data augmentation was used to improve the model’s performance. And it had good average accuracy for insulators and vibration dampers. Yang et al. [87] presented an MSFF-KCD algorithm to detect multiscale devices. The precision of this means could maintain 86% on ARM devices, but the speed is not quick. In addition, Yang et al. [88] compared the performance of different backbone networks based on SSD [89] and used feature fusion to improve detection accuracy. For the purpose of detecting fittings, Qi et al. [90] presented an improved SSD model. This model improved the detection performance of small or dense targets in a complex background by improving Intersection over Union (IoU) and using the repulsion loss function. For the same task, Liu et al. [91] proposed a means based on R-FCN. Online hard example mining, sample adjusting, and soft NMS were employed to promote the performance of R-FCN.The above research of multidevice detection indicates that deep learning-based detection algorithms have a good generalization ability. They do without a complicated process of handcrafted features and can choose suitable and robust features to express targets. For multiscale problems, similar to the traditional algorithms, their performance is poor. So it is necessary to consider the use of multiscale technologies, such as improved anchor generation, SNIP [92], and TridentNet [93], to solve this problem. It is a good direction for the power line inspection study.Due to the confidentiality of power line inspection datasets, there is no one public dataset now, which hinders the progress of power line inspection. So only some typical deep learning-based detection algorithms for PTL inspection are listed in Table2 for comparing performance. In China, there are two power grid companies, which are State Grid Corporation of China and China Southern Power Grid. They have massive datasets of the power grid. Consequently, if they cooperatively create a public dataset, which combines data from power grids across the country, researchers could first use public datasets to testify the effectiveness of the means or the model proposed by themselves and show their results based on private datasets.Table 2
Typical deep learning-based detection algorithms for PTL inspection.
ReferenceObjectNetworkDatasetmAPGuo et al. [77]PylonYOLO+anchor cluster based onk-meansTraining set: 7636 images, test set: 815 images88.68%Wang et al. [81]Bird’s nestFaster R-CNN+modified ResNet-50Training set: 9000 images, test set: 3000 images94.09%Han et al. [84]InsulatorA new model based on modified ResNet-50Training set: 2675 images, test set: 1356 images98.30%Yang et al. [88]FittingsMSFF-KCDTraining set: 3440 images, test set: 382 images90.80%Although it is impossible to accurately compare the performance between these models, it can be seen that the performance is improved due to new modified models. However, there do not exist neural network models, which achieve simultaneously high precision and speed, in object detection. Just as these two series, the precision and the speed are focused on severally. The improved algorithms based on these two series often sacrifice real time to improve detection accuracy. Now researchers always use YOLO series to detect objects because of its excellent real-time performance. Nevertheless, PTL inspection should be able to fulfill the high requirements in both aspects. Thus, many improved algorithms are only in the research stage. Moreover, limited by computational capabilities of embedded platforms, they cannot be actually used in engineering. Besides, the deep learning model used in engineering needs to be capable of training with new data, but little research is ongoing in this area.


## 3.1. Traditional Object Detection Algorithms

Traditional object detection algorithms used on inspection robots were mostly foreground modeling-based methods over the last decades. They are classified as region selection, feature extraction, and classification. Sliding windows are used to acquire many candidate bounding boxes of the entire picture. And then, features of candidate bounding boxes are extracted. Classifiers, such as SVM [50], AdaBoost [51], and multilayer perceptron (MLP), are used to determine objects of candidate bounding boxes as the target or background. At last, NMS [52] is used to complete the detection. In the field of PTL inspection, SIFT [53], SURF [54], LBP [55], and HOG [56] are often used in traditional detection algorithms. Relevant research can be divided into two aspects: (a) pylons and (b) insulators and fittings.


### 3.1.1. Pylons

While conducting pylon detection, pylons and birds’ nests are mainly detected. For pylons, Sampedro et al. [57] advanced a power pylon detection method based on HOG and MLP. Two MLPs were trained in this method. They were used for background-foreground segmentation and the classification of four types of pylons. The system architecture is shown in Figure 4. Cerón et al. [58] advanced a pylon detection method based on a line detection method and a grid of two-dimensional feature descriptors. Pylons could be characterized well by the descriptors. Wang et al. [59] used HOG and SVM to complete the rapid detection of pylons, but this method’s dismissal alarm rate was higher than the false alarm rate. Wang et al. [60] presented a method based on HOG for detecting the pylon components from far to near. This method used HOG features of the pylon in different orientations to train the MLP. It was suitable for pylon detection in an open environment. Aiming at the bird’s nest, Zhang et al. [61] raised the means based on the coarse-fine search tactics. It used HOG and AdaBoost to detect pylons’ position and roughly determined the candidate bounding boxes of birds’ nests. Then, color features were used to subtly detect birds’ nests. Xu et al. [62] proposed a method based on HSV and texture features. This method could eliminate much interference caused by areas, which were similar to the color and texture of the bird’s nest under the complex background.Figure 4
The system architecture: (a) pylon detection stage; (b) pylon classification stage [57].
(a)(b)


### 3.1.2. Insulators and Fittings

In terms of insulators and fittings, many of the detection algorithms are designed for a single object, while only a few algorithms are for multiobjects. For insulators, Zhao et al. [63] came up with an insulator detection means based on SIFT and RANSAC. This method used RANSAC to remove abnormal SIFT features of insulators and completed the insulator detection through affine transformation. After that, Zhao and Liu [64] presented an approach based on SURF and intuitionistic fuzzy set (IFS). The steps are shown in Figure 5. The insulator’s SURF features were divided by IFS. Then, the connected regions of all categories were calculated to obtain the smallest circumscribed rectangle of each region for positioning. Prasad et al. [65] used SVM and local binary pattern histogram Fourier (LBP-HF) to complete health status detection of insulators, with a precision of 93.33%.Figure 5
The steps of the identification and localization of insulators [64].For fittings, Zhang et al. [66] proposed a fitting means based on HOG. This method used PCA to reduce the dimension of HOG features. Then, these features were identified by SVM. In contrast to other traditional algorithms, this means is more accurate. Zhang et al. [67] raised a method that could detect vibration dampers under the complex background. This method used Relief-F to weight and merge aggregate channel features (ACF) and complex frequency domain features. Ultimately, AdaBoost and NMS were used to complete the detection of vibration dampers. Feng et al. [68] employed HOG and SVM to accomplish the detection of bolts, but the accuracy of the means was low. Fan et al. [69] presented a method based on an improved HT for bolt detection. The peak selection strategy of HT was ameliorated for the improvement of detection accuracy.On the grounds of size, the targets of PTL can be classified into the pylon level, the device level, the part level, and the component level [70]. At the moment, object detection research is focused on targets of pylon level and device level. There is less research on detecting targets of the part level and component level. Traditional object detection algorithms applied to pylon level and device level generally use the framework of “feature expression” + “classifier”. Consequently, features and classifiers are vital for the precision of detecting different objects. In traditional algorithms, basic features usually cannot perfectly characterize the detected target. Features need to be improved as the detected object is replaced. This phenomenon is more obvious in the field of PTL inspection. There are nearly two thousand kinds of fittings in different specifications used in the field of PTL inspection. The features acquired from different regions cannot be unified. For the purpose of ensuring precision, it is imperative to design features for specific targets. Although the features designed for specific objects are intuitive and accessible, it has drawbacks such as low efficiency and poor generalization ability. In addition, tasks of PTL inspection require the robustness and real-time performance of algorithms. Facing the changes of light and viewing angle in the environment, handcrafted features are less robust. Besides, detection accuracy will be immensely influenced in the face of harsh operating environments. Moreover, handcrafted features are highly complex. And they are poorly real-time, which cannot satisfy needs of PTL tasks.


## 3.1.1. Pylons

While conducting pylon detection, pylons and birds’ nests are mainly detected. For pylons, Sampedro et al. [57] advanced a power pylon detection method based on HOG and MLP. Two MLPs were trained in this method. They were used for background-foreground segmentation and the classification of four types of pylons. The system architecture is shown in Figure 4. Cerón et al. [58] advanced a pylon detection method based on a line detection method and a grid of two-dimensional feature descriptors. Pylons could be characterized well by the descriptors. Wang et al. [59] used HOG and SVM to complete the rapid detection of pylons, but this method’s dismissal alarm rate was higher than the false alarm rate. Wang et al. [60] presented a method based on HOG for detecting the pylon components from far to near. This method used HOG features of the pylon in different orientations to train the MLP. It was suitable for pylon detection in an open environment. Aiming at the bird’s nest, Zhang et al. [61] raised the means based on the coarse-fine search tactics. It used HOG and AdaBoost to detect pylons’ position and roughly determined the candidate bounding boxes of birds’ nests. Then, color features were used to subtly detect birds’ nests. Xu et al. [62] proposed a method based on HSV and texture features. This method could eliminate much interference caused by areas, which were similar to the color and texture of the bird’s nest under the complex background.Figure 4
The system architecture: (a) pylon detection stage; (b) pylon classification stage [57].
(a)(b)


## 3.1.2. Insulators and Fittings

In terms of insulators and fittings, many of the detection algorithms are designed for a single object, while only a few algorithms are for multiobjects. For insulators, Zhao et al. [63] came up with an insulator detection means based on SIFT and RANSAC. This method used RANSAC to remove abnormal SIFT features of insulators and completed the insulator detection through affine transformation. After that, Zhao and Liu [64] presented an approach based on SURF and intuitionistic fuzzy set (IFS). The steps are shown in Figure 5. The insulator’s SURF features were divided by IFS. Then, the connected regions of all categories were calculated to obtain the smallest circumscribed rectangle of each region for positioning. Prasad et al. [65] used SVM and local binary pattern histogram Fourier (LBP-HF) to complete health status detection of insulators, with a precision of 93.33%.Figure 5
The steps of the identification and localization of insulators [64].For fittings, Zhang et al. [66] proposed a fitting means based on HOG. This method used PCA to reduce the dimension of HOG features. Then, these features were identified by SVM. In contrast to other traditional algorithms, this means is more accurate. Zhang et al. [67] raised a method that could detect vibration dampers under the complex background. This method used Relief-F to weight and merge aggregate channel features (ACF) and complex frequency domain features. Ultimately, AdaBoost and NMS were used to complete the detection of vibration dampers. Feng et al. [68] employed HOG and SVM to accomplish the detection of bolts, but the accuracy of the means was low. Fan et al. [69] presented a method based on an improved HT for bolt detection. The peak selection strategy of HT was ameliorated for the improvement of detection accuracy.On the grounds of size, the targets of PTL can be classified into the pylon level, the device level, the part level, and the component level [70]. At the moment, object detection research is focused on targets of pylon level and device level. There is less research on detecting targets of the part level and component level. Traditional object detection algorithms applied to pylon level and device level generally use the framework of “feature expression” + “classifier”. Consequently, features and classifiers are vital for the precision of detecting different objects. In traditional algorithms, basic features usually cannot perfectly characterize the detected target. Features need to be improved as the detected object is replaced. This phenomenon is more obvious in the field of PTL inspection. There are nearly two thousand kinds of fittings in different specifications used in the field of PTL inspection. The features acquired from different regions cannot be unified. For the purpose of ensuring precision, it is imperative to design features for specific targets. Although the features designed for specific objects are intuitive and accessible, it has drawbacks such as low efficiency and poor generalization ability. In addition, tasks of PTL inspection require the robustness and real-time performance of algorithms. Facing the changes of light and viewing angle in the environment, handcrafted features are less robust. Besides, detection accuracy will be immensely influenced in the face of harsh operating environments. Moreover, handcrafted features are highly complex. And they are poorly real-time, which cannot satisfy needs of PTL tasks.


## 3.2. Deep Learning-Based Object Detection Algorithms

Powered by deep learning, R-CNN [49] broke the deadlock of stagnation of object detection. Plenty of similar approaches emerged in the following years, and they are also applied in PTL. These algorithms are classified as the “two-stage detection” and the “one-stage detection”. In “two-stage detection” algorithms, since the speed of Faster R-CNN is relatively quick [71, 72], it is often used in PTL inspection. In “one-stage detection” algorithms, the algorithms of the YOLO series [73–76] are used much more. Object detection algorithms, which are based on deep learning, can also be classified as two aspects like traditional algorithms.


### 3.2.1. Pylons

In terms of pylons, Guo et al. [77] presented a real-time pylon detection model based on YOLO. The method used k-means to improve the anchor parameters in YOLO. Its detection rate was about twenty frames per second, and mAP was 94.09%. Wang et al. [78] verified the performance of different networks for detecting pylons. For birds’ nests, Shi et al. [79] proposed a bird’s nest detection model by employing RetinaNet [80]. This model was accurate but not real-time. No other neural network models had been tried as the backbone in this model, so there were some limitations in the experiment. Aiming at the same task, Wang et al. [81] proposed an approach, which was used to detect birds’ nests in a multiscale of high-voltage pylons. For enhancing the detection ability for birds’ nests, multiscale convolution features and region proposals were employed in this model, which was based on Faster R-CNN. The model was evaluated on a test set composed of two thousand images. And the average detection precision reached 84.55%.


### 3.2.2. Insulators and Fittings

For insulators, Liu et al. [82] employed Faster R-CNN to detect insulators. The precision reached 94%. Zhao et al. [83] proposed an approach based on improved Faster R-CNN. This approach achieved better detection of insulators by improving NMS and anchor generation in RPN. It also solved the problem of missed detection caused by insulator shielding. Han et al. [84] designed a new neural network to detect insulators. There was high average precision but low real-time performance in the model.In terms of fittings, the detection based on deep learning is no longer limited to a single target. In recent years, research on multiobject detection has emerged. The targets include a variety of fittings and insulators. Lin et al. [85] advanced a means based on improved Faster R-CNN. The means could keep a good detection performance in different resolutions, angles, and positions. Dong [86] came up with a real-time multidevice detection method based on YOLOv3. Data augmentation was used to improve the model’s performance. And it had good average accuracy for insulators and vibration dampers. Yang et al. [87] presented an MSFF-KCD algorithm to detect multiscale devices. The precision of this means could maintain 86% on ARM devices, but the speed is not quick. In addition, Yang et al. [88] compared the performance of different backbone networks based on SSD [89] and used feature fusion to improve detection accuracy. For the purpose of detecting fittings, Qi et al. [90] presented an improved SSD model. This model improved the detection performance of small or dense targets in a complex background by improving Intersection over Union (IoU) and using the repulsion loss function. For the same task, Liu et al. [91] proposed a means based on R-FCN. Online hard example mining, sample adjusting, and soft NMS were employed to promote the performance of R-FCN.The above research of multidevice detection indicates that deep learning-based detection algorithms have a good generalization ability. They do without a complicated process of handcrafted features and can choose suitable and robust features to express targets. For multiscale problems, similar to the traditional algorithms, their performance is poor. So it is necessary to consider the use of multiscale technologies, such as improved anchor generation, SNIP [92], and TridentNet [93], to solve this problem. It is a good direction for the power line inspection study.Due to the confidentiality of power line inspection datasets, there is no one public dataset now, which hinders the progress of power line inspection. So only some typical deep learning-based detection algorithms for PTL inspection are listed in Table2 for comparing performance. In China, there are two power grid companies, which are State Grid Corporation of China and China Southern Power Grid. They have massive datasets of the power grid. Consequently, if they cooperatively create a public dataset, which combines data from power grids across the country, researchers could first use public datasets to testify the effectiveness of the means or the model proposed by themselves and show their results based on private datasets.Table 2
Typical deep learning-based detection algorithms for PTL inspection.
ReferenceObjectNetworkDatasetmAPGuo et al. [77]PylonYOLO+anchor cluster based onk-meansTraining set: 7636 images, test set: 815 images88.68%Wang et al. [81]Bird’s nestFaster R-CNN+modified ResNet-50Training set: 9000 images, test set: 3000 images94.09%Han et al. [84]InsulatorA new model based on modified ResNet-50Training set: 2675 images, test set: 1356 images98.30%Yang et al. [88]FittingsMSFF-KCDTraining set: 3440 images, test set: 382 images90.80%Although it is impossible to accurately compare the performance between these models, it can be seen that the performance is improved due to new modified models. However, there do not exist neural network models, which achieve simultaneously high precision and speed, in object detection. Just as these two series, the precision and the speed are focused on severally. The improved algorithms based on these two series often sacrifice real time to improve detection accuracy. Now researchers always use YOLO series to detect objects because of its excellent real-time performance. Nevertheless, PTL inspection should be able to fulfill the high requirements in both aspects. Thus, many improved algorithms are only in the research stage. Moreover, limited by computational capabilities of embedded platforms, they cannot be actually used in engineering. Besides, the deep learning model used in engineering needs to be capable of training with new data, but little research is ongoing in this area.


## 3.2.1. Pylons

In terms of pylons, Guo et al. [77] presented a real-time pylon detection model based on YOLO. The method used k-means to improve the anchor parameters in YOLO. Its detection rate was about twenty frames per second, and mAP was 94.09%. Wang et al. [78] verified the performance of different networks for detecting pylons. For birds’ nests, Shi et al. [79] proposed a bird’s nest detection model by employing RetinaNet [80]. This model was accurate but not real-time. No other neural network models had been tried as the backbone in this model, so there were some limitations in the experiment. Aiming at the same task, Wang et al. [81] proposed an approach, which was used to detect birds’ nests in a multiscale of high-voltage pylons. For enhancing the detection ability for birds’ nests, multiscale convolution features and region proposals were employed in this model, which was based on Faster R-CNN. The model was evaluated on a test set composed of two thousand images. And the average detection precision reached 84.55%.


## 3.2.2. Insulators and Fittings

For insulators, Liu et al. [82] employed Faster R-CNN to detect insulators. The precision reached 94%. Zhao et al. [83] proposed an approach based on improved Faster R-CNN. This approach achieved better detection of insulators by improving NMS and anchor generation in RPN. It also solved the problem of missed detection caused by insulator shielding. Han et al. [84] designed a new neural network to detect insulators. There was high average precision but low real-time performance in the model.In terms of fittings, the detection based on deep learning is no longer limited to a single target. In recent years, research on multiobject detection has emerged. The targets include a variety of fittings and insulators. Lin et al. [85] advanced a means based on improved Faster R-CNN. The means could keep a good detection performance in different resolutions, angles, and positions. Dong [86] came up with a real-time multidevice detection method based on YOLOv3. Data augmentation was used to improve the model’s performance. And it had good average accuracy for insulators and vibration dampers. Yang et al. [87] presented an MSFF-KCD algorithm to detect multiscale devices. The precision of this means could maintain 86% on ARM devices, but the speed is not quick. In addition, Yang et al. [88] compared the performance of different backbone networks based on SSD [89] and used feature fusion to improve detection accuracy. For the purpose of detecting fittings, Qi et al. [90] presented an improved SSD model. This model improved the detection performance of small or dense targets in a complex background by improving Intersection over Union (IoU) and using the repulsion loss function. For the same task, Liu et al. [91] proposed a means based on R-FCN. Online hard example mining, sample adjusting, and soft NMS were employed to promote the performance of R-FCN.The above research of multidevice detection indicates that deep learning-based detection algorithms have a good generalization ability. They do without a complicated process of handcrafted features and can choose suitable and robust features to express targets. For multiscale problems, similar to the traditional algorithms, their performance is poor. So it is necessary to consider the use of multiscale technologies, such as improved anchor generation, SNIP [92], and TridentNet [93], to solve this problem. It is a good direction for the power line inspection study.Due to the confidentiality of power line inspection datasets, there is no one public dataset now, which hinders the progress of power line inspection. So only some typical deep learning-based detection algorithms for PTL inspection are listed in Table2 for comparing performance. In China, there are two power grid companies, which are State Grid Corporation of China and China Southern Power Grid. They have massive datasets of the power grid. Consequently, if they cooperatively create a public dataset, which combines data from power grids across the country, researchers could first use public datasets to testify the effectiveness of the means or the model proposed by themselves and show their results based on private datasets.Table 2
Typical deep learning-based detection algorithms for PTL inspection.
ReferenceObjectNetworkDatasetmAPGuo et al. [77]PylonYOLO+anchor cluster based onk-meansTraining set: 7636 images, test set: 815 images88.68%Wang et al. [81]Bird’s nestFaster R-CNN+modified ResNet-50Training set: 9000 images, test set: 3000 images94.09%Han et al. [84]InsulatorA new model based on modified ResNet-50Training set: 2675 images, test set: 1356 images98.30%Yang et al. [88]FittingsMSFF-KCDTraining set: 3440 images, test set: 382 images90.80%Although it is impossible to accurately compare the performance between these models, it can be seen that the performance is improved due to new modified models. However, there do not exist neural network models, which achieve simultaneously high precision and speed, in object detection. Just as these two series, the precision and the speed are focused on severally. The improved algorithms based on these two series often sacrifice real time to improve detection accuracy. Now researchers always use YOLO series to detect objects because of its excellent real-time performance. Nevertheless, PTL inspection should be able to fulfill the high requirements in both aspects. Thus, many improved algorithms are only in the research stage. Moreover, limited by computational capabilities of embedded platforms, they cannot be actually used in engineering. Besides, the deep learning model used in engineering needs to be capable of training with new data, but little research is ongoing in this area.


## 4. Visual Servo of PTL Inspection

The visual servo system takes visual information as feedback. After the relative pose between the robot and the target is computed by visual information, it can realize the robot’s pose control. Three classification standards including types of feedback [95–97], camera location [98], and calibration requirements [99] are commonly used in visual servo systems. Among them, PBVS and IBVS are usually employed in the tasks of PTL inspection. The former is based on position, and the latter is based on images. According to operation modes, inspection robots can be divided into climbing robots, UAVs, and hybrid platforms [94], as shown in Figure 6. Research on hybrid platforms is not yet mature, and visual servo control solutions are only used in the first two types of inspection robots.Figure 6
The PTL inspection robots: (a) LineScout (climbing robots); (b) SmartCopter (UAVs); (c) LineDrone (hybrid platforms) [94].
(a)(b)(c)


### 4.1. Servo Control of Climbing Robots

The visual servo control of climbing robots is most applied to autonomous line-grasping control in the process of bypassing obstacles. And visual servo is less used in other aspects of control.In terms of autonomous line grabbing, Wang et al. [100] designed a single-camera-based visual servo control method for line grasping. Although this method could meet the line-grasping requirements, it mainly depended on the mechanical structure of the robot. Zhang et al. [101] designed an IBVS controller for getting across obstacles. The image processing of this scheme was complicated, and it was sensitive to light changes. After that, Zhao et al. [102] proposed a means of combining remote control and IBVS. This method was effectively carried out in the laboratory environment. Guo et al. [103] raised an IBVS method for autonomous line-grasping control. This method used a 2D fuzzy controller. The final deflection error of a grasping line was within two degrees, and the time was less than forty seconds. Wang et al. [104] presented a line-grasping control based on hand-eye-vision. This method used a 2D fuzzy controller as shown in Figure 7. The impacts of changes in lighting, background, and other factors on the line-grasping accuracy were considered comprehensively. And it consumed twenty seconds. Aiming at a new dual-arm inspection robot designed by themselves, He et al. [105] proposed an adaptive visual servo method. This method could provide good characteristics of power lines without hand-eye calibration, and it improved the robustness of autonomously grasping the line.Figure 7
Visual servo fuzzy controller [104].The above-mentioned studies are aimed at the line-grasping problem of dual-arm PTL inspection robots, and many of them use IBVS. But the inspection robots only complete the autonomous line-grasping function in the laboratory environment, and they are not used in actual application scenarios. The problems of image noise, model errors, control delay, and environmental changes are not considered, which may be met in the actual scenarios. Besides, it takes too long for the inspection robots to grasp the line and the efficiency is low. Compared with traditional visual servo, uncalibrated visual servo has higher flexibility and adaptability, which is more suitable for inspection robots. Therefore, uncalibrated visual servo of inspection robots will be the future research direction.In other ways, aiming at the problem of autodocking charging control for inspection robots, Wu et al. [107] used IBVS for precise positioning control. A variable universe fuzzy control was employed to overcome the difficulty of insufficient control precision. Besides, Jiang et al. [106] presented a double closed-loop autonomous localization control means. It originated from the BP network and visual servo. The BP network and visual servo were, respectively, used to solve the coarse and fine positioning of the manipulator. This method could effectually boost the automation level of inspection robots, but it took too long to complete the bolt alignment. The control architecture diagram of the method is shown in Figure 8.Figure 8
The control architecture diagram of the method based on the BP network and visual servo [106].Intelligent methods have begun to be used in the visual servo of inspection robots. However, they are not fully used in visual servo because of calculation drawbacks of intelligent means. Besides, many of the above-mentioned visual servo studies only concentrate on one task. However, there were many maintenance tasks on the PTL. The visual servo scheme based on single-task control has poor applicability, so it is necessary to focus on the research of multitask visual servo control.


### 4.2. Servo Control of UAVs

Compared with the visual servo control of climbing robots, the research scope of UAVs’ visual servo control is more limited. Currently, only visual servo control of tracking power lines is available.Aiming at the problem of tracking power lines, Araar and Aouf [108] of Cranfield University proposed two solutions, which, respectively, were an IBVS method combined with the LQ-Servo and a partial PBVS (PPBVS) method. There was a better effect of PPBVS by contrast. However, this method was greatly affected by calibration errors. Xie et al. [109] raised a novel IBVS method. It could be robust to camera calibration errors without depth estimation. However, it was only in the simulation stage. Aiming at the problem of fixed-wing UAV tracking targets in the presence of wind, Mills et al. [110] designed an IBVS method. It considered the wind correction angle as the desired line angle. The result showed that it could boost the tracking response of UAVs. Rafique and Lynch [111] used IBVS combined with output feedback for the motion control of UAVs. Taking into account uncertain factors such as linear acceleration disturbance and quality, it adopted the inner-outer loop structure. The result showed that the stability of the structure is good. However, it was not tested in the outdoor PTL environment.Many studies of visual servo control of UAVs are in the simulation stage. UAVs are notably influenced by environmental factors. The influence of wind on the motion control of UAVs needs to be considered in tracking power lines. Besides, compared to climbing robots, the inspection speed of UAVs is faster. It is necessary to boost response speed of servo during tracking. On the basis of tracking power lines, UAVs also need to perform path planning. So path planning for visual servo of UAVs is a good research direction.


## 4.1. Servo Control of Climbing Robots

The visual servo control of climbing robots is most applied to autonomous line-grasping control in the process of bypassing obstacles. And visual servo is less used in other aspects of control.In terms of autonomous line grabbing, Wang et al. [100] designed a single-camera-based visual servo control method for line grasping. Although this method could meet the line-grasping requirements, it mainly depended on the mechanical structure of the robot. Zhang et al. [101] designed an IBVS controller for getting across obstacles. The image processing of this scheme was complicated, and it was sensitive to light changes. After that, Zhao et al. [102] proposed a means of combining remote control and IBVS. This method was effectively carried out in the laboratory environment. Guo et al. [103] raised an IBVS method for autonomous line-grasping control. This method used a 2D fuzzy controller. The final deflection error of a grasping line was within two degrees, and the time was less than forty seconds. Wang et al. [104] presented a line-grasping control based on hand-eye-vision. This method used a 2D fuzzy controller as shown in Figure 7. The impacts of changes in lighting, background, and other factors on the line-grasping accuracy were considered comprehensively. And it consumed twenty seconds. Aiming at a new dual-arm inspection robot designed by themselves, He et al. [105] proposed an adaptive visual servo method. This method could provide good characteristics of power lines without hand-eye calibration, and it improved the robustness of autonomously grasping the line.Figure 7
Visual servo fuzzy controller [104].The above-mentioned studies are aimed at the line-grasping problem of dual-arm PTL inspection robots, and many of them use IBVS. But the inspection robots only complete the autonomous line-grasping function in the laboratory environment, and they are not used in actual application scenarios. The problems of image noise, model errors, control delay, and environmental changes are not considered, which may be met in the actual scenarios. Besides, it takes too long for the inspection robots to grasp the line and the efficiency is low. Compared with traditional visual servo, uncalibrated visual servo has higher flexibility and adaptability, which is more suitable for inspection robots. Therefore, uncalibrated visual servo of inspection robots will be the future research direction.In other ways, aiming at the problem of autodocking charging control for inspection robots, Wu et al. [107] used IBVS for precise positioning control. A variable universe fuzzy control was employed to overcome the difficulty of insufficient control precision. Besides, Jiang et al. [106] presented a double closed-loop autonomous localization control means. It originated from the BP network and visual servo. The BP network and visual servo were, respectively, used to solve the coarse and fine positioning of the manipulator. This method could effectually boost the automation level of inspection robots, but it took too long to complete the bolt alignment. The control architecture diagram of the method is shown in Figure 8.Figure 8
The control architecture diagram of the method based on the BP network and visual servo [106].Intelligent methods have begun to be used in the visual servo of inspection robots. However, they are not fully used in visual servo because of calculation drawbacks of intelligent means. Besides, many of the above-mentioned visual servo studies only concentrate on one task. However, there were many maintenance tasks on the PTL. The visual servo scheme based on single-task control has poor applicability, so it is necessary to focus on the research of multitask visual servo control.


## 4.2. Servo Control of UAVs

Compared with the visual servo control of climbing robots, the research scope of UAVs’ visual servo control is more limited. Currently, only visual servo control of tracking power lines is available.Aiming at the problem of tracking power lines, Araar and Aouf [108] of Cranfield University proposed two solutions, which, respectively, were an IBVS method combined with the LQ-Servo and a partial PBVS (PPBVS) method. There was a better effect of PPBVS by contrast. However, this method was greatly affected by calibration errors. Xie et al. [109] raised a novel IBVS method. It could be robust to camera calibration errors without depth estimation. However, it was only in the simulation stage. Aiming at the problem of fixed-wing UAV tracking targets in the presence of wind, Mills et al. [110] designed an IBVS method. It considered the wind correction angle as the desired line angle. The result showed that it could boost the tracking response of UAVs. Rafique and Lynch [111] used IBVS combined with output feedback for the motion control of UAVs. Taking into account uncertain factors such as linear acceleration disturbance and quality, it adopted the inner-outer loop structure. The result showed that the stability of the structure is good. However, it was not tested in the outdoor PTL environment.Many studies of visual servo control of UAVs are in the simulation stage. UAVs are notably influenced by environmental factors. The influence of wind on the motion control of UAVs needs to be considered in tracking power lines. Besides, compared to climbing robots, the inspection speed of UAVs is faster. It is necessary to boost response speed of servo during tracking. On the basis of tracking power lines, UAVs also need to perform path planning. So path planning for visual servo of UAVs is a good research direction.


## 5. Problems and Prospects

Many research institutions have started to study inspection robots globally. However, the research of inspection robots is only in the preliminary stage. Some key problems remain unsolved in terms of 3D reconstruction, object detection, and visual servo control. At the same time, the research of inspection robots in other areas still needs to develop.


### 5.1. Pivotal Technical Problems

It is effective to use inspection robots to replace manual inspection. And there are clear application requirements of PTL inspection. Hence, there is a very important engineering value to develop an inspection robot system that can perform efficient operations in a complex PTL environment. At present, studies concentrates on mechanical design, environmental perception, and visual servo control in the field of PTL inspection. Nevertheless, many of the results are mechanical design-related, and few results are given in other areas. Therefore, for boosting the efficiency and operation precision of inspection robots, the following problems need to be solved.


#### 5.1.1. The Problem of Environmental Perception

Both the high-altitude environment and natural interference factors pose challenges to inspection robots. In this scenario, highly accurate environmental information is imperative for task conduction. The use of a single sensor will inevitably have perceptual limitations. For example, illumination variations affect the camera and fog affects the LiDAR. When power devices are aging or damaged, fault areas on the power devices can be hardly recognized with naked eyes, but heat or discharge will occur in the fault area. These phenomena can be detected by infrared thermal devices and UV cameras. Therefore, multisensor information fusion is indispensable for PTL inspection. As shown in Figure9, it can be carried out from the following steps:Figure 9
The flowchart of multisensor information fusion.3D fusion reconstruction of transmission line corridors is used. Firstly, a rough 3D model of the transmission line corridor is constructed by oblique photography. The data registration is completed by the ICP algorithm to form a detailed 3D model. The fault areas are approximately found by the detailed 3D model.The data fusion of infrared, visible light, and ultraviolet images is employed. After determining approximate fault areas and time synchronization, these sensors are used to shoot for image fusion. At present, image fusion still rests on the pixel stage. Feature fusion should be further considered for it.The accurate coordinate of discharge, hotspot, and visible defects in fault areas are gained by object detection. The spatial conversion relations are gained by the calibration of the visible-light camera and LiDAR. For getting 3D information, the precise coordinate of the defect area is projected into point clouds according to spatial conversion relations. Then, the 3D information is used for local 3D reconstruction.Visual servo control is applied. The current pose of the robot and objects can be obtained by local 3D reconstruction. After that, it can be provided to a 3D visual servo controller to complete the control.


#### 5.1.2. The Problem of Datasets Required for Deep Learning

With the development of AI, a growing number of fields commence research by employing deep learning. Deep learning algorithms rely on data, but there is no standard power inspection dataset for researchers to conduct algorithm research. Therefore, the establishment of a standard power inspection dataset for deep learning development promotion is urgently needed. It can be constructed from the following steps:Standards of inspection images and power devices are made. Aiming at different defects of power devices, a unified defect description and classification rules should be established. The size and sharpness of inspection images should be normalized. Standards provide convenience when instructing personnel in data collection and highly improve the dataset construction.Datasets of PTL inspection should be divided according to various regions. The geographical environment and weather conditions of each region are different. PTL will be affected by a distinct climate, which makes the type of the same power device be disparate. Different types of power devices may vary greatly in appearance and shape, which results in regional characteristics. Therefore, there are regional characteristics in PTL corridors. In addition, the background of various regions differs greatly. Hence, it is necessary to divide the whole dataset first according to the territories.According to different kinds of power devices and degrees of defects, datasets of regional PTL inspection should be further divided into different subset datasets. According to specific power devices in PTL, it is helpful to form the corresponding PTL dataset by subset datasets, which can represent the characteristics greatly and be helpful for training models. The steps are shown in Figure10.Figure 10
The steps of power inspection dataset establishment.Now, a good deep learning model is always based on high-quality annotated data. However, it is expensive to collect data, especially annotated data. In the field of PTL inspection, this phenomenon is more obvious. Consequently, researchers try to employ weakly supervised data to train models [112, 113]. Furthermore, they employ unsupervised learning to annotate data. These methods can be also used in object detection of PTL inspection [114] for reducing the cost of data annotation. It will be a good research direction of PTL inspection.


#### 5.1.3. The Problem of Applying a Light Network Model Suitable for Embedded Platforms

The training of models usually runs on servers. The computational ability should be considered when deep learning-based object algorithms are used. Therefore, setting up a light-weight network in advance is significant. Then the model, transferred from the server to the embedded platform, can be additionally trained in actual scenarios. At present, different backbones such as SSD and YOLOv4 can be used in MobileNetv3. Furthermore, fine-tuning can also be employed.


#### 5.1.4. The Problem of Control Efficiency and Robustness of the Visual Servo System

Now, the whole control process of grasping the line or tracking lines takes much time. And the efficiency is low. In this regard, improving visual sampling frequency and image processing speed is able to ameliorate its real-time capability. In addition, subtle structural changes of inspection robots may occur due to wind and drag of vibration dampers, which leads to errors. The uncalibrated visual servo has a high tolerance for these errors. Therefore, it can be adopted for improving the robustness.


### 5.2. Prospects

In the application scenarios of PTL inspection, not only the foundation capabilities of inspection robots but also further expansion should be realized according to requirements. It is helpful to realize the vital engineering value of inspection robots. The facets of future directions are as follows:


#### 5.2.1. The Development of Mechanical Design

The core of inspection robots is the mechanical design. And a good one can greatly improve the efficiency. For example, compared to LineScout [115], LineRanger [116] takes much less time to cross obstacles because of the ingenious and passive obstacle-crossing mechanism. However, the mechanical design of PTL inspection robots is not refined now, and robots cannot balance inspection and operation. So far, in contrast to other inspection robots, climbing robots are more practical due to their in-contact measurements. The engineering value of climbing robots is greater when they can succeed in crossing pylons. So the mechanical design of inspection robots still needs to be improved.


#### 5.2.2. The Development of Energy Harvesting

Battery capacities directly influence the duration of PTL inspection. Regarding a certain load capacity of robots, a high-capacity battery, such as lithium battery, can increase maximum endurance of robots, but it limits the weight of other mechanical components. So online charging is important. For increasing the inspection endurance, inspection robots can acquire electric energy from power line autonomously. So an induction power supply system should be developed.


#### 5.2.3. The Development of Electromagnetic Protection

EMI on inspection robots is more intense with increasing voltage, especially the transient discharge process of the equipotential working. However, the present inspection robots are weak to EMI. Therefore, it is an inevitable problem during the inspection. The entire robot should conduct electricity for avoiding electrical discharge within components of robots. Shielding is an effective protection measurement aimed at EMI, and high-quality shields should be designed. However, inspection robots cannot be fully shielded, which means that other optimization technologies, such as routing, filtering, and grounding techniques, should be considered. Consequently, how to design electromagnetic protection is a crucial challenge.


#### 5.2.4. The Development of Modularity

It is inevitable that inspection robots will be damaged in a bad working environment. And the modularity development of inspection robots is significant for rapid maintenance. The modularity of inspection robot is convenient for the maintainer to find the fault location and replace the fault module of robots. Moreover, it is easy to install and disassemble inspection robots by modularity, which also improves the efficiency of online and offline switchover. Furthermore, it helps to switch robot configurations in different inspection scenarios, such as overhead ground wire and multibundled conductors.


#### 5.2.5. The Development of Reliability

The working environment of inspection robots is harsh, such as the power line area with long span and steep slope. There is a requirement for good robustness to accurately complete the task. Consequently, inspection robots should have the ability to cross the most complex area in PTL. For the purpose of satisfying the needs of multiple and complicated tasks, effective load capacity and operation control technology should be improved. What is more, before online operation, robots should be tested for endurance, such as wear of the mechanical system, battery performance, and EMI robustness.


#### 5.2.6. The Development of Cooperation

This cooperation not only refers to the cooperation among a variety of inspection robots. It also includes the cooperation between inspection robots and the ground control system (GCS). The multirobot system can promote the performance of PTL inspection. In addition, in the case of teleoperation, the efficiency of inspection can be also affected by the cooperation between inspection robots and GCS. Appropriate deployments of GCS points in a long PTL corridor are the key to the cooperation between inspection robots and GCS. The cooperation includes real-time bidirectional data transmission, the energy consumption of disassembly, installation, and teleoperation.


#### 5.2.7. The Development of Friendly Operability and Autonomy

The final goal is to achieve autonomous patrol of inspection robots. However, autonomous control and teleoperation need to coexist in the process of development. Although inspection robots could automatically get across obstacles such as spacers and suspension clamps, there is a lack of autonomous ability in specific operations. The specific operations include dealing with damaged power lines and removing foreign bodies. For fulfilling tasks securely and briskly, teleoperation should be used. Meanwhile, the friendly man-machine interface of inspection robots should be developed for simple teleoperation.


## 5.1. Pivotal Technical Problems

It is effective to use inspection robots to replace manual inspection. And there are clear application requirements of PTL inspection. Hence, there is a very important engineering value to develop an inspection robot system that can perform efficient operations in a complex PTL environment. At present, studies concentrates on mechanical design, environmental perception, and visual servo control in the field of PTL inspection. Nevertheless, many of the results are mechanical design-related, and few results are given in other areas. Therefore, for boosting the efficiency and operation precision of inspection robots, the following problems need to be solved.


### 5.1.1. The Problem of Environmental Perception

Both the high-altitude environment and natural interference factors pose challenges to inspection robots. In this scenario, highly accurate environmental information is imperative for task conduction. The use of a single sensor will inevitably have perceptual limitations. For example, illumination variations affect the camera and fog affects the LiDAR. When power devices are aging or damaged, fault areas on the power devices can be hardly recognized with naked eyes, but heat or discharge will occur in the fault area. These phenomena can be detected by infrared thermal devices and UV cameras. Therefore, multisensor information fusion is indispensable for PTL inspection. As shown in Figure9, it can be carried out from the following steps:Figure 9
The flowchart of multisensor information fusion.3D fusion reconstruction of transmission line corridors is used. Firstly, a rough 3D model of the transmission line corridor is constructed by oblique photography. The data registration is completed by the ICP algorithm to form a detailed 3D model. The fault areas are approximately found by the detailed 3D model.The data fusion of infrared, visible light, and ultraviolet images is employed. After determining approximate fault areas and time synchronization, these sensors are used to shoot for image fusion. At present, image fusion still rests on the pixel stage. Feature fusion should be further considered for it.The accurate coordinate of discharge, hotspot, and visible defects in fault areas are gained by object detection. The spatial conversion relations are gained by the calibration of the visible-light camera and LiDAR. For getting 3D information, the precise coordinate of the defect area is projected into point clouds according to spatial conversion relations. Then, the 3D information is used for local 3D reconstruction.Visual servo control is applied. The current pose of the robot and objects can be obtained by local 3D reconstruction. After that, it can be provided to a 3D visual servo controller to complete the control.


### 5.1.2. The Problem of Datasets Required for Deep Learning

With the development of AI, a growing number of fields commence research by employing deep learning. Deep learning algorithms rely on data, but there is no standard power inspection dataset for researchers to conduct algorithm research. Therefore, the establishment of a standard power inspection dataset for deep learning development promotion is urgently needed. It can be constructed from the following steps:Standards of inspection images and power devices are made. Aiming at different defects of power devices, a unified defect description and classification rules should be established. The size and sharpness of inspection images should be normalized. Standards provide convenience when instructing personnel in data collection and highly improve the dataset construction.Datasets of PTL inspection should be divided according to various regions. The geographical environment and weather conditions of each region are different. PTL will be affected by a distinct climate, which makes the type of the same power device be disparate. Different types of power devices may vary greatly in appearance and shape, which results in regional characteristics. Therefore, there are regional characteristics in PTL corridors. In addition, the background of various regions differs greatly. Hence, it is necessary to divide the whole dataset first according to the territories.According to different kinds of power devices and degrees of defects, datasets of regional PTL inspection should be further divided into different subset datasets. According to specific power devices in PTL, it is helpful to form the corresponding PTL dataset by subset datasets, which can represent the characteristics greatly and be helpful for training models. The steps are shown in Figure10.Figure 10
The steps of power inspection dataset establishment.Now, a good deep learning model is always based on high-quality annotated data. However, it is expensive to collect data, especially annotated data. In the field of PTL inspection, this phenomenon is more obvious. Consequently, researchers try to employ weakly supervised data to train models [112, 113]. Furthermore, they employ unsupervised learning to annotate data. These methods can be also used in object detection of PTL inspection [114] for reducing the cost of data annotation. It will be a good research direction of PTL inspection.


### 5.1.3. The Problem of Applying a Light Network Model Suitable for Embedded Platforms

The training of models usually runs on servers. The computational ability should be considered when deep learning-based object algorithms are used. Therefore, setting up a light-weight network in advance is significant. Then the model, transferred from the server to the embedded platform, can be additionally trained in actual scenarios. At present, different backbones such as SSD and YOLOv4 can be used in MobileNetv3. Furthermore, fine-tuning can also be employed.


### 5.1.4. The Problem of Control Efficiency and Robustness of the Visual Servo System

Now, the whole control process of grasping the line or tracking lines takes much time. And the efficiency is low. In this regard, improving visual sampling frequency and image processing speed is able to ameliorate its real-time capability. In addition, subtle structural changes of inspection robots may occur due to wind and drag of vibration dampers, which leads to errors. The uncalibrated visual servo has a high tolerance for these errors. Therefore, it can be adopted for improving the robustness.


## 5.1.1. The Problem of Environmental Perception

Both the high-altitude environment and natural interference factors pose challenges to inspection robots. In this scenario, highly accurate environmental information is imperative for task conduction. The use of a single sensor will inevitably have perceptual limitations. For example, illumination variations affect the camera and fog affects the LiDAR. When power devices are aging or damaged, fault areas on the power devices can be hardly recognized with naked eyes, but heat or discharge will occur in the fault area. These phenomena can be detected by infrared thermal devices and UV cameras. Therefore, multisensor information fusion is indispensable for PTL inspection. As shown in Figure9, it can be carried out from the following steps:Figure 9
The flowchart of multisensor information fusion.3D fusion reconstruction of transmission line corridors is used. Firstly, a rough 3D model of the transmission line corridor is constructed by oblique photography. The data registration is completed by the ICP algorithm to form a detailed 3D model. The fault areas are approximately found by the detailed 3D model.The data fusion of infrared, visible light, and ultraviolet images is employed. After determining approximate fault areas and time synchronization, these sensors are used to shoot for image fusion. At present, image fusion still rests on the pixel stage. Feature fusion should be further considered for it.The accurate coordinate of discharge, hotspot, and visible defects in fault areas are gained by object detection. The spatial conversion relations are gained by the calibration of the visible-light camera and LiDAR. For getting 3D information, the precise coordinate of the defect area is projected into point clouds according to spatial conversion relations. Then, the 3D information is used for local 3D reconstruction.Visual servo control is applied. The current pose of the robot and objects can be obtained by local 3D reconstruction. After that, it can be provided to a 3D visual servo controller to complete the control.


## 5.1.2. The Problem of Datasets Required for Deep Learning

With the development of AI, a growing number of fields commence research by employing deep learning. Deep learning algorithms rely on data, but there is no standard power inspection dataset for researchers to conduct algorithm research. Therefore, the establishment of a standard power inspection dataset for deep learning development promotion is urgently needed. It can be constructed from the following steps:Standards of inspection images and power devices are made. Aiming at different defects of power devices, a unified defect description and classification rules should be established. The size and sharpness of inspection images should be normalized. Standards provide convenience when instructing personnel in data collection and highly improve the dataset construction.Datasets of PTL inspection should be divided according to various regions. The geographical environment and weather conditions of each region are different. PTL will be affected by a distinct climate, which makes the type of the same power device be disparate. Different types of power devices may vary greatly in appearance and shape, which results in regional characteristics. Therefore, there are regional characteristics in PTL corridors. In addition, the background of various regions differs greatly. Hence, it is necessary to divide the whole dataset first according to the territories.According to different kinds of power devices and degrees of defects, datasets of regional PTL inspection should be further divided into different subset datasets. According to specific power devices in PTL, it is helpful to form the corresponding PTL dataset by subset datasets, which can represent the characteristics greatly and be helpful for training models. The steps are shown in Figure10.Figure 10
The steps of power inspection dataset establishment.Now, a good deep learning model is always based on high-quality annotated data. However, it is expensive to collect data, especially annotated data. In the field of PTL inspection, this phenomenon is more obvious. Consequently, researchers try to employ weakly supervised data to train models [112, 113]. Furthermore, they employ unsupervised learning to annotate data. These methods can be also used in object detection of PTL inspection [114] for reducing the cost of data annotation. It will be a good research direction of PTL inspection.


## 5.1.3. The Problem of Applying a Light Network Model Suitable for Embedded Platforms

The training of models usually runs on servers. The computational ability should be considered when deep learning-based object algorithms are used. Therefore, setting up a light-weight network in advance is significant. Then the model, transferred from the server to the embedded platform, can be additionally trained in actual scenarios. At present, different backbones such as SSD and YOLOv4 can be used in MobileNetv3. Furthermore, fine-tuning can also be employed.


## 5.1.4. The Problem of Control Efficiency and Robustness of the Visual Servo System

Now, the whole control process of grasping the line or tracking lines takes much time. And the efficiency is low. In this regard, improving visual sampling frequency and image processing speed is able to ameliorate its real-time capability. In addition, subtle structural changes of inspection robots may occur due to wind and drag of vibration dampers, which leads to errors. The uncalibrated visual servo has a high tolerance for these errors. Therefore, it can be adopted for improving the robustness.


## 5.2. Prospects

In the application scenarios of PTL inspection, not only the foundation capabilities of inspection robots but also further expansion should be realized according to requirements. It is helpful to realize the vital engineering value of inspection robots. The facets of future directions are as follows:


### 5.2.1. The Development of Mechanical Design

The core of inspection robots is the mechanical design. And a good one can greatly improve the efficiency. For example, compared to LineScout [115], LineRanger [116] takes much less time to cross obstacles because of the ingenious and passive obstacle-crossing mechanism. However, the mechanical design of PTL inspection robots is not refined now, and robots cannot balance inspection and operation. So far, in contrast to other inspection robots, climbing robots are more practical due to their in-contact measurements. The engineering value of climbing robots is greater when they can succeed in crossing pylons. So the mechanical design of inspection robots still needs to be improved.


### 5.2.2. The Development of Energy Harvesting

Battery capacities directly influence the duration of PTL inspection. Regarding a certain load capacity of robots, a high-capacity battery, such as lithium battery, can increase maximum endurance of robots, but it limits the weight of other mechanical components. So online charging is important. For increasing the inspection endurance, inspection robots can acquire electric energy from power line autonomously. So an induction power supply system should be developed.


### 5.2.3. The Development of Electromagnetic Protection

EMI on inspection robots is more intense with increasing voltage, especially the transient discharge process of the equipotential working. However, the present inspection robots are weak to EMI. Therefore, it is an inevitable problem during the inspection. The entire robot should conduct electricity for avoiding electrical discharge within components of robots. Shielding is an effective protection measurement aimed at EMI, and high-quality shields should be designed. However, inspection robots cannot be fully shielded, which means that other optimization technologies, such as routing, filtering, and grounding techniques, should be considered. Consequently, how to design electromagnetic protection is a crucial challenge.


### 5.2.4. The Development of Modularity

It is inevitable that inspection robots will be damaged in a bad working environment. And the modularity development of inspection robots is significant for rapid maintenance. The modularity of inspection robot is convenient for the maintainer to find the fault location and replace the fault module of robots. Moreover, it is easy to install and disassemble inspection robots by modularity, which also improves the efficiency of online and offline switchover. Furthermore, it helps to switch robot configurations in different inspection scenarios, such as overhead ground wire and multibundled conductors.


### 5.2.5. The Development of Reliability

The working environment of inspection robots is harsh, such as the power line area with long span and steep slope. There is a requirement for good robustness to accurately complete the task. Consequently, inspection robots should have the ability to cross the most complex area in PTL. For the purpose of satisfying the needs of multiple and complicated tasks, effective load capacity and operation control technology should be improved. What is more, before online operation, robots should be tested for endurance, such as wear of the mechanical system, battery performance, and EMI robustness.


### 5.2.6. The Development of Cooperation

This cooperation not only refers to the cooperation among a variety of inspection robots. It also includes the cooperation between inspection robots and the ground control system (GCS). The multirobot system can promote the performance of PTL inspection. In addition, in the case of teleoperation, the efficiency of inspection can be also affected by the cooperation between inspection robots and GCS. Appropriate deployments of GCS points in a long PTL corridor are the key to the cooperation between inspection robots and GCS. The cooperation includes real-time bidirectional data transmission, the energy consumption of disassembly, installation, and teleoperation.


### 5.2.7. The Development of Friendly Operability and Autonomy

The final goal is to achieve autonomous patrol of inspection robots. However, autonomous control and teleoperation need to coexist in the process of development. Although inspection robots could automatically get across obstacles such as spacers and suspension clamps, there is a lack of autonomous ability in specific operations. The specific operations include dealing with damaged power lines and removing foreign bodies. For fulfilling tasks securely and briskly, teleoperation should be used. Meanwhile, the friendly man-machine interface of inspection robots should be developed for simple teleoperation.


## 5.2.1. The Development of Mechanical Design

The core of inspection robots is the mechanical design. And a good one can greatly improve the efficiency. For example, compared to LineScout [115], LineRanger [116] takes much less time to cross obstacles because of the ingenious and passive obstacle-crossing mechanism. However, the mechanical design of PTL inspection robots is not refined now, and robots cannot balance inspection and operation. So far, in contrast to other inspection robots, climbing robots are more practical due to their in-contact measurements. The engineering value of climbing robots is greater when they can succeed in crossing pylons. So the mechanical design of inspection robots still needs to be improved.


## 5.2.2. The Development of Energy Harvesting

Battery capacities directly influence the duration of PTL inspection. Regarding a certain load capacity of robots, a high-capacity battery, such as lithium battery, can increase maximum endurance of robots, but it limits the weight of other mechanical components. So online charging is important. For increasing the inspection endurance, inspection robots can acquire electric energy from power line autonomously. So an induction power supply system should be developed.


## 5.2.3. The Development of Electromagnetic Protection

EMI on inspection robots is more intense with increasing voltage, especially the transient discharge process of the equipotential working. However, the present inspection robots are weak to EMI. Therefore, it is an inevitable problem during the inspection. The entire robot should conduct electricity for avoiding electrical discharge within components of robots. Shielding is an effective protection measurement aimed at EMI, and high-quality shields should be designed. However, inspection robots cannot be fully shielded, which means that other optimization technologies, such as routing, filtering, and grounding techniques, should be considered. Consequently, how to design electromagnetic protection is a crucial challenge.


## 5.2.4. The Development of Modularity

It is inevitable that inspection robots will be damaged in a bad working environment. And the modularity development of inspection robots is significant for rapid maintenance. The modularity of inspection robot is convenient for the maintainer to find the fault location and replace the fault module of robots. Moreover, it is easy to install and disassemble inspection robots by modularity, which also improves the efficiency of online and offline switchover. Furthermore, it helps to switch robot configurations in different inspection scenarios, such as overhead ground wire and multibundled conductors.


## 5.2.5. The Development of Reliability

The working environment of inspection robots is harsh, such as the power line area with long span and steep slope. There is a requirement for good robustness to accurately complete the task. Consequently, inspection robots should have the ability to cross the most complex area in PTL. For the purpose of satisfying the needs of multiple and complicated tasks, effective load capacity and operation control technology should be improved. What is more, before online operation, robots should be tested for endurance, such as wear of the mechanical system, battery performance, and EMI robustness.


## 5.2.6. The Development of Cooperation

This cooperation not only refers to the cooperation among a variety of inspection robots. It also includes the cooperation between inspection robots and the ground control system (GCS). The multirobot system can promote the performance of PTL inspection. In addition, in the case of teleoperation, the efficiency of inspection can be also affected by the cooperation between inspection robots and GCS. Appropriate deployments of GCS points in a long PTL corridor are the key to the cooperation between inspection robots and GCS. The cooperation includes real-time bidirectional data transmission, the energy consumption of disassembly, installation, and teleoperation.


## 5.2.7. The Development of Friendly Operability and Autonomy

The final goal is to achieve autonomous patrol of inspection robots. However, autonomous control and teleoperation need to coexist in the process of development. Although inspection robots could automatically get across obstacles such as spacers and suspension clamps, there is a lack of autonomous ability in specific operations. The specific operations include dealing with damaged power lines and removing foreign bodies. For fulfilling tasks securely and briskly, teleoperation should be used. Meanwhile, the friendly man-machine interface of inspection robots should be developed for simple teleoperation.


## 6. Conclusion

For promoting researchers to develop advanced research of PTL inspection, studies of 3D reconstruction, object detection, and visual servo of PTL inspection are summarized. The extant problems, such as perception limitation and lack of datasets, are correspondingly discussed. What is more, several aspects are posed, which are promising research directions in the field of PTL inspection. In a word, the present perception technologies and control technologies cannot meet the requirement of automatic and accurate power line inspection. With the emergence of these research results, the automation degree of inspection robots will be highly boosted.



---

*Source: 5559231-2021-03-31.xml*