Visual inspection for transformer insulation defects by a patrol robot fish based on deep learning

IET Science, Measurement & TechnologyVolume 15, Issue 7 p. 606-618 ORIGINAL RESEARCH PAPEROpen Access Visual inspection for transformer insulation defects by a patrol robot fish based on deep learning Hongxin Ji, School of Aerospace Engineering, Tsinghua University, Beijing, ChinaSearch for more papers by this authorXiwang Cui, Corresponding Author cuixiwang2010@126.com School of Instrument Science and Opto-electronics Engineering, Beijing Information Science and Technology University, Beijing, China Correspondence Xiwang Cui, School of Instrument Science and Opto-electronics Engineering, Beijing Information Science and Technology University, Beijing, China 100192. Email: cuixiwang2010@126.comSearch for more papers by this authorWeiyan Ren, School of Aerospace Engineering, Tsinghua University, Beijing, ChinaSearch for more papers by this authorLiqing Liu, State Grid Tianjin Electric Power Research Institute, Tianjin, ChinaSearch for more papers by this authorWei Wang, State Grid Tianjin Electric Power Research Institute, Tianjin, ChinaSearch for more papers by this author Hongxin Ji, School of Aerospace Engineering, Tsinghua University, Beijing, ChinaSearch for more papers by this authorXiwang Cui, Corresponding Author cuixiwang2010@126.com School of Instrument Science and Opto-electronics Engineering, Beijing Information Science and Technology University, Beijing, China Correspondence Xiwang Cui, School of Instrument Science and Opto-electronics Engineering, Beijing Information Science and Technology University, Beijing, China 100192. Email: cuixiwang2010@126.comSearch for more papers by this authorWeiyan Ren, School of Aerospace Engineering, Tsinghua University, Beijing, ChinaSearch for more papers by this authorLiqing Liu, State Grid Tianjin Electric Power Research Institute, Tianjin, ChinaSearch for more papers by this authorWei Wang, State Grid Tianjin Electric Power Research Institute, Tianjin, ChinaSearch for more papers by this author First published: 23 April 2021 https://doi.org/10.1049/smt2.12062 Funding Information National Natural Science Foundation of China; Grant/Award No.: 51907102, China Postdoctoral Science Foundation; Grant/Award No.: 2018M640128 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract In order to effectively detect the internal insulation defects of large transformers, a miniature patrol robot fish is designed and used to observe the pressboard inside the transformer, which can visually inspect the insulation condition of the pressboard. In the process of visual inspection by a patrol robot fish, insulation defects observed in the transformer (such as discharge carbon marks, pressboard cracks, etc.) are small size, indistinctive colour contrast and different shapes. The key purpose of the patrol robot fish is to identify the defect targets and defect types intelligently and quickly from the images taken by the camera on the fish. Considering that there are abundant insulation defects in transformers and a lack of samples for learning, a vision detection method based on deep learning network is proposed in this study. The proposed method integrates the variable autoencoder into the traditional Faster-RCNN target detection network and constructs an improved Faster-RCNN that enhances feature extraction. This method expands the small-scale training sample set and improves the generalisation performance of the model. In order to verify the effectiveness of the proposed method, the improved network is trained and tested, and the test results show that the improved network training model can accurately identify and mark the carbon marks on the pressboard surface. 1 INTRODUCTION Although the transformer monitoring and diagnosis technology has made remarkable progress, detection of the internal insulation of a transformer is still an important and difficult task. Due to the closeness of the metal shell of a large transformer, it is difficult to effectively detect the internal condition of the large transformer. The current traditional method requires the transformer oil to be emptied so that technicians can enter the interior of the transformer through a manhole above the transformer for manual inspection [1-5]. When the manual detection is completed, a large amount of transformer oil needs to be injected back into the transformer. In addition, when the technician enters a large transformer for manual inspection, there are many key parts of the transformer that need to be carefully observed. Therefore, the technician needs to spend a considerable amount of time to detect the transformer in the narrow space, and the task is relatively difficult. At present, detecting the internal insulation of a transformer faces four major problems: (1) Slow inspection: Most of the traditional internal inspection processes are carried out by people by entering the transformer or by hanging over. The inspection process includes oil drainage, oil filling and oil filtration. The whole process lasts at least 15 days, resulting in a long power outage and serious economic losses. (2) Inaccuracy: Due to the complex internal structure and limited space of the transformer, manual inspection makes it difficult to accurately determine the type, location and severity of internal defects, thus this inspection method is inaccurate and blind. (3) Risks: The space in the transformer is small and narrow, so that the technician runs the risk of colliding with the components inside the transformer, resulting in personnel injury and secondary damage to the equipment. (4) Uneconomical: Before determining the type and severity of the defects in the transformer, it will incur higher costs in the transportation, pumping, lifting, maintenance and other operations. The small patrol robot has the advantages of having a small size and flexible movement and can replace the technician to enter the transformer from the manhole. The transformer oil does not need to be drained completely and the operators need not enter the dangerous transformer. This method improves the detection safety and allows the transformer resume its operation in time, which is of great significance for intelligent, fast and accurate maintenance of the large transformer. Therefore, some research institutions have carried out related research of the transformer patrol robot. In 2018, ABB launched its first oil-immersed transformer internal patrol robot named Txplore [6, 7]. Txplore is rectangular in shape and 18×20×24 cm in size. Four propellers are arranged on the shell of the robot. Two propellers in the horizontal direction drive the forward movement and steering of the robot, and two propellers in the vertical direction control the upward and downward movement of the robot. Multiple cameras are installed on the Txplore. Shenyang University of Technology has developed a spherical transformer robot with a diameter of 19 cm, named SSTIR, with reference to the structure of underwater robots [8, 9]. The SSTIR is equipped with two fuel injection thrusters in the vertical direction. Since the SSTIR has a slight positive buoyancy, the vertical thrusters provide only downward thrust. Two horizontal thrusters provide forward and steering thrust. A camera is installed in the middle of the robot [10]. At present, when the transformer inspection robot moves in the transformer oil, its vision camera constantly takes pictures of the internal environment, which are transmitted to the computer workstation through wireless signals. Then, the technician manually observes whether there are target defects on the pictures. Few studies have been carried out on robots working in transformer oil. Referring to underwater vehicles, our team designed a new type of transformer patrol robot with fish fins. The robot's fish body is a spherical sealing structure. Two micro-propellers are installed on both sides of the body to control the forward movement and turn of the fish. A visual camera is equipped on the upper part of the fish body. A swim bladder device is installed below the fish body and can accurately control the amount of oil absorbed or expelled by detecting the location of the piston, so as to accurately control the floating speed and sinking speed of the robot fish. There are four fins, two fins are mounted on the upper end of the fish body and the other two fins are mounted on the lower end of the fish body. The mechanical fins can control the deflection direction and the angle of the fish. Through the combination of the fins and the swim bladder device, a gliding movement of the robot fish can be realised even without power, which effectively reduces the power consumption and increases the inspection range of the robot fish. The camera of the patrol robot fish constantly takes pictures of the internal environment, which is transmitted to the computer workstation through wireless signals. The patrol robot fish has a relatively large patrol range, and only relying on the human eye to continuously observe the photos taken, it is easy to miss some small target defects and has huge workload. In response to this problem, this paper carried out the intelligent autonomous inspection based on the vision of the patrol robot fish and hopes that the defect targets can be identified automatically by artificial intelligence to improve work efficiency. The detection precision of insulation defects depends on the target feature extraction from images taken by the patrol robot fish. Feature extraction and classification are the focus of traditional target detection methods. Therefore, many features extraction methods have been proposed in order to improve the deformation resistance and expression ability of features, and many forms of classifiers have been proposed in order to inprove the speed and precision of classifiers [11]. Lowe [12, 13] proposed the Scale Invariant Feature Transform (SIFT) by using the gradient information near key points of the image to describe the moving target. Dalal proposed the Histogram of Oriented Gradient (HOG) method, which was applied to solve the pedestrian detection problem of static images [14]. Felzenszwalb et al. proposed a De-formable part model (DPM) by combing the HOG and Support Vector Machine (SVM) [15]. These methods have become the classical target detection methods. Although the traditional target detection methods have achieved good development, high classification errors are generated due to the insufficient expression ability of extracted features to the target and poor feature separability. Meanwhile, the traditional target detection methods need to design features extraction manually, which are not suitable for different data. Compared with the features designed manually, the features acquired by deep learning are much richer and have stronger feature expression ability. Common deep learning models include restricted boltzmann machine (RBM) [16], autoencoder (AE) [17] and convolutional neural networks (CNN) [18]. With the development of deep learning, researchers have found that the precision of target detection using CNN can be greatly improved. This is because CNN extracts high-level features and improves the expression ability of features. In addition, the convolutional neural network integrates feature extraction, feature selection and feature classification into one model. Through the end-to-end training, CNN conducts overall functional optimisation and enhances features separability. Compared with traditional target detection methods, the detection method based on CNN has greatly improved precision and duration [19]. For this reason, this study adopts deep learning to visually detect the defects inside the transformer. The transformer is completely enclosed by the metal shell, so that the image acquisition by the patrol robot fish in the transformer will be affected easily, by too strong or insufficient camera light supplementation, occlusion, low contrast, blurred motion and tilted angle of view, etc. These problems have led to poor sample quality (i.e. defective image samples) and the lack of “positive samples”. However, the current deep learning training usually requires a large number of samples. In order to solve the problem of sample imbalance in the actual system, a visual detection method of the patrol robot fish based on an improved Faster-RCNN was proposed in this study. In this method, the input layer of VGG16 deep network in the traditional target detection model is replaced by a VAE network, and then a self-encoding convolutional neural network with enhanced feature extraction is proposed. This method expands the small-scale training sample set and improves the generalisation ability of the deep network model. The improved target network is trained and tested experimentally. The test results show that the improved network training model can accurately identify the carbon marks on the pressboard surface. 2 STRUCTURE MODEL OF THE TRANSFORMER PATROL ROBOT FISH The overall structure of the transformer patrol robot fish is shown in Figure 1, mainly including: robot fish body, robot swim bladder device, visual device, robot fish control system and robot fish fin device. The robot fish body is a spherical sealing structure. Two micro-propellers are installed on both sides of the body to control the forward movement and turn of the fish. A visual camera is equipped on the upper part of the fish body to inspect the insulation condition of the transformer. The control system is located in the middle of the robot fish body which mainly includes a motor drive and control module, a sensor acquisition module, a pose detection module, a wireless transmission module and a power supply module. A swim bladder is installed below the fish body and includes a piston oil cylinder, displacement sensors and a piston lift. Among them, the displacement sensor can accurately measure the amount of oil absorbed or expelled by detecting the location of the piston, so as to accurately control the floating speed and sinking speed of the robot fish. There are four fins, two fins are mounted on the upper end of the fish body and the other two fins are mounted on the lower end of the fish body. The mechanical fins can control the deflection direction and the angle of the fish. Through the combination of the fins and the swim bladder, a gliding movement of the robot fish can be realised even without power, which effectively reduces the power consumption and increases the inspection range of the robot fish. FIGURE 1Open in figure viewerPowerPoint Schematic diagram of the transformer patrol robot fish The physical picture of the robot fish is shown in Figure 2. The robot fish is spherical with a diameter of 11 cm. There are a pair of fins at the upper and lower ends of the robot fish with a length (L) of 11 cm and a width (W) of 5 cm. The total width (TW) of the two fins is 18 cm and the total height (TH) of the upper and lower fins is 13 cm. The robot fish body is 3D printed with transparent resin, so the whole body appears transparent light blue. The installation position of the camera is 3D printed with colourless and transparent resin. Besides the robot fish, all other parts are 3D printed with highly ductile resin material. FIGURE 2Open in figure viewerPowerPoint Physical picture of the transformer patrol robot fish Effective pixels of the vision camera installed in the patrol robot fish are 4 million. Horizontal field of view (HFOV) is 122o, vertical field of view (VFOV) is 93o and diagonal field of view (DFOV) is 150o. LED lights with adjustable brightness are used to fill in the light for the vision camera. Under conditions of strong supplementary light, the vision camera can take pictures in the dark . The pictures taken by the vision camera are then transmitted to the computer workstation via the 5.8G wireless digital image transfer module. The sealing performance and motion performance of the robot fish are tested in this study. The prototype has preliminarily realised the functions of rise up, sink down, plane 360-degree rotation, unpowered glide in transformer oil. 3 AN IMPROVED FAST-RCNN VISUAL TARGET DETECTION ALGORITHM 3.1 Typical Faster-RCNN Faster-RCNN was proposed by Girshich et al. [20] in 2016 and has been widely used in the field of target detection at present. Faster-RCNN is an improved version based on the RCNN, SPP-Net and Fast-RCNN. Faster-RCNN consists of feature extraction network, regional recommendation network and classification regression network. The typical structure of Faster-RCNN is shown in Figure 3. FIGURE 3Open in figure viewerPowerPoint Typical structure of Faster-RCNN As shown in Figure 3, the steps of using Faster-RCNN for target detection are as follows: First, the whole image is input into the input layer of the first CNN, and a feature map is obtained through convolution. Then, the feature map obtained by convolution is input into the Regional Proposal Network (RPN) to obtain feature information of the candidate box. The features of the candidate box are extracted and input into the classification regression network. Softmax classifier is used to identify calibrated categories. For the candidate boxes belonging to a certain feature, their position is further adjusted by regression. 3.2 Network structure and algorithm of the improved faster-RCNN 3.2.1 Feature extraction network based on VAE-CNN and VGG16 CNN is a kind of deep learning network with outstanding performance in image feature extraction. The structure of CNN consists of convolution, activation and pooling. When dealing with classification task, CNN establishes the mapping relationship between input data and specific feature space. After output image label through a fully connected network, the specific classification is realised [21]. VGG16 network is a CNN-based deep learning framework proposed by Synonyan and Zisserman in 2015 [22]. The network has realised the training of 1.3 million 224×224×3 images belonging to 1000 categories, and the classification precision rate is 92.7%. The parameters of the VGG16 model are open source, this makes VGG16 available for migration as a pre-training model in many classification models. The specific parameters of VGG16 can be seen in [22]. The CNN in VGG16 has a strong feature extraction capability, but there are two problems in the defect detection of an electric power system at present. One problem is that the dimension of the input image is much larger than 224*224*3 and the feature is more complex. The other problem is the lack of positive samples. Therefore, the feature extraction capability of VGG16 network will be limited, and the generalisation of the model is difficult to be guaranteed due to the lack of training samples. In recent years, the generation model has achieved remarkable results in small sample expansion [23]. Among them, variational auto-encoder CNN (VAE-CNN) can ensure the effect of feature extraction in small samples of complex data structures [24]. A self-encoding CNN detection model based on enhanced features is proposed by replacing part network in VGG16 with VAEs in this study, as shown in Figure 4. This detection model consists of two parts: variational encoder and CNN. FIGURE 4Open in figure viewerPowerPoint Improved feature extraction network The most important part of the variational encoder is the generation of data samples. For each input image xi∈X, its specific distribution p(xi|z) can be obtained by encoding the image. The recognition model q(z|xi) is adopted to approximate the true posterior probability p(xi|z). KL divergence is adopted to measure the similarity degree of the two distributions: K L q | | p = ∫ q t log q t p t d t = E q log q − E q log p (1) After obtaining the distribution of real data according to the decoder, the data Z generated by the generator can recover as much original information as possible by decoding. The loss function of this process is expressed as: L o s s V A E = M S E + K L , (2)where MSE is the mean square error of identifying model and real data. 3.2.2 Region proposal network RPN based on the feature extraction network generates the recommendation region by classifying and coordinate regressing the image background and detection target. Therefore, the input of RPN is the feature map obtained by convolution, and the output of RPN is the generated rectangular recommendation box. RPN is fully connected to the feature map of the last convolutional layer through a sliding window. Each sliding window generates a corresponding low-dimensional vector. The rectangular recommendation box generated by RPN through window sliding is also called anchor box. The recommended region is obtained by modifying any anchor frame with four parameters, and the modified formula is expressed as: x = w o t x + x o y = h o t y + y o w = w o exp t w h = h o exp t h , (3)where, x, y, w, h represent the central horizontal coordinates, vertical coordinate, width and height of the predicted recommended area variables, respectively; tx, ty, tw, th represent the four correction parameters. At the beginning of the training, RPN determines the recommended window conforming to the calibration from a large number of windows by setting the threshold. The loss function of the network includes the classification part and the regression part, expressed as: L o s s R P N = 1 N c l s ∑ i L c l s p i , p i ∗ + λ 1 N r e g ∑ i p i ∗ L r e g t i , t i ∗ , (4)where pi represents the probability that the ith predicted anchor frame; p i ∗ represents the actual label of the ith anchor frame; p i ∗ L r e g represents whether regression correction is made to the anchor frame without foreground object. p i ∗ = 0 or p i ∗ = 1 is logical indicator; ti represents the correction parameter of the predicted recommendation box; t i ∗ represents the correction parameter of the anchor frame relative to the actual recommendation box; Lcls represents the logarithmic loss of dichotomies; Lreg represents regression loss of coordinate correction parameter. L c l s p i , p i ∗ = − log p i p i ∗ + 1 − p i 1 − p i ∗ L r e g t i , t i ∗ = ∑ i ∈ x , y , w , h s m o o t h L 1 t i − t i ∗ (5)where, smoothL1 is smooth regression function which is used to filter the predicted correction parameters, as calculated by: s m o o t h L 1 x = 0.5 x 2 if x ≤ 1 x − 0.5 otherwise . (6) 3.2.3 Classified regression network Classification regression network is the key network for the final target detection. The classification regression network takes the feature graph output by the feature network and the recommended area box output by RPN as inputs and unifies the size of all features in the channel through the RoI pooling layer and generates the feature graph of the fixed area. Finally, two full connection layers are used to predict the defect category confidence of fixed recommended area frame and the regression of recommended area frame. In this network, two connected layers in a series are used to combine the feature maps of the candidate regions with a fixed size, then the feature vectors representing the candidate regions are obtained. The feature vector was input into the defect classification layer and the coordinate refinement layer respectively, and the score value and correction parameter of the candidate area belonging to each defect category are obtained. The Softmax layer is used to normalise the score value, and then the corresponding confidence degree is obtained. The coordinate information of the candidate area is obtained by using the correction parameter. Softmax classifier is selected as the output function of the network to predict the defect category of the zone box [25]. The Softmax layer follows the full connection layer and receives the output of full connection layer, then calculates the prediction probability distribution of the region as follows: p u i | m = e m i ∑ k N c e m k , (7)where, p(ui|m) represents the probability of category ui under the condition of softmax layer input feature vector m; mi represents the ith input data of the eigenvector m; Nc represents the total number of defect categories. For all values of i, argmax p(ui|m) is the predicted defect area of the softmax layer output. The final loss function includes classification loss and regression loss, expressed as: L o s s c l s r e g = L ′ c l s p , u + λ L l o c t u , v ∗ L ′ c l s p , u , (8)where, L c l s ′ is forecast defect category loss, expressed as: L c l s ′ p , u = − log p u i | m , (9)where, p is confidence coefficient, u represents the actual category tag corresponding to the input recommendation area. Lloc is the regression loss of the position correction parameter in the predicted area, expressed as: L l o c v u , v i ∗ = ∑ i ∈ x , y , w , h s m o o t h l L 1 v u , v i ∗ , (10)where, vu represents the final recommended area box correction parameter for the prediction; v i ∗ represents the correction parameter of the recommended area relative to the actual calibrated area. 4 MODEL TRAINING PROCESS The transformer defect detection can be realised through the proposed improved algorithm framework of faster CNN. The implementation process is described in detail as follows: Training sample set construction Transformer defect images are obtained through a high voltage experiment. The location and category of the defect are manually marked by a calibration software. The image matrix of the marked image is used as variable X and the XML file formed by calibration is used as variable Y. X and Y form the training sample. The cross validation method was adopted to divide the training samples into training validation sets and test sets, and then input them into the model for training. 2. Determine the network structure and training algorithm The training process of the model adopts the method of combining alternate training and joint training which is the same as the original network. The training process is divided into four steps. First, the regional recommendation network and feature extraction network are trained separately, in which the model parameters of VGG16 are directly imported by the pre-training model. The training samples are sampled using Mini-batch, each of which contained N anchor frames randomly extracted from an image. BP algorithm and STOCHASTIC gradient descent algorithm are used to optimise the loss functions of all anchors and carry out end-to-end training. Second, the output of the regional recommendation network in step (1) is taken as the input of the detection network to conduct pre-training classification regression network. After the training, the unique parameters of shared convolutional layer and classified regression network are updated. Third, retrain the area recommended network, fix network shared part of the parameters and only update its unique parameters. Fourth, according to the retraining results, the entire firm-RCNN is fine-tuned, the parameters of shared parts of the network are fixed and only the parameters of unique parts of the firm-RCNN are updated. 3. Hyperparameter setting The training network hyperparameter setting mainly includes input and output layers of each network, convolution kernel size, dropout inactivation rate, regular term coefficient and learning rate. Among them, the dimensions of input and output layers are determined by the number of variables. After many tests, the network modelling effect is good when the convolution kernel is set as 5*5, the pooling kernel size is set as 2*2, the sliding window of the anchor frame is set as 3*3 and the learning rate adopts variable learning settings ranging from 1E-4 to 0.1. 5 ANALYSIS OF THE TRANSFORMER DEFECT DETECTION EFFECT 5.1 Sample acquisition and calibration In order to verify the detection effect of the proposed improved Faster-RCNN in transformer defect recognition, 100 transformer defect images are obtained through a high voltage experiment, which is used to construct the initial small sample defect recognition library. Some defect examples are shown in Figure 5. FIGURE 5Open in figure viewerPowerPoint Some typical defect examples The calibration of training samples is an important part in the process of constructing training samples. The samples obtained by the high voltage experiment are original defect graphs. The defects in the graphs need to be marked with Label Img software to obtain the real labels of defects in the training sample graphs. The label format of defect graph is Y = {' carbon mark ‘, 'carbon mark’…, ‘Carbon mark’}. In practice, defects only need to be detected and there is no need to classify defect types. Therefore, label format of Y has only one catego