Research on 3D reconstruction technology of large‐scale substation equipment based on NeRF

IET Science, Measurement & TechnologyVolume 17, Issue 2 p. 71-83 ORIGINAL RESEARCHOpen Access Research on 3D reconstruction technology of large-scale substation equipment based on NeRF Shaotong Pei, Corresponding Author Shaotong Pei [email protected] orcid.org/0000-0002-9496-9386 Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Correspondence Shaotong Pei, Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China. Email: [email protected] Contribution: Conceptualization, Funding acquisition, Project administration, SupervisionSearch for more papers by this authorRui Yang, Rui Yang Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of ChinaSearch for more papers by this authorYunpeng Liu, Yunpeng Liu Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of ChinaSearch for more papers by this authorWenxuan Xu, Wenxuan Xu Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Contribution: Formal analysis, ​Investigation, Methodology, Software, ValidationSearch for more papers by this authorGonghao Zhang, Gonghao Zhang Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Contribution: ​Investigation, ValidationSearch for more papers by this author Shaotong Pei, Corresponding Author Shaotong Pei [email protected] orcid.org/0000-0002-9496-9386 Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Correspondence Shaotong Pei, Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China. Email: [email protected] Contribution: Conceptualization, Funding acquisition, Project administration, SupervisionSearch for more papers by this authorRui Yang, Rui Yang Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of ChinaSearch for more papers by this authorYunpeng Liu, Yunpeng Liu Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of ChinaSearch for more papers by this authorWenxuan Xu, Wenxuan Xu Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Contribution: Formal analysis, ​Investigation, Methodology, Software, ValidationSearch for more papers by this authorGonghao Zhang, Gonghao Zhang Hebei Provincial Key Laboratory of Power Transmission Equipment Security Defense, North China Electric Power University, Yonghua North Street 619, Baoding City, People's Republic of China Contribution: ​Investigation, ValidationSearch for more papers by this author First published: 26 November 2022 https://doi.org/10.1049/smt2.12131AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The high-quality 3D model is the key to digital twin and visualization. Here, the neural radiation field (NeRF) is applied to greatly improve the speed and accuracy of 3D reconstruction of large-scale substation equipment. The incremental SFM algorithm is used to accurately position the camera in open scenes, and the frame extraction experiment indicates that the single angle increment should not be greater than 8°. The reconstruction test is carried out on the power transformer and high-voltage experimental equipment following the circular shooting track for instant-NGP (a variant of NeRF). Compared with the traditional 3D reconstruction method, the speed of reconstruction is accelerated by 23–55 times, and the fidelity is improved by 2–3 times. Also, the deployment scheme of NeRF in the digital twin technology of large-scale substation equipment is proposed. The experimental results verify the effectiveness and feasibility of NeRF in practical application. 1 INTRODUCTION With the emergence of the concept of "digital twin" in power grid, visualization, digitization and informatization have become the directions of development for the energy equipment technology. Data visualization under the digital twin architecture requires the 3D display of sensing and simulation data based on the 3D model of power equipment [1-3]. Therefore, the establishment of an accurate 3D model of power equipment can provide richer display space for sensing data and can endow stronger analysis ability, so as to assist operation and maintenance personnel to make rapid and accurate judgments in their work [4, 5]. Hence, it is critical to build a digital platform to improve the reconstruction speed and the accuracy of power equipment via the method of 3D reconstruction with low cost, high efficiency, and few restrictions. There are two main 3D reconstruction methods at present, the 3D software modelling and lidar scanning [6]. Using 3D software modelling features strong controllability, but it requires laborious workload and is expensive. Also, it fails in accomplishing the large-scale task related to substation, and has been gradually phased out. Lidar scanning can directly obtain point cloud data of the surface contour of the model. Although point cloud data can be obtained flexibly, it is disordered and requires separate semantic segmentation, which brings difficulties to subsequent visualization and interaction. Heterogeneous data fusion between point cloud data and thermal images taken by infrared camera is also a major technical difficulty at present. The lidar is expensive, and the equipment covers a large area. Also, the proper operation of lidar is subject to many environmental constraints. These all limit the development of lidar scanning method [7-11]. 3D feature recovery and extraction using images shot by camera have aroused wide attention in recent years. In this method, cameras used can be divided into ordinary monocular camera, RGB-D depth camera, binocular camera etc., according to its principle of imaging. Compared with lidar, this method has the advantages of low cost and good controllability. However, the current massive state monitoring multi-source heterogeneous data of the power equipment cannot be automatically synthesized and displayed in 3D. Jiang Qian [12] proposed a fully automated spatial synthesis and 3D display method to deal with this problem. Owing to the in-depth research of deep neural network and the enhancement of GPU graphics processing capability, neural network is increasingly used to reconstruct models and extract features from complex 3D data [13, 14]. Neural network has excellent fitting and approximation ability, and has achieved excellent results in tasks such as image registration, model reconstruction, real-time rendering, and point cloud repairing. By virtue of the powerful fitting ability of MLP neural network, Ben Mildenhall proposed a new 3D reconstruction algorithm based on viewpoint synthesis, the neural radiation field (NeRF) [15]. NeRF establishes the connection between 2D projection view and 3D model using neural network directly, and achieves excellent reconstruction and rendering effects. On the basis of improved speed of training and rendering, Alex Yu proposed pixelNeRF, which modulates the architecture of NeRF based on image inputs by full convolution. The pixelNeRF can greatly reduce the number of images required for training (1–10 images), and can achieve efficient reconstruction of vehicles [16]. Jonathan T. Barron significantly improved the overall effect by changing the way of sampling the light [17]. Zhengqi Li proposed neural scene flow fields, a new representation that models the dynamic scene as a time-variant continuous function, and realized the dynamic video-based spatiotemporal view synthesis and the dynamic reconstruction of the human body [18]. Different from the data in the form of point cloud, NeRF only uses the input image as the supervision information, and the neural network is used as the implicit function to fit the scene. After the training is completed, the voxel model with strong spatial correlation is reconstructed, followed by the high-fidelity reconstruction of power equipment and the real photo-level rendering. The excellent 3D reconstruction effects exhibited by NeRF arouse interests in further optimization to improve the generalization ability, training speed and rendering effect of the algorithm [19-29]. Lingjie Liu proposed neural sparse voxel fields (NSVF) [30], a group of voxel boundary implicit fields organized by Sparse Voxel octree, to model partial characteristics in each unit, and the training speed was increased by over 10 times. Based on this work, Thomas Muller proposed the instant Neural Graphics Primitives (instant-NGP) [31] and adopted the hashing coding method to optimize the model, which endows the NeRF algorithms the accuracy and real-time responsive required by the digital twin technology. Figure 1 shows the evolution of all the above technologies. FIGURE 1Open in figure viewerPowerPoint Development of 3D reconstruction technology for power equipment The feature information in the reconstruction tasks for power equipment is more complicated than the standard data set, including not only low-frequency image components, such as macroscopic shape, but also high-frequency image components, such as surface texture, instrument reading, nameplate information, sharp edge, and discharge tip. Therefore, the feasibility of 3D reconstruction technology based on NeRF in power scene has not been proved. The environment of the 3D reconstruction tasks in the actual power scene is more complex than the standard data set obtained in the laboratory. In the lab, multiple cameras are often used to capture the physical scene, and the coordinates of cameras can be acquired accurately. In the inspection and maintenance of substation equipment, aerial photography of UAV or handheld device is often used, which results in the following problems: (1) Positioning: NeRF reconstruction relies on the precise positioning of the camera. The location of the camera in open space is usually determined by the inspection task, which has great randomness. Meanwhile, the positioning of the camera is hard, and accurate coordinates and poses are difficult to obtain. (2) Path: In standard data sets, the camera coverage is usually hemispherical or spherical. In real shooting using hemispherical or spherical shooting paths, the paths need to be divided into four to five layers and are shot layer-by-layer. The pitching angle and deflection angle of the camera head need to be adjusted constantly. Here, the incremental SFM algorithm is used to recover the camera pose and 3D information directly from the images in order to solve the problem of difficult camera positioning. An achievable circular shooting path is adopted instead of the hemispherical shooting path. Based on these, a large number of 3D reconstruction tests are carried out on power transformers and high-voltage experimental equipment adopting instant-NGP, with the integration of two criteria (the degree of fidelity and timeliness). The feasibility of NeRF in power scenes is verified, and an application scheme based on NeRF and its variant algorithm in digital twin technology is proposed. 2 BACKGROUND It is a kind of reverse modelling task to carry out data collection and 3D reconstruction in the power production site after the power equipment has been in operation for a period of time. Data generated by different acquisition methods and reverse reconstruction algorithms can be divided into point cloud data and voxel grid [13]. Point cloud data is a collection of 3D coordinate points in space. This method uses a group of 3D points on the surface of the model to represent the surface of the shape, which can be expressed in the matrix of n×d. In the matrix, n represents the nth point on the surface of the 3D model, and d represents the number of the characteristic dimensions of each point (including point coordinates, colour, and depth). Point cloud data can be obtained only through handheld or airborne lidar large-area scanning, but the main source restricting the speed and accuracy of mainstream 3D reconstruction is the lack of semantic and sequential information between point cloud data [32]. A voxel grid is a raster map with fixed resolution in 3D space, which is analogous to a pixel in 2D space. Some methods used in 2D image processing, such as convolution, coding and regularization, are applicable to 3D voxel grid. Compared with point cloud data, voxel grids are usually obtained by monocular cameras through complex camera reconstruction algorithms and cannot be obtained directly. The novelty of NeRF algorithm lies in that it establishes implicit connections from 2D to 3D between multi-view and voxel grids, and does not require in-depth information supervision. It uses MLP network to fit accurate neural implicit functions for high-resolution geometric shapes, merely using input multi-view as supervision information as shown in Figure 2. In this way, the free view generation and photorealistic light and shadow rendering for a complex scene can be achieved. Neural implicit functions are smooth and continuous; thus, rendering space with infinitely high resolution can be theoretically achieved. The algorithm flow is shown in Figure 3. FIGURE 2Open in figure viewerPowerPoint The principle of the multilayer perceptron FIGURE 3Open in figure viewerPowerPoint Principle of 3D reconstruction of neural radiation fields The 5D coordinate vector ( x , y , z , θ , ϕ ) $( {x,\;y,\;z,\;\theta ,\;\phi } )$ is taken as the input, which is constituted by the 3D-coordinate X = ( x , y , z ) $X = ( {x,\;y,\;z} )$ of the 3D scene, and the observation angle D = ( θ , ϕ ) $D = ( {\theta ,\;\phi } )$ of the camera. The colour c = ( R , G , B ) $c = ( {R,\;G,\;B} )$ of the scene and the grid body density σ are taken as the output. The following implicit field mappings are established based on deep fully connected neural network (MLP): F θ : X , D → c , σ . $$\begin{equation}{F_\theta }:\left( {X,D} \right) \to \left( {c,{{\sigma }}} \right).\end{equation}$$ (1) In the above mapping relation, the density σ of the grid body reflects the probability of the existence of an entity in the space point, which is only related to the 3D-coordinate X of the point, and has nothing to do with the observation direction. However, the colour c of the point in the scene depends on the spatial position X and the observation angle d of the camera. Therefore, we set Fθ as follows. The input 3D-coordinate X is encoded as a hidden vector of volume density σ and 256D through eight layers (Relu, 256 channels) of full connection layers. The implicit vector is then spliced with the input perspective and regressed to the RGB colour value c through a full connection layer (Relu, 128 channels), as shown in Figure 2. The volume density σ represents the probability that the ray will terminate after arriving at this point. Therefore, for the ray from a certain perspective o in the direction of D, which reaches the point r (t) = o + td at time t, the colour integral along this direction will generate the final colour value: C r = ∫ t n t f T t σ r t c r t , d d t , $$\begin{equation}C\left( r \right) = \int _{{t_n}}^{{t_f}} T\left( t \right)\sigma \left( {r\left( t \right)} \right)c\left( {r\left( t \right),d} \right)dt,\end{equation}$$ (2)where T t = exp − ∫ t n t σ r s d s , $$\begin{equation}T\left( t \right) = \exp \left( { - \int _{{t_n}}^t \sigma \left( {r\left( s \right)} \right)ds} \right),\end{equation}$$ (3) In the actual rendering method, the light can be divided into N parts equidistant from the nearest and furthest distance, and a point is uniformly sampled from each part to obtain N points. For N points, discrete integral approximation is carried out. The formula is as follows: C ̂ r = ∑ i = 1 N T i 1 − exp − σ i δ i c i , $$\begin{equation}\hat C\left( r \right) = \mathop \sum \limits_{i = 1}^N {T_i}\left( {1 - \exp \left( { - {\sigma _i}{\delta _i}} \right)} \right){c_i},\end{equation}$$ (4)where T i = exp − ∑ j = 1 i − 1 σ i δ i . $$\begin{equation}{T_i} = \exp \left( { - \mathop \sum \limits_{j = 1}^{i - 1} {\sigma _i}{\delta _i}} \right).\end{equation}$$ (5) These are the core parts of NeRF, but such a depth model cannot achieve the high-precision as required by power equipment reconstruction, so the algorithm is further improved following two steps: Location coding Deep network is inclined to learn low-frequency signals in space. If the signals jump sharply, it is difficult to fit the deep network only by using coordinate representation as input. Inspired by the transformer mechanism, a special mapping method is adopted here, the location coding. The coordinate representation is mapped to the higher dimensions by high-frequency function. This is realized through multiperiod sine and cosine functions, where the coordinate L takes 10, and the direction L takes 4. Using this method, MLP has the ability to approximate high-frequency functions more easily. γ p = sin ( 2 0 π p , cos 2 0 π p , … , cos 2 L − 1 π p ) $$\begin{equation}\gamma \left( p \right) = \left( {{\rm{sin\;}}({2^0}\pi p} \right)\;,\;\cos \left( {{2^0}\pi p} \right),\; \ldots ,\;\cos \left( {{2^{L - 1}}\pi p} \right))\;\end{equation}$$ (6) 2. Hierarchical volume sampling Using uniform N sample points for each ray is sometimes unnecessary, because different regions contribute differently to the final colour output. Therefore, sampling should be conducted on those regions that contribute more. Write C in the new form: C c ̂ r = ∑ i = 1 N c w i c i , $$\begin{equation}\widehat {{C_c}}\left( r \right) = \mathop \sum \limits_{i = 1}^{{N_c}} {w_i}{c_i},\end{equation}$$ (7)where w i = T i 1 − exp − σ i δ i . $$\begin{equation}{w_i} = {T_i}\left( {1 - \exp \left( { - {\sigma _i}{\delta _i}} \right)} \right).\end{equation}$$ (8) As a result, the corresponding contribution weight of each point can be obtained. The two networks are trained simultaneously, one for rough rendering and the other for fine rendering. The latter uses the contribution weights of sample points obtained by the former for re-weighting sampling, which can invest more computational force in the region with larger contribution value to achieve more accurate rendering results. Finally, loss is defined as the sum of loss of two networks: L = ∑ r ∈ R C c ̂ r − C r 2 2 + C f ̂ r − C r 2 2 $$\begin{equation}L = \sum_{r \in R} {\widehat {{C_c}}\left( r \right) - C\left( r \right)_2^2 + \widehat {{C_f}}\left( r \right) - C\left( r \right)_2^2} \end{equation}$$ (9) The process of training MLP network is also the process of NeRF network fitting 3D entities. NeRF has achieved remarkable results in 3D reconstruction and viewpoint synthesis, making it one of the hottest research topics in 2021. The reconstruction of NeRF is excellent, but its biggest weakness lies in the speed of algorithm training and rendering. When facing new scenes, it takes about 30 h per training session and about 15 s (0.07 fps) per rendering frame on average. Digital platforms typically require the reconstruction to be completed in minutes and the viewing rate to be real-time (12.5 FPS). Therefore, great efforts have been done to improve the algorithm based on the principles of NeRF. 3 METHODOLOGY Improved NeRF algorithm based on high-resolution hash codes: Instant-NGP In order to further reduce training and rendering costs and to speed up training, a coding method based on hash table is applied to instant-NGP algorithm. A multi-resolution hash table with trainable feature vectors is used to encode the position of small MLPS, and the parameters are completed by random gradient descent. This compact encoding structure allows the use of smaller MLP networks while maintaining reconstruction accuracy, thus significantly reducing the number of floating points and memory access operations. Through the parallel computation of GPU, comprehensive acceleration of several orders of magnitude can be obtained, and 1920 × 1080 resolution rendering can be completed in tens of milliseconds. For a given MLP network M=(y, Φ), INPG algorithm creates the coded input y = enc ( x , θ ) $y = {\rm{enc}}\;( {x,\;\theta } )$ . Both the weight parameter Ф and the coding parameter θ can be trained. These hyperparameters are arranged into independent L layers, and each layer contains the feature vector T of dimension f. The typical values of the hyperparameters are shown in Table 1. As shown in Figure 4, in the 3D voxel space, for a given input coordinate X, the surrounding voxel units at the resolution level of L are selected, and the integer coordinates of their corner points are taken as the index of the hash table. For all the corner points obtained, the corresponding F-dimensional feature vectors from the hash table θ are linearly interpolated according to the relative positions of x in the respective lth voxel unit. The results of each layer are sequentially series with the auxiliary parameter ξ∈RE to get the encoded series vector, which is used as the input of MLP network. In the process of training and coding, the loss gradient propagates back through MLP, series vectors, and interpolation space successively, and finally accumulates in the feature vectors found in the hash table. TABLE 1. Hash code parameters and their typical values Parameter Symbol Value Number of levels L 16 Max. entries per level (hash table size) T [214, 224] Number of feature dimensions per entry F 2 Coarsest resolution Nmin 16 Finest resolution Nmax [512, 524288] FIGURE 4Open in figure viewerPowerPoint 3D multi-resolution hash code: (1) hashing of voxel vertices; (2) lookup; (3) linear interpolation. (4) concatenation; (5) neural network The multi-resolution hashing coding method has strong adaptability. The algorithm can map the level grid to the corresponding fixed size eigenvector array. At low resolution, an array of grid points to feature vectors can be treated as a 1-to-1 mapping, while at the high resolution, the array can be treated as a hash table and can be indexed using a spatial Hash Function. Multiple grid points are used to number each array entry. This kind of coding method can average the training gradient, so that the largest gradient is dominant. Therefore, the hash table automatically prioritizes the sparse regions with the most important fine details. At the same time, such coding method can be mapped to modern GPU well, avoiding the inherent execution divergence in tree traversal. Meanwhile, it can query hash tables of all resolutions in parallel, which greatly improves the computing efficiency. Camera pose solution method based on incremental SFM technology In order to solve the problem that handheld cameras and airborne cameras cannot be located accurately in open space, incremental SFM technology and sparse reconstruction is introduced. The overall technical principle of incremental SFM is shown in Figure 6. Feature matching can be carried out between disordered images, and geometric correction and triangulation can be also implemented to restore the sparse point cloud structure. Also, the relative attitude of the camera can be re-estimated through the existing cloud; then local and global BA optimization can be carried out. Gradually, perspective or image can be added to the existing structure to conduct triangulation and attitude estimation, and BA optimization can be carried out to modify the structural data. Finally, all the camera parameters and sparse 3D point cloud are exported, and the calculated sparse point cloud and camera pose are shown in Figure 5 FIGURE 5Open in figure viewerPowerPoint Camera posture localization at different frame extraction rates. (a) All positioning is successful (left); (b) the failure rate is 11% (middle); (c) the failure rate is 30% (right) FIGURE 6Open in figure viewerPowerPoint Incremental SFM technique and sparse reconstruction FIGURE 7Open in figure viewerPowerPoint The technical route of 3D reconstruction Based on the incremental SFM technique, the camera pose estimation necessary for NeRF 3D reconstruction is accomplished. The incremental SFM technique requires a large number of images from different perspectives. We conducted a frame extraction test to determine the minimum number of images required for positioning. The complete technical process based on incremental SFM technology and instant-NGP is shown in Figure 7. 4 EXPERIMENTS Positioning test The key step for incremental SFM technique is the feature matching of images. To ensure the completion of feature matching, there should be enough similar features among images. In the process of shooting, the differences between each photograph and its close neighbours (such as the adjacent positions on the motion track) should not exceed a certain threshold. To obtain the empirical value of this threshold, frame extraction test was carried out in the video of power transformer ring shot (360°). The average rotation angle of each frame was increased from 1° to 20°, and the proportion of pictures that failed to locate in the total number of corresponding extracted frames was recorded. The rule of distribution is shown in Figure 5. We found that when the frame extraction rate reaches 1.33% (10° rotation angle), the percentage of successful positioning begins to decline significantly. Therefore, the 8° is taken as the empirical value of a considering a 20% margin. This value is important for 3D reconstruction through a camera cluster. It suggests that when the central power equipment is reconstructed with surround fixed cameras, it is advisable to have 45 or more cameras around the power equipment. The above conclusions are verified by a large number of repeated experiments. Here are some more advices: It is necessary to ensure that the photographer moves at a constant speed when the uniform frame extraction method is adopted from the patrol video; otherwise, the camera points will be too dense or too sparse, and the reconstruction will fail predominately in the areas of sparse points. For UAV, ground-type inspection robot, and orbit-type robot, the accurate positioning and control of movement parameters can be ensured by RTK spatial positioning technology. For the fixed cameras, they should be evenly distributed. Ensure the central symmetry of the track during shooting. Circular track, inner and outer double ring track, upper and lower double ring track and polygon track can meet this requirement. Training In order to verify the feasibility of the above technical scheme, on the basis of the incremental SFM technology to solve the camera pose, experiments of camera capture and 3D reconstruction are carried out for the power transformer and the high-voltage test equipment. For the power transformer, HoloLens wearable devices are adopted for ring photography, and 45 photos from different perspectives are used as reconstruction objects. The main hardware and software parameters of our experimental platform are as follows: Power equipment: High voltage test equipment and 110 KV oil-immersed transformer; Wearable devices: HoloLens augmented reality glasses with the main camera resolution at 2048 × 1152; Computing host: CPU: Intel (R) Core (TM) [email protected] GHz, GPU: NVIDIA GeForce RTX 2080Ti 11G; Environment construction: Visual Studio 2019, CUDA v10.2, CMake v3.21, Python 3.7, and OptiX 7.3. To understand the training situation of MLP network, the decrease of the value of loss function with training time was visualized, as shown in Figure 9. The reconstruction process for power equipment based on instant-NGP needs to go through a fast convergence period of 1 min, a slow convergence period of 3 min, and then enters a stable period. At the end of 4 min, the model converges with slight image noise fluctuations. The results of real-time rendering during training are shown in Figures 8 and 10.