# 3D Reconstruction Method of Virtual and Real Fusion Based on Machine Learning

Authors: Wenyao Zhu; Shuyue Zhou
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

With the continuous development of computer vision technology, people are paying more and more attention to the method of using computers to simulate actual 3D scenes, and the requirements for 3D reconstruction technology are getting higher and higher. Virtual and real fusion refers to combining the virtual environment generated by the computer with the actual scenes around the user through photoelectric display, sensors, computer graphics, multimedia, and other technologies. This is a technology that can obtain more convenient and direct expressions, and it is also a technique for expressing content more abundantly and accurately. The key to virtual and real fusion technology is the registration of virtual objects and real scenes. It means that the system should be able to correctly estimate the position and posture of the camera in the real world, and then place the virtual object where it should be. Machine learning is a multifield interdisciplinary subject that specializes in how computers simulate or realize human learning behaviors. It is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are in all the fields of artificial intelligence. This article introduces the virtual-real fusion 3D reconstruction method based on machine learning, compares the performance of the method with other algorithms through experiments, and draws the following conclusion: the algorithm in this study is the fastest, with an average speed of 72.9% under different times. To evaluate the image acquisition indicators of each algorithm, the algorithm in this study has the lowest error rate. The matching accuracy of each algorithm is tested, and it is found that the average matching accuracy of the algorithm in this study is about 0.87, which is the highest.


---

## Body

## 1. Introduction

### 1.1. Background

The twenty-first century is the information century and the Internet century. Computer technology is increasingly inseparable from people’s daily life and has become a part of their lives. Whether it is military, education, etc., which are related to the country’s centuries-old plan, or life, shopping, and people’s other livelihoods, computer technology plays a very important role. With the vigorous development of applications such as virtual reality, 3D animation, three-dimensional measurement, virtual teaching, photomicrography, computer fluid mechanics, and military simulation, the research on the fusion of virtual and real three-dimensional reconstruction technology under different backgrounds has become more and more important. The research of 3D reconstruction has experienced the initial modeling of small and medium-sized objects, followed by the modeling of large buildings. The three-dimensional reconstruction of indoor scenes has been developed for many years in the field of scientific research. In this process, various implementation methods have been proposed and improved. Many scholars have proposed various 3D reconstruction methods. Among them, the three-dimensional reconstruction technology based on the machine learning theory algorithm has the advantages of small error, high degree of restoration, and good stability.


### 1.2. Significance

As a new technology in the Internet era, 3D reconstruction technology is developing rapidly. Three-dimensional reconstruction is an important communication bridge between the computer and the real scene and plays an important role in the interaction between virtual reality and humans and computers. At the same time, it can effectively guide human life and is widely used in various fields. Three-dimensional reconstruction is a reverse engineering from reality to the virtual world. The combination of virtual and real means to combine the virtual world of 3D reconstruction with reality to present a good visual effect, which is of great significance to industries such as medicine, artificial intelligence, and education. Machine learning is the core of artificial intelligence, which is the basic method to make computers intelligent. Machine learning mainly studies the methods of computer simulation or implementation of human learning actions to learn new knowledge and skills. The virtual-real fusion 3D reconstruction technology based on machine learning can shorten the image matching time, reduce the error rate of image matching, and increase the accuracy of image matching. The research of virtual and real 3D reconstruction technology based on machine learning can improve the efficiency of 3D reconstruction and bring a better experience to technicians, which has important practical significance.


### 1.3. Related Work

The emergence of 3D reconstruction technology has had a significant impact on the fields of artificial intelligence, robotics, and unmanned driving. So far, many scholars have researched it. Watanabe T used a focused ion beam scanning electron microscope to examine the three-dimensional organization of GAG chains in the Achilles tendons of mature rats embedded with epoxy resin. After staining with cupra iron blue, he specifically dyed the GAG chain. He also used 250 serial backscattered electron images (at 10 nm intervals) in the longitudinal section for reconstruction. The three-dimensional image shows that the GAG chain forms a ring-shaped network structure, each ring surrounds the collagen fibrils in the d-band and merges with adjacent rings to form a planar network. The disadvantage of this study is the lack of practical data support [1]. Zhu et al. set up two sets of experiments to compare and analyze the degree of agreement between the three-dimensional reconstruction data and the actual tumor classification found after surgery, as well as the difference between the estimated liver volume of the two-dimensional and three-dimensional images. The average intraoperative blood loss, operation time, and extubation time were significantly reduced, but there was no statistical difference in the postoperative hospital stay and the total complication rate between the two groups. Among the patients in the observation group, the coincidence rate of the bismuth Colette classification by three-dimensional reconstruction and the actual results was 86.7%. There was no statistical difference between the estimated hepatectomy volume calculated by the two-dimensional and three-dimensional techniques. It is concluded that the three-dimensional reconstruction can provide accurate preoperative evaluation for hilar cholangiocarcinoma. It has certain guiding significance for the surgical treatment of hilar cholangiocarcinoma [2]. Ficker Tomá outlines various microreconstruction techniques based on optical sectioning. He used graphical and numerical methods to compare the 3D copy of this special reconstruction frequency method with the 3D copy of the confocal method. Based on comparative studies, it was concluded that the quality of conventional replicas of surfaces with moderately high irregular textures is acceptable, and it is almost the same quality as the confocal replica, but the disadvantage of this experiment is the lack of detailed design [3]. Zhang et al. introduced the high-precision reconstruction method of sinusoidal motion and the factors that affect the reconstruction accuracy. First, he established a stereo vision reconstruction error model that considered delay time, frequency, amplitude, and parallax. He analyzed theoretically and experimentally the accuracy of sinusoidal motion reconstruction considering subpixel interpolation in the entire cycle. Peak recognition is the key to sinusoidal motion reconstruction, with the highest accuracy. But the experiment is more complicated and not practical [4]. Min et al. studied the impact of 23G minimally invasive vitrectomy assisted by optical coherence tomography three-dimensional reconstruction on patients with proliferative diabetic retinopathy. The follow-up time was about 6 months. The operation time, intraoperative complication rate, and postoperative complication rate of the observation group were significantly lower than those of the control group. The BCVA of the observation group was significantly higher than that of the control group, while the intraocular pressure and retinal thickness were significantly lower than that of the control group. In summary, with the aid of OCT three-dimensional imaging, 23G surgery for PDR can improve the efficacy and reduce complications [5]. Terao et al. have developed an accurate method for three-dimensional reconstruction of microstructures to analyze mass transfer phenomena in the microporous layer (MPL). The method he proposed involves the use of focused ion beam scanning electron microscopy tomography. He used the reconstruction results to perform numerical calculations on the structure and mass transfer characteristics of MPL and found that these results are in good agreement with the experimental results, which proves the accuracy of the new method. The disadvantage of this method is that it has higher requirements for the experimental environment [6]. In recent years, there has been a large amount of literature outlining the use of three-dimensional (3D) reconstruction and printing techniques. However, precise guidance articles describing the step-by-step method of reconstructing 3D images from computed tomography (CT) or magnetic resonance imaging (MRI) are still limited. To solve these problems, Chen D described a detailed agreement. This agreement will allow readers to easily perform 3D reconstructions in their future research. It allows investigation of appropriate surgical anatomy and allows innovative design of new screw fixation techniques or preoperative surgical planning. The experiment is innovative but the description of the agreement is not complete [7].


### 1.4. Innovation

The innovations of this article are: (1) This article combines machine learning with virtual and real 3D reconstruction technology. This article introduces the virtual and real 3D reconstruction algorithm based on machine learning and explains the algorithm process. (2) This article designs a comparative experiment. The virtual-real fusion 3D reconstruction algorithm based on machine learning is compared with other commonly used algorithms to verify the superiority of the performance of the virtual-real fusion 3D reconstruction algorithm based on machine learning.


## 1.1. Background

The twenty-first century is the information century and the Internet century. Computer technology is increasingly inseparable from people’s daily life and has become a part of their lives. Whether it is military, education, etc., which are related to the country’s centuries-old plan, or life, shopping, and people’s other livelihoods, computer technology plays a very important role. With the vigorous development of applications such as virtual reality, 3D animation, three-dimensional measurement, virtual teaching, photomicrography, computer fluid mechanics, and military simulation, the research on the fusion of virtual and real three-dimensional reconstruction technology under different backgrounds has become more and more important. The research of 3D reconstruction has experienced the initial modeling of small and medium-sized objects, followed by the modeling of large buildings. The three-dimensional reconstruction of indoor scenes has been developed for many years in the field of scientific research. In this process, various implementation methods have been proposed and improved. Many scholars have proposed various 3D reconstruction methods. Among them, the three-dimensional reconstruction technology based on the machine learning theory algorithm has the advantages of small error, high degree of restoration, and good stability.


## 1.2. Significance

As a new technology in the Internet era, 3D reconstruction technology is developing rapidly. Three-dimensional reconstruction is an important communication bridge between the computer and the real scene and plays an important role in the interaction between virtual reality and humans and computers. At the same time, it can effectively guide human life and is widely used in various fields. Three-dimensional reconstruction is a reverse engineering from reality to the virtual world. The combination of virtual and real means to combine the virtual world of 3D reconstruction with reality to present a good visual effect, which is of great significance to industries such as medicine, artificial intelligence, and education. Machine learning is the core of artificial intelligence, which is the basic method to make computers intelligent. Machine learning mainly studies the methods of computer simulation or implementation of human learning actions to learn new knowledge and skills. The virtual-real fusion 3D reconstruction technology based on machine learning can shorten the image matching time, reduce the error rate of image matching, and increase the accuracy of image matching. The research of virtual and real 3D reconstruction technology based on machine learning can improve the efficiency of 3D reconstruction and bring a better experience to technicians, which has important practical significance.


## 1.3. Related Work

The emergence of 3D reconstruction technology has had a significant impact on the fields of artificial intelligence, robotics, and unmanned driving. So far, many scholars have researched it. Watanabe T used a focused ion beam scanning electron microscope to examine the three-dimensional organization of GAG chains in the Achilles tendons of mature rats embedded with epoxy resin. After staining with cupra iron blue, he specifically dyed the GAG chain. He also used 250 serial backscattered electron images (at 10 nm intervals) in the longitudinal section for reconstruction. The three-dimensional image shows that the GAG chain forms a ring-shaped network structure, each ring surrounds the collagen fibrils in the d-band and merges with adjacent rings to form a planar network. The disadvantage of this study is the lack of practical data support [1]. Zhu et al. set up two sets of experiments to compare and analyze the degree of agreement between the three-dimensional reconstruction data and the actual tumor classification found after surgery, as well as the difference between the estimated liver volume of the two-dimensional and three-dimensional images. The average intraoperative blood loss, operation time, and extubation time were significantly reduced, but there was no statistical difference in the postoperative hospital stay and the total complication rate between the two groups. Among the patients in the observation group, the coincidence rate of the bismuth Colette classification by three-dimensional reconstruction and the actual results was 86.7%. There was no statistical difference between the estimated hepatectomy volume calculated by the two-dimensional and three-dimensional techniques. It is concluded that the three-dimensional reconstruction can provide accurate preoperative evaluation for hilar cholangiocarcinoma. It has certain guiding significance for the surgical treatment of hilar cholangiocarcinoma [2]. Ficker Tomá outlines various microreconstruction techniques based on optical sectioning. He used graphical and numerical methods to compare the 3D copy of this special reconstruction frequency method with the 3D copy of the confocal method. Based on comparative studies, it was concluded that the quality of conventional replicas of surfaces with moderately high irregular textures is acceptable, and it is almost the same quality as the confocal replica, but the disadvantage of this experiment is the lack of detailed design [3]. Zhang et al. introduced the high-precision reconstruction method of sinusoidal motion and the factors that affect the reconstruction accuracy. First, he established a stereo vision reconstruction error model that considered delay time, frequency, amplitude, and parallax. He analyzed theoretically and experimentally the accuracy of sinusoidal motion reconstruction considering subpixel interpolation in the entire cycle. Peak recognition is the key to sinusoidal motion reconstruction, with the highest accuracy. But the experiment is more complicated and not practical [4]. Min et al. studied the impact of 23G minimally invasive vitrectomy assisted by optical coherence tomography three-dimensional reconstruction on patients with proliferative diabetic retinopathy. The follow-up time was about 6 months. The operation time, intraoperative complication rate, and postoperative complication rate of the observation group were significantly lower than those of the control group. The BCVA of the observation group was significantly higher than that of the control group, while the intraocular pressure and retinal thickness were significantly lower than that of the control group. In summary, with the aid of OCT three-dimensional imaging, 23G surgery for PDR can improve the efficacy and reduce complications [5]. Terao et al. have developed an accurate method for three-dimensional reconstruction of microstructures to analyze mass transfer phenomena in the microporous layer (MPL). The method he proposed involves the use of focused ion beam scanning electron microscopy tomography. He used the reconstruction results to perform numerical calculations on the structure and mass transfer characteristics of MPL and found that these results are in good agreement with the experimental results, which proves the accuracy of the new method. The disadvantage of this method is that it has higher requirements for the experimental environment [6]. In recent years, there has been a large amount of literature outlining the use of three-dimensional (3D) reconstruction and printing techniques. However, precise guidance articles describing the step-by-step method of reconstructing 3D images from computed tomography (CT) or magnetic resonance imaging (MRI) are still limited. To solve these problems, Chen D described a detailed agreement. This agreement will allow readers to easily perform 3D reconstructions in their future research. It allows investigation of appropriate surgical anatomy and allows innovative design of new screw fixation techniques or preoperative surgical planning. The experiment is innovative but the description of the agreement is not complete [7].


## 1.4. Innovation

The innovations of this article are: (1) This article combines machine learning with virtual and real 3D reconstruction technology. This article introduces the virtual and real 3D reconstruction algorithm based on machine learning and explains the algorithm process. (2) This article designs a comparative experiment. The virtual-real fusion 3D reconstruction algorithm based on machine learning is compared with other commonly used algorithms to verify the superiority of the performance of the virtual-real fusion 3D reconstruction algorithm based on machine learning.


## 2. 3D Reconstruction Method of Virtual and Real Fusion Based on Machine Learning

### 2.1. 3D Reconstruction and Fusion of Virtual and Real

#### 2.1.1. Overview of 3D Reconstruction and Virtual-Real Fusion

(1) Three-dimensional reconstruction. Generally speaking, the 3D reconstruction technology first uses the vision sensor to obtain the image information in the real world and then obtains the 3D information of the object through techniques such as information matching in the 3D reconstruction. This means that 3D reconstruction is a computer vision method that uses 2D projection to recover 3D information. There are many types of 3D reconstruction techniques, but generally speaking, they can be divided into two types: non-contact method and non-contact method. The non-contact method can be divided into active vision and passive vision. There are more subdivided types, so I will not repeat them here [8].(2) Fusion of virtual and real. The fusion of virtuality and reality, as the name suggests, refers to the fusion of computer-generated virtual objects with real scenes. The most important thing in the fusion of virtual and real is the presentation of the fusion of virtual and real. The virtual and real fusion presentation includes the rendering of the virtual model and the occlusion processing of the virtual and real targets.For the occlusion processing part, the result of virtual and real fusion must correctly simulate the relationship between occlusion and occlusion of virtual objects in the actual scene. The correct occlusion relationship is an important condition for achieving excellent fusion effects. However, most of the existing virtual and real fusion systems only overlap the virtual object with the actual scene image, so the virtual image always obstructs the actual scene. Once there is a real object that is closer to the virtual object, the wrong occlusion relationship will be highlighted and will seriously affect the perception.The rendering of the virtual model affects the integration of the virtual target in the real environment, including information such as color and lighting. Due to the rapid development of computer graphics technology, many types of rendering technologies have emerged, and people can choose according to the desired effect of the image [9].


#### 2.1.2. Three-Dimensional Reconstruction and Virtual-Real Fusion Structure Design Method

The design of 3D reconstruction and virtual-real interaction is shown in Figure1. The system consists of a client host, a camera, and a server. The modeling service sends instructions to the client. Multiple client programs extract images collected by multiple cameras. Then the client sends the data to the server and the server uses the relevant 3D reconstruction algorithm to reconstruct the space to realize the interaction of virtual elastic objects. In this process, a three-dimensional reconstruction method with the combination of virtual and real can be realized.Figure 1
Three-dimensional reconstruction and virtual–real interaction design.


#### 2.1.3. Virtual and Real Fusion Algorithm in 3D Reconstruction

Virtual objects can be viewed as fluids with restricted motion. The interaction between the virtual object and the reconstructed object can be calculated by the microcompressible smooth particle dynamics method [10].The first step is to assume the density of any particlea of the object:(1)ρa=∑iφbiρ0Wxai,h+ma∑jWxaj,h,where j is the neighboring particle index of the virtual object, i is the neighboring mesh vertex index of the reconstructed object, W is the kernel function based on SPH, and Vbi is the volume of the neighboring particle at the i-th boundary, where(2)φbiρ0=ρ0Vbi.This formula is the contribution measurement function. For a pair of adjacent virtual object particlea and reconstructed object mesh vertex j. The interaction force between j and a includes normal force F1 and tangential force F2:(3)F1=−maφjρ0⋅paρa2Wxaj,h,F2=μmaφjρ0⋅vjaρj2∇2Wxaj,h.Here, μ is the friction coefficient of the object, v is the speed, where:(4)vj=xc−xpΔt.In the equation, xc is the position of the object in the current frame and xp represents the position of the object in the previous frame. However, the constraints of the virtual object on the reconstructed object cannot be determined only by the interaction force. Therefore, we propose a position and velocity correction method that preserves momentum. Figure 2 is a schematic diagram of this method [11]:Figure 2
Virtual and real interaction method.


### 2.2. 3D Reconstruction Method of Image Edge Based on Machine Learning

#### 2.2.1. Image Edge Contour Detection

Before detecting the edge of the image, the first step for the image is image preprocessing. Not preprocessing may make image analysis difficult. The spatial method is used to process and filter, and then the image is smoothed. The calculation formula is as follows:(5)fx,y=∑il+βx,y.In the formula,βx,y represents additive white noise and x,y represents the point set in the area. This formula can overcome the blurring of image details caused by filtering [12].The calculation formula for the pixel value at the center of the area is as follows:(6)p=dgG2HF,where g represents the size of the convolution coefficient and F represents the pixel value of the area.


#### 2.2.2. Obtain Three-Dimensional Coordinates

Using the stereo matching method, the position deviation between the corresponding points is calculated and the disparity map is obtained [13]. First, the matching window is selected as the size of the matching window will affect the matching quality. If the set window is too small, the calculation will become difficult and it will become difficult to identify some functions in the window and compare errors. Therefore, the matching window is defined as [14]:(7)Y=Yn−1⊗Bkh,where Y is the pixel point, h is the parameter of the window size, and ⊗ is the image base point.


#### 2.2.3. Based on Machine Learning Image Edge 3D Reconstruction Process

Figure3 is a process diagram of the three-dimensional reconstruction. The three-dimensional reconstruction of the edge of the image should be reconstructed according to the steps in the figure. The smoothed three-dimensional coordinates are obtained by the calculation method of the fitting function. Assuming that the global fitting function of the approximate point set is x, the point set is split and the approximate fitting function is as follows:(8)ηx=∑cpx∂,where ηx is the fitting function, ∂ means the number of basis functions, and p means the number of point sets [15].Figure 3
Three-dimensional reconstruction process.


### 2.3. Model Matching 3D Reconstruction Technology Based on Machine Learning

This section introduces the 3D reconstruction method based on model matching. Based on the segmentation of the two-dimensional image, the image of the indoor scene elements is extracted and then the three-dimensional model is matched through the convolutional neural network based on machine learning. Finally, the extracted matching model is combined with the virtual three-dimensional model constructed by the computer [16] and the final three-dimensional reconstruction product is obtained.


#### 2.3.1. Foundation of Convolutional Neural Network

A convolutional neural network is an efficient recognition method that has been developed in recent years and has attracted widespread attention. It is one of the more popular deep learning architectures in recent years, which is inspired by the visual perception of the human brain in the real world. The convolutional neural network can obtain the effective information of the original image. That is, without or after less preprocessing, some reliable rules can be distinguished from the original image [17]. Machine learning is a theory that learns knowledge and skills through a series of computer research and continues to improve. The idea of solving the problem is roughly shown in Figure 4. First, the sensor performs data preprocessing, then features extraction, and finally performs prediction or recognition instructions. However, most of the machine learning is a manual selection of features, which not only requires time and effort. The characteristics of manual selection are not necessarily effective, and the selection is almost random. Therefore, people put forward the concept of deep learning on this basis. Deep learning itself is a field of machine learning.Figure 4
Basic problem solving process of machine learning.The following introduces a very important unit module convolution layer in the convolutional neural network [18], the following is the formula for convolution forward calculation:(9)xjl=F∑i=mjxjl−1∗ki,jl+bjl.The following is the residual calculation formula of the convolutional layer:(10)δjl=βjl+1F,μjl⋅upδjl+1.Among them, thel + 1 layer is extended to an operation of the same size as the l layer. The following is the gradient calculation formula of the convolutional layer:(11)∂E∂bj=∑u,vδjluv,∂E∂ki,jl=∑u,vδjluvpil−1uv.


#### 2.3.2. Model Matching Method Based on Convolutional Neural Network

Model matching refers to the establishment of a corresponding relationship between image pairs based on the extracted features. That is, the imaging points of the same physical space point in two different images are mapped one by one.The next stage of the work is to compare the 3D model that is most similar to the known image blocks in the 3D model library. It configures it in an appropriate position to reconstruct the three-dimensional scene. A model matching method based on a convolutional neural network is proposed here. Since the detailed classification categories contained in different scene elements are different, to achieve the purpose of classification and matching, the network structure of different elements themselves is designed [19]. The following is a summary of the important modules of the network structure.(1) Pooling. As a result of pooling, features are reduced. The parameter reduction is to maintain a certain invariance (such as rotation, translation, etc.) in the process of image processing at each layer. Its error generation in feature extraction mainly comes from two aspects: one is the variance of the estimated value caused by the limited size of the neighborhood. The second is the deviation of the estimated value caused by the parameter error of the convolutional layer. According to the characteristics of the problem in this article, we choose max pooling as the downsampling layer here.(2) Dropout layer. The dropout method is a breakthrough in the field of deep learning. Figure 5 is a schematic diagram of Dropout. It refers to temporarily suspending the state of certain neurons with a certain probability during the training process so that these neurons are temporarily separated from the information transmission process of this neural network. These neurons are temporarily ignored, but their weights must be maintained and continue to participate in the work during the next sample input [20].Figure 5
Dropout diagram.(3) Activation function. The activation function is very important for the artificial neural network model to learn and understand very complex and non-linear functions. They introduce nonlinear characteristics into our network. Its main purpose is to convert the input signal of a node in the A-NN model into an output signal. This output signal is now used as the input for the next layer in the stack. In this chapter, an activation function is added after each convolutional layer of the network structure [21]. The RELU function is selected here, and its formula is:(12)fx=max0,x.The advantage of this function is high efficiency and a small amount of calculation.


#### 2.3.3. Interactive Image Semantic Segmentation and Annotation

To segment the RGBD image obtained by the user into labeled image blocks, we use a conditional random field (CRF) model to solve the labeling problem. The conditional random field energy function for labelC is as follows:(13)EC=∑iE1Ci:Xi+λE2Ci:Xj,where E1Ci:Xi represents the possibility that pixel i belongs to the category and is the feature of pixel i. E2Ci:Xj is a compatibility item, which measures the consistency of the categories between two adjacent pixels.(1) Principle of semantic segmentation. Semantic segmentation is a typical computer vision problem. It involves taking some raw data (e.g., flat images) as input and converting them into a mask with highlighted regions of interest. What is evaluated is the possibility that the feature of pixel i belongs to the category label [22]. It is composed of the sum of two data items, the appearance model and the geometric model, which are obtained by the color information and the depth map information obtained by the depth camera:(14)E1Ci:Xi=EaCi:Xia+EgCi:Xig.The former item represents the appearance item and the latter item represents the geometric item. For the local appearance feature, the local color information is used to compute the geometric feature. The appearance term is calculated as follows:(15)EaCi:Xia=−log1−αppCi|Xia+αppCi|Xia.We extract SIFT features from each pixel in the RGB image and the depth image. Then connect them into RGBD SIFT features, and use a two-layer neural network and a supervised learning method of cross-entropy loss function to calculate. We combine the classification results of all pixels as the result of image segmentation, and then use RGB data to perform K-means operation [23]. The calculation formula is as follows:(16)pCi|Xia=1/−dXia,Ci+ε∑j1/−dXia,Ci+ε,where dXia,Ci represents the distance between the current RGB pixel and the nearest cluster center and ε is a minimal parameter.The calculation formula for geometric terms is as follows:(17)EgCi:Xig=−log1−αgpCi|Xig+αgpCi|Xig.pCi|Xig is trained from the depth data of the scene data set and pCi|Xig is the segmentation result of the previous layer [24].The basic plane image is used to extract local geometric features instead of local depth data because the basic plane image has less noise than the depth data. The information extraction of the basic floor plan uses the more efficient RANSAC algorithm.Compatibility items are used to enhance the smoothness of labels between adjacent pixels. The calculation method is as follows:(18)E2Ci:Cj=dCi≠CjsimFi:Fj.Among them,Fi=r,g,b,tT, where Fi is the connection vector of the RGB value and the depth value at pixel i. The similarity calculation method between two pixels is:(19)simFi:Fj=exp−Fi−Fj22d2,where, d is the average distance between features [25]. Fi and Fj respectively refer to two pixels and the similarity of these two pixels is calculated by exp.


## 2.1. 3D Reconstruction and Fusion of Virtual and Real

### 2.1.1. Overview of 3D Reconstruction and Virtual-Real Fusion

(1) Three-dimensional reconstruction. Generally speaking, the 3D reconstruction technology first uses the vision sensor to obtain the image information in the real world and then obtains the 3D information of the object through techniques such as information matching in the 3D reconstruction. This means that 3D reconstruction is a computer vision method that uses 2D projection to recover 3D information. There are many types of 3D reconstruction techniques, but generally speaking, they can be divided into two types: non-contact method and non-contact method. The non-contact method can be divided into active vision and passive vision. There are more subdivided types, so I will not repeat them here [8].(2) Fusion of virtual and real. The fusion of virtuality and reality, as the name suggests, refers to the fusion of computer-generated virtual objects with real scenes. The most important thing in the fusion of virtual and real is the presentation of the fusion of virtual and real. The virtual and real fusion presentation includes the rendering of the virtual model and the occlusion processing of the virtual and real targets.For the occlusion processing part, the result of virtual and real fusion must correctly simulate the relationship between occlusion and occlusion of virtual objects in the actual scene. The correct occlusion relationship is an important condition for achieving excellent fusion effects. However, most of the existing virtual and real fusion systems only overlap the virtual object with the actual scene image, so the virtual image always obstructs the actual scene. Once there is a real object that is closer to the virtual object, the wrong occlusion relationship will be highlighted and will seriously affect the perception.The rendering of the virtual model affects the integration of the virtual target in the real environment, including information such as color and lighting. Due to the rapid development of computer graphics technology, many types of rendering technologies have emerged, and people can choose according to the desired effect of the image [9].


### 2.1.2. Three-Dimensional Reconstruction and Virtual-Real Fusion Structure Design Method

The design of 3D reconstruction and virtual-real interaction is shown in Figure1. The system consists of a client host, a camera, and a server. The modeling service sends instructions to the client. Multiple client programs extract images collected by multiple cameras. Then the client sends the data to the server and the server uses the relevant 3D reconstruction algorithm to reconstruct the space to realize the interaction of virtual elastic objects. In this process, a three-dimensional reconstruction method with the combination of virtual and real can be realized.Figure 1
Three-dimensional reconstruction and virtual–real interaction design.


### 2.1.3. Virtual and Real Fusion Algorithm in 3D Reconstruction

Virtual objects can be viewed as fluids with restricted motion. The interaction between the virtual object and the reconstructed object can be calculated by the microcompressible smooth particle dynamics method [10].The first step is to assume the density of any particlea of the object:(1)ρa=∑iφbiρ0Wxai,h+ma∑jWxaj,h,where j is the neighboring particle index of the virtual object, i is the neighboring mesh vertex index of the reconstructed object, W is the kernel function based on SPH, and Vbi is the volume of the neighboring particle at the i-th boundary, where(2)φbiρ0=ρ0Vbi.This formula is the contribution measurement function. For a pair of adjacent virtual object particlea and reconstructed object mesh vertex j. The interaction force between j and a includes normal force F1 and tangential force F2:(3)F1=−maφjρ0⋅paρa2Wxaj,h,F2=μmaφjρ0⋅vjaρj2∇2Wxaj,h.Here, μ is the friction coefficient of the object, v is the speed, where:(4)vj=xc−xpΔt.In the equation, xc is the position of the object in the current frame and xp represents the position of the object in the previous frame. However, the constraints of the virtual object on the reconstructed object cannot be determined only by the interaction force. Therefore, we propose a position and velocity correction method that preserves momentum. Figure 2 is a schematic diagram of this method [11]:Figure 2
Virtual and real interaction method.


## 2.1.1. Overview of 3D Reconstruction and Virtual-Real Fusion

(1) Three-dimensional reconstruction. Generally speaking, the 3D reconstruction technology first uses the vision sensor to obtain the image information in the real world and then obtains the 3D information of the object through techniques such as information matching in the 3D reconstruction. This means that 3D reconstruction is a computer vision method that uses 2D projection to recover 3D information. There are many types of 3D reconstruction techniques, but generally speaking, they can be divided into two types: non-contact method and non-contact method. The non-contact method can be divided into active vision and passive vision. There are more subdivided types, so I will not repeat them here [8].(2) Fusion of virtual and real. The fusion of virtuality and reality, as the name suggests, refers to the fusion of computer-generated virtual objects with real scenes. The most important thing in the fusion of virtual and real is the presentation of the fusion of virtual and real. The virtual and real fusion presentation includes the rendering of the virtual model and the occlusion processing of the virtual and real targets.For the occlusion processing part, the result of virtual and real fusion must correctly simulate the relationship between occlusion and occlusion of virtual objects in the actual scene. The correct occlusion relationship is an important condition for achieving excellent fusion effects. However, most of the existing virtual and real fusion systems only overlap the virtual object with the actual scene image, so the virtual image always obstructs the actual scene. Once there is a real object that is closer to the virtual object, the wrong occlusion relationship will be highlighted and will seriously affect the perception.The rendering of the virtual model affects the integration of the virtual target in the real environment, including information such as color and lighting. Due to the rapid development of computer graphics technology, many types of rendering technologies have emerged, and people can choose according to the desired effect of the image [9].


## 2.1.2. Three-Dimensional Reconstruction and Virtual-Real Fusion Structure Design Method

The design of 3D reconstruction and virtual-real interaction is shown in Figure1. The system consists of a client host, a camera, and a server. The modeling service sends instructions to the client. Multiple client programs extract images collected by multiple cameras. Then the client sends the data to the server and the server uses the relevant 3D reconstruction algorithm to reconstruct the space to realize the interaction of virtual elastic objects. In this process, a three-dimensional reconstruction method with the combination of virtual and real can be realized.Figure 1
Three-dimensional reconstruction and virtual–real interaction design.


## 2.1.3. Virtual and Real Fusion Algorithm in 3D Reconstruction

Virtual objects can be viewed as fluids with restricted motion. The interaction between the virtual object and the reconstructed object can be calculated by the microcompressible smooth particle dynamics method [10].The first step is to assume the density of any particlea of the object:(1)ρa=∑iφbiρ0Wxai,h+ma∑jWxaj,h,where j is the neighboring particle index of the virtual object, i is the neighboring mesh vertex index of the reconstructed object, W is the kernel function based on SPH, and Vbi is the volume of the neighboring particle at the i-th boundary, where(2)φbiρ0=ρ0Vbi.This formula is the contribution measurement function. For a pair of adjacent virtual object particlea and reconstructed object mesh vertex j. The interaction force between j and a includes normal force F1 and tangential force F2:(3)F1=−maφjρ0⋅paρa2Wxaj,h,F2=μmaφjρ0⋅vjaρj2∇2Wxaj,h.Here, μ is the friction coefficient of the object, v is the speed, where:(4)vj=xc−xpΔt.In the equation, xc is the position of the object in the current frame and xp represents the position of the object in the previous frame. However, the constraints of the virtual object on the reconstructed object cannot be determined only by the interaction force. Therefore, we propose a position and velocity correction method that preserves momentum. Figure 2 is a schematic diagram of this method [11]:Figure 2
Virtual and real interaction method.


## 2.2. 3D Reconstruction Method of Image Edge Based on Machine Learning

### 2.2.1. Image Edge Contour Detection

Before detecting the edge of the image, the first step for the image is image preprocessing. Not preprocessing may make image analysis difficult. The spatial method is used to process and filter, and then the image is smoothed. The calculation formula is as follows:(5)fx,y=∑il+βx,y.In the formula,βx,y represents additive white noise and x,y represents the point set in the area. This formula can overcome the blurring of image details caused by filtering [12].The calculation formula for the pixel value at the center of the area is as follows:(6)p=dgG2HF,where g represents the size of the convolution coefficient and F represents the pixel value of the area.


### 2.2.2. Obtain Three-Dimensional Coordinates

Using the stereo matching method, the position deviation between the corresponding points is calculated and the disparity map is obtained [13]. First, the matching window is selected as the size of the matching window will affect the matching quality. If the set window is too small, the calculation will become difficult and it will become difficult to identify some functions in the window and compare errors. Therefore, the matching window is defined as [14]:(7)Y=Yn−1⊗Bkh,where Y is the pixel point, h is the parameter of the window size, and ⊗ is the image base point.


### 2.2.3. Based on Machine Learning Image Edge 3D Reconstruction Process

Figure3 is a process diagram of the three-dimensional reconstruction. The three-dimensional reconstruction of the edge of the image should be reconstructed according to the steps in the figure. The smoothed three-dimensional coordinates are obtained by the calculation method of the fitting function. Assuming that the global fitting function of the approximate point set is x, the point set is split and the approximate fitting function is as follows:(8)ηx=∑cpx∂,where ηx is the fitting function, ∂ means the number of basis functions, and p means the number of point sets [15].Figure 3
Three-dimensional reconstruction process.


## 2.2.1. Image Edge Contour Detection

Before detecting the edge of the image, the first step for the image is image preprocessing. Not preprocessing may make image analysis difficult. The spatial method is used to process and filter, and then the image is smoothed. The calculation formula is as follows:(5)fx,y=∑il+βx,y.In the formula,βx,y represents additive white noise and x,y represents the point set in the area. This formula can overcome the blurring of image details caused by filtering [12].The calculation formula for the pixel value at the center of the area is as follows:(6)p=dgG2HF,where g represents the size of the convolution coefficient and F represents the pixel value of the area.


## 2.2.2. Obtain Three-Dimensional Coordinates

Using the stereo matching method, the position deviation between the corresponding points is calculated and the disparity map is obtained [13]. First, the matching window is selected as the size of the matching window will affect the matching quality. If the set window is too small, the calculation will become difficult and it will become difficult to identify some functions in the window and compare errors. Therefore, the matching window is defined as [14]:(7)Y=Yn−1⊗Bkh,where Y is the pixel point, h is the parameter of the window size, and ⊗ is the image base point.


## 2.2.3. Based on Machine Learning Image Edge 3D Reconstruction Process

Figure3 is a process diagram of the three-dimensional reconstruction. The three-dimensional reconstruction of the edge of the image should be reconstructed according to the steps in the figure. The smoothed three-dimensional coordinates are obtained by the calculation method of the fitting function. Assuming that the global fitting function of the approximate point set is x, the point set is split and the approximate fitting function is as follows:(8)ηx=∑cpx∂,where ηx is the fitting function, ∂ means the number of basis functions, and p means the number of point sets [15].Figure 3
Three-dimensional reconstruction process.


## 2.3. Model Matching 3D Reconstruction Technology Based on Machine Learning

This section introduces the 3D reconstruction method based on model matching. Based on the segmentation of the two-dimensional image, the image of the indoor scene elements is extracted and then the three-dimensional model is matched through the convolutional neural network based on machine learning. Finally, the extracted matching model is combined with the virtual three-dimensional model constructed by the computer [16] and the final three-dimensional reconstruction product is obtained.


### 2.3.1. Foundation of Convolutional Neural Network

A convolutional neural network is an efficient recognition method that has been developed in recent years and has attracted widespread attention. It is one of the more popular deep learning architectures in recent years, which is inspired by the visual perception of the human brain in the real world. The convolutional neural network can obtain the effective information of the original image. That is, without or after less preprocessing, some reliable rules can be distinguished from the original image [17]. Machine learning is a theory that learns knowledge and skills through a series of computer research and continues to improve. The idea of solving the problem is roughly shown in Figure 4. First, the sensor performs data preprocessing, then features extraction, and finally performs prediction or recognition instructions. However, most of the machine learning is a manual selection of features, which not only requires time and effort. The characteristics of manual selection are not necessarily effective, and the selection is almost random. Therefore, people put forward the concept of deep learning on this basis. Deep learning itself is a field of machine learning.Figure 4
Basic problem solving process of machine learning.The following introduces a very important unit module convolution layer in the convolutional neural network [18], the following is the formula for convolution forward calculation:(9)xjl=F∑i=mjxjl−1∗ki,jl+bjl.The following is the residual calculation formula of the convolutional layer:(10)δjl=βjl+1F,μjl⋅upδjl+1.Among them, thel + 1 layer is extended to an operation of the same size as the l layer. The following is the gradient calculation formula of the convolutional layer:(11)∂E∂bj=∑u,vδjluv,∂E∂ki,jl=∑u,vδjluvpil−1uv.


### 2.3.2. Model Matching Method Based on Convolutional Neural Network

Model matching refers to the establishment of a corresponding relationship between image pairs based on the extracted features. That is, the imaging points of the same physical space point in two different images are mapped one by one.The next stage of the work is to compare the 3D model that is most similar to the known image blocks in the 3D model library. It configures it in an appropriate position to reconstruct the three-dimensional scene. A model matching method based on a convolutional neural network is proposed here. Since the detailed classification categories contained in different scene elements are different, to achieve the purpose of classification and matching, the network structure of different elements themselves is designed [19]. The following is a summary of the important modules of the network structure.(1) Pooling. As a result of pooling, features are reduced. The parameter reduction is to maintain a certain invariance (such as rotation, translation, etc.) in the process of image processing at each layer. Its error generation in feature extraction mainly comes from two aspects: one is the variance of the estimated value caused by the limited size of the neighborhood. The second is the deviation of the estimated value caused by the parameter error of the convolutional layer. According to the characteristics of the problem in this article, we choose max pooling as the downsampling layer here.(2) Dropout layer. The dropout method is a breakthrough in the field of deep learning. Figure 5 is a schematic diagram of Dropout. It refers to temporarily suspending the state of certain neurons with a certain probability during the training process so that these neurons are temporarily separated from the information transmission process of this neural network. These neurons are temporarily ignored, but their weights must be maintained and continue to participate in the work during the next sample input [20].Figure 5
Dropout diagram.(3) Activation function. The activation function is very important for the artificial neural network model to learn and understand very complex and non-linear functions. They introduce nonlinear characteristics into our network. Its main purpose is to convert the input signal of a node in the A-NN model into an output signal. This output signal is now used as the input for the next layer in the stack. In this chapter, an activation function is added after each convolutional layer of the network structure [21]. The RELU function is selected here, and its formula is:(12)fx=max0,x.The advantage of this function is high efficiency and a small amount of calculation.


### 2.3.3. Interactive Image Semantic Segmentation and Annotation

To segment the RGBD image obtained by the user into labeled image blocks, we use a conditional random field (CRF) model to solve the labeling problem. The conditional random field energy function for labelC is as follows:(13)EC=∑iE1Ci:Xi+λE2Ci:Xj,where E1Ci:Xi represents the possibility that pixel i belongs to the category and is the feature of pixel i. E2Ci:Xj is a compatibility item, which measures the consistency of the categories between two adjacent pixels.(1) Principle of semantic segmentation. Semantic segmentation is a typical computer vision problem. It involves taking some raw data (e.g., flat images) as input and converting them into a mask with highlighted regions of interest. What is evaluated is the possibility that the feature of pixel i belongs to the category label [22]. It is composed of the sum of two data items, the appearance model and the geometric model, which are obtained by the color information and the depth map information obtained by the depth camera:(14)E1Ci:Xi=EaCi:Xia+EgCi:Xig.The former item represents the appearance item and the latter item represents the geometric item. For the local appearance feature, the local color information is used to compute the geometric feature. The appearance term is calculated as follows:(15)EaCi:Xia=−log1−αppCi|Xia+αppCi|Xia.We extract SIFT features from each pixel in the RGB image and the depth image. Then connect them into RGBD SIFT features, and use a two-layer neural network and a supervised learning method of cross-entropy loss function to calculate. We combine the classification results of all pixels as the result of image segmentation, and then use RGB data to perform K-means operation [23]. The calculation formula is as follows:(16)pCi|Xia=1/−dXia,Ci+ε∑j1/−dXia,Ci+ε,where dXia,Ci represents the distance between the current RGB pixel and the nearest cluster center and ε is a minimal parameter.The calculation formula for geometric terms is as follows:(17)EgCi:Xig=−log1−αgpCi|Xig+αgpCi|Xig.pCi|Xig is trained from the depth data of the scene data set and pCi|Xig is the segmentation result of the previous layer [24].The basic plane image is used to extract local geometric features instead of local depth data because the basic plane image has less noise than the depth data. The information extraction of the basic floor plan uses the more efficient RANSAC algorithm.Compatibility items are used to enhance the smoothness of labels between adjacent pixels. The calculation method is as follows:(18)E2Ci:Cj=dCi≠CjsimFi:Fj.Among them,Fi=r,g,b,tT, where Fi is the connection vector of the RGB value and the depth value at pixel i. The similarity calculation method between two pixels is:(19)simFi:Fj=exp−Fi−Fj22d2,where, d is the average distance between features [25]. Fi and Fj respectively refer to two pixels and the similarity of these two pixels is calculated by exp.


## 2.3.1. Foundation of Convolutional Neural Network

A convolutional neural network is an efficient recognition method that has been developed in recent years and has attracted widespread attention. It is one of the more popular deep learning architectures in recent years, which is inspired by the visual perception of the human brain in the real world. The convolutional neural network can obtain the effective information of the original image. That is, without or after less preprocessing, some reliable rules can be distinguished from the original image [17]. Machine learning is a theory that learns knowledge and skills through a series of computer research and continues to improve. The idea of solving the problem is roughly shown in Figure 4. First, the sensor performs data preprocessing, then features extraction, and finally performs prediction or recognition instructions. However, most of the machine learning is a manual selection of features, which not only requires time and effort. The characteristics of manual selection are not necessarily effective, and the selection is almost random. Therefore, people put forward the concept of deep learning on this basis. Deep learning itself is a field of machine learning.Figure 4
Basic problem solving process of machine learning.The following introduces a very important unit module convolution layer in the convolutional neural network [18], the following is the formula for convolution forward calculation:(9)xjl=F∑i=mjxjl−1∗ki,jl+bjl.The following is the residual calculation formula of the convolutional layer:(10)δjl=βjl+1F,μjl⋅upδjl+1.Among them, thel + 1 layer is extended to an operation of the same size as the l layer. The following is the gradient calculation formula of the convolutional layer:(11)∂E∂bj=∑u,vδjluv,∂E∂ki,jl=∑u,vδjluvpil−1uv.


## 2.3.2. Model Matching Method Based on Convolutional Neural Network

Model matching refers to the establishment of a corresponding relationship between image pairs based on the extracted features. That is, the imaging points of the same physical space point in two different images are mapped one by one.The next stage of the work is to compare the 3D model that is most similar to the known image blocks in the 3D model library. It configures it in an appropriate position to reconstruct the three-dimensional scene. A model matching method based on a convolutional neural network is proposed here. Since the detailed classification categories contained in different scene elements are different, to achieve the purpose of classification and matching, the network structure of different elements themselves is designed [19]. The following is a summary of the important modules of the network structure.(1) Pooling. As a result of pooling, features are reduced. The parameter reduction is to maintain a certain invariance (such as rotation, translation, etc.) in the process of image processing at each layer. Its error generation in feature extraction mainly comes from two aspects: one is the variance of the estimated value caused by the limited size of the neighborhood. The second is the deviation of the estimated value caused by the parameter error of the convolutional layer. According to the characteristics of the problem in this article, we choose max pooling as the downsampling layer here.(2) Dropout layer. The dropout method is a breakthrough in the field of deep learning. Figure 5 is a schematic diagram of Dropout. It refers to temporarily suspending the state of certain neurons with a certain probability during the training process so that these neurons are temporarily separated from the information transmission process of this neural network. These neurons are temporarily ignored, but their weights must be maintained and continue to participate in the work during the next sample input [20].Figure 5
Dropout diagram.(3) Activation function. The activation function is very important for the artificial neural network model to learn and understand very complex and non-linear functions. They introduce nonlinear characteristics into our network. Its main purpose is to convert the input signal of a node in the A-NN model into an output signal. This output signal is now used as the input for the next layer in the stack. In this chapter, an activation function is added after each convolutional layer of the network structure [21]. The RELU function is selected here, and its formula is:(12)fx=max0,x.The advantage of this function is high efficiency and a small amount of calculation.


## 2.3.3. Interactive Image Semantic Segmentation and Annotation

To segment the RGBD image obtained by the user into labeled image blocks, we use a conditional random field (CRF) model to solve the labeling problem. The conditional random field energy function for labelC is as follows:(13)EC=∑iE1Ci:Xi+λE2Ci:Xj,where E1Ci:Xi represents the possibility that pixel i belongs to the category and is the feature of pixel i. E2Ci:Xj is a compatibility item, which measures the consistency of the categories between two adjacent pixels.(1) Principle of semantic segmentation. Semantic segmentation is a typical computer vision problem. It involves taking some raw data (e.g., flat images) as input and converting them into a mask with highlighted regions of interest. What is evaluated is the possibility that the feature of pixel i belongs to the category label [22]. It is composed of the sum of two data items, the appearance model and the geometric model, which are obtained by the color information and the depth map information obtained by the depth camera:(14)E1Ci:Xi=EaCi:Xia+EgCi:Xig.The former item represents the appearance item and the latter item represents the geometric item. For the local appearance feature, the local color information is used to compute the geometric feature. The appearance term is calculated as follows:(15)EaCi:Xia=−log1−αppCi|Xia+αppCi|Xia.We extract SIFT features from each pixel in the RGB image and the depth image. Then connect them into RGBD SIFT features, and use a two-layer neural network and a supervised learning method of cross-entropy loss function to calculate. We combine the classification results of all pixels as the result of image segmentation, and then use RGB data to perform K-means operation [23]. The calculation formula is as follows:(16)pCi|Xia=1/−dXia,Ci+ε∑j1/−dXia,Ci+ε,where dXia,Ci represents the distance between the current RGB pixel and the nearest cluster center and ε is a minimal parameter.The calculation formula for geometric terms is as follows:(17)EgCi:Xig=−log1−αgpCi|Xig+αgpCi|Xig.pCi|Xig is trained from the depth data of the scene data set and pCi|Xig is the segmentation result of the previous layer [24].The basic plane image is used to extract local geometric features instead of local depth data because the basic plane image has less noise than the depth data. The information extraction of the basic floor plan uses the more efficient RANSAC algorithm.Compatibility items are used to enhance the smoothness of labels between adjacent pixels. The calculation method is as follows:(18)E2Ci:Cj=dCi≠CjsimFi:Fj.Among them,Fi=r,g,b,tT, where Fi is the connection vector of the RGB value and the depth value at pixel i. The similarity calculation method between two pixels is:(19)simFi:Fj=exp−Fi−Fj22d2,where, d is the average distance between features [25]. Fi and Fj respectively refer to two pixels and the similarity of these two pixels is calculated by exp.


## 3. 3D Reconstruction Technology Experiment Based on Virtual and Real Fusion Based on Machine Learning

### 3.1. Test System Design

#### 3.1.1. Hardware Platform Configuration

The experimental system of this study uses Kinect sensor as the input device to obtain depth data, and the processing part uses a computer, and the computer reconstructs the obtained image data. This system only uses one Kinect device, no additional auxiliary equipment is needed, and the hardware platform is easy to build. In this experiment, holding the Kinect device to scan the object, or fixing the Kinect sensor, and rotating the target to realize the complete collection of object information. The hardware platform and configuration are shown in Table1. Kinect sensors are superior in terms of performance and price, and computers with higher running memory are more suitable for 3D reconstruction.Table 1
Hardware configuration.
EquipmentTo configureKinectMicrosoft kinect for XboxKinect power adapterComputerOperating system:Windows 7 64 bitCPU:i5-4200U,1.6GHZRAM: 4GThe following focuses on the Kinect depth sensor, which is an important equipment in the experiment.(1) Kinect. Kinect is a fusion of many current advanced machine vision technologies, which can collect the color and depth information of the object at the same time. Compared with other devices, Kinect is not only cheap but also can obtain high-resolution depth maps and color maps at the same time. The hardware structure is mainly divided into three parts. They are a system-on-chip, three cameras, and a four-element microphone array. The basic parameters of Kinect are shown in Table 2:Table 2
Kinect basic parameters.
Characteristic specificationParameterMaximum frame rate30 fpsDepth range40 cm–400 cmDepth image resolution640 × 480, 11 bitColor image resolution640 × 480, 8 bitKinect can obtain 640∗ 480 pixel depth information and color information at a frame rate of up to 30 fps. By combining the infrared CMOS sensor and the color camera, the three-dimensional information of the target can be projected onto the screen. Its working principle is shown in Figure 6:Figure 6
Kinect working principle diagram.The shooting frame rate of Kinect has reached 30 fps, which will not be a shortcoming of the system frame rate. The processing speed of the system is often lower than the input speed of the image, so the processing speed determines the frame rate of the final system. The main hardware factors that affect the frame rate are the video memory capacity and the number of cores of the graphics card. After testing on different devices, the results are shown in Table3:Table 3
System frame rate.
Memory size (GB)Memory size (MB)Number of CUDA coresFrame rate (fps)4512192Cannot execute410245761282048134421


#### 3.1.2. Software Platform Configuration

The development environment used in this article is Win7 64 bit, the development platform is Visual Studio 2010, and the interface is designed with Qt4.8. Although Microsoft provides a software development kit Kinect SDK and related drivers, these software development kits are mainly for somatosensory control, gesture recognition, tracking, and voice control. The latest Kinect SDK 1.7 is more powerful and also includes the realization of algorithms such as KinectFusion. The focus of this article is to achieve the three-dimensional reconstruction of the target object. The existing open-source library contains a large number of resources, which is convenient for learning and using.


#### 3.1.3. System Structure Design

Although different methods use different 3D reconstruction techniques, the main process of 3D reconstruction is roughly the same. The system function module is shown in Figure7:Figure 7
3D reconstruction function module.The six functional modules we designed for the 3D reconstruction system are image acquisition module, registration module, surface extraction module, preprocessing module, texture mapping module, and output display module. Aiming at these six functional modules, the software architecture of the reconstruction system is designed. In this architecture, the Kinect sensor is first used to obtain depth image information and color image information. Since the image contains noise information, the data needs to be preprocessed. The depth information is coordinate-converted into 3D point cloud data and then filtering and denoising operation is performed, and then the registration and alignment operation is performed on the 3D point cloud data set under multiple viewing angles. After registration, the point cloud data is fused according to the corresponding spatial conversion relationship, and the color RGB image is used for texture mapping, so as to output a complete 3D reconstruction model of the target object.


#### 3.1.4. Comparative Test Design

The experiment is based on the built hardware and software platform using Kinect sensors and computers to analyze the performance of the above-mentioned virtual-real fusion 3D reconstruction technology is based on machine learning, and the experiment selects other 3D reconstruction techniques for comparison to obtain the performance data of each algorithm, such as speed, error rate, correct rate, and other information, so as to understand whether the 3D reconstruction fusion method based on machine learning is more superior. The comparison algorithms selected in this experiment are SGM, SFM, and CMMs (coordinate measuring machines). These algorithms belong to different kinds of algorithms in 3D reconstruction and they are used frequently. The performance data of each algorithm is obtained through experiments, and the performance of the virtual and real 3D reconstruction algorithm based on machine learning is understood.


### 3.2. Test Results and Analysis

#### 3.2.1. Image Reconstruction Speed

Testing the speed of image reconstruction in the figure below for each algorithm, increasing the number of experiments, calculating the average value of the speed under different number of experiments, and getting the speed data of each algorithm is shown in Figure8:Figure 8
The speed of different algorithms.According to the data in Figure8, the speed of the 3D reconstruction algorithm based on machine learning in this article first slowly increases with the increase of the number of trials. After the number of times is greater than 30, the speed increases, and this algorithm is always faster than the other several algorithms. The average speed under different test times is 72.9%. Therefore, it is concluded that the 3D reconstruction algorithm based on machine learning can greatly reduce the image reconstruction time and increase the efficiency of 3D reconstruction processing.


#### 3.2.2. The Evaluation Value of Each Index in the Image Test Set

In this part of the experiment, the experiment chose to compare 3 quantitative parameters, namely the non-texture area matching error percentage (untex), all pixel matching error percentage (all), parallax discontinuous area matching error percentage (disc) and matching, and the time used (time). The evaluation values of these indicators of the four algorithms are all measured, and the data obtained is shown in Figure9:Figure 9
Evaluation values of various indicators in the image test set of different algorithms.It can be seen from Figure9 whether it is the test result of the nontexture area matching error percentage, all pixel matching error percentage, or parallax discontinuous area matching error percentage. The error percentage of the 3D reconstruction algorithm based on machine learning is the lowest, among them, and 3d reconstruction algorithm based on machine learning has the lowest error rate on untex index and the highest error rate on disc. But the error rate for each indicator was less than 10%, which proves that the 3D reconstruction algorithm based on machine learning has a low error rate and high efficiency.


#### 3.2.3. Accuracy Comparison

In this part of the experiment, we put the reconstructed object indoors through the matching of indoor objects, tables, lamps, sofas, chairs, and beds to understand the matching accuracy of each algorithm and recorded the data. The results are shown in Figure10:Figure 10
Matching accuracy of different algorithms.It can be seen from Figure10 that the 3D reconstruction algorithm based on machine learning has the highest matching accuracy no matter which object is in the room. Among them, the chair matching accuracy is the highest and the bed matching accuracy is the lowest. But the accuracy was better than 0.8 for any object. After calculation, for different objects, the average matching accuracy of the 3D reconstruction algorithm based on machine learning is about 0.87. The average matching accuracy of other algorithms SGM, SFM, and CMMS for different objects is about 0.84, 0.84, and 0.83, respectively. It can be seen that the accuracy of the matching rate of several algorithms is relatively close, but the accuracy of the 3D reconstruction algorithm based on machine learning is the best.


## 3.1. Test System Design

### 3.1.1. Hardware Platform Configuration

The experimental system of this study uses Kinect sensor as the input device to obtain depth data, and the processing part uses a computer, and the computer reconstructs the obtained image data. This system only uses one Kinect device, no additional auxiliary equipment is needed, and the hardware platform is easy to build. In this experiment, holding the Kinect device to scan the object, or fixing the Kinect sensor, and rotating the target to realize the complete collection of object information. The hardware platform and configuration are shown in Table1. Kinect sensors are superior in terms of performance and price, and computers with higher running memory are more suitable for 3D reconstruction.Table 1
Hardware configuration.
EquipmentTo configureKinectMicrosoft kinect for XboxKinect power adapterComputerOperating system:Windows 7 64 bitCPU:i5-4200U,1.6GHZRAM: 4GThe following focuses on the Kinect depth sensor, which is an important equipment in the experiment.(1) Kinect. Kinect is a fusion of many current advanced machine vision technologies, which can collect the color and depth information of the object at the same time. Compared with other devices, Kinect is not only cheap but also can obtain high-resolution depth maps and color maps at the same time. The hardware structure is mainly divided into three parts. They are a system-on-chip, three cameras, and a four-element microphone array. The basic parameters of Kinect are shown in Table 2:Table 2
Kinect basic parameters.
Characteristic specificationParameterMaximum frame rate30 fpsDepth range40 cm–400 cmDepth image resolution640 × 480, 11 bitColor image resolution640 × 480, 8 bitKinect can obtain 640∗ 480 pixel depth information and color information at a frame rate of up to 30 fps. By combining the infrared CMOS sensor and the color camera, the three-dimensional information of the target can be projected onto the screen. Its working principle is shown in Figure 6:Figure 6
Kinect working principle diagram.The shooting frame rate of Kinect has reached 30 fps, which will not be a shortcoming of the system frame rate. The processing speed of the system is often lower than the input speed of the image, so the processing speed determines the frame rate of the final system. The main hardware factors that affect the frame rate are the video memory capacity and the number of cores of the graphics card. After testing on different devices, the results are shown in Table3:Table 3
System frame rate.
Memory size (GB)Memory size (MB)Number of CUDA coresFrame rate (fps)4512192Cannot execute410245761282048134421


### 3.1.2. Software Platform Configuration

The development environment used in this article is Win7 64 bit, the development platform is Visual Studio 2010, and the interface is designed with Qt4.8. Although Microsoft provides a software development kit Kinect SDK and related drivers, these software development kits are mainly for somatosensory control, gesture recognition, tracking, and voice control. The latest Kinect SDK 1.7 is more powerful and also includes the realization of algorithms such as KinectFusion. The focus of this article is to achieve the three-dimensional reconstruction of the target object. The existing open-source library contains a large number of resources, which is convenient for learning and using.


### 3.1.3. System Structure Design

Although different methods use different 3D reconstruction techniques, the main process of 3D reconstruction is roughly the same. The system function module is shown in Figure7:Figure 7
3D reconstruction function module.The six functional modules we designed for the 3D reconstruction system are image acquisition module, registration module, surface extraction module, preprocessing module, texture mapping module, and output display module. Aiming at these six functional modules, the software architecture of the reconstruction system is designed. In this architecture, the Kinect sensor is first used to obtain depth image information and color image information. Since the image contains noise information, the data needs to be preprocessed. The depth information is coordinate-converted into 3D point cloud data and then filtering and denoising operation is performed, and then the registration and alignment operation is performed on the 3D point cloud data set under multiple viewing angles. After registration, the point cloud data is fused according to the corresponding spatial conversion relationship, and the color RGB image is used for texture mapping, so as to output a complete 3D reconstruction model of the target object.


### 3.1.4. Comparative Test Design

The experiment is based on the built hardware and software platform using Kinect sensors and computers to analyze the performance of the above-mentioned virtual-real fusion 3D reconstruction technology is based on machine learning, and the experiment selects other 3D reconstruction techniques for comparison to obtain the performance data of each algorithm, such as speed, error rate, correct rate, and other information, so as to understand whether the 3D reconstruction fusion method based on machine learning is more superior. The comparison algorithms selected in this experiment are SGM, SFM, and CMMs (coordinate measuring machines). These algorithms belong to different kinds of algorithms in 3D reconstruction and they are used frequently. The performance data of each algorithm is obtained through experiments, and the performance of the virtual and real 3D reconstruction algorithm based on machine learning is understood.


## 3.1.1. Hardware Platform Configuration

The experimental system of this study uses Kinect sensor as the input device to obtain depth data, and the processing part uses a computer, and the computer reconstructs the obtained image data. This system only uses one Kinect device, no additional auxiliary equipment is needed, and the hardware platform is easy to build. In this experiment, holding the Kinect device to scan the object, or fixing the Kinect sensor, and rotating the target to realize the complete collection of object information. The hardware platform and configuration are shown in Table1. Kinect sensors are superior in terms of performance and price, and computers with higher running memory are more suitable for 3D reconstruction.Table 1
Hardware configuration.
EquipmentTo configureKinectMicrosoft kinect for XboxKinect power adapterComputerOperating system:Windows 7 64 bitCPU:i5-4200U,1.6GHZRAM: 4GThe following focuses on the Kinect depth sensor, which is an important equipment in the experiment.(1) Kinect. Kinect is a fusion of many current advanced machine vision technologies, which can collect the color and depth information of the object at the same time. Compared with other devices, Kinect is not only cheap but also can obtain high-resolution depth maps and color maps at the same time. The hardware structure is mainly divided into three parts. They are a system-on-chip, three cameras, and a four-element microphone array. The basic parameters of Kinect are shown in Table 2:Table 2
Kinect basic parameters.
Characteristic specificationParameterMaximum frame rate30 fpsDepth range40 cm–400 cmDepth image resolution640 × 480, 11 bitColor image resolution640 × 480, 8 bitKinect can obtain 640∗ 480 pixel depth information and color information at a frame rate of up to 30 fps. By combining the infrared CMOS sensor and the color camera, the three-dimensional information of the target can be projected onto the screen. Its working principle is shown in Figure 6:Figure 6
Kinect working principle diagram.The shooting frame rate of Kinect has reached 30 fps, which will not be a shortcoming of the system frame rate. The processing speed of the system is often lower than the input speed of the image, so the processing speed determines the frame rate of the final system. The main hardware factors that affect the frame rate are the video memory capacity and the number of cores of the graphics card. After testing on different devices, the results are shown in Table3:Table 3
System frame rate.
Memory size (GB)Memory size (MB)Number of CUDA coresFrame rate (fps)4512192Cannot execute410245761282048134421


## 3.1.2. Software Platform Configuration

The development environment used in this article is Win7 64 bit, the development platform is Visual Studio 2010, and the interface is designed with Qt4.8. Although Microsoft provides a software development kit Kinect SDK and related drivers, these software development kits are mainly for somatosensory control, gesture recognition, tracking, and voice control. The latest Kinect SDK 1.7 is more powerful and also includes the realization of algorithms such as KinectFusion. The focus of this article is to achieve the three-dimensional reconstruction of the target object. The existing open-source library contains a large number of resources, which is convenient for learning and using.


## 3.1.3. System Structure Design

Although different methods use different 3D reconstruction techniques, the main process of 3D reconstruction is roughly the same. The system function module is shown in Figure7:Figure 7
3D reconstruction function module.The six functional modules we designed for the 3D reconstruction system are image acquisition module, registration module, surface extraction module, preprocessing module, texture mapping module, and output display module. Aiming at these six functional modules, the software architecture of the reconstruction system is designed. In this architecture, the Kinect sensor is first used to obtain depth image information and color image information. Since the image contains noise information, the data needs to be preprocessed. The depth information is coordinate-converted into 3D point cloud data and then filtering and denoising operation is performed, and then the registration and alignment operation is performed on the 3D point cloud data set under multiple viewing angles. After registration, the point cloud data is fused according to the corresponding spatial conversion relationship, and the color RGB image is used for texture mapping, so as to output a complete 3D reconstruction model of the target object.


## 3.1.4. Comparative Test Design

The experiment is based on the built hardware and software platform using Kinect sensors and computers to analyze the performance of the above-mentioned virtual-real fusion 3D reconstruction technology is based on machine learning, and the experiment selects other 3D reconstruction techniques for comparison to obtain the performance data of each algorithm, such as speed, error rate, correct rate, and other information, so as to understand whether the 3D reconstruction fusion method based on machine learning is more superior. The comparison algorithms selected in this experiment are SGM, SFM, and CMMs (coordinate measuring machines). These algorithms belong to different kinds of algorithms in 3D reconstruction and they are used frequently. The performance data of each algorithm is obtained through experiments, and the performance of the virtual and real 3D reconstruction algorithm based on machine learning is understood.


## 3.2. Test Results and Analysis

### 3.2.1. Image Reconstruction Speed

Testing the speed of image reconstruction in the figure below for each algorithm, increasing the number of experiments, calculating the average value of the speed under different number of experiments, and getting the speed data of each algorithm is shown in Figure8:Figure 8
The speed of different algorithms.According to the data in Figure8, the speed of the 3D reconstruction algorithm based on machine learning in this article first slowly increases with the increase of the number of trials. After the number of times is greater than 30, the speed increases, and this algorithm is always faster than the other several algorithms. The average speed under different test times is 72.9%. Therefore, it is concluded that the 3D reconstruction algorithm based on machine learning can greatly reduce the image reconstruction time and increase the efficiency of 3D reconstruction processing.


### 3.2.2. The Evaluation Value of Each Index in the Image Test Set

In this part of the experiment, the experiment chose to compare 3 quantitative parameters, namely the non-texture area matching error percentage (untex), all pixel matching error percentage (all), parallax discontinuous area matching error percentage (disc) and matching, and the time used (time). The evaluation values of these indicators of the four algorithms are all measured, and the data obtained is shown in Figure9:Figure 9
Evaluation values of various indicators in the image test set of different algorithms.It can be seen from Figure9 whether it is the test result of the nontexture area matching error percentage, all pixel matching error percentage, or parallax discontinuous area matching error percentage. The error percentage of the 3D reconstruction algorithm based on machine learning is the lowest, among them, and 3d reconstruction algorithm based on machine learning has the lowest error rate on untex index and the highest error rate on disc. But the error rate for each indicator was less than 10%, which proves that the 3D reconstruction algorithm based on machine learning has a low error rate and high efficiency.


### 3.2.3. Accuracy Comparison

In this part of the experiment, we put the reconstructed object indoors through the matching of indoor objects, tables, lamps, sofas, chairs, and beds to understand the matching accuracy of each algorithm and recorded the data. The results are shown in Figure10:Figure 10
Matching accuracy of different algorithms.It can be seen from Figure10 that the 3D reconstruction algorithm based on machine learning has the highest matching accuracy no matter which object is in the room. Among them, the chair matching accuracy is the highest and the bed matching accuracy is the lowest. But the accuracy was better than 0.8 for any object. After calculation, for different objects, the average matching accuracy of the 3D reconstruction algorithm based on machine learning is about 0.87. The average matching accuracy of other algorithms SGM, SFM, and CMMS for different objects is about 0.84, 0.84, and 0.83, respectively. It can be seen that the accuracy of the matching rate of several algorithms is relatively close, but the accuracy of the 3D reconstruction algorithm based on machine learning is the best.


## 3.2.1. Image Reconstruction Speed

Testing the speed of image reconstruction in the figure below for each algorithm, increasing the number of experiments, calculating the average value of the speed under different number of experiments, and getting the speed data of each algorithm is shown in Figure8:Figure 8
The speed of different algorithms.According to the data in Figure8, the speed of the 3D reconstruction algorithm based on machine learning in this article first slowly increases with the increase of the number of trials. After the number of times is greater than 30, the speed increases, and this algorithm is always faster than the other several algorithms. The average speed under different test times is 72.9%. Therefore, it is concluded that the 3D reconstruction algorithm based on machine learning can greatly reduce the image reconstruction time and increase the efficiency of 3D reconstruction processing.


## 3.2.2. The Evaluation Value of Each Index in the Image Test Set

In this part of the experiment, the experiment chose to compare 3 quantitative parameters, namely the non-texture area matching error percentage (untex), all pixel matching error percentage (all), parallax discontinuous area matching error percentage (disc) and matching, and the time used (time). The evaluation values of these indicators of the four algorithms are all measured, and the data obtained is shown in Figure9:Figure 9
Evaluation values of various indicators in the image test set of different algorithms.It can be seen from Figure9 whether it is the test result of the nontexture area matching error percentage, all pixel matching error percentage, or parallax discontinuous area matching error percentage. The error percentage of the 3D reconstruction algorithm based on machine learning is the lowest, among them, and 3d reconstruction algorithm based on machine learning has the lowest error rate on untex index and the highest error rate on disc. But the error rate for each indicator was less than 10%, which proves that the 3D reconstruction algorithm based on machine learning has a low error rate and high efficiency.


## 3.2.3. Accuracy Comparison

In this part of the experiment, we put the reconstructed object indoors through the matching of indoor objects, tables, lamps, sofas, chairs, and beds to understand the matching accuracy of each algorithm and recorded the data. The results are shown in Figure10:Figure 10
Matching accuracy of different algorithms.It can be seen from Figure10 that the 3D reconstruction algorithm based on machine learning has the highest matching accuracy no matter which object is in the room. Among them, the chair matching accuracy is the highest and the bed matching accuracy is the lowest. But the accuracy was better than 0.8 for any object. After calculation, for different objects, the average matching accuracy of the 3D reconstruction algorithm based on machine learning is about 0.87. The average matching accuracy of other algorithms SGM, SFM, and CMMS for different objects is about 0.84, 0.84, and 0.83, respectively. It can be seen that the accuracy of the matching rate of several algorithms is relatively close, but the accuracy of the 3D reconstruction algorithm based on machine learning is the best.


## 4. Discussion

The three-dimensional model is more superior than the two-dimensional image because the three-dimensional image has the advantage of more than one dimension, so the appearance and structure of the object can be expressed more clearly. For e-commerce, for example, businesses can construct a three-dimensional model to represent the appearance of the product, so that customers can gain a clearer and more intuitive understanding of the product. At the same time, 3D models can also be used in 3D movies, TV animations, medical imaging, and other fields. Also, the digital geometric processing of three-dimensional geometric data has become an important technology in the study of computer graphics, computer vision, and image processing. As a new form of information carrier, three-dimensional geometric models have gradually changed the way people explore and understand the world due to their unparalleled advantages such as high fidelity and intuitive visualization. The combination of 3D reconstruction technology and virtual and real fusion can combine the scene reconstructed by calculation with the real scene. This plays an important role in many fields such as augmented reality and artificial intelligence. Theories such as convolutional neural networks and deep learning in machine learning can be used in 3D reconstruction technology to improve the efficiency and accuracy of 3D reconstruction.


## 5. Conclusion

This article introduces the background significance of virtual and real fusion and 3D reconstruction technology, as well as the combination of the two. This article introduces the theory of machine learning, combines the theory with 3D reconstruction, and introduces the 3D reconstruction algorithm based on machine learning. Finally, this article designs experiments and designs the hardware and software and basic architecture of the system. This article compares the 3D reconstruction algorithm based on machine learning with several other common algorithms and draws a conclusion as follows: (1) the 3D reconstruction algorithm based on machine learning is the fastest, with an average speed of 72.9% under different times. (2) Evaluating the image acquisition indicators of each algorithm and finding that the error rate of the 3D reconstruction algorithm based on machine learning is low. (3) Testing the matching accuracy of each algorithm. The average matching accuracy of the 3D reconstruction algorithm based on machine learning is about 0.87, which is the highest. Experiments in this article verify the superiority of the performance of the 3D reconstruction algorithm based on machine learning, but there are a few performance indexes in the test. If more indexes are added, it will be more convincing, AND due to practical reasons, the equipment available for the experiment is limited. If given the chance, a better job can be made.



---

*Source: 7158504-2022-05-19.xml*