# Refocusing Algorithm Based on Nonlinear SAI System

Authors: Bin Liu; Yue Luo; Yi-Hua Pan; Wen-Min Yan; Xin-Yu Zhang
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Synthetic aperture imaging (SAI) technology gets the light field information of the scene through the camera array. With the large virtual aperture, it can effectively acquire the information of the partially occluded object in the scene, and then we can focus on the arbitrary target plane corresponding to the reference perspective through the refocus algorithm. Meanwhile, objects that deviate from the plane will be blurred to varying degrees. However, when the object to be reconstructed in the scene is occluded by the complex foreground, the optical field information of the target cannot be effectively detected due to the limitation of the linear array. In order to deal with these problems, this paper proposes a nonlinear SAI method. This method can obtain the occluded object’s light field information reliably by using the nonlinear array. Experiments are designed for the nonlinear SAI, and refocusing is performed for the occluded objects with different camera arrays, different depths, and different distribution intervals. The results demonstrate that the method proposed in this paper is advanced than the traditional SAI method based on linear array.


---

## Body

## 1. Introduction

Synthetic aperture imaging (SAI) uses the camera array to capture the light field of the scene and reconstructs the specified object in the scene by the refocusing algorithm. In this process, the object out of the focal plane will be blurred because of the virtual large aperture of the system.Object occlusion processing is a big challenge for computer vision, such as object recognition and tracking [1] and de-occlusion imaging [2, 3]. When the object is occluded, the traditional single-view imaging cannot effectively obtain the information of the occluded object. As an important branch of light field imaging, the synthetic aperture imaging technology can overcome the limitation of traditional imaging. It can effectively synthesize the multiview images obtained by the camera array into a virtual large-aperture image. When the detected object is occluded, the synthesized virtual large aperture has a very shallow depth of field, which can effectively blur the foreground occlusion and realize the “perspective” detection of the occluded object (see Figure 1).Figure 1
Synthetic aperture imaging. (a) Original image of reference view. (b) The result of synthetic aperture imaging.
(a)
(b)Camera arrays are the main means to acquire multiview images. Currently, the related theories and applications of SAI mainly focus on the linear arrangement mode, and the nonlinear arrangement mode is rarely used in the field of SAI. When the depth difference between the foreground occlusion and the occluded object is small, the linear array is prone to the problem of serious lack of sampling information of the occluded object, which will affect the reconstruction effect of the occluded object. Therefore, a nonlinear array can be designed by changing the arrangement position of the cameras and the direction of the optical axis to capture more information of the occluded object (see Figure2). However, the current linear SAI reconstruction algorithm cannot directly adapt to the SAI reconstruction of the nonlinear array.Figure 2
The comparison of an occluded object detected by a linear array and a nonlinear array. (a) A linear array. (b) A nonlinear array.
(a)
(b)In view of the above questions, this paper proposes a novel approach of SAI based on a nonlinear array and reconstructs the image of the occluded objects. The rest of this paper is organized as follows: Section2 introduces the related works at home and abroad. Section 3 describes the algorithm, which includes details of nonlinear image rearrangement and SAI of rearranged images. Section 4 presents the experimental results and discussion. Section 5 summarizes the research.


## 2. Related Works

As we all know, if we need to reconstruct a point in the image, one of the prerequisites is that at least one camera should capture the information about the point. Recently, more and more researchers are working on removing the impact of occlusion. Some of them are interested in image optimization; they use various methods: Yang et al. presented a novel depth-free all-in-focus SAI technique based on the light field visibility analysis [4], the experimental results demonstrate the effectiveness and superiority of the approach; Pei et al. proposed a novel method to improve the image quality of synthetic aperture imaging using image matting via energy minimization by estimating the foreground and the background [5]; Yanyangshuo et al. proposed a refocusing imaging algorithm via foreground labelling [6], the algorithm can effectively remove the information of the occlusion in the scene and improve the quality of the refocused image; Lee et al. presented the concept of a practical free-viewpoint television system with purely optical depth estimation [7]. Although these methods can optimize the reconstruction results, if the camera array cannot capture enough information of the occluded object, it cannot reconstruct the object. Therefore, others studied the arrangement of the camera array after achieving the experiment of SAI by using a linear camera array [8, 9]. Many researchers study simulation of a camera array with a moving camera [10–12]. This method is apparently cheaper than the camera array, and the system affords more flexible camera arrangement. However, the position of the camera is still limited to a plane, and the virtual array formed by the system is still equivalent to a plane array.When the distance between the object and the occlusion in the optical axis direction is small or the arrangement space of the camera array is limited, the linear array performs poorly in the acquisition of the object information. For this problem, this paper carries out research on SAI reconstruction based on a nonlinear array. Nonlinear arrays can detect a wider range of scene information, and the arrangement of cameras is more flexible than linear arrays. Therefore, nonlinear arrays are widely used for 3D reconstruction of objects, free-view videos of scenes [13, 14], panoramic image generation [15, 16], and scene depth estimation, but nonlinear arrays are still rarely used in the field of synthetic aperture imaging.


## 3. SAI Algorithm Based on Nonlinear Array

### 3.1. Rearrangement of Nonlinear Images

Because the optical axes of the multiview images captured by the nonlinear array are not parallel to one another, it brings a great inconvenience to SAI. Therefore, the first step is to rearrange the multiview images. According to computer vision, the one-to-one mapping between the two camera images can be described by the homography matrix H. Relying on the transformation relationship between the images, the captured images can be rearranged so that their optical axes are parallel to that of the reference camera (see Figure3).Figure 3
Diagram of multiview image rearrangement.In Figure3, a point M in space and its projection m in the image are connected by a projection matrix P. Their relationship is written as(1)m=PM=KRtM,where K, R, and t indicate the intrinsic matrix, rotation matrix, and translation vector, respectively.Because the depth of the occluded object is unknown, we first assume a depth plane which will be reconstructed. By adjusting the assumed depth value, we can get the reconstructed image of occluded objects. In Figure3, taking the reference camera coordinate system as the reference coordinate system, the X, Y, and Z axes are the horizontal axis, vertical axis, and optical axis, respectively. We assume that the point M with a depth of z located on the optical axis of the reference camera is the point on the reconstruction plane. By calibrating, we can get Rn,r and tn,r=tn,X,tn,Y,tn,ZT (n = 1, 2, 3,…, N, where n≠r), which are the rotation matrix and translation vector of the original camera relative to the reference camera, N indicates the total number of cameras, n means the nth camera in the camera array, and r means the position of reference camera in the camera array. The translation vector from each camera to M is tn, which is defined as(2)tn=Rn,r0,0,zT+tn,r.After rearrangement, the position of each rearranged camera is different from the reference camera only on theX axis. The distance between each camera and reference camera, tn,X′, is defined as(3)tn,X′=1,0,0Rn,r−1tn,r.Thus, the translation vectortn′ of each rearranged camera to M is(4)tn′=tn,X′,0,zT.The projection matrix of each cameraPn and the rearranged camera Pn′ is shown as equations (5) and (6):(5)Pn=KRn,rtn,(6)Pn′=KEtn′,where K indicates the intrinsic matrix of the camera and E indicates the identity matrix. We make H as a homography matrix from each camera to the rearranged camera, which is defined as(7)mn′=Hmn,where mn and mn′ indicate a point on the image of each original camera and the rearranged camera. According to equations (1), (5), and (6), we can calculate H as follows:(8)H=Pn′Pn+,where Pn+ is the pseudoinverse matrix of Pn.When these nonlinear images are rearranged, a lot of object information is lost. To solve this problem, we ensure that each image, and the images taken by the reference camera too, contains enough information about the object by translating the principal point of the images (see Figure4).Figure 4
(a) Original images. (b) Before translating the principal point of the images. (c) After translating the principal point of the images.
(a)
(b)
(c)According to equation (7), we can get four vertices of a rearranged image. The new principal point of the rearranged image u0,v0 is derived from the following equation:(9)u0=maxUi+minUi2,v0=maxVi+minVi2,where Ui,Vi represents a set of four vertices of a rearranged image, i = 1, 2, 3, 4.


### 3.2. SAI of Rearranged Images

According to the traditional linear SAI formula, we can get the following equation:(10)In,zs′=In,zs+α⋅n−r⋅B,where In,z is the rearranged image captured by the camera n focusing on a depth z, s is the coordinate of a point on the rearranged image, s′ is the coordinate of a point on the rearranged image projected to the depth z, B is the length of the baseline between two adjacent cameras, and α is the focus coefficient, which is defined by the following equation:(11)α=fz,where f is the focal length of the camera (see Figure 5).(12)Sz=1N∑n=1NIn,z.Figure 5
The principle of SAI.From equation (12), we can get the synthetic aperture image Sz, which focuses on depth z. However, the intervals between the cameras are unequal and the principal points of the images are translated; the traditional linear SAI algorithm is no longer applicable. Therefore, to realize the nonlinear SAI algorithm for rearranged images, equation (10) is reformulated as the following equation:(13)In,zs′=In,zs+Δs+α⋅ΔB,where Δs is the translation distance of the principal point and ΔB is the distance between the camera n and the reference camera. Equation (13) can be evaluated to get the synthetic aperture image Sz, which focuses on the depth z. Therefore, the synthetic aperture image focusing on the occluded object is obtained by adjusting the depth z.Through the above analysis, it is easy to get the flowchart of the algorithm proposed in this paper (see Figure6).Figure 6
Flowchart of nonlinear synthetic aperture imaging.


## 3.1. Rearrangement of Nonlinear Images

Because the optical axes of the multiview images captured by the nonlinear array are not parallel to one another, it brings a great inconvenience to SAI. Therefore, the first step is to rearrange the multiview images. According to computer vision, the one-to-one mapping between the two camera images can be described by the homography matrix H. Relying on the transformation relationship between the images, the captured images can be rearranged so that their optical axes are parallel to that of the reference camera (see Figure3).Figure 3
Diagram of multiview image rearrangement.In Figure3, a point M in space and its projection m in the image are connected by a projection matrix P. Their relationship is written as(1)m=PM=KRtM,where K, R, and t indicate the intrinsic matrix, rotation matrix, and translation vector, respectively.Because the depth of the occluded object is unknown, we first assume a depth plane which will be reconstructed. By adjusting the assumed depth value, we can get the reconstructed image of occluded objects. In Figure3, taking the reference camera coordinate system as the reference coordinate system, the X, Y, and Z axes are the horizontal axis, vertical axis, and optical axis, respectively. We assume that the point M with a depth of z located on the optical axis of the reference camera is the point on the reconstruction plane. By calibrating, we can get Rn,r and tn,r=tn,X,tn,Y,tn,ZT (n = 1, 2, 3,…, N, where n≠r), which are the rotation matrix and translation vector of the original camera relative to the reference camera, N indicates the total number of cameras, n means the nth camera in the camera array, and r means the position of reference camera in the camera array. The translation vector from each camera to M is tn, which is defined as(2)tn=Rn,r0,0,zT+tn,r.After rearrangement, the position of each rearranged camera is different from the reference camera only on theX axis. The distance between each camera and reference camera, tn,X′, is defined as(3)tn,X′=1,0,0Rn,r−1tn,r.Thus, the translation vectortn′ of each rearranged camera to M is(4)tn′=tn,X′,0,zT.The projection matrix of each cameraPn and the rearranged camera Pn′ is shown as equations (5) and (6):(5)Pn=KRn,rtn,(6)Pn′=KEtn′,where K indicates the intrinsic matrix of the camera and E indicates the identity matrix. We make H as a homography matrix from each camera to the rearranged camera, which is defined as(7)mn′=Hmn,where mn and mn′ indicate a point on the image of each original camera and the rearranged camera. According to equations (1), (5), and (6), we can calculate H as follows:(8)H=Pn′Pn+,where Pn+ is the pseudoinverse matrix of Pn.When these nonlinear images are rearranged, a lot of object information is lost. To solve this problem, we ensure that each image, and the images taken by the reference camera too, contains enough information about the object by translating the principal point of the images (see Figure4).Figure 4
(a) Original images. (b) Before translating the principal point of the images. (c) After translating the principal point of the images.
(a)
(b)
(c)According to equation (7), we can get four vertices of a rearranged image. The new principal point of the rearranged image u0,v0 is derived from the following equation:(9)u0=maxUi+minUi2,v0=maxVi+minVi2,where Ui,Vi represents a set of four vertices of a rearranged image, i = 1, 2, 3, 4.


## 3.2. SAI of Rearranged Images

According to the traditional linear SAI formula, we can get the following equation:(10)In,zs′=In,zs+α⋅n−r⋅B,where In,z is the rearranged image captured by the camera n focusing on a depth z, s is the coordinate of a point on the rearranged image, s′ is the coordinate of a point on the rearranged image projected to the depth z, B is the length of the baseline between two adjacent cameras, and α is the focus coefficient, which is defined by the following equation:(11)α=fz,where f is the focal length of the camera (see Figure 5).(12)Sz=1N∑n=1NIn,z.Figure 5
The principle of SAI.From equation (12), we can get the synthetic aperture image Sz, which focuses on depth z. However, the intervals between the cameras are unequal and the principal points of the images are translated; the traditional linear SAI algorithm is no longer applicable. Therefore, to realize the nonlinear SAI algorithm for rearranged images, equation (10) is reformulated as the following equation:(13)In,zs′=In,zs+Δs+α⋅ΔB,where Δs is the translation distance of the principal point and ΔB is the distance between the camera n and the reference camera. Equation (13) can be evaluated to get the synthetic aperture image Sz, which focuses on the depth z. Therefore, the synthetic aperture image focusing on the occluded object is obtained by adjusting the depth z.Through the above analysis, it is easy to get the flowchart of the algorithm proposed in this paper (see Figure6).Figure 6
Flowchart of nonlinear synthetic aperture imaging.


## 4. Experiment Results and Discussions

This section describes in detail the experiments conducted to verify the performance of the proposed method. In Section4.1, a linear array and a nonlinear one are compared for their performance. In Section 4.2, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. In Section 4.3, the reconstruction performance between unevenly distributed and uniformly distributed nonlinear arrays is researched. In Section 4.4, the experimental results are qualitatively and quantitatively analyzed.In the experiment, this paper uses a single camera fixed on the mechanical arm and an electric turntable to simulate the image acquisition process of the nonlinear array. The electric turntable drives the camera to revolve around the scene. A BASLER camera, model acA1600-20gc, with an image resolution of 1624 × 1234 is used. The electric turntable is 42BYG250Bk-B, produced by RED STAR YANG TECHNOLOGY (see Figure7).Figure 7
The experiment system.


### 4.1. Experiment 1: The Comparison between Nonlinear Array and Linear Array

In Experiment 1, the synthetic aperture imaging reconstruction results of the linear array and arc array are compared. The scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees. The arc array is composed of 41 equally spaced viewpoints with a radius of about 500 mm. The middle viewpoint is used as the reference view, and nine viewpoints are extracted from 41 viewpoints to represent their relative position (see Figure8). Three scenes are selected for the experiments, that is, three different foreground occlusions and the occluded objects, and the results of synthetic aperture imaging under linear and nonlinear arrays are shown in Figure 9. The occluded objects from the first line to the third line are “Centrum”, “Move Free,” and “云南白药气雾剂”.Figure 8
Schematic diagram of the camera position of the nonlinear array.Figure 9
SAI reconstruction results of the linear array and nonlinear array. (a) The object. (b) The occluded object. (c) Linear array reconstruction results. (d) Nonlinear array reconstruction results.
(a)
(b)
(c)
(d)At present, research studies on synthetic aperture imaging at home and abroad are based on linear arrays. The method of Vaish V et al. [3] is a typical synthetic aperture imaging method by using a linear array, and the subsequent studies are based on the method to optimize the image. Therefore, in order to study the effect of the camera array arrangement mode on the synthetic aperture imaging, the synthetic aperture imaging results between the nonlinear method in this paper and the linear method in [3] are compared.In Figures9(c) and 9(d), the imaging details are enlarged and displayed in the black box. Because the distance between foreground occlusion and the occluded object is small, the data of the occluded object collected by using the linear array is insufficient and the occluded object cannot be reconstructed well. It can be seen from the imaging results that compared with the method in [3], the method proposed in this paper can display more information about the occluded object.


### 4.2. Experiment 2: The Effect of Different Distances between Foreground Occlusion and the Occluded Object on SAI

In Experiment 2, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. Two sets of scenes are selected for the experiment: one is a card occluded by a greeting card; the other is a box occluded by a bookmark, and two cones of the same size are placed beside the bookmark. As shown in Figure10, the length of each object in the scene is 57 mm for the card, 123 mm for the greeting card, 73 mm for the box, 75 mm for the bookmarks, and 79 mm for the cones.Figure 10
The scene of Experiment 2. (a) The card. (b) The card occluded by a greeting card. (c) The box. (d) The box occluded by a bookmark.
(a)
(b)
(c)
(d)Experiment 2 and Experiment 1 have the same experimental conditions. The performance of linear SAI and nonlinear SAI is compared by changing the distance between foreground occlusion and the occluded object; the distance is set as 60 mm, 80 mm, 100 mm, 120 mm, 140 mm, and 160 mm. The experimental results are shown in Figure11.Figure 11
Reconstruction results of SAI at different distances. The first column shows linear SAI reconstruction results of the occluded card. The second column shows nonlinear SAI reconstruction results of the occluded card. The third column shows linear SAI reconstruction results of the occluded box. The fourth column shows nonlinear SAI reconstruction results of the occluded box. (a) 60 mm, (b) 80 mm, (c) 100 mm, (d) 120 mm, (e) 140 mm, and (f) 160 mm.
(a)
(b)
(c)
(d)
(e)
(f)


### 4.3. Experiment 3: Comparison of Reconstruction Performance between Unevenly Distributed and Uniformly Distributed Nonlinear Arrays

In Experiment 3, the performance of a nonlinear array with uneven distribution and even distribution is researched. The experimental scenario is a box labeled “Centrum” occluded by a larger box labeled “抹茶” and some leaves (see Figure12). Such scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees, and the radius of the arc array is 500 mm. The arc array consists of 41 equally spaced viewpoints, which are labeled 1, 2, 3,…, 41 from left to right.Figure 12
Different views of the nonlinear array. (a) The unoccluded box. (b) The first view. (c) The reference view. (d) The 41st view.
(a)
(b)
(c)
(d)Experiment 3 and Experiment 1 have the same experimental conditions. In the experiment, 11 unevenly distributed views and 11 evenly distributed views are selected from 41 views (see Table1). The nonlinear SAI results of unevenly distributed views and evenly distributed views are shown in Figure 13.Table 1
View selection list.
Distribution
The selected views
Even
1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41
Uneven
1, 2, 5, 10, 12, 24, 33, 35, 39, 40, 41Figure 13
(a) Nonlinear SAI result of evenly distributed views. (b) Nonlinear SAI result of unevenly distributed views.
(a)
(b)


### 4.4. Analysis of Experimental Results

In this paper, the peak signal-to-noise ratio (PSNR) and mean structural similarity index measure (MSSIM) are used to quantitatively evaluate the experimental results [17]. The higher the values of PSNR and MSSIM are, the better the occluded objects are reconstructed. The evaluation results of Experiment 1 are shown in Table 2.Table 2
The evaluation results of Experiment 1.
The occluded object
PSNR
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
Centrum
8.0454
9.3179
0.4120
0.5387
Move Free
7.9543
13.4980
0.3783
0.6952
云南白药气雾剂
6.4895
11.0235
0.3290
0.4946It can be seen from Table2 that the evaluation value of the nonlinear array is higher than that of the linear array. This is because the nonlinear array can capture more information about the occluded object, which can show more details in the synthetic aperture imaging results. The evaluation results of Experiment 2 are shown in Tables 3 and 4; Table 3 shows the evaluation of the card reconstruction results, and Table 4 shows the evaluation of the box reconstruction results.Table 3
The evaluation of the card reconstruction results.
The distance between foreground occlusion and the occluded object (mm)
MSSIM
MSSIM
MSSIM
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
60
9.5608
10.5514
0.6313
0.6607
80
9.4015
11.1107
0.6309
0.6871
100
9.9350
12.1599
0.6452
0.7467
120
10.2295
13.3547
0.6642
0.8035
140
10.3917
14.1182
0.6759
0.8124
160
10.8802
16.1748
0.6953
0.8499Table 4
The evaluation of the box reconstruction results.
The distance between foreground occlusion and the occluded object (mm)
PSNR
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
60
10.7350
12.3870
0.3091
0.3701
80
11.2018
13.7788
0.3336
0.4278
100
11.9426
16.2440
0.3738
0.5204
120
13.3036
16.8048
0.5162
0.6472
140
14.5448
17.8792
0.5256
0.7030
160
14.7767
18.0432
0.5572
0.7005It can be seen from Tables3 and 4 that the image reconstruction quality improves as the distance between foreground occlusion and the occluded object increases. This is because when the distance increases, the interference of the foreground occlusion to the occluded object decreases and the information obtained by the camera array increases, which improves the image reconstruction quality. In addition, the nonlinear array can get good reconstruction results especially when the distance between the foreground occlusion and the occluded object is small.It can be seen from Table5 that although the number of views is the same, the reconstruction results of unevenly distributed views are better than those of evenly distributed views. Because the uneven distribution array can capture pictures according to the occlusion situation in Figure 8, fewer pictures are captured in front of the object with serious occlusion, while more pictures are captured on the left and right sides with less occlusion, so the uneven distribution array can detect more information of the object and has better reconstruction quality.Table 5
The evaluation results of Experiment 3.
Distribution
PSNR
MSSIM
Even
12.6729
0.5265
Uneven
14.4916
0.6772


## 4.1. Experiment 1: The Comparison between Nonlinear Array and Linear Array

In Experiment 1, the synthetic aperture imaging reconstruction results of the linear array and arc array are compared. The scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees. The arc array is composed of 41 equally spaced viewpoints with a radius of about 500 mm. The middle viewpoint is used as the reference view, and nine viewpoints are extracted from 41 viewpoints to represent their relative position (see Figure8). Three scenes are selected for the experiments, that is, three different foreground occlusions and the occluded objects, and the results of synthetic aperture imaging under linear and nonlinear arrays are shown in Figure 9. The occluded objects from the first line to the third line are “Centrum”, “Move Free,” and “云南白药气雾剂”.Figure 8
Schematic diagram of the camera position of the nonlinear array.Figure 9
SAI reconstruction results of the linear array and nonlinear array. (a) The object. (b) The occluded object. (c) Linear array reconstruction results. (d) Nonlinear array reconstruction results.
(a)
(b)
(c)
(d)At present, research studies on synthetic aperture imaging at home and abroad are based on linear arrays. The method of Vaish V et al. [3] is a typical synthetic aperture imaging method by using a linear array, and the subsequent studies are based on the method to optimize the image. Therefore, in order to study the effect of the camera array arrangement mode on the synthetic aperture imaging, the synthetic aperture imaging results between the nonlinear method in this paper and the linear method in [3] are compared.In Figures9(c) and 9(d), the imaging details are enlarged and displayed in the black box. Because the distance between foreground occlusion and the occluded object is small, the data of the occluded object collected by using the linear array is insufficient and the occluded object cannot be reconstructed well. It can be seen from the imaging results that compared with the method in [3], the method proposed in this paper can display more information about the occluded object.


## 4.2. Experiment 2: The Effect of Different Distances between Foreground Occlusion and the Occluded Object on SAI

In Experiment 2, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. Two sets of scenes are selected for the experiment: one is a card occluded by a greeting card; the other is a box occluded by a bookmark, and two cones of the same size are placed beside the bookmark. As shown in Figure10, the length of each object in the scene is 57 mm for the card, 123 mm for the greeting card, 73 mm for the box, 75 mm for the bookmarks, and 79 mm for the cones.Figure 10
The scene of Experiment 2. (a) The card. (b) The card occluded by a greeting card. (c) The box. (d) The box occluded by a bookmark.
(a)
(b)
(c)
(d)Experiment 2 and Experiment 1 have the same experimental conditions. The performance of linear SAI and nonlinear SAI is compared by changing the distance between foreground occlusion and the occluded object; the distance is set as 60 mm, 80 mm, 100 mm, 120 mm, 140 mm, and 160 mm. The experimental results are shown in Figure11.Figure 11
Reconstruction results of SAI at different distances. The first column shows linear SAI reconstruction results of the occluded card. The second column shows nonlinear SAI reconstruction results of the occluded card. The third column shows linear SAI reconstruction results of the occluded box. The fourth column shows nonlinear SAI reconstruction results of the occluded box. (a) 60 mm, (b) 80 mm, (c) 100 mm, (d) 120 mm, (e) 140 mm, and (f) 160 mm.
(a)
(b)
(c)
(d)
(e)
(f)


## 4.3. Experiment 3: Comparison of Reconstruction Performance between Unevenly Distributed and Uniformly Distributed Nonlinear Arrays

In Experiment 3, the performance of a nonlinear array with uneven distribution and even distribution is researched. The experimental scenario is a box labeled “Centrum” occluded by a larger box labeled “抹茶” and some leaves (see Figure12). Such scene information is captured by an arc array (nonlinear array) with a central angle of 80 degrees, and the radius of the arc array is 500 mm. The arc array consists of 41 equally spaced viewpoints, which are labeled 1, 2, 3,…, 41 from left to right.Figure 12
Different views of the nonlinear array. (a) The unoccluded box. (b) The first view. (c) The reference view. (d) The 41st view.
(a)
(b)
(c)
(d)Experiment 3 and Experiment 1 have the same experimental conditions. In the experiment, 11 unevenly distributed views and 11 evenly distributed views are selected from 41 views (see Table1). The nonlinear SAI results of unevenly distributed views and evenly distributed views are shown in Figure 13.Table 1
View selection list.
Distribution
The selected views
Even
1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41
Uneven
1, 2, 5, 10, 12, 24, 33, 35, 39, 40, 41Figure 13
(a) Nonlinear SAI result of evenly distributed views. (b) Nonlinear SAI result of unevenly distributed views.
(a)
(b)


## 4.4. Analysis of Experimental Results

In this paper, the peak signal-to-noise ratio (PSNR) and mean structural similarity index measure (MSSIM) are used to quantitatively evaluate the experimental results [17]. The higher the values of PSNR and MSSIM are, the better the occluded objects are reconstructed. The evaluation results of Experiment 1 are shown in Table 2.Table 2
The evaluation results of Experiment 1.
The occluded object
PSNR
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
Centrum
8.0454
9.3179
0.4120
0.5387
Move Free
7.9543
13.4980
0.3783
0.6952
云南白药气雾剂
6.4895
11.0235
0.3290
0.4946It can be seen from Table2 that the evaluation value of the nonlinear array is higher than that of the linear array. This is because the nonlinear array can capture more information about the occluded object, which can show more details in the synthetic aperture imaging results. The evaluation results of Experiment 2 are shown in Tables 3 and 4; Table 3 shows the evaluation of the card reconstruction results, and Table 4 shows the evaluation of the box reconstruction results.Table 3
The evaluation of the card reconstruction results.
The distance between foreground occlusion and the occluded object (mm)
MSSIM
MSSIM
MSSIM
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
60
9.5608
10.5514
0.6313
0.6607
80
9.4015
11.1107
0.6309
0.6871
100
9.9350
12.1599
0.6452
0.7467
120
10.2295
13.3547
0.6642
0.8035
140
10.3917
14.1182
0.6759
0.8124
160
10.8802
16.1748
0.6953
0.8499Table 4
The evaluation of the box reconstruction results.
The distance between foreground occlusion and the occluded object (mm)
PSNR
MSSIM
Linear array
Nonlinear array
Linear array
Nonlinear array
60
10.7350
12.3870
0.3091
0.3701
80
11.2018
13.7788
0.3336
0.4278
100
11.9426
16.2440
0.3738
0.5204
120
13.3036
16.8048
0.5162
0.6472
140
14.5448
17.8792
0.5256
0.7030
160
14.7767
18.0432
0.5572
0.7005It can be seen from Tables3 and 4 that the image reconstruction quality improves as the distance between foreground occlusion and the occluded object increases. This is because when the distance increases, the interference of the foreground occlusion to the occluded object decreases and the information obtained by the camera array increases, which improves the image reconstruction quality. In addition, the nonlinear array can get good reconstruction results especially when the distance between the foreground occlusion and the occluded object is small.It can be seen from Table5 that although the number of views is the same, the reconstruction results of unevenly distributed views are better than those of evenly distributed views. Because the uneven distribution array can capture pictures according to the occlusion situation in Figure 8, fewer pictures are captured in front of the object with serious occlusion, while more pictures are captured on the left and right sides with less occlusion, so the uneven distribution array can detect more information of the object and has better reconstruction quality.Table 5
The evaluation results of Experiment 3.
Distribution
PSNR
MSSIM
Even
12.6729
0.5265
Uneven
14.4916
0.6772


## 5. Conclusions

This paper proposes an algorithm to achieve nonlinear SAI. The algorithm reconstructs the occluded object by nonlinear image rearrangement and SAI of rearranged images. Experiment work on linear arrays and nonlinear arrays demonstrates that a nonlinear array produces better SAI results. Then, the effect of different distances between foreground occlusion and the occluded object on SAI is studied. The experimental results show that the reconstruction performance of the nonlinear array is better than that of the linear array in different distances. Finally, this paper discusses the performance of the nonlinear array with uneven distribution and even distribution. The experimental results show that the quality of reconstruction can be improved by increasing the number of views at the sparse part of occlusion. The future work will focus on synthetic aperture imaging for 3D objects using a circular array.



---

*Source: 7178348-2020-03-24.xml*