# Epipolar Plane Image Rectification and Flat Surface Detection in Light Field

Authors: Lipeng Si; Hao Zhu; Qing Wang
Journal: Journal of Electrical and Computer Engineering
 (2017)

---

## Abstract

Flat surface detection is one of the most common geometry inferences in computer vision. In this paper we propose detecting printed photos from original scenes, which fully exploit angular information of light field and characteristics of the flat surface. Unlike previous methods, our method does not need a prior depth estimation. The algorithm rectifies the mess epipolar lines in the epipolar plane image (EPI). Then feature points are extracted from light field data and used to compute an energy ratio in the depth distribution of the scene. Based on the energy ratio, a feature vector is constructed and we obtain robust detection of flat surface. Apart from flat surface detection, the kernel rectification algorithm in our method can be expanded to inclined plane refocusing and continuous depth estimation for flat surface. Experiments on the public datasets and our collections have demonstrated the effectiveness of the proposed method.


---

## Body

## 1. Introduction

With the rapid development of light field theory [1, 2], light field cameras such as Lytro [3] and Raytrix [4] are now available for consumer and industrial use. Different from 2D image captured by traditional camera, light field camera records extra angular information of the real world and it provides more possibilities for many traditional computer vision tasks [5–8].Flat surface detection is such a prominent task to make planar structure inference from natural scenes. One potential application of this issue is detection of printed photo in face-based verification [9]. Face identification has been widely applied in industrial world. However, a common problem of such system suffers whether the face is a real one or just a printed photo of the authorized face. The main reason is the loss of depth information when the camera records the real world. Traditional methods always assume that the printed faces contain detectable texture patterns or require a user interaction to solve this problem [10]. However, these methods are unreliable or inefficient. Depth estimation [11] can be another option before the authentication, but it may bring other problems in depth estimation such as occlusion [12] and shading [13].In this paper, we analyze the variant and invariant features of flat surface in EPI representation and propose an algorithm to detect the flat surface without depth estimation, which fully exploits angular information of the light field and the characteristics of flat surface. Our main contributions are as follows:(i)
An algorithm to rectify EPI for a flat surface, which can also be expanded to other tasks such as refocusing in an inclined plane.(ii)
A framework to detect the flat surface in light field by a two-stage algorithm without depth estimation.The rest of this paper is organized as follows. In Section2, we review the background and the previous works about flat surface detection. In Section 3, the detailed algorithm is described. And we give the experimental results in Section 4. Our conclusion and future work are arranged in Section 5. Some proofs of related properties are provided in Appendix.


## 2. Background and Related Work

A light field is a function defined in 4D space namedf(x,y,u,v) [1] to describe light rays in physical world, where (x,y) and (u,v) describe the distribution of the light in spatial and angular space, respectively. Under the two-plane parametrization, when we fix one spatial dimension and one angular dimension (x∗,u∗) (or (y∗,v∗)), the EPI appears. For each point in the real world, there is a corresponding epipolar line in the EPI, and the slope of this line has a linear relationship with the depth of this point.As a basic problem in computer vision, flat surface detection has been researched for decades but it is not well developed. Most of techniques detect flat surface using a prior depth estimation. Zhao et al. [11] proposed to detect flat surface using the disparity map of the scene; however this method depends on the accuracy of disparity estimation and is sensitive to the alignment errors of disparity estimation. Raghavendra et al. [15] proposed a similar strategy. Instead of the disparity map used in [11], they proposed to obtain a rough depth map by using the focus measure. Undoubtedly, the method also suffers the same problem, that is, inaccuracy of disparity estimation. Ghasemi and Vetterli [14] proposed to extract energy feature vector based on the change of gradient of EPI and to distinguish the flat surface from nonflat one by using a Bayes classifier. The method computes the slope of all epipolar lines and then takes the variance; it is still a depth-based method.Different from the traditional “depth map to plane fitting” strategy [11, 15, 16], we detect the Lisad-1 feature point [17] in 4D light field and then fit the function of the flat surface by using several feature points in light field directly; finally the robustness of the function which we build is tested by another several feature points. If the scene is a flat surface, the function which is built from several feature points should also be suitable for other points and vice versa.


## 3. The Proposed Approach for Detecting Flat Surface

It is well observed that, for a flat surface which is parallel to the camera plane, all epipolar lines in the EPI have the same slope since they are in the same depth. However, this invariant property went when the flat surface is tilted to different angles. By analyzing the properties of flat surface function, it is noticed that no matter what angle the flat surface is tilted to, the difference of slope stays the same (this property is discussed in Section3.3 and its proof can be found in Appendix). Based on this invariant property, we first propose to rectify the slope of epipolar lines in EPI into a same value S0 and then to detect the Lisad-1 feature points [17] of the rectified EPI (the most important advantage of the Lisad-1 feature points is that it provides the slope of each feature point, and the extraction of their depth does not suffer any occlusion or shading problems as they are salient points). The slope of each feature point ought to be equal to S0 if the plane function is true. Finally, we combine the energy ratio from different EPIs as an energy vector and employ a classifier to distinguish flat surface from natural scenes.


### 3.1. Epipolar Plane Image Rectification

On a flat surface, the depth value of a point has a linear relationship with its 2D image coordinates. When the point is in an EPI with a fixedy∗, its depth can be expressed by a linear function only on x (Figure 1), (1)dx=ax+b,where d(x) is the depth of the point (x,y∗). And we can derive the slope of each point by (2)Sx=1disx=1fs/dx=ax+bfs,where dis(x) is the disparity of the point (x,y∗), S(x) is the slope of the point x in EPI with the fixed y∗, f is the focal length of the lens, and s is the baseline between two views. As the camera parameters f and s are constants, they are ignored. Equation (2) can be rewritten as (3)Sx=ax+b.Figure 1
The relationship between the horizontal axisx in image and its depth is linear if the scene is a flat surface.If the linear function is determined, the slope of each point in EPI can be known and the slope of epipolar lines in EPI can be normalize into a same value. Suppose one point(x,u) in the original EPI; its slope is S(x∗), where x∗ satisfies the following function: (4)x∗+u1Sx∗=x.If the slope we hope to normalize to is S0 (it is called target slope later), the target point of shearing (x,u) is (x′,u), where x′ meets the following function: (5)x′=x∗+u1S0.In a brief summary, for a point(x′,u) in the rectified EPI, the corresponding point in the original EPI is (6)x′-u1S0+u1Sx′-u1/S0,u.We can refer to Figure 2 to understand the procedure of rectification. And Figure 3 gives an example of the original EPI and the rectified EPI. We can see that only one slope is remained in EPI after the rectification.Figure 2
The relationship between two points in the original EPI and the rectified EPI. The black line refers to the location of the epipolar line in the original EPI, and the red dashed line refers to the location of the epipolar line in the rectified EPI. After the rectification, the slope of the red dashed line is the same as the slope of the right black line.Figure 3
(a) is the original EPI, and the slopes of different coordinates are different. (b) is the rectified EPI, and there is only one slope remained in this EPI.
(a)
(b)


### 3.2. Line Function Determination

By substituting the depth range[dmin,dmax] and the image size [1,N] into (1), 4 constraints are determined to search the value of the parameters (a,b), (7)a+b≥dmin,a+b≤dmax,aN+b≥dmin,aN+b≤dmax.By solving these constraints, the searching space of (a,b) is obtained (it is labeled with red in Figure 4). Then (a,b) is generated by dividing the searching space into discrete grids.Figure 4
With the 4 constraints mentioned in (7), the solution space of the parameters (a,b) is obtained and it is labeled with red.We rectify the EPI by using each possible combination of(a,b). A Lisad-1 feature [17] is the local extrema in scale-depth spaces by convolving the EPIs with scale variation kernels. As the Lisad-1 feature point provides the slope information, the feature points are extracted and the ratio r of the feature points which have the same slope with the target slope S0 in all feature points is computed. (8)r=∑i=1FgSFi=S0F,gx=1x=10x≠1,where F is the set of the Lisad-1 feature points extracted from the rectified EPI. |F| is the size of the set F. S(Fi) is the slope of the ith feature point.For a flat surface, if(a,b) is known, all epipolar lines should have the same slope value with the target slope S0 (see Figure 3) after rectification. So we select the optimal (a,b) which results in the largest ratio r.


### 3.3. The Proposed Strategy

Practically, we can not use the ratio of only one EPI to detect the flat surface since it is too regional to represent the whole scene. We take the following two useful properties into consideration to solve the problem:(i)
The value ofa should be a constant with different fixed y∗ if the scene is a flat surface.(ii)
The value ofb should have a linear relationship with the variable y if the scene is a flat surface.These two properties are obvious and can be proved easily (see Appendix). With these two properties, we formulate our strategy as a plane fitting stage and a feature extracting stage.In the plane fitting stage, the plane function (the commona and the function of b with y) is fitted by using a series of parameters (a,b) calculated from several EPIs. And in the feature extracting stage, the parameters (a,b) of each EPI are computed by using the function of the plane. If the scene is a flat surface, the plane function that we build from previous EPIs should also be suitable for other EPIs; that is to say the slope value in all rectified EPIs should be equal to the value S0 and vice versa. This is the core idea of our strategy. The detailed description can be seen in Algorithm 1.Algorithm 1:The proposed strategy for flat surface detection.
(1) Input:Light field LF, slope range [dmin,dmax], the
number of training samplesK1, the number of testing
samplesK2
(2) Output:The energy ratio R of the feature points which
have the same slope value withS0 in all feature points
which come from all rectified EPIs.
(3)/∗Plane fitting stage∗/
(4) Choose K1 EPIs from LF randomly.
(5) Calculate the best parameters (ai,bi) of each EPI by
using the algorithm mentioned in Section3.2.
(6) Select the a0 which has the most frequency in all ai as
the commona. And fit the linear function of b with y
by using thebi whose ai equals to a0.
(7) /∗Feature extracting stage∗/
(8) Choose K2 EPIs from LF randomly.
(9) Calculate the parameters (ai,bi) of each EPI by using
the commona and the linear function of b with y.
(10) Rectify each EPI with its (ai,bi) by using the algorithm
mentioned in Section3.1.
(11) Extract the Lisad-1 feature points of all rectified EPIs.
(12) Count the ratio R of the feature points which have
same slope value withS0 for each EPI, and contruct a
the feature vector by combining these ratios in a
descending order.
(13) Employ a classifier to distinguish flat surface from
natural scenes.


### 3.4. Expand to Inclined Plane Refocus

The traditional method shears the EPI [3, 18] to achieve refocusing; the displacement of each point is the same value as the plane we hope to refocus on is a plane which is parallel to the camera plane, in which all points in the plane are in a same depth. (9)Lαx,u=L0x+u1-1α,u,where L0 denotes the input EPI and Lα denotes the sheared EPI by a value of α.Under the framework of our algorithm, we can obtain the line function of each EPI after the plane fitting stage, and then we shear each point in EPI with a different displacement according to the line function(10)La,bx,u=L0x+Δx,u,Δx=u1ax+b,where L0 denotes the input EPI and La,b denotes the sheared EPI by two parameters (a,b) of the line function.In other words, we just need to set the target slopeS0 as +∞, in which the epipolar line is perpendicular to the horizontal axis. Two refocus results of our method can be seen in Figure 5; as the data captured by us is a 3D light field (1 angular dimension), there is only defocus blur in horizontal direction and no defocus blur in vertical blur.Figure 5
(a) is the refocus result by using the correct inclined plane parameters, and it is an all-in-focus image. (b) is the result by using wrong parameters, and it is noticed that there is only defocus blur in horizontal direction.
(a)
(b)


### 3.5. Depth Estimation for Inclined Plane

Similarly, if the scene is a flat surface, we can estimate its depth with the commona and the function of b obtained from the plane fitting stage. The detailed description is as follows:(i)
Calculate the parameters(a,b) of the line function for each EPI.(ii)
Calculate the disparity of each point in EPI according to(11)disx=1ax+b.We detect the flat surface by using a small set of EPIs, and we fit the function of this plane at the same time. With the function of the flat surface, the depth map can be obtained by substituting the coordinate into the function of the plane.Different from the traditional multilabeling methods [19], our depth estimation results are continuous since we know the function of the flat surface. Two of our results are shown in Figure 6.Figure 6
(a) and (c) are two light fields captured from the printed Medieval dataset at different degrees. (b) and (d) are the corresponding depth maps estimated by using our plane function (red represents large depth value and blue represents small depth value). It can be seen that the depth map is continuous, and there are no noticeable “stairs” in this result.
(a)
(b)
(c)
(d)


## 3.1. Epipolar Plane Image Rectification

On a flat surface, the depth value of a point has a linear relationship with its 2D image coordinates. When the point is in an EPI with a fixedy∗, its depth can be expressed by a linear function only on x (Figure 1), (1)dx=ax+b,where d(x) is the depth of the point (x,y∗). And we can derive the slope of each point by (2)Sx=1disx=1fs/dx=ax+bfs,where dis(x) is the disparity of the point (x,y∗), S(x) is the slope of the point x in EPI with the fixed y∗, f is the focal length of the lens, and s is the baseline between two views. As the camera parameters f and s are constants, they are ignored. Equation (2) can be rewritten as (3)Sx=ax+b.Figure 1
The relationship between the horizontal axisx in image and its depth is linear if the scene is a flat surface.If the linear function is determined, the slope of each point in EPI can be known and the slope of epipolar lines in EPI can be normalize into a same value. Suppose one point(x,u) in the original EPI; its slope is S(x∗), where x∗ satisfies the following function: (4)x∗+u1Sx∗=x.If the slope we hope to normalize to is S0 (it is called target slope later), the target point of shearing (x,u) is (x′,u), where x′ meets the following function: (5)x′=x∗+u1S0.In a brief summary, for a point(x′,u) in the rectified EPI, the corresponding point in the original EPI is (6)x′-u1S0+u1Sx′-u1/S0,u.We can refer to Figure 2 to understand the procedure of rectification. And Figure 3 gives an example of the original EPI and the rectified EPI. We can see that only one slope is remained in EPI after the rectification.Figure 2
The relationship between two points in the original EPI and the rectified EPI. The black line refers to the location of the epipolar line in the original EPI, and the red dashed line refers to the location of the epipolar line in the rectified EPI. After the rectification, the slope of the red dashed line is the same as the slope of the right black line.Figure 3
(a) is the original EPI, and the slopes of different coordinates are different. (b) is the rectified EPI, and there is only one slope remained in this EPI.
(a)
(b)


## 3.2. Line Function Determination

By substituting the depth range[dmin,dmax] and the image size [1,N] into (1), 4 constraints are determined to search the value of the parameters (a,b), (7)a+b≥dmin,a+b≤dmax,aN+b≥dmin,aN+b≤dmax.By solving these constraints, the searching space of (a,b) is obtained (it is labeled with red in Figure 4). Then (a,b) is generated by dividing the searching space into discrete grids.Figure 4
With the 4 constraints mentioned in (7), the solution space of the parameters (a,b) is obtained and it is labeled with red.We rectify the EPI by using each possible combination of(a,b). A Lisad-1 feature [17] is the local extrema in scale-depth spaces by convolving the EPIs with scale variation kernels. As the Lisad-1 feature point provides the slope information, the feature points are extracted and the ratio r of the feature points which have the same slope with the target slope S0 in all feature points is computed. (8)r=∑i=1FgSFi=S0F,gx=1x=10x≠1,where F is the set of the Lisad-1 feature points extracted from the rectified EPI. |F| is the size of the set F. S(Fi) is the slope of the ith feature point.For a flat surface, if(a,b) is known, all epipolar lines should have the same slope value with the target slope S0 (see Figure 3) after rectification. So we select the optimal (a,b) which results in the largest ratio r.


## 3.3. The Proposed Strategy

Practically, we can not use the ratio of only one EPI to detect the flat surface since it is too regional to represent the whole scene. We take the following two useful properties into consideration to solve the problem:(i)
The value ofa should be a constant with different fixed y∗ if the scene is a flat surface.(ii)
The value ofb should have a linear relationship with the variable y if the scene is a flat surface.These two properties are obvious and can be proved easily (see Appendix). With these two properties, we formulate our strategy as a plane fitting stage and a feature extracting stage.In the plane fitting stage, the plane function (the commona and the function of b with y) is fitted by using a series of parameters (a,b) calculated from several EPIs. And in the feature extracting stage, the parameters (a,b) of each EPI are computed by using the function of the plane. If the scene is a flat surface, the plane function that we build from previous EPIs should also be suitable for other EPIs; that is to say the slope value in all rectified EPIs should be equal to the value S0 and vice versa. This is the core idea of our strategy. The detailed description can be seen in Algorithm 1.Algorithm 1:The proposed strategy for flat surface detection.
(1) Input:Light field LF, slope range [dmin,dmax], the
number of training samplesK1, the number of testing
samplesK2
(2) Output:The energy ratio R of the feature points which
have the same slope value withS0 in all feature points
which come from all rectified EPIs.
(3)/∗Plane fitting stage∗/
(4) Choose K1 EPIs from LF randomly.
(5) Calculate the best parameters (ai,bi) of each EPI by
using the algorithm mentioned in Section3.2.
(6) Select the a0 which has the most frequency in all ai as
the commona. And fit the linear function of b with y
by using thebi whose ai equals to a0.
(7) /∗Feature extracting stage∗/
(8) Choose K2 EPIs from LF randomly.
(9) Calculate the parameters (ai,bi) of each EPI by using
the commona and the linear function of b with y.
(10) Rectify each EPI with its (ai,bi) by using the algorithm
mentioned in Section3.1.
(11) Extract the Lisad-1 feature points of all rectified EPIs.
(12) Count the ratio R of the feature points which have
same slope value withS0 for each EPI, and contruct a
the feature vector by combining these ratios in a
descending order.
(13) Employ a classifier to distinguish flat surface from
natural scenes.


## 3.4. Expand to Inclined Plane Refocus

The traditional method shears the EPI [3, 18] to achieve refocusing; the displacement of each point is the same value as the plane we hope to refocus on is a plane which is parallel to the camera plane, in which all points in the plane are in a same depth. (9)Lαx,u=L0x+u1-1α,u,where L0 denotes the input EPI and Lα denotes the sheared EPI by a value of α.Under the framework of our algorithm, we can obtain the line function of each EPI after the plane fitting stage, and then we shear each point in EPI with a different displacement according to the line function(10)La,bx,u=L0x+Δx,u,Δx=u1ax+b,where L0 denotes the input EPI and La,b denotes the sheared EPI by two parameters (a,b) of the line function.In other words, we just need to set the target slopeS0 as +∞, in which the epipolar line is perpendicular to the horizontal axis. Two refocus results of our method can be seen in Figure 5; as the data captured by us is a 3D light field (1 angular dimension), there is only defocus blur in horizontal direction and no defocus blur in vertical blur.Figure 5
(a) is the refocus result by using the correct inclined plane parameters, and it is an all-in-focus image. (b) is the result by using wrong parameters, and it is noticed that there is only defocus blur in horizontal direction.
(a)
(b)


## 3.5. Depth Estimation for Inclined Plane

Similarly, if the scene is a flat surface, we can estimate its depth with the commona and the function of b obtained from the plane fitting stage. The detailed description is as follows:(i)
Calculate the parameters(a,b) of the line function for each EPI.(ii)
Calculate the disparity of each point in EPI according to(11)disx=1ax+b.We detect the flat surface by using a small set of EPIs, and we fit the function of this plane at the same time. With the function of the flat surface, the depth map can be obtained by substituting the coordinate into the function of the plane.Different from the traditional multilabeling methods [19], our depth estimation results are continuous since we know the function of the flat surface. Two of our results are shown in Figure 6.Figure 6
(a) and (c) are two light fields captured from the printed Medieval dataset at different degrees. (b) and (d) are the corresponding depth maps estimated by using our plane function (red represents large depth value and blue represents small depth value). It can be seen that the depth map is continuous, and there are no noticeable “stairs” in this result.
(a)
(b)
(c)
(d)


## 4. Experimental Results

### 4.1. Experimental Setup

To better analyze the performance of our algorithm, we select two different datasets. The HCI light field dataset [20] and its printed edition are selected to analyze the properties of energy ratio. It is noticed that the printed photos are tilted to different degrees in order to better evaluate our algorithm. The experimental environment and printed light fields can be seen in Figure 7. Apart from this, the EPFL light field dataset [14] is selected to do a comparison with the previous work. As this dataset is captured by a Lytro camera, the experimental results on this dataset can better reflect the pros and cons of the algorithm.Figure 7
The left top one is our experimental environment. Others are printed light fields. Note that the scene in the left top is just one of the different situations, and our datasets contain 5 different tilt angles.We implement the algorithm in the Matlab 2014b, on OS X 10.11.1 with 8 gigabytes of RAM and 2.7 GHz of processor. The running time of our implementation for a9×9×768×768×3 light field is measured in seconds but does not excel the time complexity of [14]. This time can be accelerated to microseconds by using GPU.In the stage of determining line function, we divide the searching space of(a,b) into an 11×11 grid. The target slope S0 we hope to normalize to is not important in our rectification; actually it can be an arbitrary value. In the plane fitting stage, we select 11 EPIs to obtain the function of the plane and select another 15 EPIs in the feature extracting stage. The SVM classifier is employed to distinguish the flat surface from the natural scenes.


### 4.2. Analysis of the Energy Ratio

We compute the energy ratio for each EPI in the first dataset and plot their distribution in Figure9. The horizontal axis is the line number, and the vertical axis is the energy ratio for each EPI. It can be seen that the energy ratio of natural scenes may reach a high value sometimes, but mostly it is very small and far away from the flat surface. Apart from this, the energy ratio distribution of flat surface is not only large but also stable; in contrast, natural scenes do not meet these properties.It is noticed that there are always some EPIs which have large energy ratios in original scenes (the first half of the blue curve in Figure8). The main reason for this phenomenon is that these EPIs are selected to fit the plane function, and the fitted plane function is more suitable for these areas. However, it may not be suitable for other EPIs (the second half of the blue curve in Figure 8), and this is the reason why we select other EPIs to combine the feature vectors.Figure 8
(a)–(d) are four failed samples of our algorithm, where (a) and (b) are misclassified as flat surface, and (c) and (d) are misclassified as natural scenes. The green points are the Lisad-1 points. (e) and (f) are the corresponding distributions of energy ratio.
(a)
(b)
(c)
(d)
(e)
(f)Figure 9
The energy ratio distribution for different light fields.
(a)
(b)
(c)


### 4.3. Further Experiments and Comparison

In order to better evaluate the accuracy of our algorithm in real data captured by light field camera, we run our code in another public light field dataset captured by Ghasemi and Vetterli [14]. The dataset consists of 50 light fields of printed photos and 50 light fields of natural scenes.To test and verify our algorithm in a classification setting, we used a SVM model with cross validation [21]. The results can be seen in Table 1.Table 1
The detection accuracy comparison.
Ours
Ghasemi and Vetterli [14]
Printed (P)
94%
92%
Natural (N)
94%
86%Our detection precision is clearly superior to Ghasemi and Vetterli’s method [14]. This improvement is prominent especially in the detection of natural scenes, where 7 natural scenes are misclassified as flat surface in [14] and only 3 in our method. We further analyze these failed data and find that most textures of the scene lie on one continuous depth plane and there are a few textures on others. The feature points which come from this continuous plane play a more vital role and lead to a higher energy ratio. In Figure 8(a), most feature points lie on the cardboard and a few points on the foreground, the fitted plane is close to the cardboard plane, and it leads to high energy ratio (the blue curve in Figure 8(e)). Then for the flat surface, the value of wrong classified samples is 3 as well. By analyzing these samples, it is noticed that the number of feature points is too little to estimate the plane function accurately. In Figure 8(c), there are more feature points in the bottom of the scene and a few points in the top which lead to a wrong fitting of the parameters b in Section 3.3. Figures 8(e) and 8(f) show the energy ratio distribution of these failed samples.


## 4.1. Experimental Setup

To better analyze the performance of our algorithm, we select two different datasets. The HCI light field dataset [20] and its printed edition are selected to analyze the properties of energy ratio. It is noticed that the printed photos are tilted to different degrees in order to better evaluate our algorithm. The experimental environment and printed light fields can be seen in Figure 7. Apart from this, the EPFL light field dataset [14] is selected to do a comparison with the previous work. As this dataset is captured by a Lytro camera, the experimental results on this dataset can better reflect the pros and cons of the algorithm.Figure 7
The left top one is our experimental environment. Others are printed light fields. Note that the scene in the left top is just one of the different situations, and our datasets contain 5 different tilt angles.We implement the algorithm in the Matlab 2014b, on OS X 10.11.1 with 8 gigabytes of RAM and 2.7 GHz of processor. The running time of our implementation for a9×9×768×768×3 light field is measured in seconds but does not excel the time complexity of [14]. This time can be accelerated to microseconds by using GPU.In the stage of determining line function, we divide the searching space of(a,b) into an 11×11 grid. The target slope S0 we hope to normalize to is not important in our rectification; actually it can be an arbitrary value. In the plane fitting stage, we select 11 EPIs to obtain the function of the plane and select another 15 EPIs in the feature extracting stage. The SVM classifier is employed to distinguish the flat surface from the natural scenes.


## 4.2. Analysis of the Energy Ratio

We compute the energy ratio for each EPI in the first dataset and plot their distribution in Figure9. The horizontal axis is the line number, and the vertical axis is the energy ratio for each EPI. It can be seen that the energy ratio of natural scenes may reach a high value sometimes, but mostly it is very small and far away from the flat surface. Apart from this, the energy ratio distribution of flat surface is not only large but also stable; in contrast, natural scenes do not meet these properties.It is noticed that there are always some EPIs which have large energy ratios in original scenes (the first half of the blue curve in Figure8). The main reason for this phenomenon is that these EPIs are selected to fit the plane function, and the fitted plane function is more suitable for these areas. However, it may not be suitable for other EPIs (the second half of the blue curve in Figure 8), and this is the reason why we select other EPIs to combine the feature vectors.Figure 8
(a)–(d) are four failed samples of our algorithm, where (a) and (b) are misclassified as flat surface, and (c) and (d) are misclassified as natural scenes. The green points are the Lisad-1 points. (e) and (f) are the corresponding distributions of energy ratio.
(a)
(b)
(c)
(d)
(e)
(f)Figure 9
The energy ratio distribution for different light fields.
(a)
(b)
(c)


## 4.3. Further Experiments and Comparison

In order to better evaluate the accuracy of our algorithm in real data captured by light field camera, we run our code in another public light field dataset captured by Ghasemi and Vetterli [14]. The dataset consists of 50 light fields of printed photos and 50 light fields of natural scenes.To test and verify our algorithm in a classification setting, we used a SVM model with cross validation [21]. The results can be seen in Table 1.Table 1
The detection accuracy comparison.
Ours
Ghasemi and Vetterli [14]
Printed (P)
94%
92%
Natural (N)
94%
86%Our detection precision is clearly superior to Ghasemi and Vetterli’s method [14]. This improvement is prominent especially in the detection of natural scenes, where 7 natural scenes are misclassified as flat surface in [14] and only 3 in our method. We further analyze these failed data and find that most textures of the scene lie on one continuous depth plane and there are a few textures on others. The feature points which come from this continuous plane play a more vital role and lead to a higher energy ratio. In Figure 8(a), most feature points lie on the cardboard and a few points on the foreground, the fitted plane is close to the cardboard plane, and it leads to high energy ratio (the blue curve in Figure 8(e)). Then for the flat surface, the value of wrong classified samples is 3 as well. By analyzing these samples, it is noticed that the number of feature points is too little to estimate the plane function accurately. In Figure 8(c), there are more feature points in the bottom of the scene and a few points in the top which lead to a wrong fitting of the parameters b in Section 3.3. Figures 8(e) and 8(f) show the energy ratio distribution of these failed samples.


## 5. Conclusion

In this paper, we propose an algorithm to rectify the EPI of a flat surface, which normalizes the mess slope of epipolar lines in EPI into a same slope. And this algorithm can be easily expanded to the inclined plane refocus and the continuous depth estimation for flat surface. Then, we propose a framework for flat surface detection, which learns a function of the flat surface by using several EPIs and tests this function by using another several EPIs. The results show that our algorithm performs well for most scenes, and the more complex the scene, the better the performance.We may continue to study the limits of the algorithms, such as in terms of low texture scene which leads to the insufficient feature points.



---

*Source: 6142795-2017-09-19.xml*