# Omnigradient Based Total Variation Minimization for Enhanced Defocus Deblurring of Omnidirectional Images

Authors: Yongle Li; Jingtao Lou
Journal: International Journal of Optics
 (2014)

---

## Abstract

We propose a new method of image restoration for catadioptric defocus blur using omnitotal variation (Omni-TV) minimization based on omnigradient. Catadioptric omnidirectional imaging systems usually consist of conventional cameras and curved mirrors for capturing 360° field of view. The problem of catadioptric omnidirectional imaging defocus blur, which is caused by lens aperture and mirror curvature, becomes more severe when high resolution sensors and large apertures are used. In an omnidirectional image, two points near each other may not be close to one another in the 3D scene. Traditional gradient computation cannot be directly applied to omnidirectional image processing. Thus, omnigradient computing method combined with the characteristics of catadioptric omnidirectional imaging is proposed. Following this Omni-TV minimization is used as the constraint for deconvolution regularization, leading to the restoration of defocus blur in an omnidirectional image to obtain all sharp omnidirectional images. The proposed method is important for improving catadioptric omnidirectional imaging quality and promoting applications in related fields like omnidirectional video and image processing.


---

## Body

## 1. Introduction

Owing to the advantage of one-shot seamless panoramic imaging, catadioptric omnidirectional imaging systems consisting of conventional cameras and curved mirrors for capturing 360° field of view are widely used in many vision applications, such as aerial photographic reconnaissance, intelligent transportation system, robot navigation, surveillance, medical applications, and video conferencing [1–5]. The problem of catadioptric omnidirectional imaging defocus blur, which is caused by lens aperture and mirror curvature, becomes more severe when high resolution sensors and large apertures are used. To solve the defocus problem, the classical Wiener filter and Richardson-Lucy deconvolution method [6] can be applied to obtain the clear images. In recent years, the restoration methods based on regularization theory, like Tikhonov model and Total Variation (TV) minimization model [7, 8], have become more popular. The TV minimization based regularization restoration approach aims to preserve more image details and restrain noise impact and ringing artifacts. Nevertheless, the traditional TV computation is not appropriate for omnidirectional image processing. The classical operators cannot be directly applied to this type of images because of the catadioptric projection [9, 10]. Thus, targeting omnidirectional images, a novel gradient computation is needed. The remainder of this paper is structured as follows. In Section 2, to make this paper self-contained, the basic principles and defocus blur analysis of a single viewpoint catadioptric omnidirectional imaging are briefly summarized. In Section 3, the omnigradient computation method combined with the characteristics of catadioptric omnidirectional imaging is proposed. For deconvolution restoration, Omni-TV minimization is proposed as the regularization constraint. Experimental results in Section 4 show that the proposed approach is effective for omnidirectional image defocus deblurring and has an important impact on improving catadioptric omnidirectional imaging quality and promoting other applications in related fields.


## 2. Analysis of Catadioptric Omnidirectional Imaging Defocus Blur

To make this paper self-contained, the basic principles and defocus blur analysis of a single viewpoint catadioptric omnidirectional imaging are briefly summarized. The system of central catadioptric omnidirectional imaging with hyperboloid mirror is considered for single viewpoint constraints [11]. Figure 1 shows an illustration of a Cartesian frame ROZ where o is the location of the effective viewpoint which is the origin and one focus of the hyperboloid mirror, p
0, is the location of the effective pinhole which is the other focus of the hyperboloid.Figure 1
Defocus sketch map of a catadioptric omnidirectional imaging system.Considering a pointm
0 (x
0
,
y
0
,
z
0) on the mirror and a point w in the world, according to the single viewpoint constraint, if the extended ray w
m
0 passes through the effective viewpoint o, the ray of light from w which is reflected by the mirror at m
0 passes directly through the centre of the lens p
0. The hyperboloid can be expressed as
(1)
z
-
c
2
c
2
k
-
2
/
k
-
x
2
+
y
2
c
2
2
/
k
=
1
,
where c and k are the parameters of the hyperboloid equation and k
>
2 is a constant. The lens plane is located at z
=
2
c and rays of light w
m
0 and m
0
p
0 are the principal incident ray and reflected ray, respectively. The image plane is located at z
=
2
c
+
v, where w
i
0 is the image of world point w and v is the distance from the image plane to lens plane.Suppose that another ray of light fromw which is reflected by the mirror at m
1 passes through the point p
1 on the lens and finally arrives at the image plane at the point w
i
1. Similarly, another ray of light from w which is reflected by the mirror at m
2 passes through the point p
2 on the lens and eventually arrives at the image plane at the point w
i
2. In general, these two rays of light will not be imaged at the same point on the image plane as the principal ray. This causes defocus blur to occur [12]. The defocus simulation results shown in Figure 2 are implemented using ZEMAX [13] which is one of optics simulation tools.Figure 2
Simulation results of catadioptric omnidirectional imaging defocus.


## 3. Omnitotal Variation Minimization Deconvolution Algorithm

### 3.1. Image Restoration Model

The image degradation model can be expressed as(2)
g
x
,
y
=
h
x
,
y
*
f
x
,
y
+
n
x
,
y
,
where g
(
x
,
y
) is the captured image, f
(
x
,
y
) is the original clear image, h
(
x
,
y
) is PSF, * is a convolution operator, and n
(
x
,
y
) is image noise. Image deconvolution is an inverse problem with applications in remote sensing, medical image, and, more generally, image restoration. The challenge in many inverse problems is that they are ill-posed even if the point spread function is known. To cope with the ill-posed nature of these problems, a large number of techniques have been developed, most of them using regularization. At the heart of regularization is a priori knowledge expressed by the prior or regularization term.TV regularization was introduced by Rudin et al. in [8] and has become popular in recent years [6]. Recently, the range of applications of TV-based methods has been successfully extended to inpainting, blind deconvolution, and processing of vector-valued images. Arguably, the success of TV regularization relies on a good balance between the ability to describe piecewise smooth images and the complexity of the resulting algorithms. The TV regularizer favors images of bounded variation, without penalizing possible discontinuities. Furthermore, the TV regularizer is convex, thus opening the door to research on efficient algorithms for computing optimal or nearly optimal solutions [7].The TVJ
TV
(
f
) can be defined as the sum of gradient amplitudes over the entire image f as
(3)
J
TV
f
=
∑
∇
f
x
,
y
=
∑
G
x
2
+
G
y
2
1
/
2
=
∑
∂
f
∂
x
2
+
∂
f
∂
y
2
1
/
2
,
where G
x
=
∂
f
/
∂
x, G
y
=
∂
f
/
∂
y, ∇
f
x
,
y is the gradient amplitude in the pixel position (
x
,
y
) of the image f. According to the law of Lagrange, the image restoration based on TV minimization can be expressed as the following minimization problem:
(4)
f
*
=
arg
⁡
min
⁡
⁡
h
*
f
-
g
2
+
λ
J
TV
f
,
where λ is a constant which can control the convergence of the deconvolution algorithm and λ
J
TV
(
f
) is the constraint, also called the image prior. The clear image f
* can be obtained by minimizing the reconstruction error h
*
f
-
g
2.


### 3.2. Omnitotal Variation

We need to especially note that, because of the distortions observed in such catadioptric omnidirectional images as shown in Figure3, traditional image processing techniques are no longer appropriate and need to be adapted to the new sensor geometry. In this way, classical operators cannot be directly applied to this type of images. As we know, the spatial resolution of catadioptric omnidirectional images decreases gradually from the periphery to the centre of the image. Because of catadioptric projection, the two distances, which have the same size between two points in real world, are different when the two points are imaged in the periphery and the centre of the image, respectively. That is, two points near each other in the omnidirectional image do not have the same relation in the real world.Figure 3
Catadioptric camera and omnidirectional image.Thus, traditional gradient computing methods cannot reflect the characteristics of catadioptric omnidirectional imaging and the intensity variability between the nearby pixels. We propose a new gradient computing method. First, a cylindrical panoramic image is obtained by the backward projection of an omnidirectional image. Then, the nearby pixels in the horizontal and vertical directions are established in the cylindrical panoramic image. Finally, the nearby pixels are forward projected onto the omnidirectional image to find the corresponding positions to calculate the gradient.As shown in Figure4(a), in terms of the single viewpoint constraints of catadioptric omnidirectional imaging and the method of light ray tracking, the point P
i in an omnidirectional image is backward projected to the point P
c in a cylindrical panoramic image passed through the point P
m on the mirror, where O
c and O
-
c are the two foci of the hyperbolic mirror, respectively, and R is the radius of the cylinder. As shown in Figure 4(b), the point I
(
i
,
j
) in the omnidirectional image is backward projected to the point P
(
u
,
v
) in a cylindrical panoramic image. Then, nearby pixels P
(
u
+
1
,
v
) and P
(
u
,
v
+
1
) of the point P
(
u
,
v
) in the horizontal and vertical directions are established in the panoramic image. P
(
u
+
1
,
v
) and P
(
u
,
v
+
1
) are forward projected to I
(
i
+
s
1
,
j
+
t
1
) and I
(
i
+
s
2
,
j
+
t
2
) in the omnidirectional image. Thus, the omnigradient amplitude in the pixel I
(
i
,
j
) can be expressed as
(5)
∇
I
i
,
j
=
I
i
+
s
1
,
j
+
t
1
-
I
i
,
j
2
+
I
i
+
s
2
,
j
+
t
2
-
I
i
,
j
2
.Sketch map of Omni-TV computation: (a) backward projection; (b) Omni-TV computation method.
(a)
(b)The Omni-TV of omnidirectional imageI is defined as
(6)
J
Omni-TV
I
=
∑
i
,
j
∇
I
i
,
j
=
∑
i
,
j
I
i
+
s
1
,
j
+
t
1
-
I
i
,
j
2
+
I
i
+
s
2
,
j
+
t
2
-
I
i
,
j
2
.
Combining (4) and (6), the deblurred omnidirectional image can be obtained.


## 3.1. Image Restoration Model

The image degradation model can be expressed as(2)
g
x
,
y
=
h
x
,
y
*
f
x
,
y
+
n
x
,
y
,
where g
(
x
,
y
) is the captured image, f
(
x
,
y
) is the original clear image, h
(
x
,
y
) is PSF, * is a convolution operator, and n
(
x
,
y
) is image noise. Image deconvolution is an inverse problem with applications in remote sensing, medical image, and, more generally, image restoration. The challenge in many inverse problems is that they are ill-posed even if the point spread function is known. To cope with the ill-posed nature of these problems, a large number of techniques have been developed, most of them using regularization. At the heart of regularization is a priori knowledge expressed by the prior or regularization term.TV regularization was introduced by Rudin et al. in [8] and has become popular in recent years [6]. Recently, the range of applications of TV-based methods has been successfully extended to inpainting, blind deconvolution, and processing of vector-valued images. Arguably, the success of TV regularization relies on a good balance between the ability to describe piecewise smooth images and the complexity of the resulting algorithms. The TV regularizer favors images of bounded variation, without penalizing possible discontinuities. Furthermore, the TV regularizer is convex, thus opening the door to research on efficient algorithms for computing optimal or nearly optimal solutions [7].The TVJ
TV
(
f
) can be defined as the sum of gradient amplitudes over the entire image f as
(3)
J
TV
f
=
∑
∇
f
x
,
y
=
∑
G
x
2
+
G
y
2
1
/
2
=
∑
∂
f
∂
x
2
+
∂
f
∂
y
2
1
/
2
,
where G
x
=
∂
f
/
∂
x, G
y
=
∂
f
/
∂
y, ∇
f
x
,
y is the gradient amplitude in the pixel position (
x
,
y
) of the image f. According to the law of Lagrange, the image restoration based on TV minimization can be expressed as the following minimization problem:
(4)
f
*
=
arg
⁡
min
⁡
⁡
h
*
f
-
g
2
+
λ
J
TV
f
,
where λ is a constant which can control the convergence of the deconvolution algorithm and λ
J
TV
(
f
) is the constraint, also called the image prior. The clear image f
* can be obtained by minimizing the reconstruction error h
*
f
-
g
2.


## 3.2. Omnitotal Variation

We need to especially note that, because of the distortions observed in such catadioptric omnidirectional images as shown in Figure3, traditional image processing techniques are no longer appropriate and need to be adapted to the new sensor geometry. In this way, classical operators cannot be directly applied to this type of images. As we know, the spatial resolution of catadioptric omnidirectional images decreases gradually from the periphery to the centre of the image. Because of catadioptric projection, the two distances, which have the same size between two points in real world, are different when the two points are imaged in the periphery and the centre of the image, respectively. That is, two points near each other in the omnidirectional image do not have the same relation in the real world.Figure 3
Catadioptric camera and omnidirectional image.Thus, traditional gradient computing methods cannot reflect the characteristics of catadioptric omnidirectional imaging and the intensity variability between the nearby pixels. We propose a new gradient computing method. First, a cylindrical panoramic image is obtained by the backward projection of an omnidirectional image. Then, the nearby pixels in the horizontal and vertical directions are established in the cylindrical panoramic image. Finally, the nearby pixels are forward projected onto the omnidirectional image to find the corresponding positions to calculate the gradient.As shown in Figure4(a), in terms of the single viewpoint constraints of catadioptric omnidirectional imaging and the method of light ray tracking, the point P
i in an omnidirectional image is backward projected to the point P
c in a cylindrical panoramic image passed through the point P
m on the mirror, where O
c and O
-
c are the two foci of the hyperbolic mirror, respectively, and R is the radius of the cylinder. As shown in Figure 4(b), the point I
(
i
,
j
) in the omnidirectional image is backward projected to the point P
(
u
,
v
) in a cylindrical panoramic image. Then, nearby pixels P
(
u
+
1
,
v
) and P
(
u
,
v
+
1
) of the point P
(
u
,
v
) in the horizontal and vertical directions are established in the panoramic image. P
(
u
+
1
,
v
) and P
(
u
,
v
+
1
) are forward projected to I
(
i
+
s
1
,
j
+
t
1
) and I
(
i
+
s
2
,
j
+
t
2
) in the omnidirectional image. Thus, the omnigradient amplitude in the pixel I
(
i
,
j
) can be expressed as
(5)
∇
I
i
,
j
=
I
i
+
s
1
,
j
+
t
1
-
I
i
,
j
2
+
I
i
+
s
2
,
j
+
t
2
-
I
i
,
j
2
.Sketch map of Omni-TV computation: (a) backward projection; (b) Omni-TV computation method.
(a)
(b)The Omni-TV of omnidirectional imageI is defined as
(6)
J
Omni-TV
I
=
∑
i
,
j
∇
I
i
,
j
=
∑
i
,
j
I
i
+
s
1
,
j
+
t
1
-
I
i
,
j
2
+
I
i
+
s
2
,
j
+
t
2
-
I
i
,
j
2
.
Combining (4) and (6), the deblurred omnidirectional image can be obtained.


## 4. Experimental Results

### 4.1. Omnigradient Computation Results

In order to verify the rationality and effectiveness of the proposed Omni-TV, the experiments give the comparative results using the computation methods of omnigradient and traditional gradient, respectively. Figure5 is the simulation test image of catadioptric omnidirectional imaging using 3ds Max. The simulation scene contains the cameraman, part of eye chart, lena, and peppers. Then, the scene is pasted onto a cylindrical surface joined from the beginning to the end. As shown in Figure 6, the results obtained by our proposed omnigradient computation can show the image edges more clearly. Figure 7 is the omnigradient computation result of a real image. In particular, in the centre part of the omnidirectional image, the edges are clearer and more regular along the radial direction.Figure 5
Simulation test image of catadioptric omnidirectional imaging.Gradient computation results of test image: (a) and (b) are traditional gradient computation result and close-up views, respectively; (c) and (d) are omnigradient computation result and close-up views, respectively.
(a)
(b)
(c)
(d)Figure 7
Gradient computation results of real image: (a) and (b) are traditional gradient computation result and close-up views, respectively; (c) and (d) are omnigradient computation result and close-up views, respectively.


### 4.2. Image Restoration Results

In order to verify the effectiveness of the deconvolution algorithm based on Omni-TV minimization, the TwIST algorithm [14] is applied, using both the Omni-TV and traditional TV as the image priors, to do the deconvolution. The signal-to-noise ratio (SNR), which can be defined as 20
log
⁡
10
⁡
(
f
-
f
2
/
f
-
f
*
2
), is used to assess the deblurred results. f and f
* are original and deblurred image; f is the average value of the original image. Figure 8 is the defocus deblurred results for the simulation test image. Figure 8(a) is the ground truth. Figure 8(b) is the blurry image. Figure 8(c) shows deblurred result using traditional TV, with SNR
=
22.87. Figure 8(d) shows deblurred result using Omni-TV, with SNR
=
24.47. Figure 9 shows the SNR comparison results when the local window (64 × 64 pixel) moves from the centre to the periphery of the image, and the deconvolution results based on Omni-TV are better than these using traditional TV. Figure 10 shows the defocus deblurred results of a real omnidirectional image captured by our experimental setup. We can observe that the proposed deconvolution algorithm using Omni-TV as the image prior can produce more visually appealing deblurred results.Defocus deblurring results of test image: (a) simulation omnidirectional image (ground truth); (b) blurry image; (c) deblurred results using TwIST algorithm, SNR = 22.87; (d) deblurred results using our proposed method (Omni-TV), SNR = 24.47.
(a)
(b)
(c)
(d)SNR comparison when the local window moves from the centre to the periphery of the image: (a) local window locations; (b) SNR comparison of cameraman; (c) SNR comparison of lena; (d) SNR comparison of peppers.
(a)
(b)
(c)
(d)Defocus deblurring results of real image captured by our experimental setup: (a) blurry omnidirectional image; (b) deblurred results using TwIST algorithm; (c) deblurred results using our proposed method (Omni-TV).
(a)
(b)
(c)


## 4.1. Omnigradient Computation Results

In order to verify the rationality and effectiveness of the proposed Omni-TV, the experiments give the comparative results using the computation methods of omnigradient and traditional gradient, respectively. Figure5 is the simulation test image of catadioptric omnidirectional imaging using 3ds Max. The simulation scene contains the cameraman, part of eye chart, lena, and peppers. Then, the scene is pasted onto a cylindrical surface joined from the beginning to the end. As shown in Figure 6, the results obtained by our proposed omnigradient computation can show the image edges more clearly. Figure 7 is the omnigradient computation result of a real image. In particular, in the centre part of the omnidirectional image, the edges are clearer and more regular along the radial direction.Figure 5
Simulation test image of catadioptric omnidirectional imaging.Gradient computation results of test image: (a) and (b) are traditional gradient computation result and close-up views, respectively; (c) and (d) are omnigradient computation result and close-up views, respectively.
(a)
(b)
(c)
(d)Figure 7
Gradient computation results of real image: (a) and (b) are traditional gradient computation result and close-up views, respectively; (c) and (d) are omnigradient computation result and close-up views, respectively.


## 4.2. Image Restoration Results

In order to verify the effectiveness of the deconvolution algorithm based on Omni-TV minimization, the TwIST algorithm [14] is applied, using both the Omni-TV and traditional TV as the image priors, to do the deconvolution. The signal-to-noise ratio (SNR), which can be defined as 20
log
⁡
10
⁡
(
f
-
f
2
/
f
-
f
*
2
), is used to assess the deblurred results. f and f
* are original and deblurred image; f is the average value of the original image. Figure 8 is the defocus deblurred results for the simulation test image. Figure 8(a) is the ground truth. Figure 8(b) is the blurry image. Figure 8(c) shows deblurred result using traditional TV, with SNR
=
22.87. Figure 8(d) shows deblurred result using Omni-TV, with SNR
=
24.47. Figure 9 shows the SNR comparison results when the local window (64 × 64 pixel) moves from the centre to the periphery of the image, and the deconvolution results based on Omni-TV are better than these using traditional TV. Figure 10 shows the defocus deblurred results of a real omnidirectional image captured by our experimental setup. We can observe that the proposed deconvolution algorithm using Omni-TV as the image prior can produce more visually appealing deblurred results.Defocus deblurring results of test image: (a) simulation omnidirectional image (ground truth); (b) blurry image; (c) deblurred results using TwIST algorithm, SNR = 22.87; (d) deblurred results using our proposed method (Omni-TV), SNR = 24.47.
(a)
(b)
(c)
(d)SNR comparison when the local window moves from the centre to the periphery of the image: (a) local window locations; (b) SNR comparison of cameraman; (c) SNR comparison of lena; (d) SNR comparison of peppers.
(a)
(b)
(c)
(d)Defocus deblurring results of real image captured by our experimental setup: (a) blurry omnidirectional image; (b) deblurred results using TwIST algorithm; (c) deblurred results using our proposed method (Omni-TV).
(a)
(b)
(c)


## 5. Conclusion

In this paper, a novel method of image restoration for catadioptric defocus blur image based on omnitotal variation minimization is proposed. Since the traditional gradient computation cannot fit the characteristic of catadioptric omnidirectional imaging, the omnigradient computing method is proposed. Then, omnitotal variation minimization is used as the condition of deconvolution regularization, which is used for the defocus blur omnidirectional image restoration to obtain all sharp omnidirectional images. Experimental results show that the proposed algorithm can produce more visually appealing deblurred results. In particular, in the centre part of the omnidirectional image, the edges are clearer and more regular along the radial direction.



---

*Source: 732937-2014-12-08.xml*