Window Image Reflection Removal by Blind Image Separation
1

Mohammad Reza Alsharif1 , Mahdi Khosravy1 and Katsumi Yamashita2
Department of Information Engineering, University of the Ryukyus, Okinawa, Japan
2
Graduate School of Engineering, Osaka Prefecture University, Osaka, Japan
E-mail : 1 asharif@ie.u-ryukyu.ac.jp, mahdikhosravy@yahoo.com
2
yamashita@eis.osakafu-u.ac.jp

Abstract

gether and mixture signals are collected by M sensors, BSS try to blindly separate the sources frommixed observed data without any prior information of
the source signals and mixing process. In mathematical model, the M × 1 vector x of observed signalsx(t) = (x1 (t) x2 (t) · · · xM (t))T is multiplicationof K × 1 vector s of unknown source signals s(t) =(s1 (t) s2 (t) · · · sK (t))T by unknown mixing matrix
A ∈ RM×K :A novel second order statistics approach to blind image separation has been deployed to remove reflection
image from window glass images. The reflection image and the image from behind the window glass are
considered as blind sources. Since sources are uncorrelated, their low frequency as well as high frequency halfbands are uncorrelated too. It is shown thatthe un-mixing matrix is obtained by simultaneous diagonalization of covariance matrices of low and highhalfbands of the mixtures. The proposed technique
has been compared with AMUSE [1] and SOBI [2] as
two other well known second-order statistics BSS techniques. Three different numerical metrics approves theefficiency dominance of the proposed technique.

where n(t) is a possible additive noise. Given only theobservation signals x(t) = (x1 (t) x2 (t) · · · xM (t))T ,the solution of BSS problem seeks for the best W ∈RK×M as un-mixing matrix to extract signals as muchas possible close to unknown source signals as follows:

Keywords: Reflection removal, Blind image separation,y(t) = W x(t)x(t) = As(t) + n(t)second-order statistics, covarriance matrix, subband linear filters.

(1)

(2)

where y(t) is K × 1 vector of extracted signals y(t) =(y1 (t) y2 (t) · · · yK (t))T . W is obtained in the waythat the estimated sources to be spatially uncorrelated
or as independent as possible, in the sense of minimizing various functions that measure independence [59]. This measure is based upon second-order statistics(SOS) or higher-order statistics (HOS). Many secondorder BSS approaches have been presented in the literature [10- 18]. One of the well known second-order BSS
methods is an algorithm for multiple unknown signals
extraction (AMUSE) [1]. AMUSE converts the problem of blind identification of mixing matrix to simultaneous diagonalization (SD) problem of two covariancematrices of mixtures and their delayed signals. The
other well known second-order BSS method is secondorder blind identification (SOBI) algorithms [2]. SOBIemploys joint diagonlization of covariance matrix ofmixtures and covariance matrices of different delayed
mixtures to obtain un-mixing matrix.
In this paper, the reflection over window glass isblindly separated from the image of the objects behind the glass by using a BSS approach by deploying second-order statistics of the sources in two different views at the same time. The proposed BSS isbased on assumption that sources are mutually uncorrelated. When sources are uncorrelated, their subbands

1 Introduction
Although images normally do not mix together in awell engineered system, blind image separation is aproblem encountered in various applications such as
document image restoration, astrophysical component
separation, analysis of functional Magnetic ResonanceImaging and removal of spurious reflections. In taking
photograph from the objects behind the window glass,
the taken photo is a mixture of the image of the objects
behind the glass and the reflection image of the objects
in front of the glass. Such event happens in window
shopping, aquarium photography and any other photography with a glass window between camera and objects.
Also in photography of inside the water from above ofwater surface, the photo is mixture of inside the waterand reflection of the sky and clouds appeared on thewater surface.
Blind source separation (BSS) [3, 4] is a well known
technique to recover multiple original sources fromtheir mixtures. While K unknown source signals upontransmission through a medium have been mixed to-1obtained by linear filters are uncorrelated too. The simultaneous diagonalization (SD) of covariance matrices of the low halfbands and high halfbands of image
mixtures has been deployed to remove glass reflectionof window images. The proposed BSS method has been
compared with AMUSE and SOBI.
The rest of the paper is as follows. In section 2, theBSS by SD of high and low halfbands has been described. Experimental results and discussions are presented in section 3. Finally, Section 4 concludes thepaper.

Consider ȳ(t) and ey(t) are respectively low and highhalfbands of the recovered signals by W. ȳ(t) and ey(t)have been obtained by a linear lowpass filter and a linear highpass filter, respectively. Throughout this section the following assumptions are made unless statedotherwise:• Assumption 1 : The mixing matrix A ∈ RM×K
is a full column rank matrix with M ≥ K, and itis the same for all frequency contents of signals.
• Assumption 2 : Sources are mutually uncorrelatedwith different autocorrelation functions.

2 BSS by simultaneous diagonalization of low and high halfbands

E [si sj ] = 0,i 6= j

E [si si ] 6= E [sj sj ]The proposed BSS is based on second-order statistics
of low and high halfbands of the uncorrelated sources.
In order to extend the theory of the proposed BSS,
first we express some properties of responses of a linear filters to uncorrelated signals. Consider a vector of signals s(t) = [s1 (t), s2 (t), · · · , sK (t)]T , andthe vector of responses of a linear filter to them ass̄(t) = [s̄1 (t), s̄2 (t), · · · , s̄K (t)]T . For responses of alinear filter to a vector of signals, the following properties can be obtained;

i 6= j(7)(8)• Assumption 3 : Autocorrelation functions of low
halfbands of the sources are different from autocorrelation functions of their high halfbands.
E [s̄i s̄i ] 6= E [esi sei ](9)

If W is the un-mixing matrix, based on assumption 2 recovered source signals are mutually uncorrelated. Also,
since low and high halfbands are obtained by linear filters, from property 1 , the covariance matrices of ȳ(t)• Property 1 If signals s(t) are mutually uncorreand ey(t) are diagonal matrices;lated, then s̄(t) the response signals of a linear fil(10)C̄ yy = E ȳȳT
ter to them are also mutually uncorrelated.
= diag (E [ȳ1 ȳ1 ] , E [ȳ2 ȳ2 ] , · · · , E [ȳK ȳK ]) ,ihE [si sj ] = 0, i 6= j =⇒ E [s̄i s̄j ] = 0 i 6= j (3)Te yy = E eyey(11)CSo, the covariance matrix of the response signals= diag (E [ey1 ye1 ] , E [ey2 ye2 ] , · · · , E [eyK yeK ]) ,is a diagonal matrix;C s̄s̄=e yy are covariance matrices of ȳ(t) andwhere C̄ yy and C
ey(t). Furthermore, since from Assumption 3 the low
and high halfbands of the sources have different autoe yy are non-zerocorrelation functions, both C̄ yy and C
distinct matrices. Taking into account y(t) = Wx(t)and considering the property 2, we havediag (E [s̄1 s̄1 ] , E [s̄2 s̄2 ] , · · · , E [s̄K s̄K ])• Property 2 Let ȳ(t) and x̄(t) are respectively
responses of a linear filter to y(t) and x(t). Ify(t)=Ux(t), U ∈ RK×M . Then, we haveȳ(t)andC ȳȳ= Ux̄(t)= UC x̄x̄ U(4)TC̄ yy = WC̄ xx WT
e yy = WCe xx WTC(5)where C x̄x̄ and C ȳȳ are respectively covariancematrices of x̄(t) and ȳ(t).

(13)

e xx are covariance matrices of x̄ andwhere C̄ xx and Ce yy are distinct matrices, it follows
ex. Since C̄ yy and C
e xx are distinct
from Eqs.(12) and (13) that C̄ xx and Cmatrices too. Moreover, Eqs.(12) and (13) convey thatsimultaneous diagonalization of two distinct matrices
e xx estimates the un-mixing matrix W up toC̄ xx and C
its re-scaled and permuted version.
For the sake of simplicity, the problem of simultaneous diagonalization is converted to generalized eigenvalue decomposition (GEVD) [1] as follows. Since, inLet’s remember the above properties and go backto the BSS problem. Vector of M mixture signalsx(t) = [x1 (t), x2 (t), · · · , xM (t)]T has been observedby M sensors. BSS looks for the best estimate of unmixing matrix WK×M in order to recover the vectorof K unknown source signals y(t) = [y1 (t), y2 (t), · · ·, yK (t)]T by Eq.(2).
y(t) = Wx(t) ≈ s(t)

(12)

(6)

2

e yy are distinct diagthe case of separation, C̄ yy and Ce yy is aonal matrices, their multiplication Ryy = C̄ yy Cdiagonal matrix;=e yy
C̄ yy C

−20

Magnitude (dB)=−10(14)On the other hand, from Eqs. (12) and (13)ihe xx WT
Ryy = WC̄ xx WT WC
e xx WT= WC̄ xx WT W C

−40
−50

−70
−80
00.05

0.1

0.15
0.2
0.25
0.3
0.35
Normalized Frequency (×π rad/sample)
Magnitude Response (dB)

0.4

0.45

0.5

00.05

0.1

0.15
0.2
0.25
0.3
0.35
Normalized Frequency (×π rad/sample)

0.4

0.45

0.5

0−10−20(15)
(16)

−30
−40
−50−60If we convert the problem to GEVD, it will allow us tomake the assumption of orthogonality as follows:−70−80• Assumption 4 : The un-mixing matrix W can becomposed of orthogonal separating vectors.

Figure 1: Frequency responses of the deployed lowpass
filter (top) and highpass filter (bottom).

By applying Assumption 4 (WT W = I) to Eq.(15),
we havee xx = W−1 Ryy W.
(17)C̄ xx C• Step 4 : The un-mixing matrix up to its re-scaled
and permuted version is given by W = V on thecondition that all eigenvalues are real and distinct.

Since Ryy is a diagonal matrix, Eq.(17) conveys that:
The un-mixing matrix W is the matrix which diagonale xx .
izes C̄ xx C

The quality of separation depends on how much theassumptions 2 and 3 are strong. The more different autocorrelation functions of the low halfbands from autocorrelation functions of the high halfbands results inbetter separation result.

Based on the result of Eq.(17) and by using diagonalization theorem [19], the un-mixing matrix W is obtained as follows.
The un-mixing matrix W is composed of eigenvectors ofe xx .
C̄ xx Ce xx is a symmetric square matrix, itsSince, C̄ xx Ceigenvectors are orthogonal, and Assumption 4 is approved.
The algorithm of blind source separation by simultaneous diagonalization of two covariance matrices ofresponses of two different linear filters to the mixture
signals can now be described by the following steps .

3Results and Discussion

To empirically evaluate potential usefulness of the proposed method, some evaluation tests. We have used
IIR elliptic lowpass and highpass filters to obtain low
and high halfbands of the mixtures, respectively. In
this simulation, in all of the cases the lowpass filter is aminimum order filter with a normalized passband frequency of 0.2, a stopband frequency of 0.25, a passband ripple of 1 dB, and a stopband attenuation of 60
dB, and the highpass filter is also a minimum orderfilter with a normalized passband frequency of 0.3, astopband frequency of 0.25, a passband ripple of 1 dB,
and a stopband attenuation of 60 dB. The above mentioned frequencies of the filters specifications are normalized with respect to sampling frequency of the signals. Since, from Niquist sampling theorem the bandwidth of the signals is maximally considered as half
the sampling frequency, to obtain halfbands of the mixtures, the optimum stop frequency of both lowpass andhighpass filters is f4S , where fS is the sampling frequency. Fig. 1 shows frequency responses of the deployed lowpass and high pass filters.
We have compared the performance of the proposed
method with two well known second-order BSS al-The Algorithm Outline:• Step 1 : Obtain x̄(t) and ex(t) the responses of two
different linear filters F̄ and Fe to observed mixture signals x(t).
e xx the covariance ma• Step 2 : Compute C̄ xx and C
trices of x̄(t) and ex(t) as well as their multiplicae xx .
tion Rxx = C̄ xx C
• Step 3 : Find the generalized eigenvector matrix
V for the generalized eigenvalue decomposition;
Rxx V = V D−30−60diag (E [ȳ1 ȳ1 ]E [ey1 ye1 ], E [ȳ2 ȳ2 ]E [ey2 ye2 ], · · ·)yK yeK ]) .
(· · · , E [ȳK ȳK ]E [eMagnitude (dB)RyyMagnitude Response (dB)
0(18)

3

ind(G)

gorithms, AMUSE (Algorithm for Multiple Unknown
Signals Extraction) [1] and SOBI (Second Order Blind
Identification) [2]. Both AMUSE and SOBI jointly
exploit the second order statistics (correlation matrices for different time delays) and temporal structure ofsources. In this simulation, the optimal delay is 1 timesample in AMUSE BSS, and number of matrices whichare jointly diagonalized at each run of SOBI algorithmis 4.
We have evaluated the method with a real experiment. Two images have been taken of the same view ofstreet from inside the window glass of a room. Both images belong to the same time of the day in different light
conditions of the room. Figure 2 (last page) shows thetaken photos and reflection removed photos as well as
extracted reflection by the above mentioned BSS methods. There exist many possible performance criterion
in the literature [20]. Most of evaluation methods are byusing sources or global matrix (multiplication of mixing matrix and demixing matrix G = AW ). But here,
since, mixing matrix and sources are not available, it
is not possible to use most of evaluation methods mentioned in literature. We have just real mixed observed
images and separated ones. In order to evaluate theseparation performance, we have used three numerical indexes for measuring similarity of recovered images obtained by each method. We have measured thecorrelation, mean absolute dsitance (MAD) and meansquared distance (MSD) between two recovered images
obtained by each method. Each of the less correlation,
the more MAD and the more MSD individually shows
the less similarity of recovered images as well as thebetter performance of separation. Table.1 shows thesenumerical evaluations result in reflection removal fromreal reflected window images of figure.2. As it is seen
all the three indexes approve higher performance of the
proposed method.

0.5
0.45
0.4
0.35
0.3
0.25
0.01

0.05

0.07

0.09

.11

0.13

Figure 3: The averaged ind(G) Vs. variance of noisein separation mixed images by AMUSE (plus - black),
SOBI (circle - pink), And SD of low and high halfbands(star - blue).
lected and mixed each other with full rank random matrix. The mixing matrix was the same for all methods.
At each noise level, the performance of the methods
have been compared by averaged result of 200 Mont
Carlo runs. Since the mixing matrix was available forevaluation, we used the ind(G) [20] the index based on
global matrix. The global matrix is obtained by multiplication of mixing and de-mixing matrices G = W A.
This performance index is as follows
PK PK |Gi,j |21
ind(G) = K(K−1)
ij max|Gi,l |2 − 1l
PK PK |Gi,j |2+ j(19)i max|Gl,j |2 − 1lwhere K is dimension of G. Indeed, this non-negativeindex is zero in the case of perfect separation, and its
the smaller value indicates the more proximity to thedesired solutions.
Fig.3 shows the averaged ind(G) Vs. variance of noisein separation of gray-scaled images. As it is seen, theproposed method is less affected by noise level than theothers and it has the highest performance in comparison
to the others.

4Table 1: Performance comparison of the proposed
method with AMUSE and SOBI in reflection removalof real window glass images of figure 2.
BSS Methods
SOBI
0.87074.5569e + 03
30.50890.03

Noise varianceIn addition to real experiment of Fig.2, the robustnessof the method to noise and different images and reflections has been artificially investigated too. To aim this,we have used a set of 15 different gray-scale images assource images and reflections. They were randomly se-AMUSE
0.87044.5892e + 03
30.5779

0.60.55

3.1 Robustness of reflection removal

Correlation :MAD :
MSD :

0.65

Proposed
0.85836.2269e + 03
36.79174conclusion

In this paper, the reflection over window glass images
has been removed by simultaneous diagonalization of
low and high halfbands of the mixtures. Decomposing
the mixtures to the set of low subband and high subband mixtures enables us to have two different views of
the mixtures. The simultaneous diagonalization in two
different views results in higher performance of separation. The proposed second order technique has been
compared with AMUSE [1] and SOBI [2] in a real experiment. Furthermore, it robustness to noise and dif-ferent sources has been compared to the others. Threedifferent numerical metrics approves dominance of the
proposed BSS to the others in removing the reflection
from window glass images.

[11] A. Belouchrani, K. A. Meraim, J. F. Cardoso andE. Moulines, “A blind source separation techniqueusing second-order statistics”. IEEE Trans. Signal Processing, vol. 45(2), pp. 434-444, February1997.

References[12] K. A. Meraim, E. Moulines, P. Loubaton, “Prediction error method for second-order blind identification.” IEEE Trans. Signal Processing, Vol. 45,
pp. 694-705, March 1997.

[1] L. Tong, V. C. Soon, Y. F. Huang, and R. Liu,
“AMUSE: a new blind identification algorithm.” inProc. IEEE ISCAS, vol.3, pp. 1784-1787 , New Orleans, LA, 1990.

[13] A. Cichocki and R. Thawonmas “On-line algorithm for blind signal extraction of arbiturarily distributed, but temporally correlated sources using
second order statistics.” Neural Processing Letters,Vol. 12, pp. 91–98, August 2000.

[2] A. Belouchrani, M. G. Amin, and K. AbedMeraim, “Direction finding in correlated noise
fields based on joint block-diagonalization of
spatio-temporal correlation matrices.” IEEE Signal
Processing Letters, 4(9), September 1997.

[14] S. Choi, A. Cichocki and A. Belouchrani “Secondorder nonstationary source separation.” Journal of
VLSI Signal Processing, Vol. 32, no. 1-2, pp. 93–104, Aug. 2002.

[3] P. Common, “Independent component analysis, A
new concept?”, Signal Process., Vol. 36, pp. 287–314, 1994.

[15] W. Liu, D. Mandic, A. Cichocki, “Blind SecondOrder Source Extraction of Instantaneous NoisyMixtures”, IEEE Transactions on Circuits and Systems II: Express Briefs 53(9), pp. 931–935, 2006.

[4] A. Cichocki, S. Amari, “Adaptive Blind Signal andImage Processing.” John Wiley & Sons, New York,USA 2000.

[16] S. Choi, A. Cichocki, A. Beluochrani, “Secondorder non-stationary source separation”, J. VLSI
Signal Process. vol. 32 (1/2) pp. 93-104, 2002.

[5] A. Hyvrinen, “ New approximations of differential
entropy for independent component analysis andprojection pursuit”, Adv. Neural Inform. Process.
Systems vol. 10, pp. 273-279, 1998.

[17] M. Khosravy, M.R. Asharif and K. Yamashita,“A Probabilistic Short-length Linear Predictability Approach to Blind Source Separation”, The
23rd International Technical Conference on Circuits/Systems, Computers and Communications(ITC-CSCC), pp. 3181–384, Yamaguchi, Japan,
2008.

[6] A. Hyvrinen, “Survey on independent componentanalysis”, Neural Comput. vol. 2, pp. 94-128, 1999.
[7] A. J. Bell, & T. J. Sejnowski, “An informationmaximization approach to blind separation andblind deconvolution”, Neural computation,vol. 7,pp. 1129–1159, 1995.

[18] M. Khosravy, M. R. Asharif and K. Yamashita, “APDF-matched Short-term Linear Predictability approach to Blind Source Separation”, International
Journal of Innovative Computing, Information andControl (IJICIC), Vol. 5, No. 11(A), pp. 36773690, November 2009.

[8] H. Mathis, T. Hoff, M. Joho, “Blind separationof mixed-kurtosis signals using an adaptive threshold nonlinearity”, in Proc. Independent ComponentAnalysis and Blind Signal Separation, Helsinki,Finland, pp. 19-22, 2000.

[19] G. L. Peterson , J. S. Sochacki, “Linear Algebraand Differential Equation”. Addison-Wesley Publishing Company, Boston, 2002.

[9] M. Khosravy, M. R. Asharif and K. Yamashita,“A PDF-matched Modification to Stones Measureof Predictability for Blind Source Separation” ,6th International Symposium on Neural Networks,Wuhan, China, Springer-Verlags Lecture Notes
in Computer Science (ISNN 2009, Part I, LNCS
5551), pp. 219-228, 2009.

[20] E. Moreau, “A generalization of jointdiagonalization criteria for source separation”,
IEEE Transactions on Signal Processing, Vol. 49,
pp. 530–541, Mar 2001.

[10] R. Gharieb and A. Cichocki, “Second-order statistics based blind source separation using a bank ofsubband filters”, Digital Signal Process. Vol. 13,
pp. 252-274, 2003.

5

Images
withreflections

AMUSE

SOBI

Proposed
Figure 2: Real window images with reflection (1st row), separated window images and reflections by AMUSE (2ndrow), by SOBI ( 3rd row) and by Simultaneous Diagonalization of low and high halfbnds ( bottom row).

6