## 3 Structural Similarity to Image Quality Assessment  
In this section, we introduce the image fidelity metric that quantifies the distortion of a test image by comparing it with a reference image assumed to have perfect quality. How to assess the quality of a distorted image is the key issue in image processing applications. A number of metrics from different aspects have been proposed, e.g., visible errors, human perception and the performance of image compression algorithm. Since the human visual system is highly sensitive to the structural information, a new philosophy to compare the structures of the reference and the distorted images is proposed in [4, 8, 9].  
Let \(x=\{x_{i}|i=1,2,\cdots,N\}\) and \(y=\{y_{i}|i=1,2,\cdots,N\}\) be the reference and distorted image signals, where \(i\) is the sample index and \(N\) is the number of signal samples. The structural similarity index between signals \(x\) and \(y\) can be defined by:  
\[SSIM(x,y)=l(x,y)c(x,y)s(x,y) \tag{1}\]  
where the three functions \(l(x,y)\), \(c(x,y)\) and \(s(x,y)\) represent the luminance, contrast and structure comparison measures between \(x\) and \(y\), respectively.  
Let \(\mu_{x}\), \(\mu_{y}\), \(\sigma_{x}^{2}\), \(\sigma_{y}^{2}\) and \(\sigma_{xy}\) be the mean of \(x\), the mean of \(y\), the variance of \(x\), the variance of \(y\), and the covariance of \(x\) and \(y\), respectively. Here the mean and the variance of a signal are treated as estimates of the luminance and the contrast of the signal. The covariance can be considered as a measurement of how much one signal is changed nonlinearly to the other signal being compared. The luminance, contrast and structure comparison measures are then given as follows:  
\[l(x,y)=\frac{2\mu_{x}\mu_{y}}{\mu_{x}^{2}+\mu_{y}^{2}},c(x,y)=\frac{2\sigma_{x }\sigma_{y}}{\sigma_{x}^{2}+\sigma_{y}^{2}},s(x,y)=\frac{\sigma_{xy}}{\sigma_{ x}\sigma_{y}} \tag{2}\]  
where  
\[\mu_{x}=\overline{x}=\frac{1}{N}\sum_{i=1}^{N}x_{i},\mu_{y}=\overline{y}= \frac{1}{N}\sum_{i=1}^{N}y_{i} \tag{3}\]  
\[\sigma_{x}^{2}=\frac{1}{N-1}\sum_{i=1}^{N}(x_{i}-\overline{x})^{2},\sigma_{y} ^{2}=\frac{1}{N-1}\sum_{i=1}^{N}(y_{i}-\overline{y})^{2} \tag{4}\]  
\[\sigma_{xy}=\frac{1}{N-1}\sum_{i=1}^{N}(x_{i}-\overline{x})(y_{i}-\overline{y}) \tag{5}\]  
The Equations (1) and (2) are combined and the SSIM index can be rewritten by  
\[SSIM(x,y)=\frac{4\mu_{x}\mu_{y}\sigma_{xy}}{(\mu_{x}^{2}+\mu_{y}^{2})(\sigma_{ x}^{2}+\sigma_{y}^{2})} \tag{6}\]  
Based on the Equation (6), the SSIM index has the following three properties:  
1. Symmetry: \(SSIM(x,y)=SSIM(y,x)\).
2. Boundedness: \(SSIM(x,y)\in[-1,1]\) since he three components of Equation (1) rang from \([0,1]\), \([0,1]\) and \([-1,1]\), respectively.
3. Unique maximum: \(SSIM(x,y)=1\) if and only if \(x=y\) (in discrete representations, \(x_{i}=y_{i}\) for all \(i=1,\cdots,N\).  
In practice, we usually want to evaluate the fidelity of a distorted video composed of some continuous images over time. Using the SSIM index, an approach to measure the statistical properties locally is more appropriate for video quality assessment. Suppose the total number of images in a video is \(M\), a video data matrix is denoted by \(X\)  
\[X=\begin{bmatrix}X_{1}\\ X_{2}\\ \vdots\\ X_{M}\end{bmatrix}=\begin{bmatrix}x_{11}&x_{12}&\cdots&x_{1N}\\ x_{21}&x_{22}&\cdots&x_{2N}\\ \vdots&\vdots&\ddots&\vdots\\ x_{M1}&x_{M2}&\cdots&x_{MN}\end{bmatrix} \tag{7}\]  
Each row of \(X\) corresponds to an image signal with \(N\) elements. With the use of SSIM index, the video fidelity index between the distorted video \(X\) and the reference video \(Y\) can be obtained as follow:  
\[SSIM(X,Y)=\frac{1}{M}\sum_{j=1}^{M}SSIM(X_{j},Y_{j}) \tag{8}\]  
where \(X_{j}=[x_{j1},x_{j2},\cdots,x_{jN}]\) and \(Y_{j}=[y_{j1},y_{j2},\cdots,y_{jN}]\).