# Infrared and Visible Image Fusion via Feature-Oriented Dual-Module Complementary
## 3 The Proposed Method
### Infrared Brightness Supplement Module
#### 3.2.1 Gradient Intensity Operator  
Given a continuous grayscale image \(I\), calculate the intensity values in the horizontal and vertical directions. Assume \(x\) and \(y\) are spatial coordinates such that the intensity components can be denoted by \(H(x,y)\) and \(V(x,y)\). Then, the grayscale image may be written as \(\mathbf{f}=[H(x,y),V(x,y)]\). The following notations are adopted: \((x,y)=(x_{1},x_{2})=\mathbf{x}\), \(\mathbf{f}=(H,V)=(f_{1},f_{2})\), \(\mathbf{y}=\mathbf{f}(x)=[f_{1}(x),f_{2}(x)]\), and \(\mathbf{x}\in\Re^{2}\), where \(\Re\) is the set of real numbers. For \(i\) and \(j=1,2\), we assume that the rank of the Jacobian matrix \(\mathbf{J}=\left(\partial f_{j}/\partial x_{i}\right)\) is two everywhere in \(\Re^{2}\). Let \(\mathbf{f}_{i}(\mathbf{x})=\left(\partial f_{1}/\partial x_{i},\partial f_{2}/\partial x _{i}\right)\). According to this definition, \(\mathbf{f}_{i}(\mathbf{x})\) is a two tuple of a real number. Moreover, we postulate that \(\mathbf{f}_{i}(\mathbf{x})\) and their first derivatives are continuous. For \(i,k=1,2\), we set  
\[\mathbf{g}_{ik}(\mathbf{x})=\mathbf{f}_{i}(\mathbf{x})\cdot\mathbf{f}_{k}(\mathbf{x}), \tag{18}\]  
where \"\(\cdot\)\" is the dot product. According to the above notations, \(\mathbf{f}_{i}(\mathbf{x})\) can be written as follows:  
\[\mathbf{p}=\frac{\partial H}{\partial x}\mathbf{h}+\frac{\partial V}{\partial\mathbf{x}} \mathbf{v}, \tag{19}\]  
Figure 2: Infrared input image, predicted image, fused image, and their corresponding heatmaps. \[\mathbf{q}=\frac{\partial H}{\partial y}\mathbf{h}+\frac{\partial V}{\partial y}\mathbf{v}, \tag{20}\]  
where \(\mathbf{h}\) and \(\mathbf{v}\) are unitary vectors associated with \(H\) and \(V\), respectively. During the calculation of the partial derivatives, two Sobel operators are used as follows:  
\[S_{x}= \begin{bmatrix}1&2&1\\ 0&0&0\\ -1&-2&-1\end{bmatrix} ,\ S_{y}= \begin{bmatrix}1&0&-1\\ 2&0&-2\\ 1&0&-1\end{bmatrix}  \tag{21}\]  
Similarly, \(\left[\mathbf{g}_{ik}(\mathbf{x})\right]_{i,k=1,2}\) can be represented by  
\[\mathbf{g}_{xx}=\mathbf{p}\cdot\mathbf{p}=\left|\frac{\partial H}{\partial x}\right|^{2}+ \left|\frac{\partial V}{\partial x}\right|^{2}, \tag{22}\]  
\[\mathbf{g}_{yy}=\mathbf{q}\cdot\mathbf{q}=\left|\frac{\partial H}{\partial y}\right|^{2}+ \left|\frac{\partial V}{\partial y}\right|^{2}, \tag{23}\]  
\[\mathbf{g}_{xy}=\mathbf{g}_{yx}=\mathbf{p}\cdot\mathbf{q}=\frac{\partial H}{\partial x}\frac {\partial H}{\partial y}+\frac{\partial V}{\partial x}\frac{\partial V}{ \partial y}. \tag{24}\]  
In image processing, we are often interested in the following two quantities [27] that are computed locally at each spatial coordinate \((x,y)\): (i) direction through \((x,y)\), along which \(f\) has the maximum rate of change; (ii) absolute value of this maximum rate of change. Therefore, we aim to find the maximization of the following form:  
\[\left(df\right)^{2}=\mathbf{g}_{xx}dxdx+\mathbf{g}_{yy}dydy+\mathbf{g}_{xy}dxdy+\mathbf{g}_{yx }dydx, \tag{25}\]  
under the condition  
\[dxdx+dydy=1.\]  
The above problem can also be formulated as finding a \(\theta\) value that maximizes the following expression:  
\[\underset{\theta}{arg\underset{\theta}{max}}\mathbf{g}_{xx}cos^{2}\theta+2\mathbf{g} _{xy}cos\theta sin\theta+\mathbf{g}_{yy}sin^{2}\theta. \tag{26}\]  
Let  
\[F(\theta)=\mathbf{g}_{xx}cos^{2}\theta+2\mathbf{g}_{xy}cos\theta sin\theta+\mathbf{g}_{yy }sin^{2}\theta, \tag{27}\]  
using the common trigonometric function formulas,  
\[sin^{2}\theta=\frac{1}{2}(1-cos2\theta), \tag{28}\]  
\[cos^{2}\theta=\frac{1}{2}(1+cos2\theta), \tag{29}\]  
\[sin\theta cos\theta=\frac{1}{2}sin2\theta. \tag{30}\]  
\(F(\theta)\) can be written as  
\[ \begin{array}{c}F(\theta)=\frac{1}{2}\Big{[}\mathbf{g}_{xx}(1+cos2\theta)+2\mathbf{ g}_{xy}sin2\theta+\mathbf{g}_{yy}(1-cos2\theta)\Big{]}\\ \phantom{\frac{1}{2}}=\frac{1}{2}\Big{[}\Big{(}\mathbf{g}_{xx}+\mathbf{g}_{yy}\Big{)}+ \Big{(}\mathbf{g}_{xx}-\mathbf{g}_{yy}\Big{)}cos2\theta+2\mathbf{g}_{xy}sin2\theta\Big{]} \end{array}