# Automatic Detection System of Deteriorated PV Modules Using Drone with Thermal Camera
## 3 System Configuration
### Thermal Image Processing
#### 3.4.2 RGB and Thermal Image Registration  
The main goal of this step is to register images \(I_{RGB}\) and \(I_{th}\) since the images have a disparity/ misalignment among them. This disparity is attributed to the slight difference in the real world coordinates of both the cameras. Failure to align the images will lead to wrong localization of the faulty PV modules. The first step towards image registration is feature detection. Maximally Stable Extremal Region (MSER) [20], one of the most popular feature detectors, was used in this step of our algorithm. MSER algorithm extracts extremal regions from an image. An extremal region \(R_{i}\) is essentially a connected component that is stable across multiple gray-level sets in an image. Simply, MSERs are regions which remain unchanged over a wide range of thresholds. Given an input image, MSER thresholds an image into multiple thresholded images \(I_{t}\) where \(t\) indicates the threshold value. The first image in the sequence would be totally black and the following images would contain white  
Figure 8: Block diagram for thermal image processing. blobs which continue to grow until all of them merge into a totally white image. The stability of these blobs are monitored by MSER algorithm across the image sequence with respect to a function. Generally, stability of a region is defined as follows:  
\[\psi\left(R_{t}\right)=\frac{A\left(R_{t}\right)}{\frac{d}{dt}A\left(R_{t}\right)} \tag{1}\]  
where \(A(R_{t})\) represents the area of the region \(R_{t}\) and \(\frac{d}{dt}A\left(R_{t}\right)\) represents the rate of change of area. A region is stable only if its area changes slightly at different threshold values. Additionally, a stable region might be rejected if its area is below a minimum area threshold (2000 in our experiments) or above a maximum area threshold (400,000 in our experiments) value. MSER features \(F_{RGB}\) and \(F_{th}\) are extracted from the RGB image \(I_{RGB}\) and the thermal image \(I_{th}\), respectively. The extracted features must be matched in order to apply homography for aligning the images. For robust matching of features, the following criterion is applied:  
\[x_{RGB}^{i}\in F_{RGB},x_{th}^{j}\in F_{th},\quad i:1\sim F_{\text{size}}, \quad j:1\sim F_{\text{size}} \tag{2}\]  
\[r=\sqrt{\left(x_{RGB}^{i}-x_{th}^{j}\right)^{2}+\left(y_{RGB}^{i}-y_{th}^{j} \right)^{2}} \tag{3}\]  
\[f(x)=\left\{M_{RGB}^{k}= F_{RGB}^{i}\text{ and }M_{th}^{k}=F_{RGB}^{j},\quad\text{if }r<th_{r}\right. \tag{4}\] \[\left.\text{invalid match},\quad\quad\quad\quad\quad\text{ otherwise}\right.\]  
where \(\left(x_{RGB}^{i},y_{RGB}^{i}\right)\) and \(\left(x_{th}^{j},y_{th}^{j}\right)\) represent the \(x\) and \(y\) value of the \(i\)th feature point of the \(F_{RGB}\) and the \(j\)th element of the \(F_{th}\), respectively. \(M_{RGB}\) and \(M_{th}\) are the set of valid feature points in \(I_{RGB}\) and \(I_{th}\), respectively. A feature point set is only valid if the distance value r between the two points is lower than the threshold distance \(th_{r}\). After matching the features, homography is applied to align the two images. Homography is the mapping between two projection planes sharing the same center of projection. It is represented as a \(3\times 3\) matrix that maps points in source image to the corresponding points in the target image. The homography matrix can be estimated by approaches such as direct linear transform (DLT) and is represented as follows:  
\[H=\left[ \begin{array}{ccc}h_{00}&h_{01}&h_{02}\\ h_{10}&h_{11}&h_{12}\\ h_{20}&h_{21}&h_{22}\end{array} \right] \tag{5}\]  
Let \(M_{RGB}=\left\{\left(x_{RGB}^{1},y_{RGB}^{1}\right),\left(x_{RGB}^{2},y_{RGB }^{2}\right),\ldots,\left(x_{RGB}^{n},y_{RGB}^{n}\right)\right\}\) be the set of matched feature points in \(M_{RGB}\) and \(M_{th}=\left\{\left(x_{th}^{1},y_{th}^{1}\right),\left(x_{th}^{2},y_{th}^{2} \right),\ldots,\left(x_{th}^{n},y_{th}^{n}\right)\right.\) be the set of corresponding feature points in \(I_{th}\).