# Study of adapted instrumentation architectures and measurements post-processing approaches for Infrared thermography applied to thermal monitoring in outdoor conditions

Jean Dumoulin, Thibaud Toullier, Laurent Mevel

► To cite this version:

Jean Dumoulin, Thibaud Toullier, Laurent Mevel. Study of adapted instrumentation architectures and measurements post-processing approaches for Infrared thermography applied to thermal monitoring in outdoor conditions. 2022 - Infrared Remote Sensing and Instrumentation XXX, Aug 2022, San Diego, United States. pp.7, 10.1117/12.2635258 . hal-03895839

HAL Id: hal-03895839

https://inria.hal.science/hal-03895839

Submitted on 13 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Study of adapted instrumentation architectures and measurements post-processing approaches for Infrared thermography applied to thermal monitoring in outdoor conditions.

Jean Dumoulin<sup>a</sup>, Thibaud Toullier<sup>a</sup>, and Laurent Mevel<sup>b</sup>

<sup>a</sup>Univ. Gustave Eiffel, Inria, COSYS-SII, I4S Team, F-44344 Bouguenais, France

<sup>b</sup>Univ. Gustave Eiffel, Inria, COSYS-SII, I4S Team, F-35042 Rennes, France

## ABSTRACT

Technological progress in uncooled infrared focal plane array sensors has contributed significantly to enlarge the scope of applications of such sensing technique in many domains: Leisure, Manufacturing, Process Survey, Building insulation diagnostic, Civil Engineering, Road works, etc... Different outdoor situations and objects of interest monitored by an in-house designed measurement architecture are presented. Designed instrumentation architectures and measurements correction from varying environmental conditions and geometrical considerations are discussed. A first step toward joint estimation of emissivity and temperature is introduced for outdoor applications. Then moving object detection by an AI approach applied on thermal image sequences is also presented and discussed. Finally conclusion and perspectives are proposed.

**Keywords:** Real site thermal monitoring, instrumentation architecture, outdoor infrared thermography, environmental conditions post-processing, spatial calibration, joint estimation of emissivity and temperature, moving object detection

## 1. INTRODUCTION

Technological progresses in uncooled infrared focal plane array sensors have bring up new opportunities towards remote and low-cost radiatives fluxes measurements and new applications in civil engineering infrastructures or critical systems monitoring. However, one concern about most infrastructures (or other real targets) is their subject to space and time environmental variable parameters in outdoor conditions. The underlying temperature estimation will therefore depend on their intrinsic thermo-optical properties, the varying environmental conditions, geometrical considerations and the chosen instrumentation' performances. As a consequence, the purpose is to identify a number of key parameters thanks to a multi-sensors instrumentation and correct the temperature estimation by merging data from those different sources. The proposed methodology presented in this paper relies on three main steps: the calibration of the used instruments, the choice of a multi-sensor instrumentation design and the post-processing of the measurements using the gathered data. Those steps can be fulfilled thanks to a in-house developed DIARITsup framework that supervises live measurements, performs Digital Twins models computations and predictions for structures (e.g. other targets) monitoring.

In this paper, a multi-sensor instrumentation design to correct measurements from infrared cameras is presented. Such correction is introduced on the conversion from radiative fluxes to temperature equations as well as the use of online data to estimate missing environmental parameters. Then, geometrical considerations are discussed and a step toward joint estimation of emissivity and temperature is introduced. Furthermore, moving objects detection by an AI based approach is also introduced. Finally, conclusion and perspectives are proposed.

<footnote>Further author information: (Send correspondence to J.D.)</footnote>

<footnote>J.D.: E-mail: jean.dumoulin@univ-eiffel.fr</footnote>
## 2. MULTI-SENSOR MODEL TEMPERATURE COMPUTATION

### 2.1 Radiometric equation model for temperature conversion

An infrared camera measures the radiative flux received on its detectors. However, this radiative flux depends on the geometry of the scene, the observed objects own properties and the meteorological conditions. To overcome the lack of knowledge about environmental and meteorological parameters, an instrumentation made of various sensors is proposed. In fact, equation (1) conjointly with Figure 1 shows the involved parameters in the simplified radiometric equation to solve to retrieve the radiance of the observed object.

<figure><img src="image_2.png" /><figcaption>Figure 1. Radiatives contributions received by the infrared camera.</figcaption></figure>

Simplified radiometric equation at pixel (i, j) for a given spectral band Δλ and direction ⃗d:

$$ 
\begin{aligned}
L_{\Delta\lambda,\text{total}}^{(i,j)}(T) = & \tau_{\text{opt}} \tau_{\text{atm}} \epsilon_{\Delta\lambda,\text{obj}}^{(i,j)} L_{\Delta\lambda,\text{obj}}^{\circ(i,j)}(T_{\text{obj}}^{(i,j)}) \\
& + \tau_{\text{opt}} \tau_{\text{atm}} (1 - \epsilon_{\Delta\lambda,\text{obj}}^{(i,j)}) (L_{\Delta\lambda,\text{env}}^{\circ} + L_{\Delta\lambda,\text{sun}}^{\circ}) \\
& + \tau_{\text{opt}} (1 - \tau_{\text{atm}}) L_{\Delta\lambda,\text{atm}}^{\circ} \\
& + (1 - \tau_{\text{opt}}) L_{\Delta\lambda,\text{opt}}^{\circ}
\end{aligned}
\quad (1) 
$$

Therefore, environmental quantities can have an important impact on the estimation of temperature. Figure 2 shows the sensitivity of the temperature conversion towards the environmental reflections and the emissivity. In this figure, the error on the estimation of the temperature is shown for different errors on the environment temperature for two different ground truths (up / down lines) and two different emissivities (left / right columns). One can see that both the quantities play an important role on the estimation of the temperature and therefore needs particular care when getting used in the previous equation.

For determining some of the environmental quantities, a weather station can be used. The reflected, optics and atmospherics luminance values can be computed by injecting some knowledge about the monitored objects, components of the infrared system and atmospheric conditions. Such step can be embedded at the infrared system level knowing that in such case it depends on the hardware and software of the manufacturer of the infrared camera used. For instance, code samples for FLIR camera conversion parameters can be found.¹ Prior to the actual instrumentation, sensors are calibrated and more particularly the camera to be able to convert the temperature from digital levels, as presented in Section 3. For handling the emissivity, a simultaneous estimation of temperature and emissivity algorithm is used and introduced in section 4.

Such complex instrumentation with a large variety of sensors needs adapted softwares to record, sometimes perform live processing and visualize acquired data. This is achieved with the designed and developed DiaritSUP framework.
<figure><img src="image_3.png" /><figcaption>Figure 2. Difference between the true object temperature and estimated one depending on the error made on the environment temperature and emissivity. Here, the true object is at 293.15K, emissivity at 0.93 (left) and 0.6 (right).
How to read: on the left the true object's temperature and emissivity. A point on the image represents the difference between the true object temperature and the estimated one with $\bar{T}_{env}$ (x-axis) and $\bar{\epsilon}_{obj}$ (y-axis) values. The images represent different environment temperatures values.</figcaption></figure>

## 2.2 DiaritSUP framework

To handle the amount of data gathered in such instrumentation, a framework called DiaritSUP has been designed and developed. It follows the concept of “system of systems”. It aims at assimilating, processing and recording data, managing computation models, returning real-time information and visualizing data. Thanks to its model-view-controller (MVC) architecture, the framework is easily extensible, can be customized and connected to existing applications. It integrates three types of modules, represented in Figure 4. “Instrumentation” with the forecast and device managers that gather data from local sensors and external data providers such as Météo-France Geoservices, Copernicus, METAR, etc. “Computation” with the model manager that launches Structural Health Monitoring (SHM) Digital Twins models at the right time. Finally, a HDF5 Handler records data and metadata in Hierarchical Data Format (HDF5) in Single-Writer-Multiple-Readers feature to save all data (computed and gathered) and enable the live visualization through the User Interface (UI) made on a webserver (see Figure 3).

<figure><img src="image_4.png" /><figcaption>Figure 3. Visualization in real-time of the collected data. For instance here local weather station paramters plot and infrared camera sensors & image.</figcaption></figure>

A typical workflow is presented in Figure 5 where the components of the framework are more detailed and the data flow is illustrated. The main communications and access to the data are done with a REST API (HTTPS). The communication between the machines can be done through different protocols such as SSH, SCP, Modbus TCP/IP, etc. The framework deployment is made on Ansible, an information technology automation tool that makes the system configuration and deployment easy thanks to simple tasks defined in a configuration file. Code is hosted on our own Gitlab instance at Inria which enables the use of Continuous Integration and Deployment tools (CI/CD) and the setup of an automatic update mechanism. Finally, for robustness purposes, examples of Vagrant configuration are provided to create virtual machines (Digital Twins) to test deployment before pushing to production.
<figure><img src="image_5.png" /><figcaption>Figure 4. Overall Model-View-Controller architecture.</figcaption></figure>

<caption>Figure 5. Main components and data flow</caption>

## 3. CALIBRATION

All sensors used in the instrumentation needs to be and are generally calibrated by the vendor with an up-to-date calibration certificate.

For the camera though, we perform the calibration in our laboratory, by using a black-body calibration source and the standard procedure usually found in the literature. This source is put in a climatic chamber which enables the control of the ambient temperature. Measurements are then performed for different black-body source temperatures that evolves in our range of interest. Then, the Planck version of the Sakuma-Hattori equation² is used for which uncertainties have been well established.³

The real field of view of the camera may also alter the estimation of the temperature. When dealing with particular region of interests, a spatial calibration has to be done. Many different methods can be found in the literature for the infrared thermography field that are mainly inspired from computer vision field.

## 4. SIMULTANEOUS ESTIMATION OF EMISSIVITY AND TEMPERATURE: NUMERICAL STUDY

Once the received flux can be estimated thanks to the gathered data and equation 1, the emissivity parameter remains unknown. The problem of estimating at the same time temperature and emissivity is a challenging task often addressed in the literature.⁴⁻⁷ An algorithm that is particularly suited for the live and in-situ estimation of temperature and emissivity is described in.⁸ The equations are derived to a linear time variant (LTV) system. Such derivation enables the use of Bayesian methods from the automation field such as Marginalized Particle Filter which aims at estimating the state vector (temperature here) and the parameters (emissivity and dynamic evolution of the temperature here) at the same time.

To simplify the writing and the problem, let assume that only the temperature and the emissivity remain to be estimated. Therefore, we can write the previous equation 1 to:

$$ 
\begin{gathered}
L_{\Delta\lambda,\text{total}}^{(i,j)}(T) - C_{\text{opt,atm}} = \tau_{\text{opt}}\tau_{\text{atm}}\epsilon_{\Delta\lambda,\text{obj}}^{(i,j)}L_{\Delta\lambda,\text{obj}}^{\circ(i,j)}(T_{\text{obj}}^{(i,j)}) + \tau_{\text{opt}}\tau_{\text{atm}}(1 - \epsilon_{\Delta\lambda,\text{obj}}^{(i,j)}) (L_{\Delta\lambda,\text{env}}^{\circ} + L_{\Delta\lambda,\text{sun}}^{\circ}) \\
\frac{L_{\Delta\lambda,\text{total}}^{(i,j)}(T)}{\tau_{\text{opt}}\tau_{\text{atm}}} - C_{\text{opt,atm}} - (L_{\Delta\lambda,\text{env}}^{\circ} + L_{\Delta\lambda,\text{sun}}^{\circ}) = \epsilon_{\Delta\lambda,\text{obj}}^{(i,j)}(L_{\Delta\lambda,\text{obj}}^{\circ(i,j)}(T_{\text{obj}}^{(i,j)}) - L_{\Delta\lambda,\text{env}}^{\circ} - L_{\Delta\lambda,\text{sun}}^{\circ})
\end{gathered}
\quad (2) $$

Therefore, the simultaneous estimation of emissivity and temperature algorithm can be applied with the estimated environmental parameters, where $C_{\text{opt,atm}}$ is the optical and atmospherical contribution sum.

An example of the estimation of temperature time evolution with such workflow using numerical simulations carried out with a modified implementation of the progressive radiosity⁹,¹⁰ algorithm is proposed in Figure 6.
<figure><img src="image_6.png" /><figcaption>Figure 6. Estimation of the temperature evolution with time with the proposed approach.</figcaption></figure>

## 5. MOVING OBJECT DETECTION: EXAMPLE ON TRAIN AXLES

As an example of AI based image processing, hot boxes detection results obtained by an improved YOLO-v4 model¹¹ are presented and discussed in this paragraph. In this example, 6365 Infrared thermal images (extracted from on site thermal images sequence) were the data used to test the Y-UC (Yolo -Uncooled Camera). The same data were used to test the baseline models. In this experiments, the target confidence threshold (conf-thresh) was taken equal to 0.45 and IoU threshold to 0.36. The results are shown in Tab. 1.

<table><caption>Table 1. Impact of improvements on the network performance on UC-based IRTIs; conf-thresh = 0.45; IOU = 0.36</caption><thead><tr><th rowspan="3">Results (%)<br>Metrics</th><th colspan="9">Dataset</th></tr><tr><th colspan="3">Freight train</th><th colspan="3">Crossing between passenger and freight trains</th><th colspan="3">New unseen trains</th></tr><tr><th>Baseline</th><th>Yolo-v4 + CBAM</th><th>Y-UC</th><th>Baseline</th><th>Yolo-v4 + CBAM</th><th>Y-UC</th><th>Baseline</th><th>Yolo-v4 + CBAM</th><th>Y-UC</th></tr></thead><tbody><tr><td>Precision</td><td>99.09</td><td>99.1</td><td>99.1</td><td>98.99</td><td>98.93</td><td>99.17</td><td>98.94</td><td>99.10</td><td>99.21</td></tr><tr><td>Recall</td><td>96.93</td><td>97.95</td><td>98.16</td><td>98.66</td><td>99.08</td><td>99.83</td><td>99.47</td><td>99.73</td><td>99.89</td></tr><tr><td>F1-score</td><td>98</td><td>98.52</td><td>98.63</td><td>98.83</td><td>98.96</td><td>99.50</td><td>99.2</td><td>99.42</td><td>99.55</td></tr></tbody></table>

Visual representations of the Y-UC results are presented in Fig. 7. Modification studied and implemented in YOLO-v4 contributed to improve the performance of the global detection. However, both integrating CBAM and optimizing PANet made more substantial improvements to the network, with precisely better recall and F1-score. On the other hand, the precision of the baseline network was found, in most cases, slightly superior than the modified networks. That means low confusing of the hot boxes with the rest of the train body. Nevertheless, with weak recall, the baseline network missed to detect some ground truth hot boxes. In contrast, the improved YOLO-v4 showed a good balance between precision and recall which justify the high F1-score.

<figure><img src="image_7.png" /><figcaption>(a) Freight train (left) - Passenger train (right)</figcaption></figure>

Figure 7. Some example results of hot axle boxes detection of freight and passenger trains from images taken from uncooled camera using Y-UC.

Based on the detected hot boxes, an axle counter was also studied. The proposed approach rely simply on the following steps:

1. Define a detection zone.

2. Run the improved YOLO-V4 detection algorithm.

3. Crop the image with respect to the bounding box.

4. Update with a new frame and repeat steps 2 and 3. Then, calculate the structural similarity index (SSIM)¹² between the cropped images. In the present study, an index superior to 0.8 was estimated sufficient to validate the count.

Fig. 8 shows some visual representations of the counting operation.
<figure><img src="image_8.png" /><figcaption>(a) Infrared image of moving freight train (left) and passenger train (right)
Figure 8. Some example results of counting the hot axle boxes</figcaption></figure>

# 6. CONCLUSION & PERSPECTIVES

A new surveillance instrumentation architecture has been designed for the thermal monitoring of transport infrastructures or any other object of interest, in outdoor conditions. It combines infrared camera sensors with in-situ or cloud-based environmental measurement data to perform measurement corrections on infrared thermal images. It also allows the integration of prediction models or its coupling with other structural health monitoring models. Its deployment includes different stages at the laboratory or in situ level. Moreover, a numerical study of a promising method for joint estimation of emissivity and temperature, in outdoor conditions, was briefly presented. Finally, for the detection of moving objects, an improved YOLO-V4 was presented. The obtained results show that the upgraded YOLO-V4 outperforms the original one. Future research work will focus on studying the robustness and performance of the new architecture currently deployed on a real transport infrastructure. In addition, the new method for joint estimation of emissivity and temperature will be evaluated in the laboratory and then in an outdoor configuration under controlled conditions. Finally, for the detection of moving objects, we plan to improve the information on the blurring characteristics in the infrared images and to compare the temperatures of the detected axles with a reference system on a new dedicated real railway test site.

# REFERENCES

[1] “Terra-ref computing pipeline.” url = https://github.com/terraref/computing-pipeline/blob/5d0d089501154c6c0de68229579c131d79e39b5e/scripts/FLIR/FlirRawToTemperature.m.

[2] Sakuma, F. and Hattori, S., “Establishing a practical temperature standard by using a narrow-band radiation thermometer with a silicon detector,” 2, p91–97.

[3] Fischer, J., Saunders, P., Sadli, M., Battuello, M., Park, C., Yuan, Z., Yoon, H., Li, W., Van der Ham, E., and Sakuma, F., “Uncertainty Budgets for Calibration of Radiation Thermometers below the Silver Point,”

[4] Zhou, S. and Cheng, J., “A Multi-Scale Wavelet-Based Temperature and Emissivity Separation Algorithm for Hyperspectral Thermal Infrared Data,” 0(0), 1–21.

[5] Snyder, W. C., Wan, Z., Zhang, Y., and Feng, Y.-Z., “Classification-Based Emissivity for Land Surface Temperature Measurement from Space,” 19(14), 2753–2774.

[6] Venafra, S., Masiello, G., Serio, C., Liuzzi, G., Blasi, M. G., and Telesca, V., “Retrieval of Surface Temperature and Emissivity from Seviri Data: Implementation of Kalman Filter Methodology to Dull Disk and Validation with in-Situ Observations,” in [EUMETSAT Meteorological Satellite Conference], 1.

[7] Krapez, J.-C., “Radiative Measurements of Temperature,” in [Thermal Measurements and Inverse Techniques], Heat Transfer, CRC Press.

[8] Toullier, T., Dumoulin, J., and Mevel, L., “A Kriging-based Interacting Particle Kalman Filter for the simultaneous estimation of temperature and emissivity in Infra-Red imaging,” in [IFAC 2020 – 21st IFAC World Congress],

[9] Sillion, F. X. and Puech, C., [Radiosity and Global Illumination], vol. 11, Morgan Kaufmann Publishers (1994).

[10] Toullier, T., Dumoulin, J., and Mevel, L., “Étude et développement d’un simulateur d’échanges radiatifs dans des scènes 3D statiques et dynamiques surveillées par thermographie infrarouge,” in [SFT 2019], (2019).

[11] Merainani, B. and al., “Toward the development of intelligent wayside hot bearings detector system : combining the thermal vision with the strength of yolo-v4,” in [Proceedings of the QIRT2022 conference on Quantitative Infrared Thermography], 33–42 (2022).

[12] Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P., “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13(4), 600–612 (2004).