# On-the-Fly Interrogation of Mobile Passive Sensors from the Fusion of Optical and Radar Data

Ali Hadj Djilani, Dominique Henry, Patrick Pons, Hervé Aubert

► To cite this version:

Ali Hadj Djilani, Dominique Henry, Patrick Pons, Hervé Aubert. On-the-Fly Interrogation of Mobile Passive Sensors from the Fusion of Optical and Radar Data. 2024 54th European Microwave Conference (EuMC), European Microwave Conference (EuMC), Sep 2024, Paris, France. pp.1118-1121, 10.23919/EuMC61614.2024.10732455 . hal-04836004

HAL Id: hal-04836004

https://laas.hal.science/hal-04836004v1

Submitted on 13 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On-the-Fly Interrogation of Mobile Passive Sensors from the Fusion of Optical and Radar Data

Ali Hadj Djilani\#§, Dominique Henry\#, Patrick Pons\#§, Herve Aubert\#§

\#LAAS-CNRS, France

§Toulouse University, France

{ahadjdjila, dhenry, ppons, aubert}@laas.fr

**Abstract** — In this paper a new method based on the fusion of optical and radar data is proposed to detect and remotely interrogate mobile and passive sensors. The sensors are detected in real time by using an optical camera, while their remote reading is carried out on the fly using a Frequency-Modulated Continuous-Wave radar. The proof-of-concept is established from the interrogation of a microfluidic temperature sensor placed on a conveyer belt.

**Keywords** — data fusion, Frequency-Modulated Continuous-Wave radar, passive and wireless sensor, detection, tracking

## I. INTRODUCTION

Over the past decades, the combination of radar and camera technologies has been extensively investigated, particularly in the automotive field, where the focus is on obstacle detection and classification [1], multi-object tracking [2], tracking of Micro Aerial Vehicles [3], and automatic traffic surveillance [4]. The integration of these two technologies is based on their distinct and complementary features. The radar provides precise information on the radial distance/range and velocity of detected objects, while the camera offers high spatial resolution, enabling visual details to be accurately distinguished. This combination delivers superior performance to that achievable with each technology used separately [5]. However, to date, the combination of radar and camera technologies has not been used to track and remotely interrogate passive and wireless sensors. Indeed, radar technology has been previously applied alone to detect, identify, and interrogate passive mobile sensors (see, e.g., [6]), but it is not suitable for real-time applications, as it usually requires large memory storage and time-consuming image processing. Furthermore, integrating these technologies with a mobile sensors can significantly reduce the number of sensors needed to monitor an environment, offering a more efficient and cost-effective solution. In this paper, we introduce a data fusion system that combines radar and camera technologies to both detect, localize, and remotely interrogate passive and wireless passive sensors in motion. Indeed, mobile sensors are detected using an optical camera, while their remote reading is carried out on-the-fly from the real-time processing of the raw data delivered by a radar. The proof-of-concept of the propped data fusion technique is established from the optical detection and radar interrogation of a microfluidic temperature sensor moving on a conveyer belt.

The article is organized as follows: Section II reports the architecture of the proposed optical/radar data fusion system. In Section III, first measurement results demonstrating the

feasibility of the on-the-fly and in-door tracking of a sensor in motion are reported. Finally, Section IV presents conclusions and prospects in line with the present research work and results.

## II. OPTICAL DETECTION AND RADAR INTERROGATION OF SENSORS

### A. System overview

The proposed remote sensing system for detecting and interrogating passive (battery-less) and wireless sensors consists of three main devices (Figure 1(a)): (1) a Frequency-Modulated Continuous-Wave (FM-CW) radar, (2) an optical camera and (3) a motorized pan-tilt. Both the radar and camera are mounted on the motorized pant-tilt, allowing for elevation and azimuth angular sweeps.

The camera has a pixel count resolution of $N_x \times N_z$ pixels. The frame rate of the camera (that is, the number of video frames per second) is denoted by $F_{RATE}$. The camera is used here to detect, locate in two-dimensional (2D) space, and identify a sensor in the scene using a specific color label placed on the sensor. Once the sensor is optically located, the main beam of the FM-CW radar is mechanically steered in the optically estimated direction of the sensor by using the pant-tilt. The sensor data are then read by the radar. In the proposed data fusion technique, a moving sensor is thus detected and tracked by the camera and interrogated on the fly by the radar.

The FM-CW radar transmits a chirp in the direction of the sensor with the repetition time $T_c$ and receives the signal backscattered by the sensor. From the transmitted and received signals, a beat frequency spectrum is then determined and the physical quantity of interest (here, a temperature) as well as the radar-to-sensor distance are estimated from echo radar level. The radar is equipped with two antennas: one for transmission ($T_x$), which is characterized by narrow beamwidths in azimuth and elevation (denoted respectively by $\phi_A$ and $\theta_A$), and another for reception ($R_x$).

The pan-tilt contributes to keep precise tracking of the sensor by adjusting the azimuth angle $\phi_P$ and elevation angle $\theta_P$ of the radar main beam in the direction of the sensor. The angular speed for azimuthal mechanical steering is denoted by $V_\phi$, while for elevation, it is denoted by $V_\theta$. In our spherical-coordinate-based 3D coordinate system, the origin (0, 0, 0) is set at the center of the mobile support of the pan-tilt, as shown in Figure 1(a). In this framework, the position $M_n$ of the sensor at time $t_n$ is defined by the coordinates $(\phi_n, \theta_n, R_n)$, where $\phi_n$ and $\theta_n$ denote respectively the azimuth and elevation angles of the
sensor location, while R_n designates the radar-to-sensor separation distance.

<figure><img src="image_2.png" /><figcaption>Fig. 1. (a) Proposed optical/radar data fusion system for the on-the-fly interrogation of moving sensors; (b) Main steps of the tracking and reading algorithm.</figcaption></figure>

Figure 1(b) illustrates the main steps for acquisition of both optical and radar data. When the sensor moves from position M_n to M_{n+1} with the velocity v, the camera captures images of the scene, and from an object detection algorithm based on HSV (Hue Saturation Value) color space [7], the contour of the colored rectangular label attached to the sensor is extracted from the images in the form of a bounding rectangle. The center of coordinates (U_L, V_L) of the rectangle is then derived, providing information on the location of the passive sensor in the image. To estimate the position of the sensor over time, the center (U_L, V_L) of the rectangle is set at the center C_0 of the camera image. To do this, we first determine the horizontal and vertical offsets in the optical image, denoted respectively by ΔU and ΔV, which represent the shifts between the coordinates of the center (U_L, V_L) of the rectangle and the coordinates of center C_0 of the image. Next, we apply a linear model to map these spatial offsets into angular offsets (Δφ_p, Δθ_p) which are ultimately used to control the pan-tilt, to correctly orient the camera and to steer the radar beam in the (optically estimated) direction of the sensor. Then, the radar transmits a chirp in the direction (φ_p, θ_p) with the repetition time T_c, receives the signal backscattered by the sensor in that direction and delivers a beat frequency f_n for which the magnitude of the spectral component reaches its highest value is then estimated

in real time and used to instantaneously approximate the radar-to-sensor distance R_n by the ratio $f_n c T_c / 4B$, where B and c denote respectively the radar bandwidth and the speed of light in vacuum.

### III. EXPERIMENTAL RESULTS AND DISCUSSION

#### A. Calibration of the optical/radar data fusion system

The specific optical/radar data fusion system used to demonstrate the feasibility of the on-the-fly and in-door tracking of a sensor in motion consists of the DK-sR-1030e radar operating at 24 GHz commercialized by IMST GmbH [8], a Logitech C270 HD webcam with resolution of 640×480 pixels, and PT-2020 pan-tilt commercialized by 2B Security Systems (see Figure 2). The transmitting antenna is a horizontally polarized lens antenna with a narrow beamwidth in azimuth and elevation plane, while the receiving antenna is vertically polarized horn (see Table 1 for numbers). The moving sensor is a depolarizing passive and wireless temperature sensor whose radar echo level varies with the temperature. It is a new version of the sensor reported by the authors in [9], as it has been manufactured entirely using three-dimensional metal additive technology. It is composed of a metallic waveguide operating in the frequency bandwidth of the radar and two identical, cross-polarized horn antennas connected through a cavity (see Figure 3). This cavity is crossed at its center by a channel filled with liquid metal (Galinstan): when the microfluidic channel is empty, the transmission coefficient of the cavity is at its maximum; as the volume of liquid metal increases within the channel due to an increase of temperature, the transmission coefficient decreases until reaching its minimum value, corresponding to a microfluidic channel entirely filled with liquid metal. Let Δl be the displacement of liquid metal in the channel when the temperature varies by ΔT. The sensitivity ΔT/Δl of the sensor is then given by $S_c/\alpha_v V_0$, where $\alpha_v$ (=11.5×10⁻⁵ K⁻¹) denotes the volumetric thermal expansion coefficient of the liquid metal (Galinstan), $S_c$ (=πr², with r =0.25mm) is the sectional surface of the microfluidic channel, and $V_0$ (=100 mm³) is the tank volume of liquid metal. The sensitivity of the sensor used here is then 17.1 °C/mm. For displacement Δl ranging from 0mm to 3.6mm with a step of 0.2mm, the sensor can detect temperature variations up to 61.5 °C with a precision of ±3.4 °C.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Photograph optical/radar data fusion system for tracking and reading mobile sensors.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Picture of the temperature sensor used in the experiment (see [9] for details). Dimensions are in mm.</figcaption></figure>

<table><caption>Table 1. Key descriptors of the optical/radar data fusion and mobile temperature sensors</caption><thead><tr><th>Device</th><th>Model</th><th colspan="2">Descriptors</th></tr></thead><tbody><tr><td rowspan="2">Webcam</td><td rowspan="2">Logitech C270 HD</td><td>Resolution</td><td>640x480 pixels</td></tr><tr><td>Frame rate F<sub>RATE</sub></td><td>1 fps</td></tr><tr><td rowspan="10">FMCW Radar</td><td rowspan="10">DK-sR-1030e from IMST GmbH</td><td>Operating frequency</td><td>24 GHz</td></tr><tr><td>Radar bandwidth B</td><td>2 GHz</td></tr><tr><td>T<sub>x</sub> antenna gain</td><td>28 dBi</td></tr><tr><td>R<sub>x</sub> antenna gain</td><td>20 dBi</td></tr><tr><td>Azimuth beamwidth φ<sub>A</sub></td><td>12°</td></tr><tr><td>Elevation beamwidth θ<sub>A</sub></td><td>9°</td></tr><tr><td>Transmitting Power P<sub>T</sub></td><td>20 dBm</td></tr><tr><td>Number of samples</td><td>256</td></tr><tr><td>Chirp duration T<sub>c</sub></td><td>15ms</td></tr><tr><td>Range resolution</td><td>7.5 cm</td></tr><tr><td rowspan="5">Pan-tilt</td><td rowspan="5">PT-2020 from 2B Security Systems</td><td>Pan Angle φ<sub>p</sub></td><td>-90° to 90°</td></tr><tr><td>Tilt Angle θ<sub>p</sub></td><td>-60° to 60°</td></tr><tr><td>Precision</td><td>0.1°</td></tr><tr><td>Pan speed V<sub>p</sub></td><td>6.1 °/s</td></tr><tr><td>Tilt speed V<sub>θ</sub></td><td>4.6 °/s</td></tr><tr><td rowspan="5">Sensor</td><td rowspan="5">Microfluidic temperature sensor [9]</td><td>Frequency bandwidth</td><td>22.8-24.8 GHz</td></tr><tr><td>Azimuth beamwidth φ<sub>S</sub></td><td>62°</td></tr><tr><td>Elevation beamwidth θ<sub>S</sub></td><td>50°</td></tr><tr><td>Displacement of liquid metal (Galinstan) in the channel</td><td>from 0 to 3.6mm</td></tr><tr><td>Full-scale measurement range of the temperature</td><td>61.5 °C</td></tr><tr><td></td><td></td><td>Measurement precision of the temperature</td><td>±3.4 °C</td></tr></tbody></table>

## B. Measurement results and discussion

To demonstrate the feasibility of the on-the-fly and in-door tracking of the temperature sensor in motion, we place the sensor on a 1.5m-long conveyor to obtain linear motion at a constant speed. As illustrated in Figure 4, we perform indoor experiments using two configurations, referred to as Scenario #1 and Scenario #2. In Scenario #1, the conveyor is positioned horizontally at 3.5 meters from optical/radar data fusion system (see Figure 4(a)). This configuration allows the sensor to move horizontally at the constant speed of 5 cm/s to track the sensor in the azimuth plane. For Scenario #2, the conveyor is positioned diagonally to allow the sensor to move in depth with the speed of 9 cm/s (see Figure 4(b)). This configuration aims to track the sensor as it moves away.

Measurements are performed by varying the level of liquid metal in the microfluidic channel from 0 mm to 2.4 mm, with a step size of 0.2 mm. The temperature is derived from the Galinstan level within the microfluidic channel, with a level of 0 mm indicating a temperature of 20°C.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Experimental indoor configurations for (a) Scenario #1 and (b) Scenario #2.</figcaption></figure>

The measurement results for Scenario #1 are shown in Figure 5, displaying the radar echo of the sensor as a function of the (optically estimated) azimuth angle φ of the sensor, and for each level of liquid metal in the microfluidic channel. As expected, the radar echo of the sensor decreases as the level of liquid metal increases in the channel. Moreover, the radar echo reaches its highest level for an azimuth angle of 2°, and it decreases as we move away from this angle. This is due to the directional characteristics of the (directive) horn antennas of the sensor. When the Galinstan level exceeds 2.0mm (i.e., for temperature higher than 54.2°), a dispersion of the measured maximum radar echo occurs. This is due to difficulties in isolating the very low radar echo of the sensor from the clutter. Above a Galinstan level of 2.4 mm (i.e., for temperature higher than 61°), this echo is completely embedded in the electromagnetic clutter, making the sensor undetectable. For Scenario #2, the results shown in Figure 6 indicate that the radar echo of the sensor decreases with the increasing level of Galinstan in the channel. However, this echo increases as the radar-to-sensor distance increases. This is because initially, the main beam of the receiving antenna of the sensor does not point in the direction of reading system, resulting in a low power received and backscattered by the sensor. As the sensor moves, the main lobes of the two antennas progressively point in the same direction, leading to an increase in the radar echo of the sensor.

Measurement uncertainty of the sensor's radar echo is now determined from using the same interrogation system (radar and antennas) and a motionless sensor. To estimate the measurement uncertainty, we have selected the maximum radar echo of the sensor for different temperature values ranging from 20°C to 61°C, with an azimuth angle set at 3.3° for Scenario #1 (Figure 7(a)), and a range of 3.8m for Scenario #2 (Figure 7(b)). These specific values of azimuth angle and range correspond to the highest achievable full-scale measurement range (or dynamic) for both scenarios. In Scenario #1, a high full-scale measurement range Δe of 14.4dB on the echo level is obtained with a measurement sensitivity of 0.47dB/°C for temperature ranging from 30.3° to 61°. When the temperature does not exceed 40.5 °C, the standard deviation of the radar echo
distribution is less than 0.1dB. Moreover, the standard deviation does not exceed 0.4dB and the average radar echo is less than -18.5dB. For Scenario#2, the full-scale measurement range $\Delta e$ is of 7.4 dB and measurement sensitivity is of 0.25dB/°C. For different temperature variation, the standard deviation of the radar echo distribution does not exceed 0.3dB.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Echo level of the sensor as a function of the (optically estimated) azimuth of the sensor at various temperatures or equivalently, for different levels of liquid metal (Galinstan) in the channel from 0 mm to 2.4 mm (Scenario #1)</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6. Echo level of the sensor as a function of the range in the (optically estimated) direction of the sensor at various temperatures or equivalently, for different levels of liquid metal (Galinstan) in the channel from 0 mm to 2.4 mm (Scenario #2)</figcaption></figure>

## IV. CONCLUSION

In this paper we proposed optical/radar data fusion system for the on-the-fly and in-door tracking of sensors in motion. The results demonstrate the capability to track and interrogate a temperature sensor effectively. Future work on further improving the proposed system performance includes replacing the current motorized platform with a faster one to track sensor with higher speed (> 9cm/s). In addition, we intend to implement dedicated software to extract 3D trajectory of sensors in real-time, while on-the-fly estimating of the temperature.

## ACKNOWLEDGMENT

The authors thank the National Agency for Research (France) for financial support through the S2LAM Project.

<figure><img src="image_8.png" /><figcaption>Fig. 7. Echo level of the sensor as a function of temperature (and liquid metal level in the Channel) for (a) Scenario #1 at radar-sensor distance of 3.8m and (b) Scenario #2 at the (optically estimated) azimuth angle of 2.5°. Error bars represent the standard deviation of the radar echo distribution.</figcaption></figure>

## REFERENCES

[1] M. Bertozzi, L. Bombini, P. Cerri, P. Medici, P. C. Antonello and M. Miglietta, "Obstacle detection and classification fusing radar and vision," *IEEE Intelligent Vehicles Symposium*, Eindhoven, Netherlands, pp. 608-613, Jun. 2008.

[2] A. Sengupta, L. Cheng and S. Cao, "Robust Multi-object Tracking Using Mm-wave Radar-Camera Sensor Fusion," *IEEE Sensors Letters*, vol. 6, no. 10, pp. 1-4, Oct. 2022.

[3] M. Caris, S. Stanko, W. Johannes, S. Sieger and N. Pohl, "Detection and tracking of Micro Aerial Vehicles with millimeter wave radar," *2016 European Radar Conference (EuRAD)*, London, UK, pp. 406-408, Oct. 2016.

[4] A. Roy, N. Gale and L. Hong, "Fusion of Doppler Radar and Video Information for Automated Traffic Surveillance," *12th International Conference on Information Fusion*, Seattle, WA, USA, pp. 1989-1996, Jul. 2009.

[5] S. Yao et al., "Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review," *IEEE Transactions on Intelligent Vehicles*, vol. 9, no. 1, pp. 2094-2128, Jan. 2024.

[6] D. Henry, J. G. D. Hester, H. Aubert, P. Pons, and M. M. Tentzeris, "Long-range wireless interrogation of passive humidity sensors using Van-Atta cross-polarization effect and different beam scanning techniques," *IEEE Trans. Microw. Theory Techn.*, vol. 65, no. 12, pp. 5345-5354, Dec. 2017.

[7] H. C. Kang, H. N. Han, H.C. Bae, M. G. Kim and Kim, Y. K., "HSV color - space - based automated object localization for robot grasping without prior knowledge", Applied Sciences, vol. 11, no. 16, p. 7593, 2021.

[8] "IMST Radar small and flexible radar modules by IMST," http://www.radar-sensor.com/, accessed: 2023-11-26.

[9] D. Henry, A. E. S. Ahmad, A. H. Djilani, P. Pons and H. Aubert, "Wireless Reading of Additively Manufactured Galinstan-based Sensor using a Polarimetric Millimeter-wave Radar Imaging Technique," *IEEE/MTT-S International Microwave Symposium (IMS)*, San Diego, CA, USA, pp. 799-802, Jun. 2023.
<figure><img src="image_9.png" /></figure>