<figure><img src="image_1.png" /></figure>

HAL
open science

# Detection Limits of Thermal-infrared Observations with Adaptive Optics. I. Observational Data

J. Sauter, W. Brandner, J. Heidt, F. Cantalloube

► To cite this version:

J. Sauter, W. Brandner, J. Heidt, F. Cantalloube. Detection Limits of Thermal-infrared Observations with Adaptive Optics. I. Observational Data. Publications of the Astronomical Society of the Pacific, 2024, 136 (9), pp.095001. 10.1088/1538-3873/ad6f45. hal-04749793

HAL Id: hal-04749793

https://hal.science/hal-04749793v1

Submitted on 24 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Detection Limits of Thermal-infrared Observations with Adaptive Optics. I. Observational Data

J. R. Sauter¹, W. Brandner¹, J. Heidt², and F. Cantalloube³

¹Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany; sauter@mpia.de

²Landesfernwarte, Zentrum für Astronomie der Universität Heidelberg, Königstuhl 12, D-69117 Heidelberg, Germany

³Université Grenoble Alpes, CNRS, IPAG, 38000 Grenoble, France

Received 2024 June 11; accepted 2024 August 14; published 2024 September 3

## Abstract

Ground-based thermal infrared observations face substantial challenges in correcting the predominant background emitted as thermal radiation from the atmosphere and the telescope itself. With the upcoming 40 m class ELTs, unprecedented sensitivities from ground will be reached, underlining the need of even more sophisticated background correction strategies. This study aims to investigate the impact of thermal backgrounds on ground-based observations and identify possible limiting factors in dedicated correction strategies. We evaluate temporal and spatial characteristics of the thermal background in direct imaging data obtained with different telescopes and observation modes. In particular, three distinct data sets, acquired using VLT/NACO and KECK/NIRC2, are analyzed. Our analysis reveals that the observations are not fully photon shot noise limited, but exhibit additional sensitivity losses caused by imperfect background compensation in the different data sets. We identify correlations between background fluctuations and the activity of the adaptive optics system. We hypothesize that the pupil modulation of the adaptive optics mirrors introduces high frequency spatial and temporal fluctuations to the background, which could ultimately constrain the detection limit if they are not compensated adequately.

*Unified Astronomy Thesaurus concepts: Astronomical instrumentation (799); High angular resolution (2167); Calibration (2179); Infrared astronomy (786); Ground-based astronomy (686)*

## 1. Introduction

Ground-based Thermal InfraRed (TIR) studies face a tough competition in the era of JWST. Despite the slightly superior angular resolution afforded by 8 to 10 m telescopes and the availability of instrument modes not offered by JWST, such as high spectral resolution with *R* in the range of 10,000 to 100,000, these observations are subject to degradation by Earth's atmosphere and the emissivity of ambient temperature telescope optics and support structures.

Ground-based observations in the TIR range are confined to specific atmospheric windows ranging from 3 to 25 µm—the *L*-, *M*-, *N*-, and *Q*-bands—while being susceptible to telluric absorption and emission features. Additionally, atmospheric turbulence induces wavefront phase (and amplitude) distortions, necessitating rapid Adaptive Optics (AO) corrections for achieving diffraction-limited resolution at 8 m class telescopes observing in the TIR (Käufl et al. 2018).

Optimally designed TIR observations strive to attain Background Limited Performance (Käufl et al. 1991), where the Signal-

to-Noise Ratio (S/N) increases as S/N ∝ t¹/². The time required to achieve a certain S/N is predicted to scale with telescope diameter *D*⁴ (National Research Council 1991), implying that transitioning from an 8 m primary mirror diameter of VLT to 39 m of ELT should yield the same S/N in approximately 1/500 of the exposure time. Thus, the *L*-band detection limit achieved by instruments like NACO (Lenzen et al. 2003; Rousset et al. 2003) or ERIS (Davies et al. 2023) at the VLT in 1 hr should be reached with METIS (Brandl et al. 2012) at the ELT in less than 10 s of exposure time. However, achieving significantly fainter detection limits requires thorough calibration and removal of various atmospheric and instrumental signatures imprinted on the data.

In this paper we aim to analyze temporal and spatial characteristics of the background signal imprinted in ground-based *L'*-band data obtained with VLT/NACO and KECK/NIRC2. The goal is to find and characterize limitations applied by the thermal background. In a second paper we derive a model for the thermal background, revise the implication on detection limits, and aim to devise optimized observing and data analysis strategies for future ground-based TIR studies.

The outline of this paper is as follows: The data and basic data reduction are described in Section 2. In Section 3 we determine the detection limits within the data sets motivating a more detailed analysis of the thermal background in Section 4. Here, we

<figure><img src="image_3.png" /></figure>

Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
investigate the background characteristics and reveal correlations with observational and instrumental parameters. We finish the paper with a general discussion and conclusion in Section 5.

## 2. Data

We analyze two VLT/NACO data sets (hereafter referred to as the 2004 and 2011 data sets) from the ESO Science Archive Facility<fnref n="4" /> and an auxiliary third KECK/NIRC2 data set from the Keck Observatory Archive.<fnref n="5" /> All three data sets comprise observations in the L'-band ($\lambda = 3.8$ µm, $\Delta\lambda = 0.7$ µm).

The VLT/NACO data sets include observations of the F8V star AF Lep, which has recently been found to have a Jovian companion (De Rosa et al. 2023; Franson et al. 2023; Mesa et al. 2023). The data were recorded with a $1024 \times 1024$ pixel² Aladdin detector at an image scale of 27.1 mas pixel⁻¹, resulting in a Field-of-View (FoV) of $28'' \times 28''$.

The 2004 data set comprises a total of 48 minutes ($96 \times 30$ s exposures) observation time obtained over two consecutive nights (2004 December 15 and 2004 December 16). The data were acquired using random dithering and 33°-discrete Angular Differential Imaging (ADI, Marois et al. 2006) applied halfway through each individual night. The data were initially published by Kasper et al. (2007).

The 2011 data set consists of 52 minutes ($156 \times 20$ s exposures) observation time obtained on 2011 October 21. The data were acquired using a four-point dithering pattern in pupil-stabilized mode, in a central $512 \times 512$ pixel² sub-window, and the cube mode of VLT/NACO. The four-point dither pattern includes two pairs of observations, where two subsequent exposures are observed at the same nod position. After each pair, the nod position is moved clockwise to the next quadrant within the detector. The pupil stabilization facilitates ADI. The ADI sequence covers a total rotation angle of 70°. In the cube mode, all integrations per exposure ($100 \times 0.2$ s integrations) are saved. The data were initially published by Rameau et al. (2013).

The VLT/NACO data sets include AO telemetry in form of the voltage covariance matrices applied to the deformable mirror and its tip-tilt mount as well as the wavefront sensor slope covariance matrices.

The KECK/NIRC2 data set includes observations of the A9/F0 star HIP 39017 and its companion (Tobin et al. 2024). The data were recorded with a $1024 \times 1024$ pixel² Aladdin detector at an image scale of 10.0 mas pixel⁻¹, resulting in a FoV of $10'' \times 10''$. It comprises a total of 45 minutes ($100 \times 27$ s exposures) of observation time observed on 2022 March 21. The data were obtained in pupil tracking ADI mode with a total rotation angle of 44°. It is the only utilized data set using a coronagraph. No chopping or nodding was applied during the observations. The data were initially published by Tobin et al. (2024).

<table><caption>**Table 1**
Atmospheric Seeing Conditions During the VLT/NACO 2004, VLT/NACO 2011, and KECK/NIRC2 2022 data Sets</caption><thead><tr><th>Data</th><th>ε (′′)</th><th>r<sub>0</sub> (cm)</th><th>τ<sub>0</sub> (ms)</th><th>S (%)</th><th>L'<sub>b</sub> (mag as<sup>-2</sup>)</th></tr></thead><tbody><tr><td>2004.1.1</td><td>0.66<sup>+0.39</sup><sub>-0.08</sub></td><td>17.0<sup>+11.4</sup><sub>-6.8</sub></td><td>9.1<sup>+3.6</sup><sub>-3.1</sub></td><td>49.0<sup>+13.6</sup><sub>-9.0</sub></td><td>3.347<sup>+0.004</sup><sub>-0.003</sub></td></tr><tr><td>2004.1.2</td><td>1.43<sup>+0.62</sup><sub>-0.46</sub></td><td>8.0<sup>+5.0</sup><sub>-3.2</sub></td><td>5.7<sup>+4.3</sup><sub>-2.1</sub></td><td>31.2<sup>+17.4</sup><sub>-30.9</sub></td><td>3.343<sup>+0.002</sup><sub>-0.001</sub></td></tr><tr><td>2004.2</td><td>0.66<sup>+0.42</sup><sub>-0.17</sub></td><td>15.5<sup>+9.8</sup><sub>-8.4</sub></td><td>9.2<sup>+6.7</sup><sub>-6.4</sub></td><td>42.4<sup>+8.7</sup><sub>-13.2</sub></td><td>3.358<sup>+0.010</sup><sub>-0.016</sub></td></tr><tr><td>2011</td><td>1.21<sup>+0.63</sup><sub>-0.31</sub></td><td>8.5<sup>+12.1</sup><sub>-5.7</sub></td><td>3.0<sup>+4.7</sup><sub>-2.2</sub></td><td>35.3<sup>+11.0</sup><sub>-33.4</sub></td><td>3.292<sup>+0.007</sup><sub>-0.007</sub></td></tr><tr><td>2022</td><td>1.08</td><td>9.4</td><td>...</td><td>...</td><td>1.984<sup>+0.008</sup><sub>-0.011</sub></td></tr></tbody></table>

**Note.** The columns represent the first and second halves of the first night in 2004, the second night in 2004, the night in 2011, and 2022. The rows indicate the corresponding median values, where the errors give the peak-to-valley range. The following parameters are given (at 500 nm wavelength): seeing ($\epsilon$), coherence length ($r_0$), coherence time ($\tau_0$), Strehl ratio ($S$), and background brightness $L'_b$.

For all three data sets we use twilight flat-fields for the data reduction obtained at the same night at multiple airmasses.

The data sets are ideal for our studies as they do not include chopping, minimizing effects from non-common light paths. Utilizing data sets with different instrumental setups (i.e., discrete and continuous ADI, nodding and no nodding, or coronagraph and no coronagraph) allows us to compare and therefore distinguish the influence of the individual setups to the observed data.

### 2.1. Observing Conditions

The observing conditions for the VLT/NACO 2004 data were good during the first half of the first night and the entire second night, but poor during the second half of the first night. The 2004 observations were obtained at airmass ranging from 1.09 to 1.27 and experienced very low ambient wind speeds between 0.6 and $3.7 \text{ m s}^{-1}$ and ambient temperatures between 10.5°C and 14.0°C.

In contrast, the conditions for the VLT/NACO 2011 data were generally moderate and highly variable. The observations occurred at smaller airmass, ranging from 1.03 to 1.04, and were accompanied by higher wind speeds ranging between 9.7 and $13.2 \text{ m s}^{-1}$ and ambient temperatures between 9.4°C and 10.0°C.

For the KECK/NIRC2 2022 data set only the average seeing is provided, indicating only moderate observing conditions. The observations occurred at small airmass, ranging from 1.10 to 1.17 with moderate wind speeds between 1 and $8 \text{ m s}^{-1}$ and ambient temperatures between -1.0°C and -0.2°C.

A summary of the seeing conditions for the three data sets is provided in Table 1.

### 2.2. Data Reduction

We apply a standard data reduction using bias and dark current subtraction as well as flat-field corrections to minimize pixel-to-pixel and large scale intensity variations. To isolate background effects originating from the atmosphere and telescope, we refrain from using additional reduction methods such as centroiding or

<footnote n="4">⁴ https://archive.eso.org/eso/eso_archive_main.html</footnote>

<footnote n="5">⁵ https://koa.ipac.caltech.edu/cgi-bin/KOA/nph-KOAlogin</footnote>
background derotation, which primarily aid in stellar light subtraction and have minimal impact on subtracted backgrounds.

When subtracting two subsequent exposures to reveal temporal changes in the background, the subtracted backgrounds in the image pairs are in the following referred to as residual backgrounds. For the 2011 data set the residual backgrounds are computed for exposures with stellar targets at the same detector position, effectively minimizing potential influences from nodding and residual stellar signals.

The stellar targets in the data are masked and then 3σ clipping over all non-masked pixels is used to eliminate outliers.

### 3. Thermal Background Detection Limits

To assess the impact of the thermal background on ground-based observations, we calculate the 5σ detection limits $L_{5\sigma}$ at accumulated observation time $t$ for the three data sets, using the background variance $\eta$ of the exposures as noise term in the standard S/N estimation (e.g., Newberry 1991):

$$L_{5\sigma}(t, r) = \frac{25 + \sqrt{625 + 100 \cdot \eta(t) \cdot \pi r^2}}{2 \cdot \xi(r)}, \quad (1)$$

where $\xi$ the normalized encircled power of the underlying Point-Spread Function (PSF) at radial distance $r$.

The background variances are measured at the outer regions of the exposures, distant from the stellar locations, ensuring minimal contamination. We set the radial distance to 4 pixel, i.e., twice the Nyquist sampling, accounting for the specific PSF shape of each data set. It is important to note that the derived absolute limits are sensitive to the assumed PSF shape and radial detection distance, hereby providing only rough estimates in this work. The focus of our analysis lies more on the relative changes within one data set and the conclusions that can be drawn from them. The respective utilized PSF shapes are measured using the calibration observation of each data set and are depicted in Figure 7 in Appendix A.

Figure 1 illustrates the derived detection limits as a function of the cumulative observations time. The respective background variances are hereby either set as the measured background intensities, thus inferring ideal photon shot noise, or measured Residual Background Variance (RBV). The derived limits of the three data sets follow qualitatively the measured background brightness (see Table 1), where higher limits are derived for the VLT/NACO data sets which comprise a lower background brightness of $3.3 \text{ mag}_L$ as $-2$ and the lower limits are derived for the KECK/NICR2 data set which comprises a higher background brightness of $2.0 \text{ mag}_L$ as $-2$.

The measured VLT/NACO background brightness roughly matches with the $3.0 \text{ mag}_L$ as $-2$ denoted in the VLT/NACO user manual.<fnref n="6" /> The KECK/NIRC2 sky brightness of $\approx 2.0 \text{ mag}_L$ as $-2$ is

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Top: residual background variance 5σ detection limits as a function of observation time for the three different data sets. The solid lines depict the measured limits, the dashed lines the median variance limits, the dashed-dotted lines the ideal median background brightness limits, and the dotted lines the median limits with applied losses from an applied linear fit. Bottom: detection losses between the ideal median background brightness and measured detection limits (solid line). Linear fits (dotted lines) are applied to the different losses showing an overall increase with observation time.</figcaption></figure>

in the range of typical background values quoted in the user manual and instrument web page.<fnref n="7" /> The differences in the derived detection limits between the two VLT/NACO data sets are related to the assumed PSF shape. For the KECK/NIRC2 data set the segmented primary mirror and pupil alignment offsets might explain the higher background brightness.

All three data sets show prominent and strongly variable excess noise with differences between the median background brightness and median variance limits of $0.03 \text{ mag}_L$ for the 2004 data set, $0.02 \text{ mag}_L$ for the 2011 data set, and $0.06 \text{ mag}_L$ for the 2022 data set. Oppositely, the background brightness shows almost no variability as depicted in Table 1. A comparison between the background variance using the measured RBV and background brightness is shown in Figure 8 in Appendix B.

The measured losses in all three data sets show increasing deviations from the ideal median background brightness limits with time. After approximately 50 minutes they range between $0.08 \text{ mag}_L$ and $0.09 \text{ mag}_L$. The measured loss offsets at zero observation time result from the excess observed for the first exposures, which are relatively high for the 2004 and 2022 data set

<footnote n="6">⁶ https://www.eso.org/sci/facilities/paranal/decommissioned/naco/doc.html</footnote>

<footnote n="7">⁷ https://www2.keck.hawaii.edu/inst/nirc2/ObserversManual.html and https://www2.keck.hawaii.edu/inst/nirc2/sensLong.html.</footnote>
as shown in Figure 8 in Appendix B. We apply linear fits to the measured losses to depict the linear trends in the individual data sets. Especially the 2011 data set which depicts strongly variable seeing conditions during the observations follows its trend very precisely. Notably, the losses of the 2004 data set tend to stay constant or even decrease in the first half of the first night and in the second night, where the seeing conditions were stable, and tend to strongly increase at the second half of the first night, where the seeing conditions were poor and very variable. The losses for the 2022 data set show only a small slope, again qualitatively matching the average seeing conditions during the observation.

The fitted linear slopes thus represent the average seeing conditions during the observations. Although very speculative, the increasing losses would ultimately constrain the achievable detection limits when extrapolated to several hours of observation time. The applied fits would hereby imply limits of 17.8 $mag_L'$ for the VLT/NACO data sets and 17.6 $mag_L'$ for the KECK/NIRC2 data set.

The measured ~50 minutes detection limits of 17.1 $mag_L'$ and 17.3 $mag_L'$ for the 2004 and 2011 VLT/NACO and 16.7 $mag_L'$ for KECK/NIRC2 data set surpass the corresponding planet brightness of 14.87 $mag_L'$ for AF Lep b (Franson et al. 2023) and 16.12 $mag_L'$ for HIP 39017 b (Tobin et al. 2024) and would thus permit the detection in the respective data sets. Nevertheless, the derived limits clearly show the dominant role of the thermal background in achievable sensitivities. The revealed additional losses and the resulting potential saturation in sensitivity hence result in a substantial limitation for ground-based observations.

# 4. Thermal Background Information Content

An ideal thermal background correction eliminates all spatial and temporal structures, leaving behind only pure, uncorrelated photon shot noise. Following Larkin (2016), intensity structures in the residual background carry valuable information or signals which should be distinguished from the pure photon shot noise. The challenge lies in interpreting this information and uncovering causal relationships. In this study, we adopt a systematic approach: First, we develop a quantitative method to measure the spatial intensity structures in the background. Then, we explore temporal correlations with ambient and instrumental parameters collected concurrently to the observations. Finally, we evaluate the correlations by comparing the different data sets to reveal possible causations.

## 4.1. Spatial Correlations

The spatial intensity structure abundance of the residual backgrounds in the VLT/NACO and KECK/NIRC2 data sets, compared to those using the 2011 and KECK/NIRC2 flat-field images, are depicted in Figure 2. To characterize these structures, we measure the RBV within a 64 × 64 pixel² box positioned far outside the stellar Airy disk at a fixed detector position. At large

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Normalized mean residual background variance as a function of applied mean filter diameter for the different data sets. The solid lines denote the subtracted exposure pairs, the dashed line the VLT/NACO 2011 integration pairs, and the dotted line ideal photon shot noise.</figcaption></figure>

enough distances, the influence of diffracted stellar light—in particular by the coronagraph in the KECK/NIRC2 data—is expected to be negligible (Roddier & Roddier 1997). The variances are evaluated after applying a mean filter of variable diameter $d_{filter}$. Deviations from an ideal, non-correlated photon shot noise $d_{filter}^{-2}$-drop reveal systematic correlations or structures within the background. The variances are measured for each subsequent exposure pair (and integration pair for the 2011 data set) and are then averaged for each mean filter diameter and normalized by the variance without any filter. The individual curves thereby depict the average spatial intensity change in the timescales of one exposure (one integration) for each data set.

From Figure 2, it is apparent that the data variances exhibit significant and qualitatively similar deviations from the flat-field variances, which almost ideally drop with increasing filter diameter. Notably, the deviations change depending on the location where the variances are measured on the detector. For the 2011 data set, where the individual 0.2 s integrations are available, less prominent deviations are visible, showing that the changes in the background appear already in sub-second timescales and seem to accumulate with time. From the figure it is thus evident that residual spatial intensity structures are prominent at all scales of the respective PSF shapes for all three data sets, possible influencing the achievable detection limits. The markedly different statistical properties between flat-field and science images can be solely attributed to the influence of the AO, as it is the only sub-instrument deactivated during flat-field observations that acts on these fast timescales.

In particular, the 2011 exposures are subtracted at the same nodding position, and in the KECK/NIRC2 data set no chopping or nodding is applied, ruling out changes in the
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Normalized cross-correlation maps for the three data sets displaying the correlation between the Residual Background Variance (RBV), the RBV Difference (RBVD), and various ambient parameters. The parameters are described in more detail in the text. The KECK/NIRC2 data lacks data on the relative wind direction, while the VLT/NACO 2011 humidity measurements did not change over the course of the observing sequence.</figcaption></figure>

optical path. Additionally, the 2004 data set employs discrete ADI without angular changes between the exposures, which excludes influences of the field rotation. The twilight flat-fields were observed at identical exposure times under similar instrumental settings, atmospheric conditions, and airmass, thereby ruling out detector or atmospheric effects.

## 4.2. Temporal Correlations with Ambient Parameters

To analyze the temporal correlations of the background with ambient parameters, we approximate the spatial structure abundance by computing the difference between the RBV at a 10 pixel mean filter diameter and the theoretical ideal photon shot noise background variance for the different exposures of the three data sets. This parameter is denoted as RBV Difference (RBVD) and is also compared to the absolute RBV (i.e., without applying the mean filter). Figure 3 displays the resulting normalized cross-correlation maps of a selection of ambient parameters: wind speed, relative wind direction, pressure, airmass, relative humidity, and temperature. The parameters are obtained from the FITS header of the different data sets (no data is available for the KECK/NIRC2 2022 wind direction, while the relative humidity remained constant at 10% over the course of the VLT/NACO 2011 observations, thereby restricting cross-correlation). The relative wind direction is calculated as the difference between the telescope orientation and the absolute wind direction. The normalized temporal cross-correlation $\bar{C}$ between two parameters x and y is given as:

$$ \bar{C}(x, y) = \frac{2 \cdot C(Z(x), Z(y))}{C(Z(x), Z(y)) + C(Z(y), Z(y))}, \quad (2) $$

where $Z(\cdot)$ represents the standard score.

From the correlation maps, it is evident that the RBV strongly correlates with the RBVD in all three data sets. At large RBVD values, where strong spatial intensity structures appear in the background, the overall background variance is consequently increased as well. However, no consistently strong correlation between an ambient parameter and the spatial background correlation is prominent in all three data sets.

Notably, wind speed and relative wind direction exhibit the highest absolute correlations with the background over the different data sets. The anti-correlation observed in the 2004 data set could result from the so-called low-wind effect, as reported for the VLT (Milli et al. 2018). In the 2022 data set, strong correlations are present between pressure, airmass, relative humidity, and temperature. Here, the temporal airmass progression coincidentally aligns with the other parameters, all following a similar linear trend linked to the beginning of dawn. This shows that the correlations must be interpreted carefully, as causalities can be complex and not readily apparent in the data.

## 4.3. Temporal Correlations with Atmospheric and AO Parameters

Analogous to Section 4.2, we investigate the temporal correlations of the background with a selection of different atmospheric and AO parameters for the two VLT/NACO data sets (no AO data is provided for the KECK/NIRC2 2022 data set). The parameters include the coherence length, seeing, coherence time, Strehl ratio, Voltage Covariance (VC) change, and Slope Covariance (SC) change. The last two parameters denote the summed absolute differences between the corresponding covariance matrix parameters of the respective exposure pairs. The results are presented in Figure 4.
<figure><img src="image_7.png" /><figcaption>**Figure 4.** Normalized cross-correlation maps for the two VLT/NACO data sets displaying the correlation between the Residual Background Variance (RBV), the RBV Difference (RBVD), and various atmospheric and adaptive optics parameters. The parameters are described in more detail in the text.</figcaption></figure>

Compared to the ambient parameters, the atmosphere and AO parameters exhibit overall consistently strong correlations with the background structure. Among all parameters, the VC change shows the strongest correlations with the RBV and RBVD. We denote the shorter exposure times and thereby higher relative photon shot noise in the 2011 data set to the smaller correlations between the VC change and RBV compared to the 2004 data set. As expected, the atmospheric and AO parameters display strong correlations with each other, since worse seeing conditions are typically more variable. However, the Strehl ratio in the 2011 data deviates from this trend, possibly due to the extremely small ratios measured in this data set. A comprehensive cross-correlation map utilizing all atmosphere, AO, and ambient parameters of the two VLT/NACO data sets can be found in Figure 9 in Appendix C.

Figure 5 illustrates the relation between the RBV and the VC change in the two VLT/NACO data sets. From the figure it is evident that both parameters show a strong linear correlation. The correlation further supports our assumption in Section 4.1 that the AO is responsible for the spatial structures observed in the residual background. In Figure 6 we present example residual background cut-outs from the three data sets. For low VC changes, corresponding to low RBV, all three data sets exhibit weak spatial structures, similar to the flat-field residuals. Conversely, at high VC changes, strong spatial intensity structures become prominent. Combining the previous results, we conclude that the permanent and high-frequency changes applied by the DM

affect the background properties, resulting in spatial intensity structures after subtracting two exposures.

## 5. Discussion and Conclusion

The measured background detection limits in all three data sets increasingly deviate from ideal photon shot noise limits as observation time extends, resulting in a sensitivity loss of 0.08 $mag_{L'}$ to 0.09 $mag_{L'}$ after approximately 50 minutes of observation. The increase in loss hereby qualitatively correlates with the seeing conditions, where bad and variable seeing conditions lead to higher losses. If extrapolated to longer timescales these losses could constrain the achievable detection limits to 17.6 $mag_{L'}$ to 17.8 $mag_{L'}$. These findings underscore the importance of conducting a detailed analysis to understand the origins of these losses and explore strategies to mitigate them effectively. In particular, the influence of ambient conditions, the telescope, and instruments is not well understood yet and requires further investigation.

Our examination revealed that residual backgrounds in twilight flat-field images exhibit negligible spatial intensity structures, whereas those in the science data show pronounced deviations. By comparing data sets from different telescopes and observation modes, we discovered that only the presence of AO corrections during science observations can explain this contrast. Furthermore, we identified a linear correlation between the RBV and the variability of the DM during the
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Top: residual background variance as a function of voltage covariance change applied to the deformable mirror for the two VLT/NACO data sets. The dashed lines depict the linear trends within the two data sets. Bottom: the same quantities plotted individually for all exposure pairs. Note: Outliers with too high variances (2004: >22 ADU² and 2011: >16.5 ADU²) are clipped to limit the dynamic range.</figcaption></figure>

observations. Notably, significant background structures emerge only when the DM exhibits high variability.

These observations highlight the complex role of the AO system in background correction. We hypothesize that variations in DM configurations between exposures modulate the observed background at high frequencies, leading to systematic residuals which are responsible for the observed sensitivity losses. Further investigations are necessary to precisely delineate the causal relationship between the DM variability and background residuals and devise strategies to mitigate or counteract these effects adequately.

Expanding our analysis to include diverse telescopes, instruments, observing conditions, and atmospheric bands

could further validate our findings and enhance our understanding of how factors such as telescope size, AO performance, and wavelength influence the thermal background and the applied correction outcomes. A detailed understanding is crucial to predict and maximize the potential of the next generation ELTs.

In a subsequent paper, we plan to delve deeper into these questions. Our goals include developing a model for the thermal background that attributes the causal relationship to AO and exploring the implications of background residuals on detection limits. Additionally, we aim to formulate optimized observing and data reduction strategies for future ground-based TIR studies.
<figure><img src="image_9.png" /><figcaption>**Figure 6.** Example residual background cut-outs for the VLT/NACO 2004 data set (top), VLT/NACO 2011 data set (center), and KECK/NIRC2 data set (bottom). The residual backgrounds for the flat-fields (left) and for low Deformable Mirror (DM) actuation (center) exhibit weak spatial intensity structures. On the other hand, the residual backgrounds for high DM actuation (right) exhibit strong spatial intensity structures. Note: the VLT/NACO 2004 flat-fields were obtained at varying rotator angles resulting in the striped features.</figcaption></figure>

## Acknowledgments

Data obtained from the ESO Science Archive Facility under ESO programmes 074.C-0323(A) and 088.C-085(A). This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExSci), under contract with the National Aeronautics and Space Administration. The KOA data were obtained under the program U187 with T. Brandt as principal investigator.

Facilities: VLT:Antu (NACO), Keck:II (NIRC2).

Software: astropy (The Astropy Collaboration et al. 2013, 2018, 2022), Numpy (Harris et al. 2020), Matplotlib (Hunter 2007), SciPy (Virtanen et al. 2020).

## Appendix A

<figure><img src="image_10.png" /><figcaption>**Figure 7.** Normalized encircled power as a function of radial distance for the three different data set calibration PSFs. The respective resolution is 27 mas pixel⁻¹ for the VLT/NACO data sets and 10 mas pixel⁻¹ for the KECK/NIRC2 data set.</figcaption></figure>
## Appendix B

<figure><img src="image_11.png" /><figcaption>**Figure 8.** Derived background variance for the different exposures in the three data sets using the Background Brightness (BB) and Residual Background Variance (RBV). All three data sets exhibit variable excess noise yielding higher RBV compared to the BB.</figcaption></figure>

## Appendix C
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /><figcaption>**Figure 9.** Normalized cross-correlation maps for the two VLT/NACO data sets displaying the correlation between the Residual Background Variance (RBV), the RBV Difference (RBVD), and different atmosphere, adaptive optics, and ambient parameters.</figcaption></figure>
ORCID iDs

J. R. Sauter https://orcid.org/0000-0002-8644-431X

W. Brandner https://orcid.org/0000-0003-1939-6351

J. Heidt https://orcid.org/0000-0002-0320-1292

F. Cantalloube https://orcid.org/0000-0002-3968-3780

References

Brandl, B. R., Lenzen, R., Pantin, E., et al. 2012, Proc. SPIE, 8446, 84461M

Davies, R., Absil, O., Agapito, G., et al. 2023, A&A, 674, A207

De Rosa, R. J., Nielsen, E. L., Wahhaj, Z., et al. 2023, A&A, 672, A94

Franson, K., Bowler, B. P., Zhou, Y., et al. 2023, ApJL, 950, L19

Harris, C. R., Millman, K. J., Van Der Walt, S. J., et al. 2020, Natur, 585, 357

Hunter, J. D. 2007, CSE, 9, 90

Kasper, M., Apai, D., Janson, M., & Brandner, W. 2007, A&A, 472, 321

Käufl, H. U., Bouchet, P., Van Dijsseldonk, A., & Weilenmann, U. 1991, ExA, 2, 115

Käufl, H.-U., Kasper, M., Arsenault, R., et al. 2018, Proc. SPIE, 10702, 107020D

Larkin, K. G. 2016, arXiv:1609.01117

Lenzen, R., Hartung, M., Brandner, W., et al. 2003, Proc. SPIE, 4841, 944

Marois, C., Lafreniere, D., Doyon, R., Macintosh, B., & Nadeau, D. 2006, ApJ, 641, 556

Mesa, D., Gratton, R., Kervella, P., et al. 2023, A&A, 672, A93

Milli, J., Mouillet, D., Beuzit, J.-L., et al. 2018, Proc. SPIE, 10703, 107032A

National Research Council 1991, Working Papers: Astronomy and Astrophysics Panel Reports (Washington, DC: National Academies Press), 59

Newberry, M. V. 1991, PASP, 103, 122

Rameau, J., Chauvin, G., Lagrange, A.-M., et al. 2013, A&A, 553, A60

Roddier, F., & Roddier, C. 1997, PASP, 109, 815

Rousset, G., Lacombe, F., Puget, P., et al. 2003, Proc. SPIE, 4839, 140

The Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, ApJ, 935, 167

The Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123

The Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33

Tobin, T. L., Currie, T., Li, Y., et al. 2024, AJ, 167, 205

Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, NatMe, 17, 261