# Planet search with the Keck/NIRC2 vortex coronagraph in the Ms band for Vega

Bin B. Ren, Nicole L. Wallack, Spencer A. Hurt, Dimitri Mawet, Aarynn L. Carter, Daniel Echeverri, Jorge Llop-Sayson, Tiffany Meshkat, Rebecca Oppenheimer, Jonathan Aguilar, et al.

<figure><img src="image_1.png" /></figure>

To cite this version:

Bin B. Ren, Nicole L. Wallack, Spencer A. Hurt, Dimitri Mawet, Aarynn L. Carter, et al. Planet search with the Keck/NIRC2 vortex coronagraph in the Ms band for Vega. Astronomy & Astrophysics, 2023, 670, 10.1051/0004-6361/202244485. insu-04479111

HAL Id: insu-04479111

https://insu.hal.science/insu-04479111

Submitted on 2 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Planet search with the Keck/NIRC2 vortex coronagraph in the Ms band for Vega

Bin B. Ren, Nicole L. Wallack, Spencer A. Hurt, Dimitri Mawet, Aarynn L. Carter, Daniel Echeverri, Jorge Llop-Sayson, Tiffany Meshkat, Rebecca Oppenheimer, Jonathan Aguilar, et al.

► To cite this version:

Bin B. Ren, Nicole L. Wallack, Spencer A. Hurt, Dimitri Mawet, Aarynn L. Carter, et al. Planet search with the Keck/NIRC2 vortex coronagraph in the Ms band for Vega. Astronomy & Astrophysics - A&A, 2023, 670, 10.1051/0004-6361/20224485. insu-04479111

HAL Id: insu-04479111

https://insu.hal.science/insu-04479111v1

Submitted on 2 Mar 2024

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire **HAL**, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Planet search with the Keck/NIRC2 vortex coronagraph in the $M_s$ band for Vega*

Bin B. Ren (任彬)<fnref n="1" />,<fnref n="2" />,<fnref n="3" /> Ⓓ, Nicole L. Wallack<fnref n="4" />,<fnref n="5" />, Spencer A. Hurt<fnref n="6" />, Dimitri Mawet<fnref n="3" />,<fnref n="7" />, Aarynn L. Carter<fnref n="8" /> Ⓓ, Daniel Echeverri<fnref n="3" />, Jorge Llop-Sayson<fnref n="3" />, Tiffany Meshkat<fnref n="9" />, Rebecca Oppenheimer<fnref n="10" /> Ⓓ, Jonathan Aguilar<fnref n="11" /> Ⓓ, Eric Cady<fnref n="7" />, Élodie Choquet<fnref n="12" /> Ⓓ, Garreth Ruane<fnref n="7" />, Gautam Vasisht<fnref n="7" />, and Marie Ygouf<fnref n="7" />

Received 12 July 2022 / Accepted 18 January 2023

## ABSTRACT

**Context.** Gaps in circumstellar disks can signal the existence of planetary perturbers, making such systems preferred targets for direct imaging observations of exoplanets.

**Aims.** Being one of the brightest and closest stars to the Sun, the photometric standard star Vega hosts a two-belt debris disk structure. Together with the fact that its planetary system is being viewed nearly face-on, Vega has been one of the prime targets for planet imaging efforts.

**Methods.** Using the vector vortex coronagraph on Keck/NIRC2 in the $M_s$ band at 4.67 µm, we report the planet detection limits from 1 au to 22 au for Vega with an on-target time of 1.8 h.

**Results.** We reach a $3 M_{\text{Jupiter}}$ limit outward of 12 au, which is nearly an order of magnitude deeper than for other existing studies. Combining our observations with existing radial velocity studies, we can confidently rule out the existence of companions more than ~8 $M_{\text{Jupiter}}$ from 22 au down to 0.1 au for Vega. Interior and exterior to ~4 au, this combined approach reaches planet detection limits down to ~2–3 $M_{\text{Jupiter}}$ using radial velocity and direct imaging, respectively.

**Conclusions.** By reaching multi-Jupiter mass detection limits, our results are expected to be complemented by the planet imaging of Vega in the upcoming observations using the *James Webb* Space Telescope to obtain a more holistic understanding of the planetary system configuration around Vega.

**Key words.** planets and satellites: detection – techniques: high angular resolution – techniques: image processing – planets and satellites: individual: Vega – stars: imaging

## 1. Introduction

Vega ($\alpha$ Lyrae, *Keoe*, *wāqi', Zhinii* 织女一), one of the historical photometric standard stars (Johnson & Morgan 1953), is an A0 V star (e.g., Johnson & Morgan 1953) that is located at 7.68 ± 0.02 pc from the Solar System (van Leeuwen 2007). Despite its proximity, Vega is not included in the *Gaia* Catalog of Nearby Stars that are within 100 pc from the Sun, since its brightness exceeds the *Gaia* limits (Gaia Collaboration 2021). With an age of 445 ± 13 Myr (Yoon et al. 2010), Vega shows a prototypical mid-infrared excess in the Infrared Astronomical

Satellite observations, situating it in an evolutionary stage that is between star formation and our Solar System (Aumann et al. 1984).

The two-belt debris disk system around Vega may result from planet-disk interactions under various planetary configurations (e.g., Matrà et al. 2020). In fact, most existing directly imaged planets are found in bright debris disk systems (e.g., HR 8799: Marois et al. 2008, β Pic: Lagrange et al. 2009, 51 Eri: Macintosh et al. 2015), making it more likely to find giant planets in debris disk systems than around stars without disks (Meshkat et al. 2017). Combined with the observational fact that giant planets are more likely to exist at 1–10 au from their host stars in both direct imaging and radial velocity surveys (Nielsen et al. 2019; Fulton et al. 2021), the proximity

<footnote n="1">¹ Université Côte d'Azur, Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, 06304 Nice, France
e-mail: bin.ren@oca.eu</footnote>

<footnote n="2">² Univ. Grenoble Alpes, CNRS, IPAG, 38000 Grenoble, France</footnote>

<footnote n="3">³ Department of Astronomy, California Institute of Technology, MC 249-17, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="4">⁴ Earth and Planets Laboratory, Carnegie Institution for Science, Washington DC 20015, USA</footnote>

<footnote n="5">⁵ Division of Geological & Planetary Sciences, California Institute of Technology, MC 150-21, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="6">⁶ Department of Earth Sciences, University of Oregon, Eugene OR 97403, USA</footnote>

<footnote n="7">⁷ Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena CA 91109, USA</footnote>

<footnote n="8">⁸ Department of Astronomy & Astrophysics, University of California, Santa Cruz, 1156 High St, Santa Cruz CA 95064, USA</footnote>

<footnote n="9">⁹ Infrared Processing and Analysis Center (IPAC), California Institute of Technology, MC 100-22, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="10">¹⁰ American Museum of Natural History, New York NY 12345, USA</footnote>

<footnote n="11">¹¹ Space Telescope Science Institute (STScI), 3700 San Martin Drive, Baltimore MD 21218, USA</footnote>

<footnote n="12">¹² Aix-Marseille Univ., CNRS, CNES, LAM, Marseille, France</footnote>

<footnote>*Contrast curve is only available at the CDS via anonymous ftp to cdsarc.cds.unistra.fr (130.79.128.5) or via https:// cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/670/A162</footnote>
# Planet search with the Keck/NIRC2 vortex coronagraph in the $M_s$ band for Vega*

Bin B. Ren (任彬)<fnref n="1" />,<fnref n="2" />,<fnref n="3" /> Ⓓ, Nicole L. Wallack<fnref n="4" />,<fnref n="5" />, Spencer A. Hurt<fnref n="6" />, Dimitri Mawet<fnref n="3" />,<fnref n="7" />, Aarynn L. Carter<fnref n="8" /> Ⓓ, Daniel Echeverri<fnref n="3" />, Jorge Llop-Sayson<fnref n="3" />, Tiffany Meshkat<fnref n="9" />, Rebecca Oppenheimer<fnref n="10" /> Ⓓ, Jonathan Aguilar<fnref n="11" /> Ⓓ, Eric Cady<fnref n="7" />, Élodie Choquet<fnref n="12" /> Ⓓ, Garreth Ruane<fnref n="7" />, Gautam Vasisht<fnref n="7" />, and Marie Ygouf<fnref n="7" />

Received 12 July 2022 / Accepted 18 January 2023

## ABSTRACT

**Context.** Gaps in circumstellar disks can signal the existence of planetary perturbers, making such systems preferred targets for direct imaging observations of exoplanets.

**Aims.** Being one of the brightest and closest stars to the Sun, the photometric standard star Vega hosts a two-belt debris disk structure. Together with the fact that its planetary system is being viewed nearly face-on, Vega has been one of the prime targets for planet imaging efforts.

**Methods.** Using the vector vortex coronagraph on Keck/NIRC2 in the $M_s$ band at 4.67 µm, we report the planet detection limits from 1 au to 22 au for Vega with an on-target time of 1.8 h.

**Results.** We reach a $3 M_{\text{Jupiter}}$ limit outward of 12 au, which is nearly an order of magnitude deeper than for other existing studies. Combining our observations with existing radial velocity studies, we can confidently rule out the existence of companions more than ~8 $M_{\text{Jupiter}}$ from 22 au down to 0.1 au for Vega. Interior and exterior to ~4 au, this combined approach reaches planet detection limits down to ~2–3 $M_{\text{Jupiter}}$ using radial velocity and direct imaging, respectively.

**Conclusions.** By reaching multi-Jupiter mass detection limits, our results are expected to be complemented by the planet imaging of Vega in the upcoming observations using the *James Webb* Space Telescope to obtain a more holistic understanding of the planetary system configuration around Vega.

**Key words.** planets and satellites: detection – techniques: high angular resolution – techniques: image processing – planets and satellites: individual: Vega – stars: imaging

## 1. Introduction

Vega ($\alpha$ Lyrae, *Keoe*, *wāqi', Zhinii* 织女一), one of the historical photometric standard stars (Johnson & Morgan 1953), is an A0 V star (e.g., Johnson & Morgan 1953) that is located at $7.68 \pm 0.02$ pc from the Solar System (van Leeuwen 2007). Despite its proximity, Vega is not included in the *Gaia* Catalog of Nearby Stars that are within 100 pc from the Sun, since its brightness exceeds the *Gaia* limits (Gaia Collaboration 2021). With an age of $445 \pm 13$ Myr (Yoon et al. 2010), Vega shows a prototypical mid-infrared excess in the Infrared Astronomical

Satellite observations, situating it in an evolutionary stage that is between star formation and our Solar System (Aumann et al. 1984).

The two-belt debris disk system around Vega may result from planet-disk interactions under various planetary configurations (e.g., Matrà et al. 2020). In fact, most existing directly imaged planets are found in bright debris disk systems (e.g., HR 8799: Marois et al. 2008, $\beta$ Pic: Lagrange et al. 2009, 51 Eri: Macintosh et al. 2015), making it more likely to find giant planets in debris disk systems than around stars without disks (Meshkat et al. 2017). Combined with the observational fact that giant planets are more likely to exist at 1–10 au from their host stars in both direct imaging and radial velocity surveys (Nielsen et al. 2019; Fulton et al. 2021), the proximity

<footnote n="1">¹ Université Côte d'Azur, Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, 06304 Nice, France
e-mail: bin.ren@oca.eu</footnote>

<footnote n="2">² Univ. Grenoble Alpes, CNRS, IPAG, 38000 Grenoble, France</footnote>

<footnote n="3">³ Department of Astronomy, California Institute of Technology, MC 249-17, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="4">⁴ Earth and Planets Laboratory, Carnegie Institution for Science, Washington DC 20015, USA</footnote>

<footnote n="5">⁵ Division of Geological & Planetary Sciences, California Institute of Technology, MC 150-21, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="6">⁶ Department of Earth Sciences, University of Oregon, Eugene OR 97403, USA</footnote>

<footnote n="7">⁷ Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena CA 91109, USA</footnote>

<footnote n="8">⁸ Department of Astronomy & Astrophysics, University of California, Santa Cruz, 1156 High St, Santa Cruz CA 95064, USA</footnote>

<footnote n="9">⁹ Infrared Processing and Analysis Center (IPAC), California Institute of Technology, MC 100-22, 1200 East California Boulevard, Pasadena CA 91125, USA</footnote>

<footnote n="10">¹⁰ American Museum of Natural History, New York NY 12345, USA</footnote>

<footnote n="11">¹¹ Space Telescope Science Institute (STScI), 3700 San Martin Drive, Baltimore MD 21218, USA</footnote>

<footnote n="12">¹² Aix-Marseille Univ., CNRS, CNES, LAM, Marseille, France</footnote>

<footnote>*Contrast curve is only available at the CDS via anonymous ftp to cdsarc.cds.unistra.fr (130.79.128.5) or via https:// cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/670/A162</footnote>
of Vega makes it one of the best systems for giant planet searches.

Before the resolved imaging of debris belts around Vega, dust structures around Vega have suggested the existence of potential planetary perturbers (e.g., Holland et al. 1998; Wilner et al. 2002) and called for the deep imaging of them. Combining the Gemini Altar adaptive optics (AO) system and the Gemini Near-infrared Imager (NIRI), Marois et al. (2006) obtained 5σ detection limits of better than 18 Mag at 3"–10" in the off-methane 1.58 µm 6.5% filter, or ~3 $M_{\text{Jupiter}}$ at 8". Using the MMT Observatory AO and the Clio camera, Hinz et al. (2006) and Heinze et al. (2008) obtained 10σ limits of better than ~12 Mag at 2"–11" at the M band, or ~10 $M_{\text{Jupiter}}$.

Recent observational studies on the Vega planetary system are in direct imaging, transiting, radial velocity, and (sub)millimeter interferometric imaging. In direct imaging, Meshkat et al. (2018) presented observations from the coronagraphic integral field spectrograph P1640 at the Palomar Observatory in J and H bands. Despite a nondetection of planets, they obtained 5σ planet detection limits from 0′.25 to 2′′, reaching a best sensitivity of ≈20 $M_{\text{Jupiter}}$ at 1′.5 or 12 au. From a complimentary approach, using the Atacama Large Millimeter/submillimeter Array (ALMA), Matrà et al. (2020) observed and modeled the outer belt from ~60 au to ~200 au. To explain the observed disk architecture, Matrà et al. (2020) discussed three mechanisms which include a single giant planet, multiple low-mass planets, and no outer planets. Combining a decade of Tillinghast Reflector Echelle Spectrograph (TRES) spectra for radial velocity and two sectors of Transiting Exoplanet Survey Satellite (TESS) photometry, and under the scenario that the planetary orbits have inclinations between 1°.5 and 11°.5, Hurt et al. (2021) obtained a nondetection of 1–10 $M_{\text{Jupiter}}$ planets within 1–10 au, while reporting a candidate Jovian signal with a period of 2.43 days in radial velocity measurements. Nevertheless, depending on the orbit orientations of the planets, the mass limit can vary by up to a factor of ~10 in Fig. 6 of Hurt et al. (2021).

To improve existing high-contrast imaging limits, test different mechanisms for the formation of the Vega debris disk system observed in ALMA, and explore beyond the nearly edge-on limitations from radial velocity and transit studies, we observed Vega in the $M_s$ band using the vortex coronagraph on Keck/NIRC2. In the $M_s$ band, planets have relatively larger planet-to-star brightness ratios than in the J or H band (e.g., Spiegel & Burrows 2012; Skemer et al. 2014), and the advantage of the $M_s$ band over shorter-wavelength bands improves the intermediate system ages for a planet of a given mass (e.g., Currie et al. 2022), both enabling us to explore around Vega for possible cooler and less massive planets than existing studies.

## 2. Observation and data reduction

We observed Vega using the Keck/NIRC2 vortex coronagraph in the $M_s$ band during two individual nights using the narrow camera with a pixel size of 9.942 mas (e.g., Service et al. 2016; Mawet et al. 2019). The first observation was on UT 2018 August 30 under program C314 (PI: D. Mawet), with the total integration time being 1991 s (≈0.181 s × 100 coadd × 110 frames), and the parallactic angle change being 74°.9. The second observation was on UT 2019 August 20 under program N097 (PI: T. Meshkat), with the total integration time being 4500 s (≈0.25 s × 150 coadd × 120 frames), and the parallactic angle change being 81°.6. We list the observation details in Table 1.

<table><caption>**Table 1.** Keck/NIRC2 vortex coronagraph observation log.</caption><thead><tr><th>Target<br>filter</th><th colspan="2">Vega<br>$M_s$ band</th></tr></thead><tbody><tr><td>UT date</td><td>2018 Aug. 30</td><td>2019 Aug. 20</td></tr><tr><td>UT start</td><td>05:15:26.35</td><td>06:35:47.62</td></tr><tr><td>UT end</td><td>07:15:00.52</td><td>10:12:23.95</td></tr><tr><td>Airmass <sup>(a)</sup></td><td>1.066 ± 0.009</td><td>1.159 ± 0.101</td></tr><tr><td>DIMM seeing <sup>(a,b)</sup></td><td>0′.42 ± 0′.11</td><td>1′.18 ± 0′.55</td></tr><tr><td>MASS seeing <sup>(a,b)</sup></td><td>0′.16 ± 0′.06</td><td>0′.18 ± 0′.08</td></tr><tr><td>Precipitable water vapor level</td><td>~7 mm</td><td>~2 mm</td></tr><tr><td>Parallactic angle change</td><td>74°.9</td><td>81°.6</td></tr><tr><td>Single integration time</td><td>0.181 s</td><td>0.25 s</td></tr><tr><td>Coadd for single frame</td><td>100</td><td>150</td></tr><tr><td>Total frame count</td><td>110</td><td>120</td></tr><tr><td>On-target integration time</td><td>1991 s</td><td>4500 s</td></tr><tr><td>Total on-target time</td><td colspan="2">1.803 h</td></tr><tr><td>Pixel scale</td><td colspan="2">9.942 mas</td></tr></tbody></table>

**Notes.** <sup>(a)</sup>The uncertainties in this paper are 1σ unless otherwise specified. <sup>(b)</sup>Calculated during this observation using http://mkwc.ifa.hawaii.edu/current/seeing/index.cgi.

Faint planetary signals are often overwhelmed by the stellar point spread function (PSF) of the central star. In order to remove the stellar PSF and reveal faint surrounding signals, we first pre-processed the data using the VIP package (Gomez Gonzalez et al. 2017), which was further customized for NIRC2 vortex observations by performing flat-fielding, bad pixel and background removal, and image centering (Xuan et al. 2018). We then reduced the preprocessed data using the Karhunen–Loève image projection (KLIP) algorithm (Soummer et al. 2012; Amara & Quanz 2012), which performs principal component analysis to capture the stellar PSF and speckles in the observation. For an image, we removed its stellar PSF and speckles by first projecting it to the KLIP components, and then subtracting the projection from the original image to obtain the residuals. Astrophysical signals including planets and disks thus reside in these residuals. To obtain the final image for each observation, we obtained the residual maps for each individual readout, then rotated and median combined them. We present the combined two-epoch result using 10% of the KLIP components with angular differential imaging (ADI: Marois et al. 2006) in Fig. 1 for demonstration purposes, in which we did not identify point sources that are beyond 5σ of the noise that is within the same angular separation from the stars.

Using VIP and taking into account self-subtraction and overfitting with ADI while using KLIP, we generated the 5σ contrast limits for each observation following Xuan et al. (2018) by varying the reduction parameters to obtain the corresponding detection limits while performing corrections for small sample statistics in Mawet et al. (2014). Specifically, VIP performs injection recovery for companions at different locations to measure the throughput from ADI and KLIP (Gomez Gonzalez et al. 2017). While measuring our throughput, we injected companions along three radial branches spread throughout the image (originating from the masked center) where the averaged throughput at each radial location in the image was determined from these multiple estimates of the throughput at different branches (Xuan et al. 2018). We computed the contrast for the entire image for each combination of inner and outer mask size and number of
of Vega makes it one of the best systems for giant planet searches.

Before the resolved imaging of debris belts around Vega, dust structures around Vega have suggested the existence of potential planetary perturbers (e.g., Holland et al. 1998; Wilner et al. 2002) and called for the deep imaging of them. Combining the Gemini Altar adaptive optics (AO) system and the Gemini Near-infrared Imager (NIRI), Marois et al. (2006) obtained 5σ detection limits of better than 18 Mag at 3"-10" in the off-methane 1.58 µm 6.5% filter, or ~3 $M_{\text{Jupiter}}$ at 8". Using the MMT Observatory AO and the Clio camera, Hinz et al. (2006) and Heinze et al. (2008) obtained 10σ limits of better than ~12 Mag at 2"-11" at the M band, or ~10 $M_{\text{Jupiter}}$.

Recent observational studies on the Vega planetary system are in direct imaging, transiting, radial velocity, and (sub)millimeter interferometric imaging. In direct imaging, Meshkat et al. (2018) presented observations from the coronagraphic integral field spectrograph P1640 at the Palomar Observatory in J and H bands. Despite a nondetection of planets, they obtained 5σ planet detection limits from 0′.25 to 2′, reaching a best sensitivity of ≈20 $M_{\text{Jupiter}}$ at 1′.5 or 12 au. From a complimentary approach, using the Atacama Large Millimeter/submillimeter Array (ALMA), Matrà et al. (2020) observed and modeled the outer belt from ~60 au to ~200 au. To explain the observed disk architecture, Matrà et al. (2020) discussed three mechanisms which include a single giant planet, multiple low-mass planets, and no outer planets. Combining a decade of Tillinghast Reflector Echelle Spectrograph (TRES) spectra for radial velocity and two sectors of Transiting Exoplanet Survey Satellite (TESS) photometry, and under the scenario that the planetary orbits have inclinations between 1°.5 and 11°.5, Hurt et al. (2021) obtained a nondetection of 1–10 $M_{\text{Jupiter}}$ planets within 1–10 au, while reporting a candidate Jovian signal with a period of 2.43 days in radial velocity measurements. Nevertheless, depending on the orbit orientations of the planets, the mass limit can vary by up to a factor of ~10 in Fig. 6 of Hurt et al. (2021).

To improve existing high-contrast imaging limits, test different mechanisms for the formation of the Vega debris disk system observed in ALMA, and explore beyond the nearly edge-on limitations from radial velocity and transit studies, we observed Vega in the $M_s$ band using the vortex coronagraph on Keck/NIRC2. In the $M_s$ band, planets have relatively larger planet-to-star brightness ratios than in the J or H band (e.g., Spiegel & Burrows 2012; Skemer et al. 2014), and the advantage of the $M_s$ band over shorter-wavelength bands improves the intermediate system ages for a planet of a given mass (e.g., Currie et al. 2022), both enabling us to explore around Vega for possible cooler and less massive planets than existing studies.

## 2. Observation and data reduction

We observed Vega using the Keck/NIRC2 vortex coronagraph in the $M_s$ band during two individual nights using the narrow camera with a pixel size of 9.942 mas (e.g., Service et al. 2016; Mawet et al. 2019). The first observation was on UT 2018 August 30 under program C314 (PI: D. Mawet), with the total integration time being 1991 s (≈0.181 s × 100 coadd × 110 frames), and the parallactic angle change being 74°.9. The second observation was on UT 2019 August 20 under program N097 (PI: T. Meshkat), with the total integration time being 4500 s (≈0.25 s × 150 coadd × 120 frames), and the parallactic angle change being 81°.6. We list the observation details in Table 1.

<table><caption>**Table 1.** Keck/NIRC2 vortex coronagraph observation log.</caption><thead><tr><th>Target<br>filter</th><th colspan="2">Vega<br>$M_s$ band</th></tr></thead><tbody><tr><td>UT date</td><td>2018 Aug. 30</td><td>2019 Aug. 20</td></tr><tr><td>UT start</td><td>05:15:26.35</td><td>06:35:47.62</td></tr><tr><td>UT end</td><td>07:15:00.52</td><td>10:12:23.95</td></tr><tr><td>Airmass <sup>(a)</sup></td><td>1.066 ± 0.009</td><td>1.159 ± 0.101</td></tr><tr><td>DIMM seeing <sup>(a,b)</sup></td><td>0′.42 ± 0′.11</td><td>1′.18 ± 0′.55</td></tr><tr><td>MASS seeing <sup>(a,b)</sup></td><td>0′.16 ± 0′.06</td><td>0′.18 ± 0′.08</td></tr><tr><td>Precipitable water vapor level</td><td>~7 mm</td><td>~2 mm</td></tr><tr><td>Parallactic angle change</td><td>74°.9</td><td>81°.6</td></tr><tr><td>Single integration time</td><td>0.181 s</td><td>0.25 s</td></tr><tr><td>Coadd for single frame</td><td>100</td><td>150</td></tr><tr><td>Total frame count</td><td>110</td><td>120</td></tr><tr><td>On-target integration time</td><td>1991 s</td><td>4500 s</td></tr><tr><td>Total on-target time</td><td colspan="2">1.803 h</td></tr><tr><td>Pixel scale</td><td colspan="2">9.942 mas</td></tr></tbody></table>

Notes. (a) The uncertainties in this paper are 1σ unless otherwise specified. (b) Calculated during this observation using http://mkwc.ifa.hawaii.edu/current/seeing/index.cgi.

Faint planetary signals are often overwhelmed by the stellar point spread function (PSF) of the central star. In order to remove the stellar PSF and reveal faint surrounding signals, we first pre-processed the data using the VIP package (Gomez Gonzalez et al. 2017), which was further customized for NIRC2 vortex observations by performing flat-fielding, bad pixel and background removal, and image centering (Xuan et al. 2018). We then reduced the preprocessed data using the Karhunen–Loève image projection (KLIP) algorithm (Soummer et al. 2012; Amara & Quanz 2012), which performs principal component analysis to capture the stellar PSF and speckles in the observation. For an image, we removed its stellar PSF and speckles by first projecting it to the KLIP components, and then subtracting the projection from the original image to obtain the residuals. Astrophysical signals including planets and disks thus reside in these residuals. To obtain the final image for each observation, we obtained the residual maps for each individual readout, then rotated and median combined them. We present the combined two-epoch result using 10% of the KLIP components with angular differential imaging (ADI: Marois et al. 2006) in Fig. 1 for demonstration purposes, in which we did not identify point sources that are beyond 5σ of the noise that is within the same angular separation from the stars.

Using VIP and taking into account self-subtraction and overfitting with ADI while using KLIP, we generated the 5σ contrast limits for each observation following Xuan et al. (2018) by varying the reduction parameters to obtain the corresponding detection limits while performing corrections for small sample statistics in Mawet et al. (2014). Specifically, VIP performs injection recovery for companions at different locations to measure the throughput from ADI and KLIP (Gomez Gonzalez et al. 2017). While measuring our throughput, we injected companions along three radial branches spread throughout the image (originating from the masked center) where the averaged throughput at each radial location in the image was determined from these multiple estimates of the throughput at different branches (Xuan et al. 2018). We computed the contrast for the entire image for each combination of inner and outer mask size and number of
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Combined two-epoch NIRC2 image of Vega in the *M*<sub>s</sub> band using ADI with 10% of the KLIP components for demonstration purposes. We did not identify point sources that were more than 5σ levels beyond the noise of a similar angular separation from the star. The pixel values correspond to lower limits of contrast values due to over- and self-subtraction with KLIP and ADI, respectively.</figcaption></figure>

principal components, where we computed up to 30 principal
components. The ADI reduction was performed with no rotation
gap; for one combination of inner and outer radii from Table 1
of Xuan et al. (2018), the full-frame reduction was performed on
the annulus zone with the region inward of the outer radius or
outward of the inner radius included. For example, for an outer
radius of 0.'5 and an inner radius of 0.'08, all pixels with radial
separations between 8 pixels and 50 pixels from the center of the
image, or between 0.'08 and 0.'5, were included in the reduction
(see the reduction details in Xuan et al. 2018). While we did not
use annular ADI, we did utilize the best contrast achieved from
our full-frame ADI at each one-pixel annulus.

To obtain the final detection limit for each angular sepa-
ration from the star with a step size of 1 pixel, we compared
the detection limits from different combinations of reduction
parameters. The reduction parameters include the frame size
(i.e., algorithmic inner and outer radii) and the number of prin-
cipal components (see Sect. 2 and Table 1 of Xuan et al. (2018)
for details on computing most optimal contrasts using five dif-
ferent frame size combinations). Therefore, while our optimal
contrast is the combination of frames processed using different
parameters, we were optimizing at each radial location. For our
observations, the number of principal components adopted for
the final contrast curve ranges from 8 to 29, with a median of 15
and a standard deviation of 7.4.

The faintest companion that can be detected from such com-
binations at 5σ level was adopted as our final detection limit.
For the 2019 data, the exposures that were used to image the
unblocked central source were saturated, we thus fit the unsatu-
rated first Airy ring to that of a theoretical model of the vortex
stellar PSF (while taking into account the PSF broadening effects
due to weather by convolving a 2D Gaussian distribution), and
used the best-fit model to generate the corresponding contrast
curve. We then combined our detection limits from the two
observations, and present them in Fig. 2. Although the 2018
observation has a shorter total on-target exposure time than

the 2019 one, their total parallactic angle change difference is
only 6.7. With the Differential Image Motion Monitor (DIMM)
seeing of 0.'41 ± 0.'11 in 2018 being more stable than that of
1.'18 ± 0.'55 in 2019, the 2018 data dominate the detection limits
in the combined dataset.

3. Analysis

3.1. Mass detection limits

3.1.1. NIRC2 imaging

Being the photometric standard, Vega's apparent magnitude is
defined as 0 in the *M*<sub>s</sub> band. Adopting a distance of 7.68 ± 0.02 pc
in van Leeuwen (2007), the absolute magnitude for Vega in the
*M*<sub>s</sub> band was 0.573 ± 0.006.

With an age of 445 ± 13 Myr (Yoon et al. 2010) and adopting the
AMES-Cond evolutionary models (Baraffe et al. 2003), we
converted the contrast to 5σ mass detection limits in Fig. 2. We
reached a detection limit of less than 5 M<sub>Jupiter</sub> beyond 9 au, and
3 M<sub>Jupiter</sub> beyond 14 au.

3.1.2. NIRC2 imaging and TRES radial velocity

Combining the 2018 NIRC2 results with the radial velocity data
from TRES in *Hurt et al.* (2021), we followed *Hurt et al.* (2021)
to obtain the mass limits assuming the planetary orbits are well
aligned with the spin axis of Vega. To explore the detectability
of companions from both direct imaging and radial velocity mea-
surements, we randomly generated 10⁶ radial velocity samples of
companion orbits following *Hurt et al.* (2021). The semi-major
axis follows a log-uniform distribution ranging from 0.1 au to
22 au. The companion mass follows a log-uniform distribution
ranging from 0.1 to 100 M<sub>Jupiter</sub>. The sine value of orbital inclina-
tion follows a uniform distribution from 1°.5 to 11°.5. The orbital
eccentricity follows a beta distribution described in *Kipping*
(2013). The argument of periastron follows a uniform distribu-
tion ranging from 0 to 2π radian. The time of periastron passage
follows a uniform distribution which was determined by the
orbital period. The stellar mass follows a Gaussian distribution
using the measurements from *Monnier et al.* (2012). In each
radial velocity sample, we scaled Gaussian noise according to the
uncertainties of the TRES measurements in *Hurt et al.* (2021).
We fit a flat line to each synthetic radial velocity curve using
*RadVal* (Fulton et al. 2018).

For all simulated radial velocity samples, we considered a
synthetic signal to be detectable in radial velocity when its *p*-value was < 0.001 (i.e., 3.3σ), while ignoring correlated noise
(Hurt et al. 2021). At a specific radial separation from the star,
we further required that a companion be detectable when its mass
is above the detection limit in Fig. 2.

We present the detection probability of companions from the
above injection-recovery procedure in Fig. 3. The detectability
of companions from combined radial velocity and direct imag-
ing follows two trends as a function of stellocentric separation.
Interior to ~4 au, the detectable planets increases with radial sep-
aration, approaching planets with ~8 M<sub>Jupiter</sub> down to ~1 M<sub>Jupiter</sub>
at ~0.1 au. Exterior to ~4 au, the detectability is dominated by
NIRC2 imaging, reaching down to ~2 M<sub>Jupiter</sub> at ~22 au. Under
the *Hurt et al.* (2021) framework, there is a possible nonabsolu-
lute detection of companions near ~22 au, which is limited by
the NIRC2 field of view in our study, since the sampled orbital
eccentricity adopted from *Kipping* (2013) can position planets
with a semi-major axis less than ~22 au beyond the 22 au angular
radius.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Combined two-epoch NIRC2 image of Vega in the *M*<sub>s</sub> band using ADI with 10% of the KLIP components for demonstration purposes. We did not identify point sources that were more than 5σ levels beyond the noise of a similar angular separation from the star. The pixel values correspond to lower limits of contrast values due to over- and self-subtraction with KLIP and ADI, respectively.</figcaption></figure>

principal components, where we computed up to 30 principal
components. The ADI reduction was performed with no rotation
gap; for one combination of inner and outer radii from Table 1
of Xuan et al. (2018), the full-frame reduction was performed on
the annulus zone with the region inward of the outer radius or
outward of the inner radius included. For example, for an outer
radius of 0.'5 and an inner radius of 0.'08, all pixels with radial
separations between 8 pixels and 50 pixels from the center of the
image, or between 0.'08 and 0.'5, were included in the reduction
(see the reduction details in Xuan et al. 2018). While we did not
use annular ADI, we did utilize the best contrast achieved from
our full-frame ADI at each one-pixel annulus.

To obtain the final detection limit for each angular sepa-
ration from the star with a step size of 1 pixel, we compared
the detection limits from different combinations of reduction
parameters. The reduction parameters include the frame size
(i.e., algorithmic inner and outer radii) and the number of prin-
cipal components (see Sect. 2 and Table 1 of Xuan et al. (2018)
for details on computing most optimal contrasts using five dif-
ferent frame size combinations). Therefore, while our optimal
contrast is the combination of frames processed using different
parameters, we were optimizing at each radial location. For our
observations, the number of principal components adopted for
the final contrast curve ranges from 8 to 29, with a median of 15
and a standard deviation of 7.4.

The faintest companion that can be detected from such com-
binations at 5σ level was adopted as our final detection limit.
For the 2019 data, the exposures that were used to image the
unblocked central source were saturated, we thus fit the unsatu-
rated first Airy ring to that of a theoretical model of the vortex
stellar PSF (while taking into account the PSF broadening effects
due to weather by convolving a 2D Gaussian distribution), and
used the best-fit model to generate the corresponding contrast
curve. We then combined our detection limits from the two
observations, and present them in Fig. 2. Although the 2018
observation has a shorter total on-target exposure time than

the 2019 one, their total parallactic angle change difference is
only 6.7. With the Differential Image Motion Monitor (DIMM)
seeing of 0.'41 ± 0.'11 in 2018 being more stable than that of
1.'18 ± 0.'55 in 2019, the 2018 data dominate the detection limits
in the combined dataset.

3. Analysis

3.1. Mass detection limits

3.1.1. NIRC2 imaging

Being the photometric standard, Vega's apparent magnitude is
defined as 0 in the *M*<sub>s</sub> band. Adopting a distance of 7.68 ± 0.02 pc
in van Leeuwen (2007), the absolute magnitude for Vega in the
*M*<sub>s</sub> band was 0.573 ± 0.006.

With an age of 445 ± 13 Myr (Yoon et al. 2010) and adopting the
AMES-Cond evolutionary models (Baraffe et al. 2003), we
converted the contrast to 5σ mass detection limits in Fig. 2. We
reached a detection limit of less than 5 M<sub>Jupiter</sub> beyond 9 au, and
3 M<sub>Jupiter</sub> beyond 14 au.

3.1.2. NIRC2 imaging and TRES radial velocity

Combining the 2018 NIRC2 results with the radial velocity data
from TRES in *Hurt et al.* (2021), we followed *Hurt et al.* (2021)
to obtain the mass limits assuming the planetary orbits are well
aligned with the spin axis of Vega. To explore the detectability
of companions from both direct imaging and radial velocity mea-
surements, we randomly generated 10⁶ radial velocity samples of
companion orbits following *Hurt et al.* (2021). The semi-major
axis follows a log-uniform distribution ranging from 0.1 au to
22 au. The companion mass follows a log-uniform distribution
ranging from 0.1 to 100 M<sub>Jupiter</sub>. The sine value of orbital inclina-
tion follows a uniform distribution from 1°.5 to 11°.5. The orbital
eccentricity follows a beta distribution described in *Kipping*
(2013). The argument of periastron follows a uniform distribu-
tion ranging from 0 to 2π radian. The time of periastron passage
follows a uniform distribution which was determined by the
orbital period. The stellar mass follows a Gaussian distribution
using the measurements from *Monnier et al.* (2012). In each
radial velocity sample, we scaled Gaussian noise according to the
uncertainties of the TRES measurements in *Hurt et al.* (2021).
We fit a flat line to each synthetic radial velocity curve using
*RadVal* (Fulton et al. 2018).

For all simulated radial velocity samples, we considered a
synthetic signal to be detectable in radial velocity when its *p*-value was < 0.001 (i.e., 3.3σ), while ignoring correlated noise
(Hurt et al. 2021). At a specific radial separation from the star,
we further required that a companion be detectable when its mass
is above the detection limit in Fig. 2.

We present the detection probability of companions from the
above injection-recovery procedure in Fig. 3. The detectability
of companions from combined radial velocity and direct imag-
ing follows two trends as a function of stellocentric separation.
Interior to ~4 au, the detectable planets increases with radial sep-
aration, approaching planets with ~8 M<sub>Jupiter</sub> down to ~1 M<sub>Jupiter</sub>
at ~0.1 au. Exterior to ~4 au, the detectability is dominated by
NIRC2 imaging, reaching down to ~2 M<sub>Jupiter</sub> at ~22 au. Under
the *Hurt et al.* (2021) framework, there is a possible nonabsolu-
lute detection of companions near ~22 au, which is limited by
the NIRC2 field of view in our study, since the sampled orbital
eccentricity adopted from *Kipping* (2013) can position planets
with a semi-major axis less than ~22 au beyond the 22 au angular
radius.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Detection limits of point sources around Vega in the *M*<sub>s</sub> band. Left: 5σ contrast, in comparison with the JWST/NIRCam F444W on-sky contrast calculated from Carter et al. (2022). Right: 5σ mass detection limit, and expected mass detection limit with NIRCam F444W. We note that the NIRCam F444W values have an assumed identical instrument performance in the JWST/ERS-1386 program (see Sect. 3.3).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Detection probability of point sources as a function of the semi-major axis of point sources for Vega, using a combination of the 2018 NIRC2 observation and the TRES radial velocity measurements from Hurt et al. (2021) (see Sect. 3.1.2). We can reach an ~8 *M*<sub>Jupiter</sub> detection limit at ~4 au, while probing down to ~2–~3 *M*<sub>Jupiter</sub> at 0.1 au and 22 au, respectively.</figcaption></figure>

## 3.2. Disk formation from mass limits

The ALMA observation in Matrà et al. (2020) resolves the outer dust belt of Vega extending from ~60 au beyond ~150 au. To explain the observed planetary system architecture, the authors have analyzed the two scenarios that involve planets: either a

chain of small planets within 70 au with mass ≥6 *M*<sub>⊕</sub>, or a sole ~5 *M*<sub>Jupiter</sub> at 50–60 au.

The combined Keck/NIRC2 *M*<sub>s</sub>-band and TRES result can exclude the existence of ~8 *M*<sub>Jupiter</sub> planets from 0.1 au to 22 au. Despite the fact that with an age of ~400 Myr there is no clear brightness difference between the hot-start and cold-start models (e.g., Spiegel & Burrows 2012), the observations presented here cannot rule out the sole giant planet which resides at 50–60 au in Matrà et al. (2020) for Vega.

For future exploration of the proposed sole giant planet at 50 au to 60 au using Keck/NIRC2 in the *M*<sub>s</sub> band, a field of view that is between 6′.5 and 7′.8 is needed. With a pixel scale of 9.942 mas, this corresponds to a half width between 654 pixels and 785 pixels, which exceeds the current 512 pixel half width of the narrow camera for NIRC2 and such an observation is thus not achievable. Alternatively, NIRC2 offers sampling setups that have pixel sizes of 20 mas or 40 mas, yet these setups are not feasible for *M*<sub>s</sub>-band imaging due to the corresponding increased sky background. Nevertheless, the planned upgrades of the NIRC2 detector electronics may permit faster readouts to enable *M*<sub>s</sub>-band imaging for such purposes.

To explore far separation imaging for Vega using the existing configuration of NIRC2, either offsetting the vortex center away from the center (e.g., the observation of HR 8799 b using the Gemini Planet Imager in Wang et al. 2018) or the NIRC2 narrow camera, or executing observations without the vortex coronagraph, could enable a half width of 785 pixels to test the sole giant planet scenario. With this 1024 × 1024 pixel field of view, the minimum permitted readout time is 0.18 s<fnref n="1" />, which could enable a nearly identical exposure sequence in the 2018 dataset presented here. Given that extra parallactic angle change is needed to cover the entire field of view for these setups, they are beyond the scope of this study.

<footnote n="1">¹ https://www2.keck.hawaii.edu/inst/nirc2/ObserversManual.html#Section2.4</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Detection limits of point sources around Vega in the *M*<sub>s</sub> band. Left: 5σ contrast, in comparison with the JWST/NIRCam F444W on-sky contrast calculated from Carter et al. (2022). Right: 5σ mass detection limit, and expected mass detection limit with NIRCam F444W. We note that the NIRCam F444W values have an assumed identical instrument performance in the JWST/ERS-1386 program (see Sect. 3.3).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Detection probability of point sources as a function of the semi-major axis of point sources for Vega, using a combination of the 2018 NIRC2 observation and the TRES radial velocity measurements from Hurt et al. (2021) (see Sect. 3.1.2). We can reach an ~8 *M*<sub>Jupiter</sub> detection limit at ~4 au, while probing down to ~2–~3 *M*<sub>Jupiter</sub> at 0.1 au and 22 au, respectively.</figcaption></figure>

## 3.2. Disk formation from mass limits

The ALMA observation in Matrà et al. (2020) resolves the outer dust belt of Vega extending from ~60 au beyond ~150 au. To explain the observed planetary system architecture, the authors have analyzed the two scenarios that involve planets: either a

chain of small planets within 70 au with mass ≥6 *M*<sub>⊕</sub>, or a sole ~5 *M*<sub>Jupiter</sub> at 50–60 au.

The combined Keck/NIRC2 *M*<sub>s</sub>-band and TRES result can exclude the existence of ~8 *M*<sub>Jupiter</sub> planets from 0.1 au to 22 au. Despite the fact that with an age of ~400 Myr there is no clear brightness difference between the hot-start and cold-start models (e.g., Spiegel & Burrows 2012), the observations presented here cannot rule out the sole giant planet which resides at 50–60 au in Matrà et al. (2020) for Vega.

For future exploration of the proposed sole giant planet at 50 au to 60 au using Keck/NIRC2 in the *M*<sub>s</sub> band, a field of view that is between 6′.5 and 7′.8 is needed. With a pixel scale of 9.942 mas, this corresponds to a half width between 654 pixels and 785 pixels, which exceeds the current 512 pixel half width of the narrow camera for NIRC2 and such an observation is thus not achievable. Alternatively, NIRC2 offers sampling setups that have pixel sizes of 20 mas or 40 mas, yet these setups are not feasible for *M*<sub>s</sub>-band imaging due to the corresponding increased sky background. Nevertheless, the planned upgrades of the NIRC2 detector electronics may permit faster readouts to enable *M*<sub>s</sub>-band imaging for such purposes.

To explore far separation imaging for Vega using the existing configuration of NIRC2, either offsetting the vortex center away from the center (e.g., the observation of HR 8799 b using the Gemini Planet Imager in Wang et al. 2018) of the NIRC2 narrow camera, or executing observations without the vortex coronagraph, could enable a half width of 785 pixels to test the sole giant planet scenario. With this 1024 × 1024 pixel field of view, the minimum permitted readout time is 0.18 s<sup>1</sup>, which could enable a nearly identical exposure sequence in the 2018 dataset presented here. Given that extra parallactic angle change is needed to cover the entire field of view for these setups, they are beyond the scope of this study.

<footnote n="1">¹ https://www2.keck.hawaii.edu/inst/nirc2/ObserversManual.html#Section2.4</footnote>
### 3.3. Implications for JWST observations

At an age of $445 \pm 13$ Myr for Vega (Yoon et al. 2010), giant planets with several Jupiter masses do not have clear brightness distinction between different formation models in the M band (e.g., Fig. 7 of Spiegel & Burrows 2012). For planets with less than ~5 $M_{Jupiter}$, their brightness is expected to peak at 4 µm to 6 µm (e.g., Fig. 6 of Spiegel & Burrows 2012).

To image planets that are brightest at these wavelengths, we applied the Keck/NIRC2 $M_s$ band which operates at a central wavelength of 4.67 µm with a bandpass of 0.24 µm<fnref n="2" />. In comparison, the NIRCam instrument onboard the James Webb Space Telescope (JWST) can cover nearly half the expected brightest wavelengths with its F444W filter (4 µm to 5 µm)<fnref n="3" />. By reaching a multi-Jupiter mass limit for 10 au to 20 au within this study, and one Jupiter mass or better beyond 20 au with NIRCam (Meshkat et al. 2018, Fig. 5 therein), we can combine Keck/NIRC2 and JWST/NIRCam to reach the deepest planet detection limit to investigate the planetary architecture for the Vega system.

Using the on-sky JWST/NIRCam F444W contrast curve from Carter et al. (2022) in the JWST ERS-1386 program (Hinkley et al. 2022), we calculated the expected on-sky contrast for Vega with NIRCam F440W in Fig. 2 for JWST GTO-1193 observations as follows. Assuming an identical instrument performance, which is an optimistic estimation given that JWST GTO-1193 uses MASK430R since it has a larger inner working angle – and thus lower throughput – than MASK335R in the JWST ERS-1386 observations in Carter et al. (2022), we scaled the exposure times with the SUB320 subarray (i.e., Observations 35 and 36) from the JWST GTO-1193 observations and we recalculated the contrast based on the exposure time difference. We then converted the JWST/NIRCam F444W contrast to point source mass following Carter et al. (2021) for Vega while adopting the same age and apparent magnitude as for NIRC2.

In terms of reaching nominal contrast, the Keck/NIRC2 $M_s$-band observations perform better than NIRCam F440W within 1" and reach a similar quality in exterior regions. In terms of mass detection limits, the Keck/NIRC2 $M_s$-band observations perform systematically better than the NIRCam F444W under the abovementioned optimistic assumptions. In addition, the degradation of contrast close to the transmission near the edges of the coronagraphic masks<fnref n="4" /> from MASK335R in JWST ERS-1386 to MASK430R in JWST GTO-1193 has been ignored. All three aspects demonstrate that the NIRC2 $M_s$-band observations presented here establish the deepest high-contrast imaging exploration of planetary companions for Vega in the probed regions. Nevertheless, we emphasize that NIRCam should be better than NIRC2 for most other systems, since the study presented here should be the best-case scenario for ground-based $M_s$-band imaging due to the brightness of Vega; for other targets that are fainter than Vega, they should have less favorable contrasts due to the relative background levels on the ground from NIRC2.

### 3.4. Planet detection toward 0.1 arcsec

To obtain a more general understanding of planetary existence, the detection of intermediate separation planets near ~0.1' are

expected to yield the most thorough discovery of planets (e.g., Nielsen et al. 2019; Fulton et al. 2021). In fact, for the detection of far-separation planets, existing direct imaging surveys equipped with extreme AO systems have experienced a degradation of contrast close to the central sources (e.g., Nielsen et al. 2019; Vigan et al. 2021; Xie et al. 2022). In comparison, for the detection of close-in planets, existing radial velocity surveys have less completeness for long orbital period planets (e.g., Wittenmyer et al. 2020; Fulton et al. 2021). As a result, in the detection probability map of companions for Vega in Fig. 3, there is a clear gap for middling separations.

To fill the gap, the concept of the vortex fiber nuller (VFN; Ruane et al. 2019; Echeverri et al. 2020) is designed to detect and characterize planets near and within 0.1'. Using nulling interferometry in the near-infrared, VFN suppresses on-axis starlight while retaining off-axis companion light with acceptable loss, which increases the signal-to-noise ratio for companions and thus better enables their detection and characterization. The VFN has already been installed for the phase 2 development of the Keck Planet Imager and Characterizer (KPIC; Jovanovic et al. 2020, Echeverri et al., in prep.) on the Keck Observatory, providing access to planets between 30 and 90 mas in the K band. Furthermore, the limitation of VFN in localizing the companions in their orbit will be further resolved with the concept of photonic lantern nuller (PLN; e.g., Xin et al. 2022). We expect that the application of VFN and PLN will fill the planet detection and characterization gap between direct imaging and radial velocity. For Vega, they will better explore possible hidden planets toward ~0.1' in Fig. 3. Nevertheless, the limitation of VFN in the K band is that it is less sensitive to evolved and cooler giant planets due to the drop in planet brightness in shorter wavelengths.

## 4. Summary

We report 1.8 h of $M_s$-band imaging observations of Vega using the Keck/NIRC2 vortex coronagraph. Despite a nondetection of companions, we have pushed the mass detection limits from existing high-contrast imaging observations in Meshkat et al. (2018) by nearly an order of magnitude smaller (see Fig. 2). Combining the NIRC2 results with an existing radial velocity study using TRES in Hurt et al. (2021), we can confidently rule out companions more massive than ~8 $M_{Jupiter}$ from 0.1 au to 22 au for Vega. Within this range, NIRC2 can reach planets that are less massive than 5 $M_{Jupiter}$ beyond 9 au.

While the NIRC2 observations presented here are sensitive to planets of 5 $M_{Jupiter}$ at 9 au down to 3 $M_{Jupiter}$ at 22 au, it is limited by the field of view and thus cannot be used to test the scenario in Matrà et al. (2020) that one multi-Jupiter mass planet at 50 au to 60 au is responsible for the planetary system architecture. Alternatively, observing without using the vortex coronagraph while performing multipoint dithering<fnref n="5" />, combined with extra parallactic angle change, may provide enough sensitivity to image such a perturber for the Vega system by pushing deeper than Heinze et al. (2008) for these separations.

Combining Keck/NIRC2 in the $M_s$ band and NIRCam in its F444W filter assuming identical performance with JWST from ERS-1386 observations (Hinkley et al. 2022; Carter et al. 2022), we can reach comparable detection limits of companions interior to 20 au. Interior to ~10 au, although companion imaging limits increase from ~5 $M_{Jupiter}$ at 10 au to ~100 $M_{Jupiter}$ at 2 au in Fig. 2, this region is where radial velocity measurements can

<footnote n="2">² https://www2.keck.hawaii.edu/inst/nirc2/filters.html</footnote>

<footnote n="3">³ https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-filters</footnote>

<footnote n="4">⁴ Fig. 3 of https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-coronagraphic-occulting-masks-and-lyot-stops</footnote>

<footnote n="5">⁵ https://www2.keck.hawaii.edu/inst/nirc2/ObserversManual.html#Section3.2.10</footnote>
### 3.3. Implications for JWST observations

At an age of $445 \pm 13$ Myr for Vega (Yoon et al. 2010), giant planets with several Jupiter masses do not have clear brightness distinction between different formation models in the M band (e.g., Fig. 7 of Spiegel & Burrows 2012). For planets with less than ~5 $M_{Jupiter}$, their brightness is expected to peak at 4 µm to 6 µm (e.g., Fig. 6 of Spiegel & Burrows 2012).

To image planets that are brightest at these wavelengths, we applied the Keck/NIRC2 $M_s$ band which operates at a central wavelength of 4.67 µm with a bandpass of 0.24 µm<fnref n="2" />. In comparison, the NIRCam instrument onboard the James Webb Space Telescope (JWST) can cover nearly half the expected brightest wavelengths with its F444W filter (4 µm to 5 µm)<fnref n="3" />. By reaching a multi-Jupiter mass limit for 10 au to 20 au within this study, and one Jupiter mass or better beyond 20 au with NIRCam (Meshkat et al. 2018, Fig. 5 therein), we can combine Keck/NIRC2 and JWST/NIRCam to reach the deepest planet detection limit to investigate the planetary architecture for the Vega system.

Using the on-sky JWST/NIRCam F444W contrast curve from Carter et al. (2022) in the JWST ERS-1386 program (Hinkley et al. 2022), we calculated the expected on-sky contrast for Vega with NIRCam F440W in Fig. 2 for JWST GTO-1193 observations as follows. Assuming an identical instrument performance, which is an optimistic estimation given that JWST GTO-1193 uses MASK430R since it has a larger inner working angle – and thus lower throughput – than MASK335R in the JWST ERS-1386 observations in Carter et al. (2022), we scaled the exposure times with the SUB320 subarray (i.e., Observations 35 and 36) from the JWST GTO-1193 observations and we recalculated the contrast based on the exposure time difference. We then converted the JWST/NIRCam F444W contrast to point source mass following Carter et al. (2021) for Vega while adopting the same age and apparent magnitude as for NIRC2.

In terms of reaching nominal contrast, the Keck/NIRC2 $M_s$-band observations perform better than NIRCam F440W within 1" and reach a similar quality in exterior regions. In terms of mass detection limits, the Keck/NIRC2 $M_s$-band observations perform systematically better than the NIRCam F444W under the abovementioned optimistic assumptions. In addition, the degradation of contrast close to the transmission near the edges of the coronagraphic masks<fnref n="4" /> from MASK335R in JWST ERS-1386 to MASK430R in JWST GTO-1193 has been ignored. All three aspects demonstrate that the NIRC2 $M_s$-band observations presented here establish the deepest high-contrast imaging exploration of planetary companions for Vega in the probed regions. Nevertheless, we emphasize that NIRCam should be better than NIRC2 for most other systems, since the study presented here should be the best-case scenario for ground-based $M_s$-band imaging due to the brightness of Vega; for other targets that are fainter than Vega, they should have less favorable contrasts due to the relative background levels on the ground from NIRC2.

### 3.4. Planet detection toward 0.1 arcsec

To obtain a more general understanding of planetary existence, the detection of intermediate separation planets near ~0.1' are

expected to yield the most thorough discovery of planets (e.g., Nielsen et al. 2019; Fulton et al. 2021). In fact, for the detection of far-separation planets, existing direct imaging surveys equipped with extreme AO systems have experienced a degradation of contrast close to the central sources (e.g., Nielsen et al. 2019; Vigan et al. 2021; Xie et al. 2022). In comparison, for the detection of close-in planets, existing radial velocity surveys have less completeness for long orbital period planets (e.g., Wittenmyer et al. 2020; Fulton et al. 2021). As a result, in the detection probability map of companions for Vega in Fig. 3, there is a clear gap for middling separations.

To fill the gap, the concept of the vortex fiber nuller (VFN; Ruane et al. 2019; Echeverri et al. 2020) is designed to detect and characterize planets near and within 0.1'1. Using nulling interferometry in the near-infrared, VFN suppresses on-axis starlight while retaining off-axis companion light with acceptable loss, which increases the signal-to-noise ratio for companions and thus better enables their detection and characterization. The VFN has already been installed for the phase 2 development of the Keck Planet Imager and Characterizer (KPIC; Jovanovic et al. 2020, Echeverri et al., in prep.) on the Keck Observatory, providing access to planets between 30 and 90 mas in the K band. Furthermore, the limitation of VFN in localizing the companions in their orbit will be further resolved with the concept of photonic lantern nuller (PLN; e.g., Xin et al. 2022). We expect that the application of VFN and PLN will fill the planet detection and characterization gap between direct imaging and radial velocity. For Vega, they will better explore possible hidden planets toward ~0.1'1 in Fig. 3. Nevertheless, the limitation of VFN in the K band is that it is less sensitive to evolved and cooler giant planets due to the drop in planet brightness in shorter wavelengths.

## 4. Summary

We report 1.8 h of $M_s$-band imaging observations of Vega using the Keck/NIRC2 vortex coronagraph. Despite a nondetection of companions, we have pushed the mass detection limits from existing high-contrast imaging observations in Meshkat et al. (2018) by nearly an order of magnitude smaller (see Fig. 2). Combining the NIRC2 results with an existing radial velocity study using TRES in Hurt et al. (2021), we can confidently rule out companions more massive than ~8 $M_{Jupiter}$ from 0.1 au to 22 au for Vega. Within this range, NIRC2 can reach planets that are less massive than 5 $M_{Jupiter}$ beyond 9 au.

While the NIRC2 observations presented here are sensitive to planets of 5 $M_{Jupiter}$ at 9 au down to 3 $M_{Jupiter}$ at 22 au, it is limited by the field of view and thus cannot be used to test the scenario in Matrà et al. (2020) that one multi-Jupiter mass planet at 50 au to 60 au is responsible for the planetary system architecture. Alternatively, observing without using the vortex coronagraph while performing multipoint dithering<fnref n="5" />, combined with extra parallactic angle change, may provide enough sensitivity to image such a perturber for the Vega system by pushing deeper than Heinze et al. (2008) for these separations.

Combining Keck/NIRC2 in the $M_s$ band and NIRCam in its F444W filter assuming identical performance with JWST from ERS-1386 observations (Hinkley et al. 2022; Carter et al. 2022), we can reach comparable detection limits of companions interior to 20 au. Interior to ~10 au, although companion imaging limits increase from ~5 $M_{Jupiter}$ at 10 au to ~100 $M_{Jupiter}$ at 2 au in Fig. 2, this region is where radial velocity measurements can

<footnote n="2">² https://www2.keck.hawaii.edu/inst/nirc2/filters.html</footnote>

<footnote n="3">³ https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-filters</footnote>

<footnote n="4">⁴ Fig. 3 of https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam-instrumentation/nircam-coronagraphic-occulting-masks-and-lyot-stops</footnote>

<footnote n="5">⁵ https://www2.keck.hawaii.edu/inst/nirc2/ObserversManual.html#Section3.2.10</footnote>
provide multi-Jupiter detection limits or better as shown in *Hurt et al. (2021)*. Future works following *Mawet et al. (2019)* and *Llop-Sayson et al. (2021)* in combining measurements including direct imaging here and radial velocity in *Hurt et al. (2021)*, as well as upcoming imaging observations with JWST GTO-1193, in addition to the VFN and PLN concepts to detect planets toward ~0.1, could allow us to obtain the most holistic understanding for the planetary system of this historical photometric standard star – Vega.

**Acknowledgements.** We thank the anonymous referee for their constructive comments that increased the clarity and reproducibility of this paper. This research is partially supported by NASA ROSES XRP, award 80NSSC19K0294. B.B.R. has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (PROTOPLANETS, grant agreement No. 101002188). É.C. has received funding from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (ESCAPE, grant agreement No 101044152). Some of the data presented herein were obtained at the W.M. Keck Observatory, which is operated as a scientific partnership among the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W.M. Keck Foundation. The authors wish to recognize and acknowledge the very significant cultural role and reverence that the summit of Maunakea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain. Part of the computations presented here was conducted in the Resnick High Performance Computing Center, a facility supported by Resnick Sustainability Institute at the California Institute of Technology.

## References

* Amara, A., & Quanz, S. P. 2012, MNRAS, 427, 948

* Aumann, H. H., Gillett, F. C., Beichman, C. A., et al. 1984, ApJ, 278, L23

* Baraffe, I., Chabrier, G., Barman, T. S., Allard, F., & Hauschildt, P. H. 2003, A&A, 402, 701

* Carter, A. L., Hinkley, S., Bonavita, M., et al. 2021, MNRAS, 501, 1999

* Carter, A. L., Hinkley, S., Kammerer, J., et al. 2022, AAS J., submitted [arXiv:2208.14990]

* Currie, T., Biller, B., Lagrange, A.-M., et al. 2022, arXiv e-prints, [arXiv:2205.05696]

* Echeverri, D., Ruane, G., Calvin, B., et al. 2020, SPIE Conf. Ser., 11446, 1144619

* Fulton, B. J., Petigura, E. A., Blunt, S., & Sinukoff, E. 2018, PASP, 130, 044504

* Fulton, B. J., Rosenthal, L. J., Hirsch, L. A., et al. 2021, ApJS, 255, 14

* Gaia Collaboration (Smart, R. L., et al.) 2021, A&A, 649, A6

* Gomez Gonzalez, C. A., Wertz, O., Absil, O., et al. 2017, AJ, 154, 7

* Heinze, A. N., Hinz, P. M., Kenworthy, M., Miller, D., & Sivanandam, S. 2008, ApJ, 688, 583

* Hinkley, S., Carter, A. L., Ray, S., et al. 2022, PASP, 134, 095003

* Hinz, P. M., Heinze, A. N., Sivanandam, S., et al. 2006, ApJ, 653, 1486

* Holland, W. S., Greaves, J. S., Zuckerman, B., et al. 1998, Nature, 392, 788

* Hurt, S. A., Quinn, S. N., Latham, D. W., et al. 2021, AJ, 161, 157

* Johnson, H. L., & Morgan, W. W. 1953, ApJ, 117, 313

* Jovanovic, N., Calvin, B., Porter, M., et al. 2020, SPIE Conf. Ser., 11447, 114474U

* Kipping, D. M. 2013, MNRAS, 434, L51

* Lagrange, A. M., Gratadour, D., Chauvin, G., et al. 2009, A&A, 493, L21

* Llop-Sayson, J., Wang, J. J., Ruffio, J.-B., et al. 2021, AJ, 162, 181

* Macintosh, B., Graham, J. R., Barman, T., et al. 2015, Science, 350, 64

* Marois, C., Lafrenière, D., Doyon, R., Macintosh, B., & Nadeau, D. 2006, ApJ, 641, 556

* Marois, C., Macintosh, B., Barman, T., et al. 2008, Science, 322, 1348

* Matrà, L., Dent, W. R. F., Wilner, D. J., et al. 2020, ApJ, 898, 146

* Mawet, D., Milli, J., Wahhaj, Z., et al. 2014, ApJ, 792, 97

* Mawet, D., Hirsch, L., Lee, E. J., et al. 2019, AJ, 157, 33

* Meshkat, T., Mawet, D., Bryan, M. L., et al. 2017, AJ, 154, 245

* Meshkat, T., Nilsson, R., Aguilar, J., et al. 2018, AJ, 156, 214

* Monnier, J. D., Che, X., Zhao, M., et al. 2012, ApJ, 761, L3

* Nielsen, E. L., De Rosa, R. J., Macintosh, B., et al. 2019, AJ, 158, 13

* Ruane, G., Echeverri, D., Jovanovic, N., et al. 2019, SPIE Conf. Ser., 11117, 1111716

* Service, M., Lu, J. R., Campbell, R., et al. 2016, PASP, 128, 095004

* Skemer, A. J., Marley, M. S., Hinz, P. M., et al. 2014, ApJ, 792, 17

* Soummer, R., Pueyo, L., & Larkin, J. 2012, ApJ, 755, L28

* Spiegel, D. S., & Burrows, A. 2012, ApJ, 745, 174

* van Leeuwen, F. 2007, A&A, 474, 653

* Vigan, A., Fontanive, C., Meyer, M., et al. 2021, A&A, 651, A72

* Wang, J. J., Graham, J. R., Dawson, R., et al. 2018, AJ, 156, 192

* Wilner, D. J., Holman, M. J., Kuchner, M. J., & Ho, P. T. P. 2002, ApJ, 569, L115

* Wittenmyer, R. A., Wang, S., Horner, J., et al. 2020, MNRAS, 492, 377

* Xie, C., Choquet, E., Vigan, A., et al. 2022, A&A, 666, A32

* Xin, Y., Jovanovic, N., Ruane, G., et al. 2022, ApJ, 938, 140

* Xuan, W. J., Mawet, D., Ngo, H., et al. 2018, AJ, 156, 156

* Yoon, J., Peterson, D. M., Kurucz, R. L., & Zagarello, R. J. 2010, ApJ, 708, 71
provide multi-Jupiter detection limits or better as shown in *Hurt et al. (2021)*. Future works following *Mawet et al. (2019)* and *Llop-Sayson et al. (2021)* in combining measurements including direct imaging here and radial velocity in *Hurt et al. (2021)*, as well as upcoming imaging observations with JWST GTO-1193, in addition to the VFN and PLN concepts to detect planets toward ~0.1, could allow us to obtain the most holistic understanding for the planetary system of this historical photometric standard star – Vega.

**Acknowledgements.** We thank the anonymous referee for their constructive comments that increased the clarity and reproducibility of this paper. This research is partially supported by NASA ROSES XRP, award 80NSSC19K0294. B.B.R. has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (PROTOPLANETS, grant agreement No. 101002188). É.C. has received funding from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (ESCAPE, grant agreement No 101044152). Some of the data presented herein were obtained at the W.M. Keck Observatory, which is operated as a scientific partnership among the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W.M. Keck Foundation. The authors wish to recognize and acknowledge the very significant cultural role and reverence that the summit of Maunakea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain. Part of the computations presented here was conducted in the Resnick High Performance Computing Center, a facility supported by Resnick Sustainability Institute at the California Institute of Technology.

## References

* Amara, A., & Quanz, S. P. 2012, MNRAS, 427, 948

* Aumann, H. H., Gillett, F. C., Beichman, C. A., et al. 1984, ApJ, 278, L23

* Baraffe, I., Chabrier, G., Barman, T. S., Allard, F., & Hauschildt, P. H. 2003, A&A, 402, 701

* Carter, A. L., Hinkley, S., Bonavita, M., et al. 2021, MNRAS, 501, 1999

* Carter, A. L., Hinkley, S., Kammerer, J., et al. 2022, AAS J., submitted [arXiv:2208.14990]

* Currie, T., Biller, B., Lagrange, A.-M., et al. 2022, arXiv e-prints, [arXiv:2205.05696]

* Echeverri, D., Ruane, G., Calvin, B., et al. 2020, SPIE Conf. Ser., 11446, 1144619

* Fulton, B. J., Petigura, E. A., Blunt, S., & Sinukoff, E. 2018, PASP, 130, 044504

* Fulton, B. J., Rosenthal, L. J., Hirsch, L. A., et al. 2021, ApJS, 255, 14

* Gaia Collaboration (Smart, R. L., et al.) 2021, A&A, 649, A6

* Gomez Gonzalez, C. A., Wertz, O., Absil, O., et al. 2017, AJ, 154, 7

* Heinze, A. N., Hinz, P. M., Kenworthy, M., Miller, D., & Sivanandam, S. 2008, ApJ, 688, 583

* Hinkley, S., Carter, A. L., Ray, S., et al. 2022, PASP, 134, 095003

* Hinz, P. M., Heinze, A. N., Sivanandam, S., et al. 2006, ApJ, 653, 1486

* Holland, W. S., Greaves, J. S., Zuckerman, B., et al. 1998, Nature, 392, 788

* Hurt, S. A., Quinn, S. N., Latham, D. W., et al. 2021, AJ, 161, 157

* Johnson, H. L., & Morgan, W. W. 1953, ApJ, 117, 313

* Jovanovic, N., Calvin, B., Porter, M., et al. 2020, SPIE Conf. Ser., 11447, 114474U

* Kipping, D. M. 2013, MNRAS, 434, L51

* Lagrange, A. M., Gratadour, D., Chauvin, G., et al. 2009, A&A, 493, L21

* Llop-Sayson, J., Wang, J. J., Ruffio, J.-B., et al. 2021, AJ, 162, 181

* Macintosh, B., Graham, J. R., Barman, T., et al. 2015, Science, 350, 64

* Marois, C., Lafrenière, D., Doyon, R., Macintosh, B., & Nadeau, D. 2006, ApJ, 641, 556

* Marois, C., Macintosh, B., Barman, T., et al. 2008, Science, 322, 1348

* Matrà, L., Dent, W. R. F., Wilner, D. J., et al. 2020, ApJ, 898, 146

* Mawet, D., Milli, J., Wahhaj, Z., et al. 2014, ApJ, 792, 97

* Mawet, D., Hirsch, L., Lee, E. J., et al. 2019, AJ, 157, 33

* Meshkat, T., Mawet, D., Bryan, M. L., et al. 2017, AJ, 154, 245

* Meshkat, T., Nilsson, R., Aguilar, J., et al. 2018, AJ, 156, 214

* Monnier, J. D., Che, X., Zhao, M., et al. 2012, ApJ, 761, L3

* Nielsen, E. L., De Rosa, R. J., Macintosh, B., et al. 2019, AJ, 158, 13

* Ruane, G., Echeverri, D., Jovanovic, N., et al. 2019, SPIE Conf. Ser., 11117, 1111716

* Service, M., Lu, J. R., Campbell, R., et al. 2016, PASP, 128, 095004

* Skemer, A. J., Marley, M. S., Hinz, P. M., et al. 2014, ApJ, 792, 17

* Soummer, R., Pueyo, L., & Larkin, J. 2012, ApJ, 755, L28

* Spiegel, D. S., & Burrows, A. 2012, ApJ, 745, 174

* van Leeuwen, F. 2007, A&A, 474, 653

* Vigan, A., Fontanive, C., Meyer, M., et al. 2021, A&A, 651, A72

* Wang, J. J., Graham, J. R., Dawson, R., et al. 2018, AJ, 156, 192

* Wilner, D. J., Holman, M. J., Kuchner, M. J., & Ho, P. T. P. 2002, ApJ, 569, L115

* Wittenmyer, R. A., Wang, S., Horner, J., et al. 2020, MNRAS, 492, 377

* Xie, C., Choquet, E., Vigan, A., et al. 2022, A&A, 666, A32

* Xin, Y., Jovanovic, N., Ruane, G., et al. 2022, ApJ, 938, 140

* Xuan, W. J., Mawet, D., Ngo, H., et al. 2018, AJ, 156, 156

* Yoon, J., Peterson, D. M., Kurucz, R. L., & Zagarello, R. J. 2010, ApJ, 708, 71