# An ALMA survey of submillimetre galaxies in the Extended Chandra Deep Field South: an unbiased study of SMG environments measured with narrowband imaging

Thomas M Cornish, Julie Wardlow, Heather Wade, David Sobral, W.N Brandt, Pierre Cox, Helmut Dannerbauer, Roberto Decarli, Bitten Gullberg,

Kirsten Knudsen, et al.

► To cite this version:

Thomas M Cornish, Julie Wardlow, Heather Wade, David Sobral, W.N Brandt, et al. An ALMA survey of submillimetre galaxies in the Extended Chandra Deep Field South: an unbiased study of SMG environments measured with narrowband imaging. Monthly Notices of the Royal Astronomical Society, 2024, 533 (2), pp.2399-2419. 10.1093/mnras/stae1868. hal-04679934

HAL Id: hal-04679934

https://hal.science/hal-04679934v1

Submitted on 18 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An ALMA survey of submillimetre galaxies in the Extended Chandra Deep Field South: an unbiased study of SMG environments measured with narrow-band imaging

Thomas M. Cornish<sup>1,2,*</sup> Julie Wardlow<sup>1,★</sup> Heather Wade,<sup>1</sup> David Sobral<sup>1,3</sup> William

N. Brandt<sup>1</sup>,<sup>4,5,6</sup> Pierre Cox<sup>1</sup>,<sup>7</sup> Helmut Dannerbauer<sup>1</sup>,<sup>8,9</sup> Roberto Decarli<sup>1</sup>,<sup>10</sup> Bitten Gullberg<sup>1</sup>,<sup>11,12</sup>

Kirsten Knudsen<sup>1</sup>,<sup>13</sup> John Stott<sup>1</sup>,<sup>1</sup> Mark Swinbank<sup>1</sup>,<sup>14</sup> Fabian Walter<sup>15</sup> and Paul van der Werf<sup>16</sup>

Accepted 2024 July 29. Received 2024 July 26; in original form 2024 January 9

## ABSTRACT

Submillimetre galaxies (SMGs) are some of the most extreme star-forming systems in the Universe, whose place in the framework of galaxy evolution is as yet uncertain. It has been hypothesized that SMGs are progenitors of local early-type galaxies, requiring that SMGs generally reside in galaxy cluster progenitors at high redshift. We test this hypothesis and explore SMG environments using a narrow-band VLT/HAWK-I+GRAAL study of H$\alpha$ and [O III] emitters around an unbiased sample of three ALMA-identified and spectroscopically confirmed SMGs at $z \sim 2.3$ and $\sim 3.3$, where these SMGs were selected solely on spectroscopic redshift. Comparing with blank-field observations at similar epochs, we find that one of the three SMGs lies in an overdensity of emission-line sources on the $\sim 4$ Mpc scale of the HAWK-I field of view, with overdensity parameter $\delta_g = 2.6_{-1.2}^{+1.4}$. A second SMG is significantly overdense only on $\lesssim 1.6$ Mpc scales and the final SMG is consistent with residing in a blank field environment. The total masses of the two overdensities are estimated to be $\log(M_h/M_\odot) = 12.1-14.4$, leading to present-day masses of $\log(M_{h,z=0}/M_\odot) = 12.9-15.9$. These results imply that SMGs occupy a range of environments, from overdense protoclusters or protogroups to the blank field, suggesting that while some SMGs are strong candidates for the progenitors of massive elliptical galaxies in clusters, this may not be their only possible evolutionary pathway.

**Key words:** galaxies: evolution – galaxies: photometry – galaxies: star formation – submillimetre: galaxies.

## 1 INTRODUCTION

Since their discovery more than two decades ago, submillimetre galaxies (SMGs; e.g. Smail, Ivison & Blain 1997; Barger et al. 1998; Hughes et al. 1998; Eales et al. 1999; Blain et al. 2002; Coppin et al. 2006; Casey, Narayanan & Cooray 2014; Hodge & da

Cunha 2020) have proven to be important laboratories for exploring galaxy formation and evolution. These galaxies are identified in (sub)millimetre surveys and have typical infrared (IR) luminosities of $L_{IR} \sim 10^{12-13} L_\odot$ corresponding to star formation rates (SFRs) of $\sim 10^{2-3} M_\odot \text{ yr}^{-1}$ (e.g. Chapman et al. 2005; Pope et al. 2006; Wardlow et al. 2011; Magnelli et al. 2012; Swinbank et al. 2014; MacKenzie et al. 2017; Michałowski et al. 2017; Rowan-Robinson et al. 2018; Cheng et al. 2019; Greenslade et al. 2020). They are massive, with stellar masses of $M_* \sim 10^{10-11} M_\odot$ (e.g. Hainline et al. 2011; Michałowski et al. 2012; Gruppioni et al. 2013; Simpson et al.

<footnote n="1"><sup>1</sup>Department of Physics, Lancaster University, Lancaster, LA1 4YB, UK</footnote>

<footnote n="2"><sup>2</sup>Department of Physics, University of Oxford, Denys Wilkinson Building, Keble Road, Oxford OX1 3RH, UK</footnote>

<footnote n="3"><sup>3</sup>BNP Paribas Corporate and Institutional Banking, Torre Ocidente Rua Galileu Galilei, 1500-392 Lisbon, Portugal</footnote>

<footnote n="4"><sup>4</sup>Department of Astronomy and Astrophysics, The Pennsylvania State University, 525 Davey Lab, University Park, PA 16802, USA</footnote>

<footnote n="5"><sup>5</sup>Institute for Gravitation and the Cosmos, The Pennsylvania State University, University Park, PA 16802, USA</footnote>

<footnote n="6"><sup>6</sup>Department of Physics, The Pennsylvania State University, University Park, PA 16802, USA</footnote>

<footnote n="7"><sup>7</sup>Institut d'Astrophysique de Paris, Sorbonne Université, CNRS UMR 7095, 98bis bvd Arago, Paris F-75014, France</footnote>

<footnote n="8"><sup>8</sup>Instituto de Astrofísica de Canarias (IAC), E-38205 La Laguna, Tenerife, Spain</footnote>

<footnote n="9"><sup>9</sup>Departamento de Astrofísica, Universidad de La Laguna, E-38206 La Laguna, Tenerife, Spain</footnote>

<footnote n="10"><sup>10</sup>INAF—Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Gobetti 93/3, Bologna I-40129, Italy</footnote>

<footnote n="11"><sup>11</sup>DTU-Space, Technical University of Denmark, Elektrovej 327, DK-2800 Kgs. Lyngby, Denmark</footnote>

<footnote n="12"><sup>12</sup>Cosmic Dawn Centre (DAWN), Denmark</footnote>

<footnote n="13"><sup>13</sup>Department of Space, Earth and Environment, Chalmers University of Technology, Gothenburg SE-412 96, Sweden</footnote>

<footnote n="14"><sup>14</sup>Centre for Extragalactic Astronomy, Department of Physics, Durham University, South Road, Durham DH1 3LE, UK</footnote>

<footnote n="15"><sup>15</sup>Max-Planck-Institut für Astronomy, Königstuhl 17, Heidelberg D-69117, Germany</footnote>

<footnote n="16"><sup>16</sup>Leiden Observatory, Leiden University, RA Leiden PO Box 9513, NL-2300, The Netherlands</footnote>

<footnote>* E-mail: thomas.cornish@physics.ox.ac.uk (TMC); j.wardlow@lancaster.ac.uk (JW)</footnote>
2014; da Cunha et al. 2015; Dudzevičiūtė et al. 2020; Pantoni et al. 2021), dust masses of $\gtrsim 10^8$M$_\odot$ (e.g. Clements, Dunne & Eales 2010; Miettinen et al. 2017; Dudzevičiūtė et al. 2020; Pantoni et al. 2021), and cold gas masses of $\sim 10^{11}$M$_\odot$ (e.g. Greve et al. 2005; Tacconi et al. 2006; Bothwell et al. 2013; Birkin et al. 2021), but with gas depletion times of just a few hundred Myr (e.g. Tacconi et al. 2006; Birkin et al. 2021). The SMG redshift distribution peaks at $z \sim 2.5$ (e.g. Chapman et al. 2005; Wardlow et al. 2011; Koprowski et al. 2014; Danielson et al. 2017; Smith et al. 2017; Stach et al. 2019; Dudzevičiūtė et al. 2020; da Cunha et al. 2021), making these massive dusty galaxies the most intense star-forming systems in the Universe during its peak epoch of star formation (Madau & Dickinson 2014). SMGs contribute up to $\sim 20$ per cent of the cosmic SFR density at $z \sim 2$ (e.g. Coppin et al. 2006; Barger et al. 2012; Swinbank et al. 2014).

The extreme properties of SMGs have long made them a good test of galaxy evolution models (e.g. Baugh et al. 2005; Lacey et al. 2008, 2010; Davé et al. 2010; Narayanan et al. 2010, 2015; Béthermin et al. 2011; Niemi et al. 2012; Hayward et al. 2021; Lovell et al. 2021), yet questions about their evolution and role in the evolution of other galaxies remain. SMGs have similar properties to those expected of the progenitors of local massive elliptical galaxies, which formed most of their stars in short bursts at $z \gtrsim 2$ (Ellis et al. 1997; Blakeslee et al. 2003). Indeed, the dust emission from SMGs is typically compact (e.g. Hodge et al. 2016; Gullberg et al. 2019), which is consistent with a scenario in which a gas-rich $z \gtrsim 2$ galaxy undergoes a compact starburst, leading to a compact quiescent galaxy, which eventually evolves into a local elliptical galaxy (Simpson et al. 2014; Toft et al. 2014; Ikarashi et al. 2015). Since local ellipticals are predominantly found in galaxy clusters (e.g. Dressler 1980) then if SMGs are indeed a progenitor phase in their formation, then it is expected that SMGs should reside in early galaxy clusters, or ‘protoclusters’, at $z \gtrsim 2$.

Galaxy protoclusters (for a review, see Overzier 2016) are typically defined as structures that will collapse and virialize to form a galaxy cluster by $z=0$. Simulations have shown that in a Lambda cold dark matter (ΛCDM) universe, protoclusters form hierarchically at the highest-density regions of the matter distribution in the universe (the ‘cosmic web’; Bond, Kofman & Pogosyan 1996) at $z \sim 4-6$ (e.g. Baugh et al. 1998; De Lucia & Blaizot 2007). As such, protoclusters are characterized by overdensities of galaxies relative to the average galaxy density in the coeval blank field. Conversely to their present-day descendants, galaxies in a protocluster are generally not bound to a single halo; they instead occupy large structures extended over megaparsec (Mpc) scales, with the main halo containing as little as 20 per cent of the member galaxies (e.g. Chiang, Overzier & Gebhardt 2013; Muldrew, Hatch & Cooke 2015).

Unfortunately, observationally identifying protoclusters is challenging. Methods of detecting galaxy clusters from their X-ray emission (e.g. Trümper 1993; Böhringer et al. 2001; Henry et al. 2006; Pacaud et al. 2016) or by searching for their imprint on the cosmic microwave background at millimetre wavelengths (e.g. Staniszewski et al. 2009; Williamson et al. 2011; Hasselfield et al. 2013; Bleem et al. 2015; Planck Collaboration XXVII 2016) via the Sunyaev–Zel’dovich effect (Sunyaev & Zeldovich 1972) are rendered impractical due to the lack of a hot intracluster medium (ICM). Similarly, searches for high concentrations of passive early-type galaxies occupying a tight ‘red sequence’ in colour–magnitude space (e.g. Gladders & Yee 2000, 2005; Muzzin et al. 2009; Wilson et al. 2009; Gilbank et al. 2011) become ineffective since the stellar populations of galaxies in protoclusters typically have not

evolved sufficiently for a significant 4000 Å break to be detected. Consequently, the majority of protocluster surveys resort to searching for overdensities of galaxies at high redshift. Such searches depend on the existence of accurate redshift information across large cosmological volumes, and several protoclusters have been discovered serendipitously through large spectroscopic surveys (e.g. Steidel et al. 1998, 2000, 2005; Cucciati et al. 2014; Lemaux et al. 2014). In lieu of expensive large-scale spectroscopic observations, an alternative method is to use wide-field narrow-band photometric surveys to search for overdensities of galaxies with strong emission lines at a particular observed-frame wavelength (e.g. $Ly\alpha$ or $H\alpha$ emitters; Venemans et al. 2002, 2005; Shimasaku et al. 2003; Matsuda et al. 2004, 2011; Palunas et al. 2004; Hatch et al. 2011; Tanaka et al. 2011; Kuiper et al. 2011b; Hayashi et al. 2012; Koyama et al. 2013; Zheng et al. 2021).

Whether SMGs commonly inhabit protoclusters or protocluster-like environments is as yet uncertain. Several examples of SMGs residing in protoclusters have been documented (e.g. Ivison et al. 2000, 2013; Smail et al. 2003; Geach et al. 2005; Daddi et al. 2009; Matsuda et al. 2011; Casey et al. 2015), but these systems were selected for detailed follow-up because of prior evidence of high galaxy densities. That is, they comprise a highly biased subset and therefore cannot be used to make inferences about the general SMG population.

Clustering studies have been used to obtain statistical measurements indicative of the whole SMG population. Results from single-dish clustering measurements suggest that on average SMGs at $z \sim 2.5$ reside in dark matter haloes of mass $\sim 10^{13}$M$_\odot$ (e.g. Hickox et al. 2012; Wilkinson et al. 2017). This is marginally lower than expected for the progenitors of massive ellipticals, and implies that SMGs are instead more likely to evolve into $2-3L^*$ galaxies in groups and small clusters. However, these halo mass measurements have typical uncertainties of $\sim 0.5$ dex due to the difficulties associated with obtaining accurate photometric redshifts for SMGs. Furthermore, these clustering measurements relied on the statistical identification of optical/near-IR counterparts to submillimetre sources detected in low-resolution single-dish surveys, which are incorrect in $\sim 30$ per cent of cases and incomplete in a further $\sim 30$ per cent (e.g. Hodge et al. 2013a). More recently, García-Vergara et al. (2020) and Stach et al. (2021) measured the clustering of SMGs which had been followed up interferometrically with the Atacama Large Millimetre/submillimetre Array (ALMA). Using a small sample of 50 ALMA-identified SMGs with spectroscopic redshifts and employing forward modelling to correct for incompleteness, García-Vergara et al. (2020) estimated halo masses that are a factor of $\sim 3.8$ lower than other studies of SMGs. From a significantly larger parent sample, Stach et al. (2021) selected a complete sample of $\sim 350$ ALMA-identified SMGs with photometric redshifts to measure halo masses consistent with the results from the single-dish studies, particularly at $z > 2$ (Hickox et al. 2012; Wilkinson et al. 2017). Overall, the picture from clustering measurements is complex, and differing results from different studies may be methodological, due to sample selection or cosmic variance. Other ways of measuring the environments of SMGs are therefore required.

Statistical photometric redshifts have identified galaxy overdensities around $\sim 5-60$ per cent of SMGs (e.g. Davies et al. 2014; Simpson et al. 2014; Smolčić et al. 2017), but these are subject to significant selection biases (e.g. see section 6 in Smolčić et al. 2017), and few overdensities have been spectroscopically confirmed. Instead, in order to determine the nature of a ‘typical’ SMG environment, and thus confirm whether SMGs really are the progenitors of massive elliptical
galaxies in local clusters, we need targeted observations of individual SMGs, but with no prior knowledge of their environments in order to avoid biases. To this end, we have conducted a wide-field narrow-band survey of the environments of three SMGs identified as part of the ALMA LESS project (ALESS; Hodge et al. 2013a). ALESS conducted ALMA follow-up observations of submillimetre sources previously detected in the LABOCA ECDFS Submillimetre Survey (LESS; Weiß et al. 2009), which in turn probed the Extended Chandra Deep Field-South (ECDFS) field with the Large APEX Bolometer Camera (LABOCA) on the Atacama Pathfinder Experiment (APEX) telescope. With our narrow-band survey, we search for overdensities of Hα or [O III] emitters around these three SMGs to assess whether they reside in protocluster-like environments. The target SMGs were selected on the basis of redshift only, and with no prior information about their environments. Our method is similar to that employed by Matsuda et al. (2011), which combined narrow-band photometry with submillimetre observations to identify an overdensity of Hα emitters around two SMGs in SSA 13. However, the SMGs targeted by Matsuda et al. (2011) were already known to be closely grouped with each other and three optically faint radio galaxies. Our study is the first to perform such an analysis around a sample of SMGs that are selected without prior knowledge of their environment.

The structure of this paper is as follows: in Section 2, we describe the SMG sample selection, our observations and data reduction, and the identification of candidate companion galaxies for each target SMG; Section 3 includes our main results, analysis and discussion; and our conclusions are presented in Section 4. Throughout this paper, we use a ΛCDM cosmology with Ωm,0 = 0.315, ΩΛ,0 = 0.685, and H0 = 67.4 km s⁻¹ Mpc⁻¹ (Planck Collaboration VI 2020). All magnitudes are presented in the AB system, where a 1 µJy source has a magnitude of 23.9 (Oke & Gunn 1983).

## 2 OBSERVATIONS AND GALAXY IDENTIFICATION

In this study, we use the High Acuity Wide-field K-band Imager (HAWK-I; Pirard et al. 2004; Casali et al. 2006; Kissler-Patig et al. 2008; Siebenmorgen et al. 2011) on the Very Large Telescope (VLT) to investigate the environments of three ALMA-identified SMGs from ALESS. As part of ALESS, extensive follow-up was undertaken to obtain spectroscopic redshifts of the SMGs (Danielson et al. 2017; Birkin et al. 2021), which enables a search for galaxies that share environments with these submillimetre sources. The wide-field imaging capabilities and narrow-band photometric filters of HAWK-I allow for an efficient emission-line survey of their environments, which are expected to span physical scales on the order of several Mpc if consistent with being protoclusters (e.g. Chiang et al. 2013; Muldrew et al. 2015; Yajima et al. 2022).

### 2.1 Sample selection

The blank-field LESS observed 0.5 × 0.5 in ECDFS with APEX/LABOCA and detected 126 sources at >3.7σ at 870 µm (Weiß et al. 2009). Each of these sources was followed up with ALMA to yield the 131 ALESS sources described in Hodge et al. (2013a), divided into a main catalogue of 99 SMGs and a supplementary catalogue of 32 SMGs. The SMGs in the main catalogue all lay within the ALMA primary beam full width at half-maximum (FWHM) of the highest-quality maps, while those in the supplementary catalogue were either extracted from outside the primary beam or from lower-quality maps (Hodge et al. 2013a).

<table><caption>**Table 1.** Details of each of the three SMGs included in our sample.</caption><thead><tr><th>SMG</th><th>z<sub>spec</sub></th><th>S<sub>870</sub> (mJy)<sup>a</sup></th><th>K<sub>s</sub><sup>b</sup> (mag)</th><th>Target line<sup>c</sup></th></tr></thead><tbody><tr><td>ALESS 5.1</td><td>3.303<sup>d</sup></td><td>7.8 ± 0.7</td><td>19.79 ± 0.01</td><td>[O III]λ5007</td></tr><tr><td>ALESS 75.2</td><td>2.294<sup>e</sup></td><td>5.0 ± 1.2</td><td>20.67 ± 0.01</td><td>Hα</td></tr><tr><td>ALESS 102.1</td><td>2.296<sup>f</sup></td><td>3.1 ± 0.5</td><td>21.07 ± 0.08</td><td>Hα</td></tr></tbody></table>

*Notes.*<sup>a</sup> Primary-beam-corrected ALMA 870 µm flux densities from Hodge et al. (2013a).

<sup>b</sup> From Simpson et al. (2014).

<sup>c</sup> The emission line used in this study to identify companion galaxies for each SMG, exploiting the fact that these lines shift into the wavelength coverage of the HAWK-I Brγ filter at the redshifts of the SMGs (see Section 2.1).

<sup>d</sup> Obtained via detection of the CO(4–3) and [C II] emission lines (Birkin et al. 2021).

<sup>e</sup> Based on Hα+[N II] and Section II detections (Danielson et al. 2017).

<sup>f</sup> Determined using a combination of Lyα, C III] and continuum measurements (Danielson et al. 2017).

Spectroscopic redshifts were obtained for 52 of the 131 ALESS SMGs by Danielson et al. (2017), and targets for our study are selected from these 52 ALMA-identified and spectroscopically confirmed SMGs. We require SMGs with spectroscopic redshifts that shift the Hα or [O III]λ5007 emission lines into the wavelength coverage of the HAWK-I Brγ filter; this requires that the SMGs are located at z = 2.299 ± 0.023 or 3.324 ± 0.060. Of the 52 ALMA-identified SMGs with spectroscopic redshifts from Danielson et al. (2017), five (ALESS 6.1, 75.2, 87.1, 102.1, and 112.1) have spectroscopic redshifts within the desired range for Hα. A sixth SMG (ALESS 5.1) has a CO-derived spectroscopic redshift of z = 3.303 (Birkin et al. 2021), which places [O III] in the Brγ coverage.

These six SMGs were the proposed targets for observations in four HAWK-I pointings (PID: 0103.A–0668). The six SMGs were selected purely based on their spectroscopic redshifts, with no consideration of their environments. Of the four pointings, only two were observed during the service-mode observations and the choice of pointings was random. The two observed pointings contain three of the six proposed targets: ALESS 5.1, ALESS 75.2, and ALESS 102.1, whose spectroscopic redshifts are z = 3.303, 2.294, and 2.296, respectively. Details of these three targeted SMGs are provided in Table 1. A total of 16 other ALESS SMGs lie within the two HAWK-I pointings, but these are not considered in this study as their redshifts are such that no bright emission lines are expected in the narrow-band filter. Indeed, those that are detected in our HAWK-I observations fail to meet our criteria for being emission-line galaxies (see Section 2.4 and Fig. 1).

### 2.2 HAWK-I data

Each pointing was imaged using the HAWK-I instrument (Pirard et al. 2004; Casali et al. 2006; Kissler-Patig et al. 2008; Siebenmorgen et al. 2011) on the VLT in the Ks (central wavelength: λc = 2.146µm; FWHM: Δλ = 0.324µm) and Brγ (λc = 2.165µm; Δλ = 0.030µm) filters (Kissler-Patig et al. 2008). The FWHM of the Brγ filter is equivalent to Δz = 0.046 (0.060) at z ~ 2.3 (3.3), corresponding to a velocity width of Δv = 4200 km s⁻¹. All three observing blocks (OBs) for the field containing ALESS 5.1 and 75.2 (hereafter Pointing 5+75) were executed on 2019 August 21, while the OBs for the field containing ALESS 102.1 (hereafter Pointing 102) were split among three separate nights from 2019 August 21 to 2020 January 01. The total exposure times for Pointing 5+75 (Pointing 102) were 7.2 ks (6.6 ks) and 900 s (660 s) in the Brγ and Ks filters, respectively. Individual exposures of 120 s (Brγ) and 30 s (Ks)
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Colour–magnitude diagrams demonstrating the criteria described in Section 2.4 for the selection of candidate narrow-band emitters in the HAWK-I pointings containing ALESS 5.1 and 75.2 (left), and ALESS 102.1 (right). All sources detected in the Brγ observations are shown and sources identified as narrow-band emitters are highlighted. Open symbols represent candidate narrow-band emitters which were removed from the sample following visual checks. Sources that are detected in Brγ but are undetected in our Ks data and have no Ks photometry in S14, are shown as lower limits. The Σ = 3 curve for the average properties and the observed-frame EW cut for each field are shown. The solid horizontal line marks a Ks–Brγ colour of zero. The target SMGs and other ALESS SMGs with counterparts in our HAWK-I data are highlighted. Two of the target SMGs (ALESS 5.1 and 75.2) and several other ALESS SMGs in these fields are not detected and are therefore not shown. The black cross in the bottom left corner of each panel shows the mean uncertainties in colour and Brγ magnitude.</figcaption></figure>

<table><caption>**Table 2.** Limiting 3σ AB magnitudes and resolution for each stacked image. Limiting magnitudes are measured using randomly placed 1.25 arcsec diameter apertures. Due to variation in the four HAWK-I detector chips, each quadrant is considered separately.</caption><thead><tr><th rowspan="2">Pointing</th><th rowspan="2">Quadrant<sup>a</sup></th><th colspan="2"><i>m</i><sup>3σ</sup><sub>lim</sub></th><th colspan="2">PSF (arcsec)</th></tr><tr><th>Brγ</th><th>K<sub>s</sub></th><th>Brγ</th><th>K<sub>s</sub></th></tr></thead><tbody><tr><td rowspan="4">Pointing 5+75</td><td>Q1</td><td>24.29</td><td>24.01</td><td>0.28</td><td>0.27</td></tr><tr><td>Q2</td><td>24.19</td><td>24.27</td><td>0.27</td><td>0.27</td></tr><tr><td>Q3</td><td>24.30</td><td>24.19</td><td>0.35</td><td>0.26</td></tr><tr><td>Q4</td><td>24.22</td><td>24.27</td><td>0.32</td><td>0.26</td></tr><tr><td rowspan="4">Pointing 102</td><td>Q1</td><td>24.09</td><td>23.85</td><td>0.38</td><td>0.30</td></tr><tr><td>Q2</td><td>24.08</td><td>24.00</td><td>0.37</td><td>0.30</td></tr><tr><td>Q3</td><td>24.19</td><td>24.01</td><td>0.44</td><td>0.30</td></tr><tr><td>Q4</td><td>24.11</td><td>24.09</td><td>0.41</td><td>0.29</td></tr></tbody></table>

Note.ª Quadrants are assigned the same labels as in Kissler-Patig et al. (2008).

were taken using the 'HAWKI img obs AutoJitter' template, with five random dither positions within a 20 arcsec box for each filter in each OB. Each pointing covers a 7.5 arcmin × 7.5 arcmin area, except for a cross-shaped gap of width 15 arcsec between the detector's four 2k × 2k Hawaii 2RG arrays. Using HAWK-I's GRAAL system (GRound layer Adaptive optics system Assisted by Lasers; Arsenault et al. 2008; Paufique et al. 2010), we achieved point spread functions (PSFs) of ~0.4 arcsec in Brγ and ~0.3 arcsec in Ks (see Table 2).

The data were reduced using a custom PYTHON-based pipeline, with each of the four detector chips treated separately. Briefly, the pipeline begins by dark-subtracting the data and subsequently using twilight flats to perform flat-fielding. We then use SExTRACTOR (Bertin & Arnouts 1996) to detect sources in each of the flattened frames and produce individual masks. A final flat field is produced for

each frame by median combining all masked frames from the same OB except the frame being flattened; the frames are then flattened using their unique final flat fields. The astrometry of each flattened frame is then calibrated by using SCAMP (Bertin 2006) to match our detected sources with sources detected in a reference K-band image, correcting for any distortions across the field of view by fitting a third-order polynomial. The reference images used for Pointing 5+75 and Pointing 102 were taken from the Taiwan ECDFS NIR Survey (TENIS; Hsieh et al. 2012) and the MUltiwavelength Survey by Yale-Chile (MUSYC; Gawiser et al. 2006b; Taylor et al. 2009), respectively. Different reference images were required for each pointing because while TENIS is deeper and has higher resolution, roughly a quarter of the Pointing 102 field of view lies outside of the TENIS coverage. Finally, the astrometrically corrected frames were median combined using SWARP (Bertin 2010). The resultant stacks in both bands were photometrically calibrated using MUSYC Ks data (Taylor et al. 2009; Simpson et al. 2014) such that they all had a zero-point magnitude of 30.0 mag and ensure a median (Ks–Brγ) colour of 0.

Source detection and photometry were conducted using SExTRACTOR (Bertin & Arnouts 1996) operating in dual-image mode; the stacked Brγ images were used to identify the positions of sources, and then photometry was extracted at these positions in both the Brγ and Ks images to ensure that any difference between the measured Brγ and Ks photometry is a purely intrinsic property of the sources and not caused by positional offsets. After masking noisy regions near the edges of the stacked images, we detected a total of 2175 sources in Pointing 5+75 and 1754 in Pointing 102. Apertures with a diameter of 1.25 arcsec were used, as this is large enough to contain the majority of the flux for all detected sources while minimizing the amount of additional background noise captured. The 3σ limiting
magnitudes measured in these 1.25 arcsec diameter apertures are provided for each filter and each HAWK-I detector chip in Table 2. To account for the variation in size of the detected sources, we then estimated total magnitudes in each filter by selecting all bright ($m_{K_s} < 19.5$) sources and (for each pointing separately) calculating the median difference between their fixed-aperture magnitudes and their magnitudes measured by SExTRACTOR in adaptively scaled (Kron 1980) apertures (MAG_AUTO; Bertin & Arnouts 1996); this difference was then added to the fixed-aperture magnitudes of all sources in the pointing to obtain their total magnitudes. The choice to use the same aperture size for all sources and apply a correction (as opposed to simply using the MAG_AUTO values) ensures that estimates of the total magnitudes are self-consistent whilst also closely matching existing photometry in the same band.

## 2.3 Ancillary data

There exists a wealth of photometric data in the ECDFS, which supplements our HAWK-I photometry. Archival TENIS (Hsieh et al. 2012), MUSYC (Taylor et al. 2009), and HAWK-I (Zibetti, private communication) $K_s$ data were collated by Simpson et al. (2014, hereafter S14) and then used to calibrate our astrometry and photometry (see Section 2.2).

In Sections 2.5 and 3.4, we fit spectral energy distributions (SEDs) to galaxies in our sample in order to first derive properties such as photometric redshift, stellar mass, and SFR. For the photometric redshifts, we use EAZY-PY<fnref n="1" /> – an updated version of the photometric redshift code EAZY (Brammer, van Dokkum & Coppi 2008) written in PYTHON (see Section 2.5) – while for the other galaxy properties we use MAGPHYS (da Cunha, Charlot & Elbaz 2008, see Section 3.4). To this end, we also make use of existing ECDFS images spanning the ultraviolet (UV) to mid-infrared (MIR; see Section 2.5). These images were sourced either from the MUSYC 2010 Public Data Release (Cardamone et al. 2010) or from TENIS (Hsieh et al. 2012). The MUSYC data set consists of $UU_{38}BVRI$ broad-band images from the Wide Field Imager (WFI) on the Max Planck Gesellschaft/European Southern Observatory (MPG/ESO) 2.2-m telescope (Hildebrandt et al. 2006); 5000 Å narrow-band and $z'$ broad-band imaging from the Mosaic-II camera on the Cerro Tololo Inter-American Observatory (CTIO) Blanco 4-m telescope (Gawiser et al. 2006a, b), with $JK_s$ broad-band imaging from the Infrared Sideport Imager on the same telescope (Taylor et al. 2009); 18 medium-band (IA427, IA445, IA464, IA484, IA505, IA527, IA550, IA574, IA598, IA624, IA651, IA679, IA709, IA738, IA767, IA797, IA827, and IA856) images taken with the Subaru telescope's Suprime-Cam (Cardamone et al. 2010); and Spitzer/Infrared Array Camera (IRAC) images at 3.6, 4.5, 5.8, and 8.0 µm (Cardamone et al. 2010; Damen et al. 2011).<fnref n="2" />

We also make use of spectroscopic redshifts in the ECDFS from studies whose areas overlap with our pointings, obtained from publicly available composite catalogues<fnref n="3" />,<fnref n="4" /> (Silverman et al. 2010). The spectroscopic redshifts used are from the VISIBLE MultiObject Spectrograph (VIMOS) VLT Deep Survey (VVDS; Le Fèvre et al. 2005); the VIMOS survey of the Great Observatories Origins Deep

Survey (GOODS) field (GOODS/VIMOS; Popesso et al. 2009; Balestra et al. 2010); the Extended Chandra Deep Field-South Survey (Silverman et al. 2010); and Treister et al. (2009). Additionally we utilize the results of the spectroscopic study conducted as part of ALESS by Danielson et al. (2017).

Finally, we make use of the Lehmer et al. (2005) Chandra point source catalogue for the identification of AGN (active galactic nuclei) in our final sample (see Section 2.5).

## 2.4 Emission-line galaxy selection

Star-forming galaxies at the same redshifts as our target SMGs ($z \sim 2.295 \pm 0.023$ for ALESS 75.2 and ALESS 102.1; $z \sim 3.324 \pm 0.030$ for ALESS 5.1) have emission lines that are redshifted into the narrow wavelength coverage of the Brγ filter. Since the Brγ filter is near the centre of the $K_s$ transmission a galaxy without line emission at these wavelengths will have a ($K_s$–Brγ) colour of zero. However, due to the narrow width of the Brγ filter relative to the $K_s$ filter, galaxies with redshifts that place an emission line in the narrow Brγ filter will have a ($K_s$–Brγ) colour that is significantly greater than zero. We employ the same methodology as previous narrow-band surveys (e.g. Moorwood et al. 2000; Geach et al. 2008; Sobral et al. 2013) to identify line-emitting galaxies. This methodology uses two parameters to select sources with a significant, physically driven narrow-band excess, as opposed to an excess due to random noise.

The first of these parameters, $\Sigma$, quantifies the significance of the narrow-band excess compared to the expected random scatter for a source with zero ($K_s$–Brγ) colour (Bunker et al. 1995). $\Sigma$ is given by:

$$ \Sigma = \frac{1 - 10^{-0.4(\text{BB-NB})}}{10^{-0.4(\text{ZP-NB})} \sqrt{\text{rms}_{\text{NB}}^2 + \text{rms}_{\text{BB}}^2}}, \quad (1) $$

where NB and BB are the apparent magnitudes in the narrow-band (Brγ) and broad-band ($K_s$) filters, respectively; ZP is the zero-point magnitude of the narrow-band images; rmsNB and rmsBB are the rms counts in 1.25 arcsec apertures for the individual narrow-band and broad-band quadrants, respectively. We require candidate line emitters have $\Sigma > 3$, which is consistent with previous narrow-band studies (e.g. Bunker et al. 1995; Sobral et al. 2013); see Fig. 1. Note that this $\Sigma$ does not correspond directly to the signal-to-noise ratio (SNR) in the Brγ filter, but is a separate quantity based on counts; $\Sigma > 3$ implicitly excludes sources with SNR $\lesssim 8$ in Brγ (for details, see e.g. Sobral et al. 2009).

In addition to having $\Sigma > 3$ line emitters are required to have an observed equivalent width (EW) $> 50$ Å. The EW is calculated for each source using:

$$ \text{EW} = \frac{f_{\text{Br}\gamma} - f_{K_s}}{f_{K_s} - f_{\text{Br}\gamma}(\Delta\lambda_{\text{Br}\gamma}/\Delta\lambda_{K_s})}, \quad (2) $$

where $\Delta\lambda_{\text{Br}\gamma}$ and $\Delta\lambda_{K_s}$ are the widths of the two filters and $f_{\text{Br}\gamma}$ and $f_{K_s}$ are the flux densities of the source in each filter. The 50 Å lower limit on EW for a source to be selected as a line emitter was chosen to lie above the $3\sigma$ scatter in ($K_s$–Brγ) colours for bright (Brγ > 19.5 mag) sources in both pointings (Fig. 1).

Before applying the selection criteria, we first account for sources that are detected with $\ge 3\sigma$ significance in the Brγ filter but $< 3\sigma$ in $K_s$. We classify these sources as non-detections in $K_s$, and replace their aperture magnitudes with the relevant $3\sigma$ limiting magnitude (see Table 2). However, several of these non-detections have counterparts in the S14 catalogue (within a 1 arcsec matching radius) and thus have $K_s$ magnitudes from either TENIS (Hsieh et al. 2012), archival HAWK-I observations (Zibetti, private communication), or

<footnote n="1">¹https://github.com/gbrammer/eazy-py</footnote>

<footnote n="2">²H-band data are also available but our pointings are not covered.</footnote>

<footnote n="3">³https://www.eso.org/sci/activities/garching/projects/goods/MasterSpectroscopy.html</footnote>

<footnote n="4">⁴http://member.ipmu.jp/john.silverman/CEDFS_vlt.html</footnote>
MUSYC (Taylor et al. 2009). For these sources, we replace our HAWK-I $K_s$ photometry with values from one of these surveys, preferentially using TENIS photometry as it is the deepest of the three (with a limiting $3\sigma$ magnitude of $m_{lim}^{3\sigma} = 24.45$ mag); if no TENIS photometry is available then we opt for the archival HAWK-I values ($m_{lim}^{3\sigma} = 24.36$ mag), using MUSYC ($m_{lim}^{3\sigma} = 22.55$ mag) only when no photometry exists for either of the other two. Note that while MUSYC $K_s$ observations are the shallowest of all the data considered here (including our own), there are 11 sources for which only MUSYC photometry is available. However, all of these sources reside in regions of Pointing 102 that are (i) outside of the coverage of the TENIS and archival HAWK-I observations, and (ii) close to the quadrant edges in our HAWK-I observations where the noise is at its greatest.

Using the $\Sigma > 3$ and EW $> 50$ Å selection criteria, 81 and 80 candidate line emitters are identified in Pointing 5+75 and Pointing 102, respectively (Fig. 1). Of these candidates, 30 are $K_s$ non-detections with no $K_s$ photometry in the S14 catalogue, and thus $\Sigma$ and EW are calculated by assuming that their $K_s$ magnitudes are equal to the $3\sigma$ limiting magnitudes of our data. Since this can only provide a lower limit for the ($K_s$–Bry) colour and thereby underestimate $\Sigma$ for these sources, we do include these sources in our sample of candidate line emitters.

Finally, we visually inspect all 161 candidate line emitters, removing stars/quasi-stellar objects and image artefacts. The final sample consists of 79 and 68 candidate line emitters in Pointing 5+75 and Pointing 102, respectively (147 sources in total).

## 2.5 Identifying line emitters associated with the SMGs

Narrow-band excess alone is not sufficient to identify star-forming galaxies in the same environments as our target SMGs; such an excess could be caused by a number of possible emission lines at different redshifts (see Fig. 3). We therefore use the available multiband photometric data covering our pointings to estimate photometric redshifts for the narrow-band emitters in our sample. The S14 catalogue contains photometric redshift estimates for sources across the ECDFS, however after cross-matching with our data (using a matching radius of 1 arcsec), a significant fraction (>30 per cent) of the line emitters identified in Section 2.4 do not have broad-band counterparts in this catalogue and thus lack any redshift information. We therefore perform our own SED fitting using EAZY-PY.

To maximize the number of sources for which we can derive photometric redshifts, we extract fixed-aperture photometry at their HAWK-I Bry positions in the UV-to-MIR images described in Section 2.3. Each image is astrometrically calibrated using SCAMP (Bertin 2006) and SWARP (Bertin 2010) to match the astrometry of our HAWK-I images, and then photometrically recalibrated so that all images have a zero-point magnitude of 30.0 mag. Photometry is extracted in fixed apertures using the PHOTUTILSPYTHON package (Bradley et al. 2022); apertures of diameter 2.0 arcsec are used for all images except those from Spitzer/IRAC, for which we use apertures of diameter 3.8 arcsec due to the larger PSF. Aperture corrections are determined for each filter by measuring the median difference between the magnitudes measured in these apertures and those measured in adaptively scaled apertures with SEXTRACTOR for bright point sources. Final corrections are applied to each filter to account for Galactic attenuation, using values from Cardamone et al. (2010) and Hsieh et al. (2012).

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Photometric redshifts derived using EAZY-PY compared to spectroscopic redshifts for all sources detected in HAWK-I Bry with archival spectroscopic redshifts (from Le Fèvre et al. 2005; Popesso et al. 2009; Treister et al. 2009; Balestra et al. 2010; Silverman et al. 2010; Danielson et al. 2017). Galaxies included in our final sample of candidate line emitters (see Section 2.4) are highlighted. The redshifts at which common extragalactic emission lines enter the Bry filter are shown using horizontal and vertical lines. Dashed lines highlight Hα and [O III], which are the emission lines of interest in this study. The diagonal line shows a one-to-one correspondence; the scatter is low and the majority of sources have photometric redshifts that are consistent with their spectroscopic redshifts.</figcaption></figure>

EAZY-PY operates using a $\chi^2$-minimization procedure in which linear superpositions of template SEDs are tested at different redshifts to find an optimal fit to the observed fluxes (Brammer et al. 2008). In keeping with other recent studies which implement EAZY-PY (e.g. Stevans et al. 2021; Finkelstein et al. 2022), we use the 'tweak_fsp_s_QSF_12_v3' set of 12 template SEDs, which cover a wide range of galaxy types and utilize a Chabrier (2003) initial mass function (IMF) and a Kriek & Conroy (2013) dust attenuation law while assuming solar metallicity. An advantage of these templates is that they include emission lines, such that a narrow-band excess can provide a relatively tight constraint on the redshift.

As discussed in Section 2.3, there have been several spectroscopic studies in the ECDFS, from which spectroscopic redshifts have been obtained for a number of galaxies across the field. Using a matching radius of 1.5 arcsec, we cross-match our data with catalogues from VVDS (Le Fèvre et al. 2005), the GOODS/VIMOS survey (Popesso et al. 2009), the ECDFS spectroscopic survey (Silverman et al. 2010), and the spectroscopic studies conducted by Treister et al. (2009) and Danielson et al. (2017). This gives spectroscopic redshifts for 163 (~4.1 per cent) of the 3929 sources detected in our HAWK-I imaging, including nine for which photometric redshifts could not be estimated due to insufficient photometry. Seven of the 163 sources with spectroscopic redshifts are emission-line galaxies selected in Section 2.4; the spectroscopic redshifts for these sources are used for our analyses. We compare the photometric and spectroscopic redshifts for our HAWK-I sources in Fig. 2. There is strong agreement between the photometric and spectroscopic redshifts, which is quantified using the normalized median absolute deviation (NMAD) of $\Delta z$:

$$ \sigma_{\text{NMAD}} = 1.48 \times \text{median} \left( \left| \frac{\Delta z - \text{median}(\Delta z)}{1 + z_{\text{spec}}} \right| \right), \quad (3) $$
<figure><img src="image_4.png" /><figcaption>**Figure 3.** The distributions of redshifts for the emission-line galaxies in Pointing 5+75 (left) and Pointing 102 (right) compared with their emission-line significance, $\Sigma$. Photometric redshifts are computed using EAZY-PY, with archival spectroscopic redshifts included where available (Section 2.5). $H\alpha$ and [O III] emitters are highlighted, and shaded regions show the redshift ranges used to select them. Peaks in the redshift distributions at these redshifts may be driven by overdensities of these line emitters. Horizontal dashed lines show the redshifts at which other common extragalactic emission lines enter the $Bry$ filter.</figcaption></figure>

<table><caption>**Table 3.** Summary of the sample at each stage of the selection process described in Sections 2.4 and 2.5.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Number per pointing</th></tr><tr><th>Pointing 5+75</th><th>Pointing 102</th><th>Total</th></tr></thead><tbody><tr><td>$Br\gamma$ detections</td><td>2175</td><td>1754</td><td>3929</td></tr><tr><td>Line emitter candidates (initial)</td><td>81</td><td>80</td><td>161</td></tr><tr><td>Line emitter candidates (confirmed)</td><td>79</td><td>68</td><td>147</td></tr><tr><td>$H\alpha$ candidates</td><td>44</td><td>11</td><td>55</td></tr><tr><td>[O III] candidates</td><td>4</td><td>2</td><td>6</td></tr></tbody></table>

where $z_{spec}$ is the spectroscopic redshift and $\Delta z = z_{spec} - z_{phot}$. We obtain $\sigma_{NMAD} = 0.062$ when considering all 152 HAWK-I detections with photometric and spectroscopic redshifts.

Only five (~3.4 per cent) of our 147 emission-line galaxies have neither spectroscopic nor photometric redshifts, the latter being due to a lack of photometry with sufficient depth. Fig. 3 shows the redshifts of the remaining 142 emission-line galaxies, compared with the significance of their narrow-band excess ($\Sigma$; equation 1). Peaks in the redshift distribution are visible at $z \sim 2.3$ (both pointings) and $z \sim 3.3$ (Pointing 5+75 only), as expected of $H\alpha$ and [O III] in the environments of the target SMGs. We select as $H\alpha$ ([O III]) emitters any galaxies for which $2.23 < z < 2.37$ (3.23 < $z < 3.41$), where these redshift ranges correspond to $3 \times$ the FWHM of the $Br\gamma$ filter when $H\alpha$ ([O III]) has redshifted to the centre. We represent these selection criteria with shaded regions in Fig. 3; the highlighted galaxies are henceforth assumed to be $H\alpha$ and [O III] emitters at similar redshifts as the target SMGs. We identify 44 $H\alpha$ emitters and 4 [O III] emitters in Pointing 5+75, and in Pointing 102 there are 11 $H\alpha$ emitters ([O III] emitters in Pointing 102 are not further considered because there is no SMG at $z \sim 3.3$ in this pointing). Table 3 summarizes the results of each step in the sample selection. We note that all of these $H\alpha$ and [O III] candidates have an SNR $> 8.5$ in the $Br\gamma$ filter as a natural consequence of our selection process (see also Section 2.4). We therefore do not expect the sizes of these samples to be significantly affected by Eddington (1913) bias.

To identify any AGN in the sample, we use a 1 arcsec matching radius to locate counterparts in the Lehmer et al. (2005) Chandra point source catalogue. None of the [O III] emitters and only one of the $H\alpha$ emitters (2.3 per cent) is an X-ray luminous AGN, which is consistent with the rate of X-ray AGN in blank-field surveys of $H\alpha$ emitters at the same redshift (e.g. 1.8 ± 1.3 per cent in Calhau et al. 2017). Since the AGN fraction is the same as in field surveys then this galaxy is kept in our sample to enable a fair like-for-like comparison between the SMG fields and blank-field $H\alpha$ emitters.

Fig. 4 shows the distributions of emission-line galaxies across the HAWK-I pointings, with $H\alpha$ and [O III] emitters highlighted. For all three target SMGs, the companion galaxies are spread across the entire field of view and therefore span several physical Mpc. This is consistent with expectations from simulations (e.g. Chiang et al. 2013; Muldrew et al. 2015; Yajima et al. 2022, see also Section 3.3), in which protoclusters are seen to extend over several Mpc, such that the entire structure is unlikely to be captured by a single HAWK-I pointing. We also note the presence of a dense clump of seven $H\alpha$ emitters (three of which are spectroscopically confirmed at $z \sim 2.3$) in the northeast of Pointing 5+75, which coincides with a photometrically identified $Ly\alpha$ blob at $z \sim 2.3$ (CDFS-LAB03; Yang et al. 2010). This system will be discussed further in Section 3.3.

Of the three SMGs targeted, only ALESS 102.1 is identified as an $H\alpha$ (or [O III]) emitter in our data. Danielson et al. (2017) did not identify ALESS 102.1 as an $H\alpha$ emitter in their spectroscopy, because the wavelength coverage with the VLT FOcal Reducer/low dispersion Spectrograph 2 (FORS2) and VIMOS instruments does not cover $H\alpha$ at $z \sim 2.3$. The original spectroscopic redshift for ALESS 5.1 is from CO(4–3) (Birkin et al. 2021) and no emission lines were observed in Keck/DEIMOS (DEep Imaging Multi-Object Spectrograph), Keck/MOSFIRE (Multi-Object Spectrograph for Infrared Exploration), or VLT/XSHOOTER observations (Danielson et al. 2017); this is likely because the redshifted [O III] line clashes with a bright $OH^-$ sky line for this source (Ramsay, Mountain & Geballe 1992). For ALESS 75.2, the original spectroscopic redshift was measured, in part, thanks to a faint $H\alpha$ line detected in Keck/MOSFIRE observations (Danielson et al. 2017), which is below the detection limit of our data.
<figure><img src="image_5.png" /><figcaption>**Figure 4.** Spatial distribution of the emission-line galaxies and the other Brγ detections in Pointing 5+75 (left) and Pointing 102 (right). Also shown are the positions of the target SMGs and other ALESS SMGs in these areas, although the redshifts of the non-target SMGs are either unknown or outside the ranges that would place the Hα or [O III] emission lines in the Brγ filter (Danielson et al. 2017; Birkin et al. 2021). Hα and [O III] candidates are indicated. While Pointing 102 does contain [O III] candidates, they are not shown here because there are no ALESS SMGs at z ~ 3.3 in this pointing. For all three SMGs, the candidate companion galaxies are distributed across the entire HAWK-I field of view, corresponding to physical spans of a few Mpc, as expected from protocluster simulations (e.g. Chiang et al. 2013; Muldrew et al. 2015; Yajima et al. 2022). Dashed circles show the boundaries of annuli used to measure radial trends in the density of companion galaxies (Section 3.3).</figcaption></figure>

# 3 RESULTS, ANALYSIS, AND DISCUSSION

## 3.1 Measuring luminosity functions

In order to quantify whether the SMGs reside in significant overdensities of Hα or [O III] emitters, a comparison to the blank field needs to be drawn. The High Redshift (z) Emission Line Survey (HiZELS; Geach et al. 2008) is a large narrow-band survey of emission-line galaxies, including Hα emitters at z = 2.23 (Sobral et al. 2013) and [O III] emitters at z = 3.24 (Khostovan et al. 2015) in the Cosmic Evolution Survey (COSMOS) field. The HiZELS results are therefore representative of Hα and [O III] emitters in regions of average density at redshifts similar to those of our target SMGs, and we use their luminosity functions as a blank-field sample for comparison with our results.

To construct luminosity functions for our Hα and [O III] emitters we bin them according to line luminosities, making corrections to the observed number counts in each bin to account for completeness, contamination from other emission lines, dust attenuation, and the shape of the narrow-band filter profile. Each of these steps is described in more detail below.

### 3.1.1 Survey volumes

Approximating the Brγ filter profile as a top-hat function with width equal to the FWHM of the filter (Δλ = 0.030μm), the Hα emission line should be detectable from zmin = 2.276 to zmax = 2.322, corresponding to a comoving volume per square degree of 5.86 × 10⁵ cMpc³ deg⁻². The redshift range within which [O III] can be detected extends from zmin = 3.294 to zmax = 3.354, which gives 1.04 × 10⁶ cMpc³ deg⁻². Accounting for the high-noise regions that were masked prior to source detection (see Section 2.2) and the gaps

between the HAWK-I detector chips, the surveyed areas in Pointing 5+75 and Pointing 102 are 0.0117 and 0.0118 deg², respectively. The volumes probed are therefore: 6859 cMpc³ for Hα in Pointing 5+75; 6891 cMpc³ for Hα in Pointing 102; 12180 cMpc³ for [O III] in Pointing 5+75; and 12230 cMpc³ for [O III] in Pointing 102. In Section 3.1.6, we correct the derived luminosity functions to account for the fact that the Brγ filter is not a perfect top-hat function, which leads to the volume probed being slightly different for sources with different luminosities.

### 3.1.2 Completeness correction

It is possible that real galaxies with weak emission lines were missed in our selection process (Section 2.4) despite actually meeting the selection criteria: the sample is incomplete at low emission-line fluxes. We correct for this using the method employed by Sobral et al. (2013), applying it separately for each quadrant of each pointing due to the variation in depth between detector chips (see Table 2). For each emission line (Hα and [O III]), we select sources that failed to meet the emission-line galaxy selection criteria (i.e. sources for which EW < 50 Å and/or Σ < 3) with redshifts within the range used to identify the targeted emission line (see Section 2.5). Due to the size of these samples, we generate ~1000 mock galaxies by randomly varying the Ks and Brγ magnitudes of the selected galaxies according to their uncertainties (assuming a Gaussian probability distribution for each magnitude) and placing them at random positions within their quadrant, removing any sources for which these changes result in them being classed as a line emitter. We then artificially inject line flux to each galaxy in this bolstered sample of non-line emitters, beginning with 10⁻²² erg s⁻¹ cm⁻² and incrementally increasing it
by 0.05 dex. Line fluxes are calculated as

$$F_{\text{line}} = \Delta\lambda_{\text{Br}\gamma} \frac{f_{\text{Br}\gamma} - f_{K_s}}{1 - (\Delta\lambda_{\text{Br}\gamma}/\Delta\lambda_{K_s})}, \quad (4)$$

where $f_{Br\gamma}$ and $f_{K_s}$ are the Brγ and Ks flux densities, respectively, in units of erg s⁻¹ cm⁻² Å⁻¹. With each increment of injected line flux, we recalculate the EW and Σ and reapply the line emitter selection criteria to determine the catalogue completeness as a function of line flux. This is used to estimate the completeness corrections for our luminosity functions. The uncertainty in the completeness at a given line flux is estimated by regenerating the mock galaxies 1000 times and measuring the standard deviation in the completeness across all iterations.

### 3.1.3 Removing [N II] contamination

The Hα emission line lies in between a doublet of [N II] lines at rest-frame wavelengths of 6548 and 6583 Å, which will contribute to the measured Brγ flux density and therefore affect the observed EW and emission-line flux. Using spectroscopic data taken with Subaru/FMOS and Keck/MOSFIRE, Sobral et al. (2015) observed an anticorrelation between the [N II]λ6583-to-Hα line flux ratio and the rest-frame EW (EW<sub>rest</sub> = EW/(1 + z)) for the Hα emitters in HiZELS, deriving the following empirical relation:

$$\frac{F_{\text{[NII]}}}{F_{\text{H}\alpha}} = -0.296 \times \log_{10}(\text{EW}_{\text{rest,H}\alpha} + \text{[NII]}) + 0.8. \quad (5)$$

We adopt this relation to apply corrections to the line fluxes of all Hα emitters in our sample, resulting in a median decrease of 11<sup>+9</sup><sub>-5</sub> per cent in emission-line flux.

### 3.1.4 Relative contributions from [O III]λ5007, [O III]λ4959, and Hβ

Thus far only the [O III]λ5007 emission line has been considered in the discussion of [O III] emitters at z ~ 3.3. However, this line is part of a doublet, with its counterpart residing at a rest-frame wavelength of 4959 Å, and there is a narrow range of redshifts (z = 3.336–3.344) in which both lines can contribute to the Brγ flux of a galaxy. Furthermore, while the Hβ emission line is sufficiently separated from the [O III] doublet to avoid contaminating the measured [O III] line flux, it is still close enough such that there is the potential for Hβ emitters to be misidentified as [O III] emitters (see Fig. 3). Therefore, rather than try and separate our sample into [O III]λ5007, [O III]λ4959 and Hβ emitters, we present a combined [O III] + Hβ luminosity function; this also allows for a consistent comparison with the blank-field [O III] + Hβ luminosity function from Khostovan et al. (2015).

We do however take into account the results of Sobral et al. (2015) when estimating the total volumes probed by the Brγ filter in the search for [O III] emitters: using spectroscopy, Sobral et al. (2015) find that for HiZELS ~50 per cent of photometrically selected [O III] + Hβ emitters at z ~ 1.4 are [O III]λ5007, ~27 per cent are [O III]λ4959, ~16 per cent are Hβ, with the remaining ~7 per cent being simultaneous detections of [O III]λ5007 and [O III]λ4959. Based on these results, Sobral et al. (2015) then add to the total volume probed (i.e. the volume probed if searching for [O III]λ5007 emitters) 16 per cent of the volume that would be probed had their search been for Hβ, and 25 per cent of the volume had they been searching for [O III]λ4959. We thus apply similar corrections to our total volume probed for [O III]λ5007 emitters.

### 3.1.5 Corrections for dust attenuation

Dust in star-forming galaxies reprocesses light emitted in the rest-frame UV and optical, and can therefore reduce the amount of Hα and [O III] flux observed. In order to estimate the intrinsic brightness of the emission lines (i.e. their integrated luminosities), one has to correct for the effect of dust attenuation. For an attenuation of A<sub>line</sub> (mag) at the emission-line wavelength, the conversion from line flux to intrinsic line luminosity is

$$L_{\text{line}} = 4\pi D_{L}^{2} F_{\text{line}} \times 10^{0.4A_{\text{line}}}, \quad (6)$$

where D<sub>L</sub> is the luminosity distance. We follow Sobral et al. (2013) and assume an attenuation at the Hα wavelength of A<sub>Hα</sub> = 1 mag, which is based on previous HiZELS studies (Garn et al. 2010; Sobral et al. 2012). Khostovan et al. (2015) do not correct for dust attenuation when plotting their luminosity functions, so we also leave our [O III]+Hβ luminosities uncorrected to ensure a consistent comparison. Khostovan et al. (2015) later go on to calculate the dust-corrected SFRs of their galaxies, where they then assume an attenuation of A<sub>[O III]+Hβ</sub> = 1.35 mag, derived by assuming A<sub>Hα</sub> = 1 mag and using a Calzetti et al. (2000) dust attenuation curve. We thus adopt the same correction when calculating our own SFRs (see Section 3.4).

### 3.1.6 Filter profile volume corrections

The comoving volumes used for our luminosity functions (Section 3.1.1) are calculated by approximating the Brγ filter as a top-hat filter with width equal to the Brγ FWHM. Since the filter profile is not a top-hat in reality, this introduces two main effects which need to be accounted for when estimating the galaxy number densities. First, bright emitters whose line falls near the edges of the Brγ filter will suffer a significant loss of line flux and thus appear to be fainter than they really are. This produces an overall bias towards faint sources in our sample. Secondly, any faint emitters close to the filter edges might be missed from our sample, and are therefore only detectable over a narrower redshift range (and thus a smaller volume) than their bright counterparts.

To correct for these effects, we follow the method used by Sobral et al. (2013) and Khostovan et al. (2015), as first proposed in Sobral et al. (2009). An initial Schechter fit is performed to the uncorrected<sup>5</sup> data. We then generate a mock sample of 10<sup>5</sup> fake sources with a luminosity distribution that is weighted by the uncorrected Schechter function. These sources are randomly assigned redshifts with a uniform distribution across the whole possible Brγ coverage. They are then convolved through the Brγ filter profile such that their luminosities decrease according to their assigned redshift (i.e. according to the position of the redshifted emission line in the filter profile) and rebinned using the same luminosity bins as for the uncorrected data. Comparing the resultant distribution to the input distribution reveals that bright sources are underestimated relative to the fainter sources, as expected. The real data are corrected using the ratio of these distributions.

<footnote n="5"><sup>5</sup>‘Uncorrected’ only in terms of the filter profile correction; the results have already been corrected for line completeness and dust attenuation by this stage.</footnote>
### 3.1.7 Fitting Schechter functions

Finally, we perform fits to the corrected data using a Schechter function (Schechter 1976):

$$ \Phi(L)dL = \ln(10)\Phi^*\left(\frac{L}{L^*}\right)^{\alpha+1}e^{-(L/L^*)}d\log L, \quad (7) $$

where $\Phi(L)$ is the number density at luminosity $L$, $\Phi^*$ is the normalization of the luminosity function, $L^*$ is the characteristic luminosity, and $\alpha$ is the slope at the faint end of the luminosity function, where the power-law component dominates.

The faintest bins (open symbols in Fig. 5) are excluded from each fit due to their low completeness. For the $H\alpha$ emitters, we take ‘low completeness’ to mean that the low-luminosity edge of the bin lies below the 30 per cent completeness limit. The line luminosities of all [O III] + H $\beta$ emitters lie above the 90 per cent completeness threshold; we therefore do not exclude any from the fit.

Due to the small number of bins left available for fitting, it is impossible to reliably constrain all three free parameters of the Schechter function simultaneously. For the $H\alpha$ luminosity functions, we therefore fix the faint-end slope, $\alpha$, to the value of -1.59 obtained by Sobral et al. (2013) at $z = 2.23$. For the [O III] + H $\beta$ emitters, we only have one available bin and thus fix both $\alpha$ and $\log(L^*/\text{erg s}^{-1})$ to the values for the $z = 3.24$ sample of [O III] + H $\beta$ from Khostovan et al. (2015), which are -1.60 and 42.83, respectively. Thus our [O III] + H $\beta$ luminosity function is effectively a renormalized version of the Khostovan et al. (2015) result, with $\Phi^*$ being the only free parameter. In addition to the $H\alpha$ luminosity functions of the individual pointings, we also construct fits to the combined sample of $H\alpha$ emitters from both SMG fields, as this provides a more general view of SMGs at $z \sim 2.3$ with improved statistics.

The best-fitting parameters for each luminosity function are summarized in Table 4. Uncertainties are estimated for each free parameter by randomly perturbing the bin heights according to their uncertainties and then recalculating the fit, and repeating this process until $10^5$ fits have been made. The $1\sigma$ confidence interval for each parameter is then estimated using the 16th and 84th percentiles of the best-fitting values.

## 3.2 Analysing luminosity functions

We next use the luminosity functions to assess whether the targeted SMGs reside in overdensities of emission-line galaxies. Fig. 5 compares the observed luminosity functions from the SMG fields to those from the blank-field surveys of Sobral et al. (2013) and Khostovan et al. (2015). The environment of ALESS 75.2 shows signs of being overdense relative to the field at $z \sim 2.3$, with most bins lying significantly above the blank-field luminosity function. Conversely the environments of ALESS 5.1 and 102.1 are broadly consistent with the blank-field luminosity functions at their respective epochs. An overdensity remains when the $H\alpha$ emitters from both fields at $z \sim 2.3$ are combined, implying that on average SMGs at this epoch reside in overdense, protocluster-like environments, which is qualitatively consistent with clustering results (e.g. Hickox et al. 2012; Wilkinson et al. 2017; Stach et al. 2021). The contrast between the individual $H\alpha$ luminosity functions suggests that there is significant variation across SMG environments, although observations of additional SMGs are required to confirm and quantify the field-to-field variation. Furthermore, as explored later in this section and

shown in Fig. 4, the field around ALESS 102.1 is itself overdense on smaller scales.

To quantitatively compare the SMG field and blank-field luminosity functions we consider the parameters of the Schechter function fits (Section 3.1.7). The parameters of Schechter function fitting are often correlated, so in Fig. 6 we show the uncertainties of the luminosity function parameters in the $L^*-\Phi^*$ plane (as described in Section 3.1.7 the faint-end slope, $\alpha$, is fixed), comparing our SMG fields with the blank fields at similar redshifts. For the individual SMG environments at $z \sim 2.3$, the fit parameters are offset from those of the blank field, although for the ALESS 102.1 region the offset is only at the $\sim 1\sigma$ level. These separations are driven by a higher $L^*$, and, in the case of ALESS 75.2, by a larger $\Phi^*$, which implies that this environment is preferentially overdense in bright line emitters compared to the blank field. Meanwhile, the environment of ALESS 5.1 exhibits an offset of $<1\sigma$ relative to the blank-field value of $\Phi^*$ at $z \sim 3.3$, implying this SMG does not reside in an overdensity.

We quantify the galaxy overdensity in each sample of $H\alpha$ and [O III] + H $\beta$ emitters in two ways. First, we calculate the ratio of the $\Phi^*$ from the best-fitting Schechter function to those from the relevant blank-field luminosity functions, $\Phi^*/\Phi^*_{\text{field}}$. This ratio tells us how much higher the ‘knee’ of each SMG-field luminosity function is relative to the blank field. These ratios are presented in Table 4. The value for the [O III] + H $\beta$ luminosity function suggests that the environment of ALESS 5.1 at $z \sim 3.3$ is $1.2^{+0.6}_{-0.4}$ times as dense as the field, i.e. it is consistent with the blank field. For the $H\alpha$ emitters at $z \sim 2.3$, $L^*$ is also a free parameter in the Schechter fits, and the $L^*-\Phi^*$ correlation means that we must first refit the data with $L^*$ fixed to the $z \sim 2.3$ blank-field value from Sobral et al. (2013), i.e. $\log(L^*/\text{erg s}^{-1}) = 42.83$. These fits give $\Phi^*/\Phi^*_{\text{field}}$ values of $3.6^{+0.6}_{-0.6}$ and $1.7^{+0.3}_{-0.3}$ for the ALESS 75.2 and ALESS 102.1 fields, respectively. The combined sample of $H\alpha$ emitters from both SMG environments suggests that the average SMG environment at $z \sim 2.3$ is $2.6^{+0.4}_{-0.4}$ times more dense than the blank field at this epoch.

To derive a more representative estimate of the galaxy overdensity in each environment, we also estimate the number of $H\alpha$ or [O III] + H $\beta$ emitters that one would expect to find in a blank field with the volumes probed by our observations. To do this, we integrate the field luminosity functions across the luminosity range covered by our data, excluding low completeness bins; that is, we integrate across the ranges $42.5 \le \log(L^*_{H\alpha}/\text{erg s}^{-1}) < 43.8$ and $42.4 \le \log(L^*_{[OIII]+H\beta}/\text{erg s}^{-1}) < 42.7$ and multiply by the volumes probed in each HAWK-I pointing to estimate the expected number counts in an equivalent blank field, $N_{\text{field}}$. Since these field galaxies would have contributed to the observed number counts, we quantify the galaxy overdensity in each environment using

$$ \delta_g = \frac{N_{\text{total}} - N_{\text{field}}}{N_{\text{field}}}, \quad (8) $$

where $N_{\text{total}}$ is the sum of the counts in our complete bins. Uncertainties in $N_{\text{total}}$ are determined by adding in quadrature the uncertainties in the bin counts. For $N_{\text{field}}$, the uncertainties are estimated by randomly permuting the blank-field Schechter parameters within their uncertainties prior to integrating, then repeating the process $10^5$ times and using the 16th and 84th percentiles of the resultant number counts to define the $1\sigma$ confidence interval.

The values of $\delta_g$ for each sample of $H\alpha$ and [O III] + H $\beta$ emitters are summarized in Table 4 along with the significance of this overdensity, $\sigma_g$. Based on these values, the environments of ALESS 5.1, 75.2, and 102.1 are overdense by factors of $0.2^{+2.5}_{-0.7}$ (0.3$\sigma_g$), $2.6^{+1.4}_{-1.2}$ (2.3$\sigma_g$), and $0.2^{+0.6}_{-0.5}$ (0.5$\sigma_g$), respectively. If the samples of
<figure><img src="image_6.png" /><figcaption>**Figure 5.** Luminosity functions of Hα (left) and [O III] + Hβ (right) emitters identified around SMGs at z ~ 2.3 and ~ 3.3, respectively. Open symbols represent bins that are highly incomplete and are thus excluded from the fits (see Section 3.1.7). The data are compared with luminosity functions from blank-field studies at similar redshifts (Sobral et al. 2013; Khostovan et al. 2015), which are highlighted with coloured solid curves. Shaded regions represent the 1σ uncertainties on each luminosity function. For our Hα luminosity functions (left), the faint-end slope is fixed to the value from the corresponding blank-field luminosity function: α = −1.59 (Sobral et al. 2013) and dashed and dotted curves show the fitted Schechter functions for the environments of ALESS 75.2 and ALESS 102.1, respectively. The thick, solid black curve shows the Schechter function obtained by fitting to the data for our combined sample of Hα emitters at z ~ 2.3 (black data points); the grey shaded region shows the 1σ confidence region for this fit. The thin black line shows another Schechter function obtained by fitting to the black data points, but with L* fixed to the blank-field value of log(L*/erg s⁻¹) = 42.87 (Sobral et al. 2013). In the right panel, the thick solid black curve and grey shaded region shows the result of scaling up the blank-field luminosity function from Khostovan et al. (2015) to fit to the single bin of [O III] + Hβ emitters from the environment of ALESS 5.1. Comparison with the blank-field luminosity functions reveals that ALESS 5.1, ALESS 75.2, and ALESS 102.1 reside in environments with overdensity parameters of δg = 0.2⁺².⁵⁻⁰.⁷, 2.6⁺¹.⁴⁻¹.², and 0.2⁺⁰.⁶⁻⁰.⁸, respectively. On average, the SMGs at z ~ 2.3 reside in environments with galaxy overdensities of δg = 1.5⁺¹.⁰⁻⁰.⁸.</figcaption></figure>

<table><caption>**Table 4.** Summary of the best-fitting Schechter parameters for companion galaxies in the environments of the target SMGs, along with comparisons to the blank field at similar redshifts. In all cases, the faint-end slope of the luminosity function, α, is fixed to the value from the relevant blank-field luminosity function and, where indicated, the characteristic luminosity, L*, is also fixed to the blank-field values. The values of Φ* field are taken from the relevant blank-field Schechter functions. All blank-field parameters are from Sobral et al. (2013) and Khostovan et al. (2015).</caption><thead><tr><th>SMG environment</th><th>log(L*/ergs<sup>-1</sup>)</th><th>log(Φ*/Mpc<sup>-3</sup>)<sup>a</sup></th><th>log(Φ*<sub>fixed L*</sub>/Mpc<sup>-3</sup>)<sup>b</sup></th><th>Φ*/Φ*<sub>field</sub><sup>a</sup></th><th>Φ*<sub>fixed L*</sub>/Φ*<sub>field</sub><sup>b</sup></th><th>δ<sub>g</sub><sup>c</sup></th><th>σ<sub>δ</sub><sup>d</sup></th></tr></thead><tbody><tr><td>ALESS 75.2</td><td>43.18<sup>+0.42</sup><sub>-0.28</sub></td><td>-2.57<sup>+0.29</sup><sub>-0.39</sub></td><td>-2.22<sup>+0.05</sup><sub>-0.09</sub></td><td>1.62<sup>+1.05</sup><sub>-0.80</sub></td><td>3.63<sup>+0.58</sup><sub>-0.59</sub></td><td>2.61<sup>+1.42</sup><sub>-1.15</sub></td><td>2.3</td></tr><tr><td>ALESS 102.1</td><td>43.28<sup>+0.50</sup><sub>-0.28</sub></td><td>-3.05<sup>+0.17</sup><sub>-0.42</sub></td><td>-2.55<sup>+0.08</sup><sub>-0.12</sub></td><td>0.54<sup>+0.22</sup><sub>-0.23</sub></td><td>1.70<sup>+0.30</sup><sub>-0.32</sub></td><td>0.21<sup>+0.55<sup>e</sup></sup><sub>-0.45</sub></td><td>0.5<sup>e</sup></td></tr><tr><td>SMGs at z ~ 2.3</td><td>43.00<sup>+0.12</sup><sub>-0.10</sub></td><td>-2.53<sup>+0.12</sup><sub>-0.17</sub></td><td>-2.36<sup>+0.04</sup><sub>-0.08</sub></td><td>1.78<sup>+0.41</sup><sub>-0.43</sub></td><td>2.63<sup>+0.41</sup><sub>-0.42</sub></td><td>1.51<sup>+0.98</sup><sub>-0.80</sub></td><td>1.9</td></tr><tr><td>ALESS 5.1</td><td>42.83 (fixed)</td><td>-</td><td>-3.22<sup>+0.09</sup><sub>-0.26</sub></td><td>-</td><td>1.23<sup>+0.58</sup><sub>-0.40</sub></td><td>0.22<sup>+2.50</sup><sub>-0.66</sub></td><td>0.3</td></tr></tbody></table>

Notes. a $Φ^*$ as measured when both $Φ^*$ and $L^*$ are free parameters (α is always fixed to the blank-field values).

b $Φ^*$<sub>fixed L*</sub> is obtained by fitting a Schechter function to the data with both $L^*$ and α fixed to the blank-field values. The values of $L^*$ are taken from the relevant blank-field Schechter functions.

c Galaxy overdensity, $δ_g = (N_{total}/N_{field}) - 1$; see Section 3.2.

d Significance of the galaxy overdensity, $δ_g$.

e ALESS 102.1 has $δ_g = 3.8^{+2.4}_{-1.8}$ (i.e. $σ_δ = 2.1$) when considering only the HAWK-I quadrant containing the SMG (Section 3.2).

Hα emitters from both pointings are considered as one, then the SMG environments at z ~ 2.3 are overdense by a factor of 1.5⁺¹.⁰⁻⁰.⁸ (1.9σδ) on average. The above uncertainties do not account for cosmic variance, which could cause a factor ~ 2 difference in number counts, as based on the Hα emitters in two equal depth HAWK-I pointings in the COSMOS and UltraDeep Survey (UDS) fields (Sobral et al. 2013). Including cosmic variance in our calculations does not change our overall conclusions that the ALESS 75.2 and the combined z ~ 2.3 SMG fields are overdense, nor does it affect the conclusions that ALESS 5.1 and ALESS 102.1 reside in environments consistent with the blank field.

While Pointing 102 as a whole is not overdense, the majority of the Hα emitters in this pointing are contained within the same quadrant as the SMG, as can be seen in Fig. 4 (see also Section 3.3). We therefore recalculate $δ_g$ for this SMG environment, this time

considering only the volume probed within that quadrant (1722 cMpc³), finding $δ_g = 3.8^{+2.4}_{-1.8} (2.1σ)$ in this area, which suggests that ALESS 102.1 actually does reside in an overdense environment with a physical scale of ~1.6 Mpc. This high concentration of galaxies surrounding the SMG could be indicative of structure formation on smaller scales than those of protoclusters and it is possible that ALESS 102.1 resides in a protogroup (e.g. Diener et al. 2013). We discuss the spatial distribution of companion galaxies in each SMG environment in more detail in Section 3.3.

The question remains as to whether the target SMGs reside in protoclusters, which will evolve into bound clusters by the present day. To learn more, we compare the overdensities in the SMG fields with previous studies of protoclusters. However, protoclusters exhibit a wide range of galaxy overdensities; a 'typical' value of $δ_g$ is not well-defined, though we highlight here structures at similar
<figure><img src="image_7.png" /></figure>

**Figure 6.** Contours showing the correlated uncertainties on the Schechter parameter fits to the luminosity functions shown in Fig. 5. For all the SMG fields, the faint-end slope, $\alpha$, is fixed to match to the blank-field luminosity functions from Sobral et al. (2013) and Khostovan et al. (2015). Single contours are at the $1\sigma$ level; the combined $z \sim 2.3$ data have $1\sigma$, $2\sigma$, and $3\sigma$ contours shown. For ALESS 5.1, only the $1\sigma$ error bars are shown as $L^*$ is fixed. Both of the luminosity functions for SMGs at $z \sim 2.3$ are separated from the corresponding blank-field luminosity function in $L^*–\Phi^*$ space, although for ALESS 102.1 this is only at the $\sim 1\sigma$ level. Increases in $L^*$ relative to the blank field, as seen for both SMGs at $z \sim 2.3$ (and for their combined luminosity function), imply that their environments may preferentially harbour brighter galaxies than those in the field. ALESS 5.1 exhibits an offset of $<1\sigma$ relative to the blank-field luminosity function at $z \sim 3.3$.

redshifts to our target SMGs. For example, $\delta_g = 2.5$ for the $z = 1.99$ protocluster in the GOODS-N field (Chapman et al. 2009). The protoclusters 4C 10.48 ($z = 2.35$) and 4C 23.56 ($z = 2.48$), which were both identified using narrow-band selection of $H\alpha$ emitters around luminous radio galaxies, were found to have overdensities of $\delta_g = 11_{-2}^{+2}$ and $4_{-3}^{+5}$, respectively (Hatch et al. 2011; Tanaka et al. 2011). Similarly, Matsuda et al. (2011) used a narrow-band search for $H\alpha$ emitters at $z = 2.23$ to identify overdensities of $\delta_g \sim 3, 2$, and 2 around a quasi-stellar object overdensity, a high-redshift radio galaxy, and an overdensity of SMGs and optically faint radio galaxies, respectively. The protocluster Cl J0227–0421 at $z = 3.29$ is overdense by a factor of $10.5 \pm 2.8$ (Lemaux et al. 2014). Two protoclusters in the COSMOS field at $z = 2.10$ and 2.47 were found to have overdensities of $\delta_g \sim 8$ and $\sim 3.3$, respectively (Yuan et al. 2014; Chiang et al. 2017). Zheng et al. (2021) confirm overdensities of $H\alpha$ emitters in two protocluster candidates, BOSS1244 and BOSS1542, with overdensity factors of $\delta_g = 5.6 \pm 0.3$ and $4.9 \pm 0.3$, respectively. Comparing $\delta_g$ for these protoclusters with our values, we posit that the environments of ALESS 75.2 and ALESS 102.1 (and thus of SMGs on average at $z \sim 2.3$) are consistent with being protoclusters, albeit at the lower-density end. For the remainder of the analyses, we assume that members of these overdensities will form larger structures by $z = 0$, although we caution that the significance of these overdensities is relatively low (1.3$\sigma_8$–2.3$\sigma_8$) and thus it is uncertain whether they will coalesce by $z \sim 0$ (e.g. Overzier et al. 2009; Angulo et al. 2012; Chiang et al. 2013).

### 3.3 Spatial distribution of line emitters

In order to investigate the role of environment in shaping the evolution of SMGs, and to assess whether the target SMGs reside in

<figure><img src="image_8.png" /></figure>

**Figure 7.** Surface density of $H\alpha$ emitters measured in 2.0 arcmin annuli centred on the two target SMGs at $z \sim 2.3$ and compared with expected values from the blank-field $H\alpha$ luminosity function (horizontal line and shaded region; Sobral et al. 2013). The shape of the field and positions of the SMGs means that coverage is incomplete with data for 74 per cent (81 per cent) of the inner, 36 per cent (31 per cent) of the middle, and 26 per cent (27 per cent) of the outer annuli for Pointing 5+75 (Pointing 102). Open symbols show the values calculated if the dense clump of $H\alpha$ emitters in the northeast of Pointing 5+75 (see Section 2.5) is included. Both $z \sim 2.3$ SMGs have high densities of $H\alpha$ emitters in the central $\sim 1$ Mpc. For ALESS 102.1, the density falls with increasing separation from the SMG, though no significant trend exists for ALESS 75.2. The existence of a significant overdensity within $\sim 2$ arcmin of ALESS 102.1 with no evidence of further extension suggests it may reside in an early galaxy group, while ALESS 75.2 appears to reside in a larger structure that extends beyond the HAWK-I coverage.

special regions within any surrounding structures, we next explore the spatial distributions of coeval line emitters across our HAWK-I pointings. Due to the small size of the $[Oiii] + H\beta$ emitter sample, we limit this part of the analysis to the $H\alpha$ emitters around ALESS 75.2 and 102.1. Fig. 7 compares the surface density of $H\alpha$ emitters as measured in annuli centred on each target SMG (where the density calculations account for masked and unobserved regions by assuming the density is the same as in the observed regions) with the surface densities one would expect based on the blank-field luminosity function from Sobral et al. (2013). The annuli have inner and outer radii increasing in increments of 2.0 arcmin, and are represented by dashed circles in Fig. 4. Note that these large annuli are necessary due to the sample sizes, but make it difficult to probe the protocluster structures in detail. We therefore also show in Fig. 8 the result of smoothing the distributions of $H\alpha$ emitters in Pointing 5+75 (left) and Pointing 102 (right) using a 2D Gaussian kernel with width corresponding to 0.5 Mpc at $z \sim 2.3$. This method of visualization clarifies where the SMGs lie in relation to any density peaks and can highlight any substructures. For simplicity we assume that there are no $H\alpha$ emitters in the unobserved region between the detector chips, or in regions of the image that have been masked, and thus the densities shown in these regions are potentially underestimated.

For ALESS 102.1, there is a noticeable decrease in the surface density of $H\alpha$ emitters as a function of radial distance from the SMG, with the innermost bin in Fig. 7 being significantly overdense relative to the field despite the environment not being overdense as a whole (see also Section 3.2). This is also clear from Fig. 8, which shows that the SMG lies $\sim 20$ arcsec from a $\sim 3.25$ arcmin ($\sim 1.6$ Mpc) density peak. Furthermore, Fig. 4 demonstrates that the
<figure><img src="image_9.png" /><figcaption>**Figure 8.** Maps showing the variation of surface overdensity, $n/n_{field}$, of H$\alpha$ emitters across Pointing 5+75 (left) and Pointing 102 (middle), and in the context of LAE density at $z \sim 2.3$ in the wider ECDFS (right). The maps are smoothed using a 2D Gaussian kernel with a width of 0.5 Mpc. Contour levels are $n/n_{field} = 0.25, 0.5, 1$ and increasing in intervals of 1 thereafter; dashed lines represent underdensities. Crosses mark the positions of the target SMGs, ALESS 75.2 and ALESS 102.1, and hatching indicates regions outside our HAWK-I coverage (including chip gaps) or that are masked (e.g. due to the presence of a bright star). Note that the smoothing implicitly assumes that no H$\alpha$ emitters reside in these regions, such that the densities here are conservative lower limits. Both SMGs are in/near H$\alpha$ density peaks, although ALESS 75.2 is not in the highest density region in Pointing 5+75. The rightmost panel shows the H$\alpha$ overdensities in the two SMG fields compared to the wider LAE density measured in the whole ECDFS (Yang et al. 2010); LAE contour levels are at $n/n_{field} = 0.3, 0.5, 1, 2$, and 3. The region of highest H$\alpha$ density in Pointing 5+75 corresponds to strong overdensity of LAEs, which contains the Ly$\alpha$ blob CDFS-LAB03 (Yang et al. 2010; see also Section 2.5) and there is an overall correlation between the H$\alpha$ and LAE overdensities in this region. Conversely, despite being in a small region of localized H$\alpha$ overdensity, ALESS 102.1 is in a region that is underdense in LAEs on the scales probed by Yang et al. (2010).</figcaption></figure>

innermost 2.0 arcmin annulus contains more than half of the H$\alpha$ emitters detected across Pointing 102. Fig. 8 includes a panel showing the location of the SMGs and our H$\alpha$ emitter density maps in the context of the overdensity of Ly$\alpha$ emitters (LAEs) at $z \sim 2.3$ mapped by Yang et al. (2010). This shows that the small-scale overdensity around ALESS 102.1 is in a broader underdense region, and it is therefore unlikely to be a condensed infalling knot within a larger structure.

In the case of ALESS 75.2 we show two results in Fig. 7: one where we include all H$\alpha$ emitters in the pointing (open squares), and one where we exclude the dense clump of H$\alpha$ emitters in the north-east (filled squares; see Section 2.5). In both cases, there is no significant trend in the H$\alpha$ surface density as a function of separation from the SMG, although it does show signs of decreasing at the outermost radii if the dense north-easterly clump is excluded. This lack of trend implies that ALESS 75.2 does not reside in a particularly special region of the structure, and/or the structure extends beyond the HAWK-I field of view. The latter hypothesis is supported by the comparison of the H$\alpha$ emitter overdensity with that of the LAEs from Yang et al. (2010) (Fig. 8, right), which shows that the whole structure around ALESS 75.2 is within a larger region of LAE overdensity. This suggests that the H$\alpha$ emitter structure likely spans a physical distance $\gtrsim 3.5$ Mpc at $z \sim 2.3$, which is consistent with the simulations of e.g. Muldrew et al. (2015), in which protoclusters are expected to extend over $\gtrsim 10$ Mpc at $z \sim 2$. The Yang et al. (2010) structure in this region includes the Ly$\alpha$ blob CDFS-LAB03, which coincides with seven H$\alpha$ emitters (see also Section 2.5). The overall picture is consistent with previous findings, in which Ly$\alpha$ blobs are found to be associated with massive dark matter haloes and filamentary large-scale structures (e.g. Geach et al. 2016; Umehata et al. 2019).

## 3.4 SMG companions: SFRs and stellar masses

In this section, we investigate the dust-corrected SFRs and stellar masses ($M_*$) of the individual galaxies around each target SMG, to

determine whether they lie on the main sequence of star formation at their epochs. This correlation between SFR and $M_*$ has been observed out to $z \sim 6$ (e.g. Brinchmann et al. 2004; Daddi et al. 2007; Elbaz et al. 2007; González et al. 2010; Speagle et al. 2014; Schreiber et al. 2015; Scoville et al. 2017) and galaxies significantly above the main sequence are usually considered to be short-lived starbursts, whereas those significantly below the main sequence are typically quenched. The position of galaxies relative to the main sequence provides insights into their evolutionary state and can be used to infer the role of any environmentally driven mechanisms enhancing or inhibiting star formation activity. Note that while ALESS 5.1 does not appear to reside in an overdense structure, the properties of the coeval [O III] + H$\beta$ emitters in its vicinity are still of interest and we thus include them in this part of the analysis.

We obtain stellar masses and SFRs for our galaxies by using the SED fitting code, MAGPHYS (da Cunha et al. 2008), to fit SEDs to the same fixed-aperture photometry used to derive photometric redshifts in Section 2.5. Fig. 9 compares the relationship between SFR and stellar mass for the H$\alpha$ and [O III] + H$\beta$ emitters that are SMG companions with the main sequence at similar epochs using the prescription from Speagle et al. (2014). These galaxies generally scatter about the main sequence at their respective epochs, following a similar trend of increasing SFR with increasing stellar mass. We thus find no significant evidence of enhanced star formation in any of these SMG environments, despite the range of overdensities that they span; this is contrary to some previous studies in which enhanced SFRs have been found in overdense environments at $z \gtrsim 1$ (e.g. Elbaz et al. 2007; Cooper et al. 2008; Lemaux et al. 2022), however it is consistent with several other studies in which no environmentally driven SFR enhancement is observed at high redshift (e.g. Scoville et al. 2013; Darvish et al. 2016; Zavala et al. 2019).

In addition to the MAGPHYS-derived SFRs, we also calculate the dust-corrected SFRs for the H$\alpha$ emitters using the scaling relation from Kennicutt (1998), modified for a Chabrier (2003) IMF:

$$ \text{SFR (M}_\odot \text{ yr}^{-1}) = 4.65 \times 10^{-42} L_{H\alpha} (\text{ergs}^{-1}), \quad (9) $$
<figure><img src="image_10.png" /><figcaption>**Figure 9.** Top: MAGPHYS-derived SFR versus stellar mass for the Hα and [O III] emitters identified in this study, compared with the z~2.3 and ~3.3 main sequence (shaded regions represent 0.2 dex scatter; Speagle et al. 2014). The target SMGs are also shown, with masses and SFRs from Danielson et al. (2017) and Birkin et al. (2021) (black points; ALESS 102.1 is connected with a black dotted line to the counterpart Hα emitter identified from our data; discussed in Section 3.4). Dashed, dotted-dashed, and dotted-dotted-dashed horizontal lines correspond to the minimum SFR sensitivity of our survey in the environments of ALESS 5.1, 75.2, and 102.1, respectively, based on estimates using the line luminosities (equation 9). The galaxies generally follow the main sequence for their respective epochs, with some scatter in all three SMG environments. Bottom: ratio of MAGPHYS-derived SFRs to the SFRs derived from line luminosities using fixed dust Hα ([O III] + Hβ) dust attenuations of 1.0 (1.35) mag, plotted as a function of stellar mass. The right-hand axis shows the corresponding dust attenuation required to make the line luminosity-derived SFR match the MAGPHYS-derived SFR for Hα emitters, A<sub>Hα</sub> (equation 11). (Analogous values for [O III] + Hβ emitters, A<sub>[OIII]</sub>, can be obtained by adding 0.14.) The observed correlation suggests that assuming a constant dust attenuation for all Hα/[O III] + Hβ emitters results in underestimated SFRs at high stellar masses; such an approximation should therefore be used with caution. The black cross in the bottom right of each panel shows the mean parameter uncertainties.</figcaption></figure>

where the Hα line flux has been corrected for contamination by the nearby [NiII] doublet (see Section 3.1.3) and we assume a dust attenuation of A<sub>Hα</sub> = 1 mag (Section 3.1.5). For the [OIII] + Hβ emitters we assume an attenuation of A<sub>[OIII]+Hβ</sub> = 1.35 mag following Khostovan et al. (2015) and use the relation between SFR and L<sub>[OIII]+Hβ</sub> from Osterbrock & Ferland (2006), similarly modified for a Chabrier (2003) IMF:

$$ \text{SFR (M}_{\odot} \text{ yr}^{-1}) = 4.32 \times 10^{-42} L_{[\text{O III}]+\text{H}\beta} (\text{erg s}^{-1}). \quad (10) $$

The bottom panel of Fig. 9 shows how the ratio of the MAGPHYS-derived and line-derived SFR estimates varies with stellar mass. Also shown is the Hα dust attenuation required for the SFR derived from equation (9), SFR<sub>Hα</sub>, to agree with the MAGPHYS-derived value,

SFR<sub>MAGPHYS</sub>, as given by:

$$ A_{H\alpha} = 2.5 \log_{10} \left( \frac{\text{SFR}_{\text{MAGPHYS}}}{\text{SFR}_{H\alpha}} \right) + 0.4. \quad (11) $$

An analagous equation for A<sub>[OIII]+Hβ</sub> can be obtained by adding 0.14 mag. It is evident that as one moves to higher stellar mass, the assumption that A<sub>Hα</sub> (A<sub>[OIII]+Hβ</sub>) = 1.0 (1.35) mag results in underestimated SFRs compared with the results from SED fitting. We therefore caution that while such an assumption may be suitable for galaxies with low-to-average stellar mass, it becomes less reliable for high-mass galaxies.

We also include the SMGs themselves in Fig. 9, with the SFRs and stellar masses for these calculated by Danielson et al. (2017) and Birkin et al. (2021) using MAGPHYS. As expected for sources selected due to their IR-brightness, the SMGs are among the most active galaxies in the observed fields. ALESS 102.1 is also one of the most massive galaxies in its environment, which suggests that if it is in a protocluster then it may be brightest cluster galaxy (BCG) progenitor, i.e. a proto-BCG. Similarly, ALESS 5.1 is massive relative to other galaxies in the surrounding region, but given the low density of this environment we deem it unlikely that this SMG is a proto-BCG. Conversely, ALESS 75.2 has a lower mass, which is not exceptional for its environment, and which points towards it being more likely to evolve into a normal cluster member. This is consistent with the spatial analysis of Hα emitters and LAEs (Section 3.3), which showed that ALESS 75.2 is offset from the densest regions of this field.

Since ALESS 102.1 has a counterpart Hα emitter in our sample, we also compare our MAGPHYS-derived SFR and stellar mass with those derived by Danielson et al. (2017). Our stellar mass of log(M<sub>⋆</sub>/M<sub>☉</sub>) = 11.49<sup>+0.18</sup><sub>-0.05</sub> is in good agreement with their value of log(M<sub>⋆</sub>/M<sub>☉</sub>) = 11.42<sup>+0.06</sup><sub>-0.06</sub>. Conversely, our SFR of log(SFR/M<sub>☉</sub>yr<sup>-1</sup>) = 2.75<sup>+0.22</sup><sub>-0.25</sub> is significantly higher than their estimate of log(SFR/M<sub>☉</sub>yr<sup>-1</sup>) = 2.12<sup>+0.25</sup><sub>-0.23</sub>. This is likely due to the inclusion of FIR and radio photometry in their SED fitting which are absent from our own fit; the dust component (and thus the SFR) is better constrained in the Danielson et al. (2017) SED fit. We therefore opt to use their values of SFR and stellar mass for this galaxy instead of our own.

## 3.5 Stellar mass functions

We next construct the stellar mass functions of the galaxies around each SMG and compare these with the blank field. The stellar mass functions are derived following a similar procedure as for the luminosity functions (see Section 3.1), minus the corrections that are only relevant to luminosity functions (dust attenuation, line flux contamination, and filter profile corrections). Completeness corrections were applied to each mass bin according to the completeness values estimated in Section 3.1.2 based on the emission-line fluxes. We then fit Schechter functions to the data:

$$ \Phi(M_{\star})dM_{\star} = \ln(10)\Phi^{*}\left(\frac{M_{\star}}{M^{*}}\right)^{\alpha+1}e^{-(M_{\star}/M^{*})}d\log M_{\star}, \quad (12) $$

where Φ(M<sub>⋆</sub>) is the number density at stellar mass M<sub>⋆</sub>, Φ<sup>*</sup> is the normalization of the stellar mass function, M<sup>*</sup> is the characteristic stellar mass, and α is the slope at the faint end of the stellar mass function. Mass bins that are less than 50 per cent complete are excluded from the fitting procedure. As with the luminosity functions, we also fix the faint-end slope α to the values derived for blank-field stellar mass functions by Sobral et al. (2014) and Khostovan et al. (2016) (i.e. α = -1.37 and -1.3 for the Hα
and [O III] + H β emitters, respectively). For [O III] + H β emitters we also fix the characteristic stellar mass, $M^*$, to the blank-field value of $log(M^*/M_\odot) = 10.96$ (Khostovan et al. 2016).

The stellar mass functions are presented in Fig. 10, with the parameters in Table 5. Uncertainties in each parameter are estimated following the same procedure as for those of the luminosity functions (see Section 3.1.7) and the correlation between the parameters and their uncertainties are shown in Fig. 11, which demonstrates that at the upper limit the characteristic stellar mass, $M^*$, is poorly constrained for all of our samples except the Hα emitters around ALESS 75.2. However, the lower limit is sufficient to show that in the $z \sim 2.3$ SMG regions the characteristic stellar mass is significantly higher than the $z \sim 2.3$ field, which suggests that the stellar mass build-up in SMG companion galaxies is further advanced than the coeval field (e.g. Muzzin et al. 2013). Due to our selection of Hα emitters the galaxies have non-negligible SFRs (though many are below the main sequence; Section 3.4). Observations using a local galaxy density estimator suggest that local environment has minimal effect on the stellar mass function of star-forming or quiescent galaxies at $z = 1.5-2$ (Papovich et al. 2018). However, there is evidence of protocluster environments being skewed towards containing galaxies with higher masses than the field (e.g. Cooke et al. 2014), consistent with our results.

## 3.6 Dark matter halo masses and evolution

We next estimate the total halo masses of the SMG environments in order to place them within the context of existing protoclusters and trace their likely evolution, focusing primarily on the overdense environments of the two SMGs at $z \sim 2.3$. Since these overdensities are unvirialized and lack a detectable ICM, the classic methods for weighing galaxy clusters cannot be used. Instead, we use two methods that have been used in protocluster studies, though the underlying assumptions required can lead to significant uncertainties, as discussed in the following subsections. The first method is detailed in Section 3.6.1 and uses the stellar-to-halo mass relation (SHMR) to estimate the high-redshift mass of the clusters (hereafter the SHMR method) and evolve it to the local Universe using the Millennium and Millennium-II simulations (McBride, Fakhouri & Ma 2009; Fakhouri, Ma & Boylan-Kolchin 2010). The second method follows Steidel et al. (1998) and assumes the region of interest is a homogeneous sphere undergoing spherical collapse and uses the overdensity parameter to estimate the $z=0$ descendant mass, which we trace back to high-redshift using the Millennium and Millennium-II simulations. This is referred to as the spherical collapse model (SCM) method and is detailed in Section 3.6.2. In Section 3.6.3, we discuss the evolution of the SMG environments compared with other systems and previous measurements.

### 3.6.1 The SHMR method for deriving halo masses

The SHMR method for estimating protocluster masses involves identifying the most massive galaxy in the structure and converting its stellar mass to a halo mass, and taking this to be the halo mass of the whole structure. This method has been employed in recent protocluster studies (e.g. Long et al. 2020; Calvi et al. 2021; Silassen et al. 2022; Ito et al. 2023) and implicitly assumes that all member galaxies occupy a single halo at the observed redshift of the structure, which may not be the case if some of the galaxies are still infalling. Nevertheless, we deem this assumption preferable to the commonly used alternative of estimating the halo masses of

each individual galaxy and summing them together (e.g. Long et al. 2020; Calvi et al. 2021), which risks ‘double-counting’ overlapping dark matter haloes to produce an overestimate of the structure halo mass. Note that we perform this calculation even for ALESS 5.1 and its surrounding [O III] + Hβ emitters despite our analyses revealing no signs of their environment being significantly overdense. This is because this method does not explicitly depend on the density of the surrounding environment, and the high stellar mass of ALESS 5.1 (see Section 3.4 and Fig. 9) suggests it may yet reside in a massive halo. We do however caution that the result obtained here likely represents an extreme upper limit on the mass of any possible structure around ALESS 5.1.

We estimate halo masses for each target SMG and their candidate companion galaxies using the SHMR from Behroozi, Wechsler & Conroy (2013). We use the relation as defined at $z=2$ for galaxies in the environments of ALESS 75.2 and 102.1, and at $z=3$ for galaxies in the environment of ALESS 5.1. For the SMGs themselves we use the stellar masses from the literature (Danielson et al. 2017; Birkin et al. 2021, see also Section 3.4). Some of our galaxies have stellar masses which lie above the range at which the SHMR is defined and for these we use the stellar-to-halo mass ratio for the largest halo mass at which the relation is defined (see fig. 7 of Behroozi et al. 2013) to convert the stellar mass to a halo mass. This affects only two Hα emitters from Pointing 5+75, along with one from Pointing 102 which we have identified as a counterpart to ALESS 102.1 (see Section 3.4 and Fig. 9). None of the [O III] + Hβ emitters have stellar masses above the range for which the SHMR is defined at $z \sim 3$, but ALESS 5.1 does lie above this range. Uncertainties on individual galaxy halo masses are estimated based on the stellar mass uncertainties and the uncertainties in the SHMR derived by Behroozi et al. (2013).

We derive halo masses of $log(M_h/M_\odot) = 11.45-14.46$ for individual Hα and [O III] + Hβ emitters, with medians of $log(M_h/M_\odot) = 12.16^{+0.21}_{-0.16}$, $11.94^{+0.46}_{-0.35}$, and $11.75^{+0.26}_{-0.12}$ for galaxies in the environments of ALESS 5.1, 75.2 and 102.1, respectively. The halo masses of the corresponding SMGs are $log(M_h/M_\odot) = 13.94^{+0.38}_{-0.21}$, $12.02^{+0.08}_{-0.25}$, and $14.39^{+0.03}_{-0.37}$, derived using their stellar masses reported in the literature (Danielson et al. 2017; Birkin et al. 2021). ALESS 5.1 and 102.1 are both the most massive galaxies in their respective environments; we therefore adopt their halo masses as the total masses of the potential structures at the observed redshifts. ALESS 75.2 is not the dominant galaxy in its environment, being surpassed in stellar (and hence inferred halo) mass by ~40 per cent of its companion Hα emitters. The most massive of these is a spectroscopically confirmed member (Popesso et al. 2009; Balestra et al. 2010) located in the Hα emitter density peak associated with the Lyα blob CDFS-LAB03 (see Fig. 8), with a halo mass of $log(M_h/M_\odot) = 14.38^{+0.01}_{-0.40}$. We thus assume this is the total mass of the surrounding structure. Since high-redshift radio galaxies are commonly found in protocluster cores (e.g. Kurk et al. 2000; Venemans et al. 2002; Kuiper et al. 2011b; Hayashi et al. 2012; Wylezalek et al. 2013; Cooke et al. 2014), we search for radio counterparts for this galaxy in the second data release from the Very Large Array 1.4 GHz survey of the ECDFS (Miller et al. 2013), for which the typical sensitivity is 7.4 µJy per 2.8 arcsec × 1.6 arcsec beam. We find no counterparts within 30 arcsec of this Hα emitter and thus rule it out as being a high-redshift radio galaxy.

The total halo masses at the observed redshift obtained using the SHMR method are thus $log(M_h/M_\odot) = 13.94^{+0.38}_{-0.21}$, $14.38^{+0.01}_{-0.40}$, and $14.39^{+0.03}_{-0.37}$ for the environments of ALESS 5.1, 75.2, and 102.1, respectively. We note that these masses may be affected by systematic uncertainties on the stellar masses
<figure><img src="image_11.png" /><figcaption>**Figure 10.** Stellar mass functions for the H$\alpha$ (left) and [O III] + H$\beta$ (right) emitters identified in this study. The data are compared with blank-field studies of emission-line galaxies at similar redshifts (Sobral et al. 2014; Khostovan et al. 2016) and shaded regions represent 1$\sigma$ uncertainties. For each of our stellar mass functions, we fix the faint-end slope to the value derived for the blank field at a similar redshift: $\alpha = -1.37$ (Sobral et al. 2014) and $\alpha = -1.3$ (Khostovan et al. 2016) for the H$\alpha$ and [O III] + H$\beta$ stellar mass functions, respectively. For the [O III] + H$\beta$ stellar mass function, we also fix the characteristic stellar mass to the blank-field value of log(M*/M$_{\odot}$) = 10.96 (Khostovan et al. 2016). For the H$\alpha$ emitters, the upper limit of M* is poorly constrained (see also Fig. 11), which leads to large uncertainties at the high mass end. There are offsets between the blank field stellar mass functions and those around our SMGs in all targeted SMG regions; these are quantified in Fig. 11 and Table 5.</figcaption></figure>

<table><caption>**Table 5.** Summary of the best-fitting stellar mass function parameters for the companion galaxies in the environments of the target SMGs. In all cases, the faint-end slope of the stellar mass function, $\alpha$, is fixed to the value from the relevant blank-field stellar mass function (Sobral et al. 2014; Khostovan et al. 2016). Where indicated, the characteristic stellar mass, M*, is also fixed to the blank-field value from Khostovan et al. (2016).</caption><thead><tr><th>SMG environment</th><th>log(M*/M<sub>☉</sub>)</th><th>log(Φ*/Mpc<sup>-3</sup>)</th></tr></thead><tbody><tr><td>ALESS 75.2</td><td>11.69<sup>+0.41</sup><sub>-0.15</sub></td><td>-3.37<sup>+0.09</sup><sub>-0.25</sub></td></tr><tr><td>ALESS 102.1</td><td>12.08<sup>+0.52</sup><sub>-0.59</sub></td><td>-4.01<sup>+0.23</sup><sub>-0.41</sub></td></tr><tr><td>SMGs at z ~ 2.3</td><td>11.85<sup>+0.30</sup><sub>-0.34</sub></td><td>-3.64<sup>+0.16</sup><sub>-0.18</sub></td></tr><tr><td>ALESS 5.1</td><td>10.96 (fixed)</td><td>-3.73<sup>+0.08</sup><sub>-0.08</sub></td></tr></tbody></table>

(due to uncertainties on star formation histories and resulting mass-to-light ratios, which is particularly relevant for young starbursts; e.g. Wardlow et al. 2011) and on the SHMR for very high-mass galaxies, which are present in the simulations from which the SHMR is derived (Behroozi et al. 2013). Indeed, predictions from halo mass functions suggest that the halo masses inferred from this method should be sufficiently rare that finding three such structures in the ~ 0.25 deg$^2$ ECDFS is unlikely (e.g. Press & Schechter 1974; Tinker et al. 2008). Therefore, we consider these SHMR-derived halo masses to be upper limits and as such they are represented by the upper bounds on Fig. 12 (the lower bounds are derived in Section 3.6.2), which compares the halo masses of the SMG environments with previously studied galaxy clusters and protoclusters.

To assess whether these SMG environments are true protoclusters, we evolve the masses derived from the SHMR method to present-day masses and compare with known galaxy clusters in the local Universe. This is done using the redshift-dependent formula for the mean mass accretion rate derived from the results of the Millennium and Millennium-II simulations (McBride et al. 2009; Fakhouri et al. 2010):

$$ \langle \dot{M} \rangle_{\text{mean}} = 46.1 \, \text{M}_{\odot} \text{yr}^{-1} \left( \frac{M_z}{10^{12} \text{M}_{\odot}} \right) (1 + 1.11z) \times \sqrt{\Omega_{m,0}(1+z)^3 + \Omega_{\Lambda,0}}, \quad (13) $$

<figure><img src="image_12.png" /><figcaption>**Figure 11.** Contours showing the correlated uncertainties on the Schechter fit parameters for the stellar mass functions shown in Fig. 10. In all cases, the faint-end slope, $\alpha$, is fixed to match to the blank-field stellar mass functions (Sobral et al. 2014; Khostovan et al. 2016), and for ALESS 5.1 M* is also fixed. Symbols and contours have the same meaning as in Fig. 6. For the H$\alpha$ emitters at z ~ 2.3 the characteristic stellar mass, M*, is effectively a lower limit due to the correlation with $\Phi^*$; this can also be seen in Fig. 10. The offset between M* for field H$\alpha$ emitters (Sobral et al. 2014) and the galaxies around z ~ 2.3 SMGs indicates that there is an excess of high-mass galaxies around the SMGs, and this is likely partially responsible for the overdensity around ALESS 75.2.</figcaption></figure>

where $M_z$ is the halo mass of the structure at its observed redshift, and $\Omega_{m,0}$ and $\Omega_{\Lambda,0}$ are the present-day density parameters for matter and the cosmological constant according to our assumed cosmology (Planck Collaboration VI 2020).

For each overdensity, we begin with the total halo masses estimated using the SHMR method and apply equation (13) to incrementally add mass in small time steps until the present day is reached. The present-day masses obtained with this method are log(M$_{h,z=0}$/M$_{\odot}$) = 15.93$^{+0.62}_{-0.33}$, 15.81$^{+0.01}_{-0.55}$, and 15.82$^{+0.04}_{-0.50}$ for the overdensities containing ALESS 5.1, 75.2, and 102.1, respectively.
<figure><img src="image_13.png" /><figcaption>**Figure 12.** A comparison of protocluster halo masses across cosmic time. Two mass estimates are obtained for the environments of ALESS 75.2 and 102.1, at both the observed redshift of the potential structure and at $z = 0$, using the methods described in Section 3.6. The evolutionary paths of these haloes across cosmic time are estimated using the redshift-dependent mean mass accretion rate formula derived from the Millennium simulations (McBride et al. 2009; Fakhouri et al. 2010). The coloured shaded regions show the possible mass ranges and evolution for each of our target SMG environments, and are labelled with the ALESS ID of the inhabiting SMG. The upper bounds of these mass ranges correspond to the masses estimated using the SHMRs from Behroozi et al. (2013), while the lower bounds are derived by assuming an SCM (see the text for details). For ALESS 5.1, only the former mass estimate is used, and is marked as an upper limit. The grey shaded region shows the expected evolution of a Coma-like cluster (Chiang et al. 2013) and coloured symbols show samples of clusters and protoclusters from CLASH, GCLASS, and CARLA, and protoclusters targeted due to their richness in DSFGs, as detailed in Section 3.6.3. We include regions showing measurements of SMG halo masses obtained from clustering studies (Hickox et al. 2012; Wilkinson et al. 2017; García-Vergara et al. 2020; Stach et al. 2021) and mark the borders between different gas regimes (Dekel & Birnboim 2006). The two $z \sim 2.3$ SMGs reside in environments consistent with protoclusters, although ALESS 102.1 may reside in a proto-group instead. The lower bounds of our mass estimates are broadly consistent with the masses obtained from SMG clustering studies, while the upper bounds imply these haloes may evolve into Coma-like structures or larger.</figcaption></figure>

These masses suggest that these structures would all evolve into some of the most massive clusters in the Universe, rivalling that of the Coma cluster (e.g. Gavazzi et al. 2009; Ho et al. 2022) and other massive clusters at $z \lesssim 1$ such as those in the Cluster Lensing And Supernova Survey with Hubble (CLASH, Postman et al. 2012; Merten et al. 2015). However, given the rarity of such massive structures seen in the local Universe, we posit that the identification of progenitor structures around all three of our target SMGs is due to the systematics in the calculations, rather than a real occurrence. In addition to the possible systematics in the stellar masses and SHMR, as previously described, we also note that the mean mass accretion rate given by equation (13) is poorly constrained for haloes with masses of $log(M_h/M_\odot) \gtrsim 14$ beyond $z \sim 0.5$. Furthermore, equation (13) alone does not account for the diversity of evolutionary paths that real dark matter haloes undergo, being the mean result for many haloes in the Millennium simulation. Therefore, as for the high-redshift SHMR-derived halo masses, we also take these SHMR-derived $z = 0$ and intermediate masses to be upper limits. This upper limit on the halo mass at the SMG redshift and the evolution to the present day is shown in Fig. 12 as the upper edges of the shaded

regions for ALESS 75.2 and 102.1, and as a single solid line for ALESS 5.1.

### 3.6.2 The SCM method for deriving halo masses

An alternative method for estimating the present-day mass of each SMG environment is obtained following Steidel et al. (1998), which approximates each SMG environment as a homogeneous sphere undergoing spherical collapse. In this case the total present-day mass is given by:

$$M_{h,z=0} = \bar{\rho}V(1 + \delta_m), \quad (14)$$

where $\bar{\rho}$ is the mean comoving matter density of the Universe, $\delta_m$ is the dark matter mass overdensity, and $V$ is the comoving volume of the structure. We refer to this method as the SCM method. Since the assumption of spherical collapse is unphysical for environments that are not overdense, we only perform this calculation for the two SMG environments at $z \sim 2.3$.

To estimate the volume of each overdensity, we assume that the structures are spherical and use the spatial extent of the structure on the sky to infer the angular diameter of the sphere containing the member galaxies. As discussed in Section 3.3, the overdensity around ALESS 75.2 extends beyond the confines of the HAWK-I pointing and therefore the size of the HAWK-I field of view can be used as a lower limit on the angular diameter of the structure. Therefore, for this environment we calculate the comoving volume for a spatial extent of $\sim 7.5$ arcmin (Kissler-Patig et al. 2008), which corresponds to a comoving volume of $V \sim 1000$ cMpc$^3$. This volume should be considered a lower limit, and thus the derived halo mass is also a lower limit. For the environment of ALESS 102.1, we assume that the structure is confined to the quadrant containing the SMG (Section 3.3), such that the angular diameter of the sphere is then $\sim 220$ arcsec (Kissler-Patig et al. 2008), which gives a comoving volume of $V \sim 110$ cMpc$^3$.

The dark matter mass overdensity is linked to the observed galaxy overdensity, $\delta_g$, via

$$\delta_m = \delta_g/b, \quad (15)$$

where $b$ is the bias parameter. To estimate the bias parameters for $H\alpha$ emitters at $z \sim 2.3$, we make use of the relationship between $b$ and $L_{H\alpha}$ derived by Cochrane et al. (2017) at $z = 2.23$. Following a similar method to that of Stott et al. (2020), we derive a linear fit to this relation and estimate $b$ at the mean value of $L_{H\alpha}$ for each SMG environment. This gives $b_{H\alpha} = 2.9_{-0.1}^{+0.2}$ and $2.8_{-0.3}^{+0.2}$ for the candidate companions of ALESS 75.2 and 102.1, respectively.

Using the above values of $b$ and $V$ along with the $\delta_g$ values calculated in Section 3.2, we obtain present-day halo masses of $log(M_{h,z=0}/M_\odot) = 13.90_{-0.10}^{+0.09}$ and $13.05_{-0.14}^{+0.12}$ for the environments of ALESS 75.2 and 102.1, respectively. Contrary to the previous present-day mass estimates, these masses suggest that the descendant of the environment of ALESS 102.1 is more akin to a galaxy group than a galaxy cluster (e.g. Han et al. 2015; Man et al. 2019), while the environment of ALESS 75.2 may evolve into a ‘Virgo-like’ or ‘Fornax-like’ cluster by $z = 0$ (e.g. Chiang et al. 2013).

The significant disparity between the present-day halo mass estimates from the SCM method compared with the SHMR method likely arises from the assumptions and uncertainties in the calculations, including the systematics discussed in Section 3.6.1 and the estimates of the structure volumes. Note that for ALESS 75.2 the derived $z = 0$ SCM halo mass is a lower limit, due to the volume used being a lower limit. Since the masses derived from
the SCM method are all lower than the those from the SHMR method and its uncertainties, we proceed with the SCM estimates and adopt the lower bound of the $1\sigma$ confidence intervals as lower limits for the present-day halo masses. Thus, the present-day halo mass estimates from the two methods gives a range of plausible evolutionary pathways for the two $z \sim 2.3$ SMG environments as shown on Fig. 12.

We also use the present-day halo mass estimates from the SCM method in combination with equation (13) to trace the evolution of these SMG environments back in time to their observed redshifts, thereby obtaining a second estimate of the total mass at these redshifts. These masses are $\log(M_h/M_\odot) = 12.93_{-0.08}^{+0.07}$ and $12.24_{-0.11}^{+0.10}$ for the potential structures around ALESS 75.2 and 102.1, respectively. This calculation provides an evolutionary track that connects the lower halo mass limit at $z \sim 2.3$ to the corresponding value at $z = 0$, and this is what defines the bottom edge of each shaded region in Fig. 12.

### 3.6.3 Halo masses and evolution

The halo mass estimates of each SMG environment are summarized in Table 6, and Fig. 12 compares the SMG halo masses and their evolution with previously studied galaxy clusters and protoclusters. As described in Sections 3.6.1 and 3.6.2, the upper and lower bounds shown for the SMG haloes are derived from the masses calculated using the SHMR method and the lower limits from the SCM method, respectively (with the exception of ALESS 5.1, for which only the SHMR method is used). Thus, this region encompasses the full range of possible halo masses and evolution for the SMGs. In Fig. 12, these are compared with clusters from CLASH (Postman et al. 2012; Merten et al. 2015); clusters from the Gemini Cluster Astrophysics Spectroscopic Survey (GCLASS; Muzzin et al. 2012; van der Burg et al. 2014); and clusters and protoclusters from the Clusters Around Radio-Loud AGN program (CARLA; Wylezalek et al. 2013; Mei et al. 2023). We also show the halo masses calculated by Casey (2016) for overdense structures known to be rich in dusty star-forming galaxies (DSFGs), including: the GOODS-N protocluster at $z = 1.99$ (Blain et al. 2004; Chapman et al. 2009); the COSMOS protoclusters at $z = 2.10$ (Yuan et al. 2014) and $z = 2.47$ (Casey et al. 2015; Chiang et al. 2015; Diener et al. 2015); the ‘Spiderweb’ protocluster, MRC 1138-256, at $z = 2.16$ (Kurk et al. 2000; Kuiper et al. 2011a); the SSA 22 protocluster at $z = 3.09$ (Steidel et al. 1998; Hayashino et al. 2004; Lehmer et al. 2009; Tamura et al. 2009; Umehata et al. 2015); and the GN20 overdensity at $z = 4.05$ (Daddi et al. 2009; Hodge et al. 2013b). In this sample of DSFG-rich protoclusters we additionally include the halo mass of the DRC at $z = 4$ (Long et al. 2020). Fig. 12 shows that the potential structures surrounding ALESS 75.2 and 102.1 are consistent with being protoclusters at $z \sim 2.3$. The upper limit for the halo mass derived for ALESS 5.1 is also consistent with this environment being a protocluster, but we emphasize that our previous analyses suggest it is unlikely to reside in such a structure.

Both the present-day and high-redshift masses obtained using the SHMR method (Section 3.6.1) are significantly higher than those obtained using the SCM method (Section 3.6.2), and higher than expected for three structures all found in a survey of $\sim 0.25$ deg$^2$ based on predictions from halo mass functions (e.g. Press & Schechter 1974; Tinker et al. 2008). This is consistent with the SHMR masses being affected by systematic effects that make them upper limits, as discussed in Section 3.6.1. The SCM-derived masses are typically lower than the masses of protoclusters at similar redshifts (see Fig. 12), but are consistent with the results of SMG clustering

studies (Hickox et al. 2012; Wilkinson et al. 2017; García-Vergara et al. 2020; Stach et al. 2021, see Fig. 12), which generally agree that SMGs reside in haloes with $\log(M_h/M_\odot) \lesssim 13$ at $z = 1-3$. However, we note that the studies by Hickox et al. (2012) and Wilkinson et al. (2017) were both conducted using single-dish observations and are thus limited by false counterpart identification and source blending. Although the high-resolution interferometric studies by García-Vergara et al. (2020) and Stach et al. (2021) are not afflicted by these limitations, they present conflicting results for the halo masses of the SMG population, likely stemming from the methodological differences described in Section 1.

Fig. 12 also includes the approximate boundaries separating different gas regimes, as proposed by Dekel & Birnboim (2006): in haloes for which $\log(M_h/M_\odot) \lesssim 12$, inflowing gas is predominantly cold and enables the growth of galaxies; in haloes above this mass threshold, these gas inflows are shock-heated resulting in strangulation of the galaxy within. However, if these massive haloes still fall below some other, redshift-dependent mass threshold (as marked by the ‘cold in hot’ boundary in Fig. 12), then penetrating cold gas may still be sustaining galaxy growth. At $z \sim 2.3$, this mass threshold is $\log(M_h/M_\odot) \sim 12.9$, while at $z = 3.3$ it is $\log(M_h/M_\odot) \sim 14.0$.

Based on our mass estimates, the halo of ALESS 5.1 is likely in the ‘cold in hot’ category at its observed redshift, particularly when noting that the mass of this halo is possibly overestimated. It is therefore probable that ALESS 5.1, along with any other galaxy that may share its halo, is undergoing growth sustained by penetrating cold gas inflows. Conversely galaxies in the environment of ALESS 75.2 are more likely to be undergoing strangulation due to shock-heating in the halo at the observed redshift; the SHMR-derived halo mass lies significantly above the limit for ‘cold in hot’ gas inflows, and the $1\sigma$ confidence interval for the SCM-derived mass only just crosses below the limit. We cannot conclude anything about the gas regime in the environment of ALESS 102.1 as the ‘cold in hot’ boundary is straddled by the mass estimates for this structure.

## 4 CONCLUSIONS

We have conducted a wide-field narrow-band survey of star-forming galaxies in the environments of three SMGs at $z \sim 2.3$ and $\sim 3.3$ to determine whether these SMGs reside in protocluster-like environments. By studying individual SMGs selecting based only on their spectroscopic redshifts we have measured ‘typical’ SMG environments. Our main conclusions are as follows:

(i) Using HAWK-I Bry and $K_s$ photometry, we identified a total of 147 candidate emission-line galaxies in the two HAWK-I pointings containing the three target SMGs. After extracting photometry from archival UV-to-NIR broad-band images, we performed SED fitting with EAZY-PY to obtain photometric redshifts for these galaxies and identified 44, 11, and 4 companion galaxies to the SMGs ALESS 75.2 ($z_{spec} = 2.294$), ALESS 102.1 ($z_{spec} = 2.296$), and ALESS 5.1 ($z_{spec} = 3.303$), respectively.

(ii) By constructing luminosity functions for each SMG environment and comparing with blank-field luminosity functions from the literature at similar redshifts, we measure overdensity parameters of $\delta_g = 0.2^{+2.5}_{-0.7}$, $2.6^{+1.4}_{-1.2}$, and $0.2^{+0.6}_{-0.5}$ across the whole $\sim 4$ Mpc HAWK-I field of view for ALESS 5.1, 75.2, and 102.1, respectively. Whilst ALESS 102.1 is not overdense on these large scales, it does sit in a $\sim 1$ Mpc region with $\delta_g = 3.8^{+2.4}_{-1.8}$. Therefore 2/3 of the target SMGs reside in overdense environments.

(iii) We considered the spatial distribution of companion Hα emitters in the environments of the two SMGs at $z \sim 2.3$ (ALESS
<table><caption>**Table 6.** Halo mass estimates for each SMG environment.</caption><thead><tr><th>SMG</th><th>log(M<sub>h</sub><sup>SHMR</sup>/M<sub>☉</sub>)<sup>a</sup></th><th>log(M<sub>h,z=0</sub><sup>SHMR</sup>/M<sub>☉</sub>)<sup>b</sup></th><th>log(M<sub>h</sub><sup>SCM</sup>/M<sub>☉</sub>)<sup>c</sup></th><th>log(M<sub>h,z=0</sub><sup>SCM</sup>/M<sub>☉</sub>)<sup>d</sup></th></tr></thead><tbody><tr><td>ALESS 5.1</td><td>13.94<sup>+0.38</sup><sub>-0.21</sub></td><td>15.93<sup>+0.63</sup><sub>-0.33</sub></td><td>–</td><td>–</td></tr><tr><td>ALESS 75.2</td><td>14.38<sup>+0.01</sup><sub>-0.40</sub></td><td>15.81<sup>+0.01</sup><sub>-0.55</sub></td><td>12.93<sup>+0.07</sup><sub>-0.08</sub></td><td>13.90<sup>+0.09</sup><sub>-0.10</sub></td></tr><tr><td>ALESS 102.1</td><td>14.39<sup>+0.03</sup><sub>-0.37</sub></td><td>15.82<sup>+0.04</sup><sub>-0.50</sub></td><td>12.24<sup>+0.10</sup><sub>-0.11</sub></td><td>13.05<sup>+0.12</sup><sub>-0.14</sub></td></tr></tbody></table>

Notes. a Halo mass derived using the SHMR method (Section 3.6.1) at the redshift of the SMG (i.e. $z \sim 2.3$ for ALESS 75.2 and 102.1, and $z \sim 3.3$ for ALESS 5.1). As discussed in Section 3.6.1, we consider these to be upper limits.

b Halo mass derived using the SHMR method and evolved to $z=0$ using equation (13); these masses are considered to be upper limits (see Section 3.6.1).

c Halo mass derived using the SCM method and traced back to the SMG redshift using equation (13) (Section 3.6.2).

d Descendant halo mass at $z=0$ derived using the SCM method (Section 3.6.2).

75.2 and 102.1) by measuring their density in annuli around the SMGs and by constructing overdensity maps. For ALESS 75.2, the companion galaxies are spread out across the entire HAWK-I field of view, spanning a few Mpc. This is consistent with simulations, in which protoclusters are seen to extend over several Mpc at $z \sim 2-3$. The SMG resides near a possible density peak of H$\alpha$ emitters, although a greater peak is seen a few arcminutes eastward which appears to be associated with a previously discovered Ly$\alpha$ blob (Yang et al. 2010). The overdensity around ALESS 102.1 is smaller ($\sim 1$ Mpc) and could instead evolve into a galaxy group locally.

(iv) Stellar masses and SFRs were obtained for the companion galaxies in each SMG environment by performing SED fitting with MAGPHYS. The galaxies are generally scattered about the star-forming main sequence at their respective epochs, with no evidence of enhanced star formation activity in either environment.

(v) Two methods were used to estimate the total halo mass of each of the two overdense SMG environments, which provided upper and lower bounds on the halo masses at the observed redshifts and evolving to the present day. These reveal that ALESS 75.2 likely resides in a protocluster, while ALESS 102.1 resides in either a protocluster or a protogroup.

(vi) We therefore surmise that 2/3 of these SMGs are strong candidates for the progenitors of massive elliptical galaxies in clusters, although the possibility remains for them to end up in galaxy groups. If these targets are indeed representative of 'typical' SMGs then this suggests that SMGs in general are likely to evolve into massive elliptical galaxies by the present day, as suggested by certain evolutionary models (e.g. Sanders et al. 1988; Hopkins et al. 2008), but with significant variation in the surrounding environments.

With this study we have demonstrated the efficacy of narrow-band surveys as a means of searching for galaxy overdensities around SMGs selected without bias towards particular environments. Future followup with larger samples of SMGs and/or spectroscopic confirmation of companion galaxies would confirm the nature of the overdensities that we have detected, and resolved analyses (e.g. with ALMA and/or JWST) will further reveal how the member galaxies evolve.

## ACKNOWLEDGEMENTS

We are extremely grateful to Ian Smail for numerous insightful discussions, which significantly improved this work. The research is based on observations collected at the European Southern Observatory under ESO programme 0103.A-0668(A). TMC received support from the Science and Technology Facilities Council (STFC; 2287406) and the Faculty of Science and Technology at Lancaster University. JW acknowledges an STFC Ernest Rutherford Fellowship (ST/P004784/2). HD acknowledges support from the Agencia

Estatal de Investigación del Ministerio de Ciencia, Innovación y Universidades (MCIU/AEI) under grant (Construcción de cúmulos de galaxias en formación a través de la formación estelar oscurecida por el polvo) and the European Regional Development Fund (ERDF) with reference (PID2022-143243NB-I00/10.13039/501100011033). The authors would also like to acknowledge the publicly available software TOPCAT (Taylor 2005), as well as the PYTHON packages NUMPY (Harris et al. 2020), SCIPY (Virtanen et al. 2020), ASTROPY (Astropy Collaboration 2013, 2018), MATPLOTLIB (Hunter 2007), and PHOTUTILS (Bradley et al. 2022).

## DATA AVAILABILITY

The data used in this publication can be accessed via the ESO Science Archive under Program ID: 0103-0668(A), PI: Wardlow.

## REFERENCES

Angulo R. E., Springel V., White S. D. M., Cole S., Jenkins A., Baugh C. M., Frenk C. S., 2012, MNRAS, 425, 2722

Arsenault R. et al., 2008, in Hubin N., Max C. E., Wizinowich P. L. eds, Proc. SPIE Conf. Ser., Vol. 7015, Adaptive Optics Systems. SPIE, Bellingham, p.701524

Astropy Collaboration, 2013, A&A, 558, A33

Astropy Collaboration, 2018, AJ, 156, 123

Balestra I. et al., 2010, A&A, 512, A12

Barger A. J., Cowie L. L., Sanders D. B., Fulton E., Taniguchi Y., Sato Y., Kawara K., Okuda H., 1998, Nature, 394, 248

Barger A. J., Wang W. H., Cowie L. L., Owen F. N., Chen C. C., Williams J. P., 2012, ApJ, 761, 89

Baugh C. M., Cole S., Frenk C. S., Lacey C. G., 1998, ApJ, 498, 504

Baugh C. M., Lacey C. G., Frenk C. S., Granato G. L., Silva L., Bressan A., Benson A. J., Cole S., 2005, MNRAS, 356, 1191

Behroozi P. S., Wechsler R. H., Conroy C., 2013, ApJ, 770, 57

Bertin E., 2006, in Gabriel C., Arviset C., Ponz D., Enrique S. eds, ASP Conf. Ser., Vol. 351, Astronomical Data Analysis Software and Systems XV. Astron. Soc. Pac., San Francisco, p. 112

Bertin E., 2010, Astrophysics Source Code Library, record ascl:1010.068

Bertin E., Arnouts S., 1996, A&AS, 117, 393

Béthermin M., Dole H., Lagache G., Le Borgne D., Penin A., 2011, A&A, 529, A4

Birkin J. E. et al., 2021, MNRAS, 501, 3926

Blain A. W., Smail I., Ivison R. J., Kneib J. P., Frayer D. T., 2002, Phys. Rep., 369, 111

Blain A. W., Chapman S. C., Smail I., Ivison R., 2004, ApJ, 611, 725

Blakeslee J. P. et al., 2003, ApJ, 596, L143

Bleem L. E. et al., 2015, ApJS, 216, 27

Böhringer H. et al., 2001, A&A, 369, 826

Bond J. R., Kofman L., Pogosyan D., 1996, Nature, 380, 603

Bothwell M. S. et al., 2013, MNRAS, 429, 3047

Bradley L. et al., 2022, astropy/photutils: https://doi.org/10.5281/zenodo.6385735
Brammer G. B., van Dokkum P. G., Coppi P., 2008, ApJ, 686, 1503

Brinchmann J., Charlot S., White S. D. M., Tremonti C., Kauffmann G., Heckman T., Brinkmann J., 2004, MNRAS, 351, 1151

Bunker A. J., Warren S. J., Hewett P. C., Clements D. L., 1995, MNRAS, 273, 513

Calhau J., Sobral D., Stroe A., Best P., Smail I., Lehmer B., Harrison C., Thomson A., 2017, MNRAS, 464, 303

Calvi R., Dannerbauer H., Arrabal Haro P., Rodríguez Espinosa J. M., Muñoz-Tuñón C., Pérez González P. G., Geier S., 2021, MNRAS, 502, 4558

Calzetti D., Armus L., Bohlin R. C., Kinney A. L., Koornneef J., Storch-Bergmann T., 2000, ApJ, 533, 682

Cardamone C. N. et al., 2010, ApJS, 189, 270

Casali M. et al., 2006, in McLean I. S., Iye M. eds, Proc. SPIE Conf. Ser. Vol. 6269, Ground-based and Airborne Instrumentation for Astronomy. SPIE, Bellingham, p.62690W

Casey C. M., 2016, ApJ, 824, 36

Casey C. M., Narayanan D., Cooray A., 2014, Phys. Rep., 541, 45

Casey C. M. et al., 2015, ApJ, 808, L33

Chabrier G., 2003, PASP, 115, 763

Chapman S. C. et al., 2005, ApJ, 622, 772

Chapman S. C., Blain A., Ibata R., Ivison R. J., Smail I., Morrison G., 2009, ApJ, 691, 560

Cheng T. et al., 2019, MNRAS, 490, 3840

Chiang Y.-K., Overzier R., Gebhardt K., 2013, ApJ, 779, 127

Chiang Y.-K. et al., 2015, ApJ, 808, 37

Chiang Y.-K. et al., 2017, ApJ, 844, L23

Clements D. L., Dunne L., Eales S., 2010, MNRAS, 403, 274

Cochrane R. K., Best P. N., Sobral D., Smail I., Wake D. A., Stott J. P., Geach J. E., 2017, MNRAS, 469, 2913

Cooke E. A., Hatch N. A., Muldrew S. I., Rigby E. E., Kurk J. D., 2014, MNRAS, 440, 3262

Cooper M. C. et al., 2008, MNRAS, 383, 1058

Coppin K. et al., 2006, MNRAS, 372, 1621

Cucciati O. et al., 2014, A&A, 570, A16

da Cunha E., Charlot S., Elbaz D., 2008, MNRAS, 388, 1595

da Cunha E. et al., 2015, ApJ, 806, 110

da Cunha E. et al., 2021, ApJ, 919, 30

Daddi E. et al., 2007, ApJ, 670, 156

Daddi E. et al., 2009, ApJ, 694, 1517

Damen M. et al., 2011, ApJ, 727, 1

Danielson A. L. R. et al., 2017, ApJ, 840, 78

Darvish B., Mobasher B., Sobral D., Rettura A., Scoville N., Faisst A., Capak P., 2016, ApJ, 825, 113

Davé R., Finlator K., Oppenheimer B. D., Fardal M., Katz N., Kereš D., Weinberg D. H., 2010, MNRAS, 404, 1355

Davies L. J. M., Bremer M. N., Stanway E. R., Husband K., Lehnert M. D., Mannering E. J. A., 2014, MNRAS, 438, 2732

De Lucia G., Blaizot J., 2007, MNRAS, 375, 2

Dekel A., Birnboim Y., 2006, MNRAS, 368, 2

Diener C. et al., 2013, ApJ, 765, 109

Diener C. et al., 2015, ApJ, 802, 31

Dressler A., 1980, ApJ, 236, 351

Dudzevičiūtė U. et al., 2020, MNRAS, 494, 3828

Eales S., Lilly S., Gear W., Dunne L., Bond J. R., Hammer F., Le Fèvre O., Crampton D., 1999, ApJ, 515, 518

Eddington A. S., 1913, MNRAS, 73, 359

Elbaz D. et al., 2007, A&A, 468, 33

Ellis R. S., Smail I., Dressler A., Couch W. J., Oemler Augustus J., Butcher H., Sharples R. M., 1997, ApJ, 483, 582

Fakhouri O., Ma C.-P., Boylan-Kolchin M., 2010, MNRAS, 406, 2267

Finkelstein S. L. et al., 2022, ApJ, 928, 52

García-Vergara C., Hodge J., Hennawi J. F., Weiss A., Wardlow J., Myers A. D., Hickox R., 2020, ApJ, 904, 2

Garn T. et al., 2010, MNRAS, 402, 2017

Gavazzi R., Adami C., Durret F., Cuillandre J. C., Ilbert O., Mazure A., Pelló R., Ulmer M. P., 2009, A&A, 498, L33

Gawiser E. et al., 2006a, ApJS, 162, 1

Gawiser E. et al., 2006b, ApJ, 642, L13

Geach J. E. et al., 2005, MNRAS, 363, 1398

Geach J. E., Smail I., Best P. N., Kurk J., Casali M., Ivison R. J., Coppin K., 2008, MNRAS, 388, 1473

Geach J. E. et al., 2016, ApJ, 832, 37

Gilbank D. G., Gladders M. D., Yee H. K. C., Hsieh B. C., 2011, AJ, 141, 94

Gladders M. D., Yee H. K. C., 2000, AJ, 120, 2148

Gladders M. D., Yee H. K. C., 2005, ApJS, 157, 1

González V., Labbé I., Bouwens R. J., Illingworth G., Franx M., Kriek M., Brammer G. B., 2010, ApJ, 713, 115

Greenslade J., Clements D. L., Petitpas G., Asboth V., Conley A., Pérez-Fournon I., Riechers D., 2020, MNRAS, 496, 2315

Greve T. R. et al., 2005, MNRAS, 359, 1165

Gruppioni C. et al., 2013, MNRAS, 432, 23

Gullberg B. et al., 2019, MNRAS, 490, 4956

Hainline L. J., Blain A. W., Smail I., Alexander D. M., Armus L., Chapman S. C., Ivison R. J., 2011, ApJ, 740, 96

Han J. et al., 2015, MNRAS, 446, 1356

Harris C. R. et al., 2020, Nature, 585, 357

Hasselfield M. et al., 2013, J. Cosmol. Astropart. Phys., 2013, 008

Hatch N. A., Kurk J. D., Pentericci L., Venemans B. P., Kuiper E., Miley G. K., Röttgering H. J. A., 2011, MNRAS, 415, 2993

Hayashi M., Kodama T., Tadaki K.-i., Koyama Y., Tanaka I., 2012, ApJ, 757, 15

Hayashino T. et al., 2004, AJ, 128, 2073

Hayward C. C. et al., 2021, MNRAS, 502, 2922

Henry J. P., Mullis C. R., Voges W., Böhringer H., Briel U. G., Gioia I. M., Huchra J. P., 2006, ApJS, 162, 304

Hickox R. C. et al., 2012, MNRAS, 421, 284

Hildebrandt H. et al., 2006, A&A, 452, 1121

Ho M., Ntampaka M., Rau M. M., Chen M., Lansberry A., Ruehle F., Trac H., 2022, Nat. Astron., 6, 936

Hodge J. A., da Cunha E., 2020, R. Soc. Open Sci., 7, 200556

Hodge J. A. et al., 2013a, ApJ, 768, 91

Hodge J. A., Carilli C. L., Walter F., Daddi E., Riechers D., 2013b, ApJ, 776, 22

Hodge J. A. et al., 2016, ApJ, 833, 103

Hopkins P. F., Hernquist L., Cox T. J., Kereš D., 2008, ApJS, 175, 356

Hsieh B.-C., Wang W.-H., Hsieh C.-C., Lin L., Yan H., Lim J., Ho P. T. P., 2012, ApJS, 203, 23

Hughes D. H. et al., 1998, Nature, 394, 241

Hunter J. D., 2007, Comput. Sci. Eng., 9, 90

Ikarashi S. et al., 2015, ApJ, 810, 133

Ito K. et al., 2023, ApJ, 945, L9

Ivison R. J. et al., 2000, ApJ, 542, 27

Ivison R. J. et al., 2013, ApJ, 772, 137

Kennicutt Robert C. J., 1998, ARA&A, 36, 189

Khostovan A. A. et al., 2015, MNRAS, 452, 3948

Khostovan A. A., Sobral D., Mobasher B., Smail I., Darvish B., Nayyeri H., Hemmati S., Stott J. P., 2016, MNRAS, 463, 2363

Kissler-Patig M. et al., 2008, A&A, 491, 941

Koprowski M. P., Dunlop J. S., Michalowski M. J., Cirasuolo M., Bowler R. A. A., 2014, MNRAS, 444, 117

Koyama Y., Kodama T., Tadaki K.-i., Hayashi M., Tanaka M., Smail I., Tanaka I., Kurk J., 2013, MNRAS, 428, 1551

Kriek M., Conroy C., 2013, ApJ, 775, L16

Kron R. G., 1980, ApJS, 43, 305

Kuiper E. et al., 2011a, MNRAS, 415, 2245

Kuiper E. et al., 2011b, MNRAS, 417, 1088

Kurk J. D. et al., 2000, A&A, 358, L1

Lacey C. G., Baugh C. M., Frenk C. S., Silva L., Granato G. L., Bressan A., 2008, MNRAS, 385, 1155

Lacey C. G., Baugh C. M., Frenk C. S., Benson A. J., Orsi A., Silva L., Granato G. L., Bressan A., 2010, MNRAS, 405, 2

Le Fèvre O. et al., 2005, A&A, 439, 845

Lehmer B. D. et al., 2005, ApJS, 161, 21

Lehmer B. D. et al., 2009, MNRAS, 400, 299

Lemaux B. C. et al., 2014, A&A, 572, A41

Lemaux B. C. et al., 2022, A&A, 662, A33
Long A. S. et al., 2020, ApJ, 898, 133

Lovell C. C., Geach J. E., Davé R., Narayanan D., Li Q., 2021, MNRAS, 502, 772

MacKenzie T. P. et al., 2017, MNRAS, 468, 4006

Madau P., Dickinson M., 2014, ARA&A, 52, 415

Magnelli B. et al., 2012, A&A, 539, A155

Man Z.-Y., Peng Y.-J., Shi J.-J., Kong X., Zhang C.-P., Dou J., Guo K.-X., 2019, ApJ, 881, 74

Matsuda Y. et al., 2004, AJ, 128, 569

Matsuda Y. et al., 2011, MNRAS, 416, 2041

McBride J., Fakhouri O., Ma C.-P., 2009, MNRAS, 398, 1858

Mei S. et al., 2023, A&A, 670, A58

Merten J. et al., 2015, ApJ, 806, 4

Michałowski M. J., Dunlop J. S., Cirasuolo M., Hjorth J., Hayward C. C., Watson D., 2012, A&A, 541, A85

Michałowski M. J. et al., 2017, MNRAS, 469, 492

Miettinen O. et al., 2017, A&A, 606, A17

Miller N. A. et al., 2013, ApJS, 205, 13

Moorwood A. F. M., van der Werf P. P., Cuby J. G., Oliva E., 2000, A&A, 362, 9

Muldrew S. I., Hatch N. A., Cooke E. A., 2015, MNRAS, 452, 2528

Muzzin A. et al., 2009, ApJ, 698, 1934

Muzzin A. et al., 2012, ApJ, 746, 188

Muzzin A. et al., 2013, ApJ, 777, 18

Narayanan D., Hayward C. C., Cox T. J., Hernquist L., Jonsson P., Younger J. D., Groves B., 2010, MNRAS, 401, 1613

Narayanan D. et al., 2015, Nature, 525, 496

Niemi S.-M., Somerville R. S., Ferguson H. C., Huang K.-H., Lotz J., Koekemoer A. M., 2012, MNRAS, 421, 1539

Oke J. B., Gunn J. E., 1983, ApJ, 266, 713

Osterbrock D. E., Ferland G. J., 2006, Astrophysics of Gaseous Nebulae and Active Galactic Nuclei. University Science Books, Sausalito, CA

Overzier R. A., 2016, A&AR, 24, 14

Overzier R. A., Guo Q., Kauffmann G., De Lucia G., Bouwens R., Lemson G., 2009, MNRAS, 394, 577

Pacaud F. et al., 2016, A&A, 592, A2

Palunas P., Teplitz H. I., Francis P. J., Williger G. M., Woodgate B. E., 2004, ApJ, 602, 545

Pantoni L. et al., 2021, MNRAS, 504, 928

Papovich C. et al., 2018, ApJ, 854, 30

Paufique J. et al., 2010, Ellerbroek B. L., Hart M., Hubin N., Wizinowich P. L., eds, Proc. SPIE Vol. 7736, Adaptive Optics Systems II. SPIE, Bellingham, p. 57

Pirard J.-F. et al., 2004, in Moorwood A. F. M., Iye M. eds, Proc. SPIE Conf. Ser. Vol. 5492, Ground-based Instrumentation for Astronomy. SPIE, Bellingham, p. 1763

Planck Collaboration VI, 2020, A&A, 641, A6

Planck Collaboration XXVII, 2016, A&A, 594, A27

Pope A. et al., 2006, MNRAS, 370, 1185

Popesso P. et al., 2009, A&A, 494, 443

Postman M. et al., 2012, ApJS, 199, 25

Press W. H., Schechter P., 1974, ApJ, 187, 425

Ramsay S. K., Mountain C. M., Geballe T. R., 1992, MNRAS, 259, 751

Rowan-Robinson M. et al., 2018, A&A, 619, A169

Sanders D. B., Soifer B. T., Elias J. H., Madore B. F., Matthews K., Neugebauer G., Scoville N. Z., 1988, ApJ, 325, 74

Schechter P., 1976, ApJ, 203, 297

Schreiber C. et al., 2015, A&A, 575, A74

Scoville N. et al., 2013, ApJS, 206, 3

Scoville N. et al., 2017, ApJ, 837, 150

Shimasaku K. et al., 2003, ApJ, 586, L111

Siebenmorgen R., Carraro G., Valenti E., Petr-Gotzens M., Brammer G., Garcia E., Casali M., 2011, The Messenger, 144, 9

Silassen N. B. et al., 2022, A&A, 665, L7

Silverman J. D. et al., 2010, ApJS, 191, 124

Simpson J. M. et al., 2014, ApJ, 788, 125 (S14)

Smail I., Ivison R. J., Blain A. W., 1997, ApJ, 490, L5

Smail I., Ivison R. J., Gilbank D. G., Dunlop J. S., Keel W. C., Motohara K., Stevens J. A., 2003, ApJ, 583, 551

Smith D. J. B., Hayward C. C., Jarvis M. J., Simpson C., 2017, MNRAS, 471, 2453

Smolčić V. et al., 2017, A&A, 597, A4

Sobral D. et al., 2009, MNRAS, 398, 75

Sobral D., Best P. N., Matsuda Y., Smail I., Geach J. E., Cirasuolo M., 2012, MNRAS, 420, 1926

Sobral D., Smail I., Best P. N., Geach J. E., Matsuda Y., Stott J. P., Cirasuolo M., Kurk J., 2013, MNRAS, 428, 1128

Sobral D., Best P. N., Smail I., Mobasher B., Stott J., Nisbet D., 2014, MNRAS, 437, 3516

Sobral D. et al., 2015, MNRAS, 451, 2303

Speagle J. S., Steinhardt C. L., Capak P. L., Silverman J. D., 2014, ApJS, 214, 15

Stach S. M. et al., 2019, MNRAS, 487, 4648

Stach S. M. et al., 2021, MNRAS, 504, 172

Staniszewski Z. et al., 2009, ApJ, 701, 32

Steidel C. C., Adelberger K. L., Dickinson M., Giavalisco M., Pettini M., Kellogg M., 1998, ApJ, 492, 428

Steidel C. C., Adelberger K. L., Shapley A. E., Pettini M., Dickinson M., Giavalisco M., 2000, ApJ, 532, 170

Steidel C. C., Adelberger K. L., Shapley A. E., Erb D. K., Reddy N. A., Pettini M., 2005, ApJ, 626, 44

Stevans M. L. et al., 2021, ApJ, 921, 58

Stott J. P. et al., 2020, MNRAS, 497, 3083

Sunyaev R. A., Zeldovich Y. B., 1972, Comments Astrophys. Space Phys., 4, 173

Swinbank A. M. et al., 2014, MNRAS, 438, 1267

Tacconi L. J. et al., 2006, ApJ, 640, 228

Tamura Y. et al., 2009, Nature, 459, 61

Tanaka I. et al., 2011, PASJ, 63, 415

Taylor M. B., 2005, in Shopbell P., Britton M., Ebert R. eds, ASP Conf. Ser. Vol. 347, Astronomical Data Analysis Software and Systems XIV. Astron. Soc. Pac., San Francisco, p. 29

Taylor E. N. et al., 2009, ApJS, 183, 295

Tinker J., Kravtsov A. V., Klypin A., Abazajian K., Warren M., Yepes G., Gottlöber S., Holz D. E., 2008, ApJ, 688, 709

Toft S. et al., 2014, ApJ, 782, 68

Treister E. et al., 2009, ApJ, 693, 1713

Trümper J., 1993, Science, 260, 1769

Umehata H. et al., 2015, ApJ, 815, L8

Umehata H. et al., 2019, Science, 366, 97

van der Burg R. F. J., Muzzin A., Hoekstra H., Wilson G., Lidman C., Yee H. K. C., 2014, A&A, 561, A79

Venemans B. P. et al., 2002, ApJ, 569, L11

Venemans B. P. et al., 2005, A&A, 431, 793

Virtanen P. et al., 2020, Nat. Methods, 17, 261

Wardlow J. L. et al., 2011, MNRAS, 415, 1479

Weiß A. et al., 2009, ApJ, 707, 1201

Wilkinson A. et al., 2017, MNRAS, 464, 1380

Williamson R. et al., 2011, ApJ, 738, 139

Wilson G. et al., 2009, ApJ, 698, 1943

Wylezalek D. et al., 2013, ApJ, 769, 79

Yajima H. et al., 2022, MNRAS, 509, 4037

Yang Y., Zabludoff A., Eisenstein D., Davé R., 2010, ApJ, 719, 1654

Yuan T. et al., 2014, ApJ, 795, L20

Zavala J. A. et al., 2019, ApJ, 887, 183

Zheng X. Z., Cai Z., An F. X., Fan X., Shi D. D., 2021, MNRAS, 500, 4354

This paper has been typeset from a TeX/Latex file prepared by the author.