# The stellar mass function of quiescent galaxies in 2 < z < 2.5 protoclusters

Adit Edward, Michael Balogh, Yannick Bahé, M Cooper, Nina Hatch, Justin

Marchioni, Adam Muzzin, Allison Noble, Gregory Rudnick, Benedetta

Vulcani, et al.

► To cite this version:

Adit Edward, Michael Balogh, Yannick Bahé, M Cooper, Nina Hatch, et al.. The stellar mass function of quiescent galaxies in 2 < z < 2.5 protoclusters. Monthly Notices of the Royal Astronomical Society, 2024, 527 (3), pp.8598-8617. 10.1093/mnras/stad3751. hal-04477721

HAL Id: hal-04477721

https://hal.science/hal-04477721v1

Submitted on 13 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# The stellar mass function of quiescent galaxies in $2 < z < 2.5$ protoclusters

Adit H. Edward,$^{1,2*}$ Michael L. Balogh,$^{1,2}$ Yannick M. Bahé,$^{3,4}$ M. C. Cooper$^{1,5}$, Nina A. Hatch$^{1,6}$

Justin Marchioni,$^{1,2}$ Adam Muzzin,$^{4,7}$ Allison Noble,$^{8}$ Gregory H. Rudnick$^{1,9}$, Benedetta Vulcani$^{1,10}$

Gillian Wilson,$^{11}$ Gabriella De Lucia$^{1,12}$, Ricardo Demarco$^{1,13}$, Ben Forrest$^{1,14}$

Michaela Hirschmann$^{1,15}$, Gianluca Castignani,$^{16,17}$ Pierluigi Cerulo,$^{18}$ Rose A. Finn$^{1,19}$

Guillaume Hewitt,$^{1,2}$ Pascale Jablonka,$^{20,21}$ Tadayuki Kodama,$^{22}$ Sophie Maurogordato,$^{23}$

Julie Nantais$^{1,24}$ and Lizhi Xie$^{1,25}$

*Affiliations are listed at the end of the paper*

Accepted 2023 December 1. Received 2023 November 28; in original form 2023 August 9

## ABSTRACT

We present an analysis of the galaxy stellar mass function (SMF) of 14 known protoclusters between $2.0 < z < 2.5$ in the COSMOS field, down to a mass limit of $10^{9.5} M_{\odot}$. We use existing photometric redshifts with a statistical background subtraction, and consider star-forming and quiescent galaxies identified from (NUV $-$ r) and (r $-$ J) colours separately. Our fiducial sample includes galaxies within 1 Mpc of the cluster centres. The shape of the protocluster SMF of star-forming galaxies is indistinguishable from that of the general field at this redshift. Quiescent galaxies, however, show a flatter SMF than in the field, with an upturn at low mass, though this is only significant at $\sim 2\sigma$. There is no strong evidence for a dominant population of quiescent galaxies at any mass, with a fraction <15 per cent at $1\sigma$ confidence for galaxies with $\log M_{\star}/M_{\odot} < 10.5$. We compare our results with a sample of galaxy groups at $1 < z < 1.5$, and demonstrate that a significant amount of environmental quenching must take place between these epochs, increasing the relative abundance of high-mass ($M_{\star} > 10^{10.5} M_{\odot}$) quiescent galaxies by a factor $\gtrsim 2$. However, we find that at lower masses ($M_{\star} < 10^{10.5} M_{\odot}$), no additional environmental quenching is required.

**Key words:** galaxies: clusters: general – galaxies: evolution.

## 1 INTRODUCTION

Observations of the galaxy stellar mass function (SMF), primarily from photometric redshift surveys, have demonstrated that most of the stellar mass in the Universe forms by $z \sim 2$ (e.g. Dickinson et al. 2003; Rudnick et al. 2003, 2006; Ilbert et al. 2013; Muzzin et al. 2013; Davidzon et al. 2017; Leja et al. 2020; McLeod et al. 2021; Santini et al. 2022; Taylor et al. 2023; Weaver et al. 2023). These studies have shown that the shape of the SMF for star-forming galaxies alone evolves only weakly with redshift below $z \sim 2$, and therefore that subsequent growth via star formation must cease for a significant number of galaxies. This process, known as quenching, leads to a gradual accumulation of non-star forming, passively evolving galaxies. Peng et al. (2010) showed that the evolution of SMFs over $0 < z < 2$ can be matched by an empirical model in which galaxies quench with a probability that is proportional to their star formation rate (SFR). Other authors have shown that this can be achieved with a quenching probability that is more fundamentally related to halo mass (Dekel & Birnboim 2006). This empirical model is often referred to as mass quenching, and is likely driven in part by energy injection due to AGN (Silk & Rees 1998; Hopkins et al. 2006) and supernova feedback (Dekel & Silk 1986; Ceverino & Klypin 2009).

Galaxies are also affected by their environment, and processes like ram pressure stripping (Gunn & Gott 1972; Poggianti et al. 2017), starvation (Larson, Tinsley & Caldwell 1980; Balogh, Navarro & Morris 2000), and galaxy mergers (Springel, Di Matteo & Hernquist 2005) can lead to environmental quenching and an excess fraction of passive galaxies in high-density regions (e.g. Blanton et al. 2005; Wetzel, Tinker & Conroy 2012), such as galaxy clusters (Lewis et al. 2002; Gómez et al. 2003) and galaxy groups (McGee et al. 2011).

Baldry et al. (2006) and Peng et al. (2010) showed that the fraction of quenched galaxies at $z = 0$ depends separably on mass and environment. The simplest interpretation is that the effectiveness of environmental quenching is independent of galaxy mass. However, observations have shown that this separability does not hold at higher redshifts (e.g. Kawinwanichakij et al. 2017; Pintos-Castro et al. 2019). For example, an analysis of clusters at $0.8 < z < 1.5$ in the GCLASS (Gemini CLuster Astrophysics Spectroscopic Survey) and GOGREEN (Gemini Observations of Galaxies in Rich Early Environments) surveys (Balogh et al. 2021) shows that the excess quenched fraction in clusters relative to the field is strongly mass dependent (Balogh et al. 2016; van der Burg et al. 2020). In particular, for massive galaxies only, the excess of quenched galaxies relative to the field is as high as it is in the local Universe. The stellar populations in these galaxies are also very old (Webb et al. 2020), indicating that they likely ceased forming stars long before they were part of a rich cluster. This is consistent with earlier work by Thomas et al. (2005), who show that most star formation in early-type galaxies located

<footnote>* E-mail: ahedward@uwaterloo.ca</footnote>
in high-density environments is expected to have happened between $3 < z < 5$. This may partly be attributed to a 'preprocessing' that occurs in groups and filaments long before galaxies are accreted into massive clusters (e.g. Reeves et al. 2021; Werner et al. 2022). Alternatively, or in addition, there may be a 'primordial' population of massive quiescent galaxies that were formed during the very earliest stages of cluster assembly (see also Poggianti et al. 2006). For quiescent galaxies with lower stellar mass, $\lesssim 10^{10.5} M_\odot$, there is strong evidence that their star formation ceased much later, upon first infall into a massive cluster (Muzzin et al. 2014; McNab et al. 2021), leading to a more gradual build up of quiescent galaxies in clusters (e.g. Gilbank & Balogh 2008). Alternatively, Baxter et al. (2022, 2023) showed that an accretion-based quenching model could work at all masses if the quenching time-scale is dependent on mass, such that massive galaxies quench more quickly and earlier than less massive galaxies.

By definition, primordial quenching would have occurred within protoclusters – the overdense, pre-virialized volumes at $z \gtrsim 2$ that will eventually collapse and form massive clusters. These volumes are very large, and only modestly overdense (Muldrew, Hatch & Cooke 2015; Chiang et al. 2017). Direct observation of the galaxy population in these regions is required to decouple the primordial quiescent population from later accretion-driven quenching. This is challenging, as it requires a survey of galaxies over a wide area that is unbiased (e.g. with respect to SFR and dust content) down to a sufficiently low stellar mass in order to study the regime at which accretion-driven quenching is dominant. The most accurate way to identify protocluster members is exploiting a highly complete, deep spectroscopic survey above $z > 2$, which does not yet exist. Though there have been spectroscopic observations of protoclusters above this redshift (e.g. Yuan et al. 2014; Diener et al. 2015; Lee et al. 2016; Wang et al. 2016; Darvish et al. 2020; McConachie et al. 2022; Ito et al. 2023), these are insufficient in completeness, spatial extent, and depth. The alternative is to use photometric redshifts. The larger uncertainties associated with these redshifts, however, mean large samples are required so that the signal from these modest overdensities can be extracted in the presence of a dominant background.

For this reason we use the data from the Cosmic Evolution Survey (COSMOS; Scoville et al. 2007), the survey with the best photometric redshifts over a cosmologically significant area. More specifically, we take advantage of the deep ($\sim 26$ AB) multiband photometry from the COSMOS2020 catalogue (Weaver et al. 2022, hereafter W22), covering $\sim 2$ deg$^2$. In this paper, we analyse the SMFs of quiescent and star-forming galaxies within 14 previously identified protoclusters in this field, selected from the catalogue of Ata et al. (2022) to be at $2.0 < z < 2.5$. In constructing the SMFs we largely follow the methodology described in Weaver et al. (2023, hereafter W23).

This paper is structured as follows. In Section 2, we discuss the galaxy sample selection and stellar mass completeness, as well as how we select protocluster members given the photometric redshift precision. Our methodology for constructing the SMFs is presented in Section 3, and the results are described in Section 4. In Section 5 we discuss the implications of our findings, including a comparison with plausibly descendent $1 < z < 1.5$ group SMFs from Reeves et al. (2021).

All magnitudes are presented in the AB magnitude system (Oke 1974). We used the 'vanilla' $\Delta$CDM cosmology model ($\Omega_m = 0.3$, $\Omega_\Lambda = 0.7$, $H_0 = 70$ km s$^{-1}$ Mpc$^{-1}$). Stellar mass estimates are taken from COSMOS2020, which assumes a Chabrier (2003) initial mass function. We present uncertainties at the $1\sigma$ level unless otherwise specified.

# 2 DATA

## 2.1 COSMOS2020 sample selection

Our data are taken from the COSMOS2020 catalogue (W22), based on detections in a combined $i_z YHKS$ image. We restrict our analysis to data within the UltraVISTA survey footprint (McCracken et al. 2012) that are not in the bright star HSC-SSP PDR2 mask nor in the bright star Suprime-Cam mask. This region corresponds to $\sim 1.278$ deg$^2$ and is flagged in the catalogue as `FLAG_COMBINED == 0`. We also limit our sources to ones with photometry measured by THE FARMER algorithm. THE FARMER, henceforth simply Farmer, is a software package that uses THE TRACTOR (Lang, Hogg & Mykytyn 2016) to model and create a full multiwavelength catalogue. Specifically, we take the photometric redshifts, mass, and rest-frame magnitude measurements from the LEPHARE (Arnouts & Ilbert 2011) in combination with Farmer. This is because this combination has been noted to have the best photo-$z$ performance (W23, W22).

This region selection leaves us with a subset of the catalogue with 746 976 entries. When we restrict this sample to galaxies between $1.8 < z < 2.7$, we are left with 105 664 entries. This choice of redshift range is informed by the precision of COSMOS2020 photometric redshifts (see Section 2.3) around our protocluster sample ($2 < z < 2.5$; see Section 2.4). We then select all objects that are above the $5\sigma$ IRAC channel 1 magnitude limit of 26 to ensure reliable stellar mass measurements. This magnitude cut removes 23 726 objects, leaving 81 938 galaxies. While this is a large cut, W23 note that $\sim 93$ per cent of these sources are below our optimistic mass limit and thus will be excluded anyway (see Section 2.2). To remove objects with poor photometric redshifts, we restrict our analysis to 'good' fits ($1p\_chi2\_best < 5$), removing another 779 galaxies (0.95 per cent). We also require $1p\_zPDF\_u68$ and $1p\_zPDF\_l68$, the upper and lower 68 percentile confidence levels of the photometric redshift, respectively, to differ by $<1.0$ to ensure that our photometric redshifts are relatively accurate, further removing 1653 galaxies (2.0 per cent) and leaving us with 79 506 galaxies in our sample.

Colour-colour diagnostics are effective at separating dusty star-forming galaxies from quiescent ones (Arnouts et al. 2007; Ilbert et al. 2013). We use rest-frame colours provided in the W22 catalogue. In Fig. 1 we show the (NUV-r) and (r-J) colour distribution for our sample. The use of rest NUV magnitudes in this diagnostic provides more sensitivity to age than the typical UVJ diagrams (Arnouts et al. 2007; Martin et al. 2007). To split the total population into quiescent and star-forming galaxies, we use the definition of Ilbert et al. (2013) where galaxies with rest-frame colours such that $(NUV - r) > 3(r - J) + 1$ and $(NUV - r) > 3.1$ (hereby referred to as NUVrJ selection) are considered quiescent. This selection approximates a cut in sSFR $\lesssim 10^{-11}$ yr$^{-1}$ (Ilbert et al. 2013; Davidzon et al. 2017), and is shown as the green line in Fig. 1. Note that at this redshift, the bimodality in colour distribution is still apparent, though the two populations are not completely disjoint. We also caution that galaxies with very recently terminated star formation may still be classified as star-forming using the NUVrJ method (e.g. McConachie et al. 2022).

## 2.2 Mass completeness limit

To find the mass completeness of the subset of COSMOS data used in this analysis, we take a similar approach to W23, as originally presented in Pozzetti et al. (2010). W23 use the `IRAC_CH1` limiting magnitude to estimate the mass completeness, as described
<figure><img src="image_2.png" /><figcaption>**Figure 1.** We show the NUVrJ colour–colour diagram for all galaxies in our sample between 1.8 < z < 2.7 above our mass limit (see Section 2.2). The solid line shows the division between the quiescent and star-forming galaxies (Ilbert et al. 2013), with the quiescent population being above this line and the star forming one below it. We use this definition throughout this analysis. N<sub>q</sub> and N<sub>sf</sub> are the number of quiescent and star-forming galaxies identified by this criterion, respectively.</figcaption></figure>

further below. This will overestimate the completeness, because red objects detected in IRAC\_CH1 may be missed in the detection image (Davidzon et al. 2017, W22). Indeed, a comparison with the deeper CANDELS (The Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey; Grogin et al. 2011; Koekemoer et al. 2011) catalogue shows that, at the determined 95 per cent mass limit, only 75 per cent of CANDELS sources are recovered. A more conservative choice is to use the K<sub>s</sub> band limit; this will underestimate the completeness because the deep Subaru/HSC photometry will allow the detection of galaxies below that limit. We therefore take the approach of showing our results relative to both mass limits. Although IRAC\_CH1 is a significantly better tracer of stellar mass for z ≳ 2.5, at the redshifts of interest here K<sub>s</sub> is still acceptable.

Following W23, we bin the galaxies in redshift and identify a cutoff magnitude m<sub>cutoff</sub> that corresponds to the 30th percentile of

magnitudes in that bin for each band. We then consider all galaxies with a magnitude fainter than m<sub>cutoff</sub> and re-scale their masses so that their apparent magnitude in a given band matches the limiting magnitude:

$$
\log_{10} \left( \frac{M_{\text{rescale}}}{M_{\odot}} \right) = \log_{10} \left( \frac{M_{*}}{M_{\odot}} \right) + 0.4(m - m_{\text{lim}}), \quad (1)
$$

Where *m* is the magnitude in a given band, and *m*<sub>lim</sub> is the limiting magnitude in that band (26.0 for IRAC\_CH1 and 25.7 for UVISTA\_Ks\_MAG (W23, W22)). We then take our limiting mass *M*<sub>lim</sub> to be the 95th percentile of the re-scaled mass distribution in each bin and fit a polynomial to these *M*<sub>lim</sub> as a function of redshift. We do this for the total, star-forming, and quiescent populations.

The mass completeness of our sample compared to the one presented in W23 is shown in Fig. 2. Given that our analysis is restricted to protoclusters between 2 < z < 2.5 and the furthest associated galaxies should be at roughly z = 2.7 (see Section 2.3), we conservatively restrict our analysis to galaxies above the mass limit at this redshift. Using this, we obtain a IRAC\_CH1-based mass completeness limit of log<sub>10</sub>(M<sub>lim</sub>/M<sub>☉</sub>) = 9.1 for the total population, 9.5 for the quiescent population, and 9.1 for the star-forming population.

We take our aforementioned mass completeness values as our optimistic mass completeness limit. We follow the same procedure in the K<sub>s</sub> magnitude band (which has a limiting magnitude of 25.7) to give conservative mass completeness limits of log<sub>10</sub>(M<sub>lim</sub>/M<sub>☉</sub>) = 9.9 for both the total and star-forming populations, and 10.0 for the quiescent population. We summarize our mass limits, both IRAC\_CH1 based and K<sub>s</sub> based in Table 1. Our final sample for galaxies with log<sub>10</sub>(M<sub>*</sub>/M<sub>☉</sub>) > 9.5, above the optimistic mass completeness limit for quiescent galaxies, consists of 27 531 galaxies, of which 1890 are quiescent and 25 641 are star-forming.

2.3 Photometric redshifts

We now consider the uncertainties on the photometric redshifts for the redshift range of interest (1.8 < z < 2.7). In the following, we define Δz to be half the difference between the upper and lower 68 per cent confidence limits from the LePhare code as provided by

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Stellar mass completeness as a function of redshift for our COSMOS2020 subsample in the left panel, with the sample restricted to star-forming and quiescent galaxies in the middle and right panels, respectively. The colour indicates the number of galaxies in each bin of redshift and stellar mass. The mass completeness is determined following W23, based on either the IRAC\_CH1 magnitude limit (optimistic, shown as the red line) or the K<sub>s</sub> magnitude limit (conservative, shown as the magenta line), as described in the text. This is compared with the IRAC\_CH1-based completeness from W23, shown as the yellow-green line. At the furthest redshift considered in this analysis (z ~ 2.7), we are complete down to ~10<sup>10</sup> M<sub>☉</sub> in our most conservative limit, and complete down to ~10<sup>9.5</sup> M<sub>☉</sub> in our optimistic limit.</figcaption></figure>
<table><caption>**Table 1.** Stellar mass limits for each population. We first show both the IRAC\_CH1 -based and $K_s$-based mass limits, evaluated at $z = 2.7$ for each population. We also show each mass limit as a polynomial function of redshift.</caption><thead><tr><th rowspan="2">Population</th><th colspan="2">log<sub>10</sub>(M<sub>lim</sub>/M<sub>☉</sub>) at z = 2.7</th><th colspan="2">log<sub>10</sub>(M<sub>lim</sub>/M<sub>☉</sub>) fit</th></tr><tr><th>IRAC\_CH1</th><th>K<sub>s</sub></th><th>IRAC\_CH1</th><th>K<sub>s</sub></th></tr></thead><tbody><tr><td>Quiescent</td><td>9.5</td><td>10.0</td><td>-0.11(1 + z)<sup>2</sup> + 1.08(1 + z) + 6.93</td><td>-0.11(1 + z)<sup>2</sup> + 1.28(1 + z) + 6.83</td></tr><tr><td>Star-forming</td><td>9.1</td><td>9.9</td><td>-0.07(1 + z)<sup>2</sup> + 0.89(1 + z) + 6.85</td><td>-0.09(1 + z)<sup>2</sup> + 1.27(1 + z) + 6.46</td></tr><tr><td>Total</td><td>9.1</td><td>9.9</td><td>-0.06(1 + z)<sup>2</sup> + 0.80(1 + z) + 7.04</td><td>-0.08(1 + z)<sup>2</sup> + 1.19(1 + z) + 6.63</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Figure 3.** Left: The dependence of $\Delta z/(1+z)$ as a function of redshift and for galaxies in COSMOS2020 between $1.8 < z < 2.7$ where $\Delta z$ is defined to be the mean distance between the upper $1\sigma$ limit and the lower one. The dashed lines are the overall upper $2\sigma$ of the identified sample over the whole redshift range shown, while the solid lines are the upper $2\sigma$ limit in bins of redshift. We note a slight increase in $\Delta z$ as a function of redshift. At the midpoint of $z = 2.25$ the upper $2\sigma$ value is $\Delta z \approx 0.2$ for star-forming galaxies and 0.3 for quiescent ones. Right: Similar, but for $\Delta z/(1+z)$ as a function of stellar mass. There is a modest decrease with increasing mass, for quiescent galaxies.</figcaption></figure>

W22. These uncertainties have been shown to represent the scatter between photometric and spectroscopic redshifts well (see W22, fig. 13). Although the outlier fraction becomes large (~20 per cent) at the magnitude limit of the sample, most of these outliers are high redshift ($z_{\text{spec}} > 3$) galaxies with $z_{\text{phot}} < 1$, and thus do not impact our sample selection. However we caution that spectroscopy at $2 < z < 3$ is very challenging from the ground, particularly for quiescent galaxies, and samples are therefore biased. Therefore the photometric redshift uncertainties at the magnitude limit cannot be considered to be as well characterized as for the rest of the sample.

In Fig. 3, we show $\Delta z/(1+z)$ as a function of redshift and mass for galaxies in our sample after the selections described in Section 2.1. The precision of the photometric redshifts do not depend significantly on redshift or mass in this redshift regime. For the redshift range of our sample, $1.8 < z < 2.7$, 95 per cent of all star-forming galaxies have $\Delta z/(1+z) < 0.06$, which corresponds to $\Delta z \approx 0.2$ at $z = 2.25$, the midpoint redshift in this range. When considering quiescent galaxies, 95 per cent of all entries have a $\Delta z/(1+z) < 0.09$, which corresponds to $\Delta z \approx 0.3$ at $z = 2.25$.

## 2.4 Cluster membership

While many $z > 2$ protocluster candidates have been identified in the literature, in general it is not possible to know for certain whether these are true protoclusters in the sense that they will evolve into massive ($>10^{14} M_{\odot}$) virialized structures by $z = 0$. Recently, Ata et al. (2022) analysed constrained N-body (dark matter only)

simulations of the COSMOS density field, with initial fluctuations at $z = 100$ chosen to evolve into the 3D structure within the central square degree of the COSMOS field, as defined by extensive spectroscopic redshifts. From fifty randomly selected realizations of these initial conditions, the simulations are evolved to $z = 0$ to predict the final state of all protocluster candidates in this field. For the present analysis, we consider only those protoclusters that have a high probability (generally $> 80$ per cent, with one exception) of evolving into massive clusters by $z = 0$ based on their analysis; these are listed in Table 2. We start with a summary of each protocluster, though more details can be found in Ata et al. (2022):

**ZFOURGE/ZFIRE:** This system was first discovered using a near-IR imaging survey with five custom medium-bandwidth filters (Spitler et al. 2012), and was then confirmed by a spectroscopic follow up (Yuan et al. 2014). It was measured to have a velocity dispersion of $\sigma = 552 \pm 52$ km s$^{-1}$ (Yuan et al. 2014). In all 50 runs of the constrained simulations, this protocluster was found to evolve into a Coma-like cluster of mass $M_{\text{vir}} = (1.2 \pm 0.3) \times 10^{15} h^{-1} M_{\odot}$, where $h = \frac{H_0}{100\text{kms}^{-1}\text{Mpc}^{-1}}$.

**CC2.2:** This protocluster was spectroscopically confirmed by Darvish et al. (2020), following up a large relative overdensity at this location (Darvish et al. 2017). Darvish et al. (2020) estimates a virial mass of $M_{\text{vir}} = (1-2) \times 10^{14} M_{\odot}$ for this structure at its observed redshift, $z \approx 2.2$. In the constrained simulations, a cluster is found at this location 42 out of 50 times, with an associated mass of $M_{\text{vir}} = (4.2 \pm 1.9) \times 10^{14} h^{-1} M_{\odot}$ by $z = 0$.
<table><caption>**Table 2.** A revised version of table 1 from Ata et al. (2022), providing a list of successful protocluster candidates in the COSMOS2020 field. Each candidate was identified in constrained simulations in the COSMOS field as the location of an overdensity that is likely (>50 per cent) to evolve into a protocluster (Ata et al. 2022).</caption><thead><tr><th>Protocluster candidate</th><th>RA (deg)</th><th>Dec (deg)</th><th>Redshift</th><th>Projected z = 0 Mass</th></tr></thead><tbody><tr><td>ZFOURGE/ZFIRE</td><td>150.094</td><td>2.251</td><td>2.095</td><td>(1.2 ± 0.3) × 10<sup>15</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>CC2.2</td><td>150.197</td><td>2.003</td><td>2.232</td><td>(4.2 ± 1.9) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>Hyperion 1</td><td>150.093</td><td>2.404</td><td>2.468</td><td>—</td></tr><tr><td>Hyperion 2</td><td>149.976</td><td>2.112</td><td>2.426</td><td>—</td></tr><tr><td>Hyperion 3</td><td>149.999</td><td>2.253</td><td>2.444</td><td>—</td></tr><tr><td>Hyperion 4</td><td>150.255</td><td>2.342</td><td>2.469</td><td>(2.5 ± 0.5) × 10<sup>15</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>Hyperion 5</td><td>150.229</td><td>2.338</td><td>2.507</td><td>—</td></tr><tr><td>Hyperion 6</td><td>150.331</td><td>2.242</td><td>2.492</td><td>—</td></tr><tr><td>Hyperion 7</td><td>149.958</td><td>2.218</td><td>2.423</td><td>—</td></tr><tr><td>COSTCO J100026.4</td><td>150.110</td><td>2.161</td><td>2.298</td><td>(4.6 ± 2.2) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>COSTCO J095924.0</td><td>149.871</td><td>2.229</td><td>2.047</td><td>(6.1 ± 2.5) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>COSTCO J100031.0</td><td>150.129</td><td>2.275</td><td>2.160</td><td>(5.3 ± 2.6) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>COSTCO J095849.4</td><td>149.706</td><td>2.024</td><td>2.391</td><td>(6.6 ± 2.3) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr><tr><td>COSTCO J095945.1</td><td>149.938</td><td>2.091</td><td>2.283</td><td>(4.3 ± 2.4) × 10<sup>14</sup> h<sup>-1</sup> M<sub>☉</sub></td></tr></tbody></table>

Hyperion (1-7): The Hyperion protoclusters were individually found by several studies (Casey et al. 2015; Chiang et al. 2015; Diener et al. 2015; Lee et al. 2016; Wang et al. 2016) before a connection between them was made by Cucciati et al. (2018), which found the system had an estimated total mass of $4.8 \times 10^{15} M_{\odot}$ over a volume of ~60 x 60 x 150 cMpc$^3$ at $z \sim 2.45$. It was originally hypothesized that this collection of seven density peaks will evolve into a supercluster by $z=0$, with the various peaks virializing by redshift $z \sim 0.8 - 1.6$ (Cucciati et al. 2018). However, the constrained simulations show that by $z=0$, four virialized clusters emerge and form a filamentary group of clusters with a total mass of $M_{vir} = (2.5 \pm 0.5) \times 10^{15} h^{-1} M_{\odot}$ spanning $(65 \pm 10) h^{-1} M_{\odot}$. This projected structure is expected to be similar in spatial extent and mass to the Coma/A1367 filament in the local Universe (Fontanelli 1984).

COSTCO protoclusters: The Constrained Simulations of The COSmos field (COSTCO) protoclusters are a set of protoclusters found purely through the constrained simulations suite presented in Ata et al. (2022). While they do not have strong overdensities throughout $2 < z < 2.52$, they are extended structures that collapse into Virgo-like clusters ($\sim 10^{14.5} M_{\odot}$) by $z=0$. COSTCO J100026.4+020940 has previously been identified as an overdensity (Lee et al. 2016). Recently, Dong et al. (2023) noted that the large scale gas associated with this protocluster has been heated far higher than expected.

COSTCO J095945.1+020528 is found to collapse into a cluster only 27 out of 50 times, though in 40 of those simulations it still results in a substantial overdensity at $z=0$. COSTCO J095945.1+020528 is just south of Hyperion and might become a substructure of it. Tidal disruptions by Hyperion may be the reason why this does not collapse into an independent virialized structure in all cases (Ata et al. 2022).

We identify candidate cluster members by selecting all galaxies within a projected radius $dR$ and a photometric redshift range $dz$. For sufficiently large $dR$, these volumes for neighbouring clusters will partially overlap. Since the $dz$ must be large enough to accommodate the significant photometric redshift uncertainties, the volume will be much larger than the physical volume occupied by the cluster, and will include many non-cluster members. These must be corrected statistically, which requires an accurate volume calculation. This is done using a Monte Carlo approach. We take a 'box' of Cartesian

<table><caption>**Table 3.** The number of quiescent, star-forming, and total galaxies in each selection of $dR$ and $dz$. Most of the analysis in this paper is based on the fiducial sample A, as a good balance between completeness and purity. Selected results for the other samples are provided in Appendix B and Table B1.</caption><thead><tr><th>Selection</th><th>Alias</th><th>dR, dz</th><th>N<sub>q</sub></th><th>N<sub>sf</sub></th><th>N<sub>tot</sub></th></tr></thead><tbody><tr><td>A</td><td>Fiducial</td><td>1.0, 0.2</td><td>32</td><td>431</td><td>463</td></tr><tr><td>B</td><td>Core</td><td>0.5, 0.2</td><td>10</td><td>118</td><td>128</td></tr><tr><td>C</td><td>Wide</td><td>1.5, 0.2</td><td>51</td><td>833</td><td>884</td></tr><tr><td>D</td><td>Core-complete</td><td>0.5, 0.3</td><td>13</td><td>162</td><td>175</td></tr></tbody></table>

space surrounding the clusters, and uniformly populate it with $10^7$ points. We first remove all the points outside the UltraVISTA rectangle (McCracken et al. 2012), or in masked regions. We then take the fraction of points inside protocluster cylinders and multiply it by the volume of the box to measure the volume of the protoclusters. For example, for our fiducial cluster volume (see Table 3) which has properties $dR = 1$ Mpc and $dz = 0.2$, there are ~7750 points inside the protocluster volume. This gives us a volume of ~192 000 ± 2200 Mpc$^3$ assuming a Poisson counting error. This precision of ~1 per cent is sufficient that it does not dominate our error budget.

# 3 STELLAR MASS FUNCTIONS

## 3.1 Methodology

To determine the observed number densities, we bin our data by mass and weight each bin by dividing the count by bin size and volume corresponding to the region in question. We take the uncertainty of this to be simply the square root of each count for the respective bins divided by the associated volume.

To fit the unbinned data above the stellar mass limit, we closely follow the parametric maximum-likelihood method (Sandage, Tammann & Yahil 1979). We will fit our data with a double (Schechter 1976) function, as defined in Baldry, Glazebrook & Driver (2008), in terms of $M = \log_{10}(M_*/M_{\odot})$:

$$ \phi(M) = \ln(10) \cdot \exp(-10^{(M-M^*)}) \cdot 10^{(M-M^*)} \cdot \left[ \phi_1^* \cdot 10^{(M-M^*)\alpha_1} + \phi_2^* \cdot 10^{(M-M^*)\alpha_2} \right], \quad (2) $$
<table><caption>**Table 4.** The mass limits and cluster-centric radius for each work shown in Fig. 12. We present the cluster-centric radius in units of physical Mpc, where available.</caption><thead><tr><th>Work</th><th>Mass limit [log<sub>10</sub>(M<sub>*</sub>/M<sub>☉</sub>)]</th><th>Clustercentric radius</th></tr></thead><tbody><tr><td>Ando, Shimasaku & Momose (2020)</td><td>9.5</td><td>0.5 Mpc</td></tr><tr><td>Ando et al. (2022)</td><td>10</td><td>0.5 Mpc</td></tr><tr><td>Balogh et al. (2016)</td><td>10.5</td><td>1 Mpc</td></tr><tr><td>Cooke et al. (2016)</td><td>10.7</td><td>~1 Mpc</td></tr><tr><td>Ito et al. (2023)</td><td>10.3</td><td>1 Mpc</td></tr><tr><td>Kawinwanichakij et al. (2017)</td><td>10.2</td><td></td></tr><tr><td>Lee-Brown et al. (2017)</td><td>10.2</td><td>0.6 Mpc</td></tr><tr><td>McConachie et al. (2022)</td><td>10.5</td><td>2.3 Mpc</td></tr><tr><td>Nantais et al. (2017)</td><td>10.3</td><td>1 Mpc</td></tr><tr><td>Pintos-Castro et al. (2019)</td><td>10.2</td><td>0.4 R/R<sub>200</sub></td></tr><tr><td>Quadri et al. (2012)</td><td>10</td><td>0.4 Mpc</td></tr><tr><td>Rodríguez-Muñoz et al. (2019)</td><td>10</td><td>0.2 Mpc</td></tr><tr><td>van der Burg et al. (2020)</td><td>9.5</td><td>1 Mpc</td></tr></tbody></table>

where $\phi(M)$ is the number of galaxies per Mpc$^3$ per dex and $M^* = \log(M^*/M_☉)$ is the characteristic mass. The parameters $\alpha_1$ and $\alpha_2$ are the high- and low-mass slopes, respectively, with corresponding normalizations $\phi_1^*$ and $\phi_2^*$. This is effectively adding together two Schechter functions with the same $M^*$. We then assign a probability to each galaxy, as first presented in Oegerle, Hoessel & Ernst (1986) and Malumuth & Kriss (1986):

$$p_i \equiv p(M_i) = \frac{\phi(M_i)}{\int_{M_{\text{lim}}}^{\infty} \phi(M) dM} \quad (3)$$

The likelihood of any given model is defined as the sum of the logarithms of the individual probabilities for each galaxy considered. We determine the parameters $M^*, \alpha_1, \alpha_2$ and the ratio $\phi_2^*/\phi_1^*$ via an MCMC chain. The overall normalization $\phi_1^*$ is set by forcing the integral of the function above the mass limit to equal the number density of galaxies in the sample. We set our uniform priors to be $\alpha_1 \subseteq [-3, 1.5]$, $\alpha_2 \subseteq [-3, -1]$, $M^* \subseteq [9.5, 12]$, and $\phi_2^*/\phi_1^* \subseteq [0, 0.5]$. While these priors for $\alpha_1$ and $M^*$ are broad and uninformed, the choice of $\alpha_2$ and $\phi_2^*/\phi_1^*$ are specifically motivated to ensure the second component corresponds to any low mass upturn, rather than other possible deviations from a single Schechter function at high mass.

To measure the protocluster SMFs, we measure the SMF in a volume centred on the protoclusters (see Section 4.1). However, as described in Section 4.1, this region is heavily contaminated with field galaxies. To accommodate this, we adjust equation (2):

$$\phi(M) = \phi_f(M) + \phi_c(M), \quad (4)$$

where the f and c subscripts are for the field and cluster contributions to the protocluster volume, respectively. Both $\phi_f(M)$ and $\phi_c(M)$ are double Schechter functions as in equation (2). We measure $\phi_f$ for the full field sample (see the next section). We then can measure $\phi_c$ by fitting for equation (4), determining the parameters $M_c^*, \alpha_{1,c}, \alpha_{2,c}$ and the ratios $\phi_{2,c}^*/\phi_{1,c}^*$ and $\phi_{1,f}^*/\phi_{1,c}^*$. We then determine $\phi_{1,c}^*$ in the same way we set $\phi_1^*$.

This allows us to determine intrinsic protocluster SMF to each (unbinned) population. We also consider the binned data for each population, measured by subtracting the field component in each bin. This is described further in Section 4.2.

## 3.2 Field stellar mass function and comparison to literature

In Fig. 4, we show the derived field SMFs observed in our subsample of the COSMOS2020 survey at $2 < z < 2.5$. This definition of the

<figure><img src="image_5.png" /><figcaption>**Figure 4.** We compare the total field SMF at $2 < z < 2.5$ (solid line) with measurements from McLeod et al. (2021), Santini et al. (2022), Muzzin et al. (2013), and W23. The results are generally consistent with one another, though there is some variation at the high-mass end. Note there is a small difference in redshift ranges considered, as McLeod et al. (2021) and Santini et al. (2022) are presented for data between 2.25 < z < 2.75, while Muzzin et al. (2013), W23 and this work are between 2 < z < 2.5. See Appendix A2 for a presentation using different redshift bins. The hatched region represents the mass range between our conservative and optimistic mass completeness limits, as discussed in Section 2.2.</figcaption></figure>

'field' is simply everything in our sample, which includes both low- and high-density regions. We recover closely the result presented in W22, as expected since we are using the same catalogue. This also agrees reasonably well with the total SMFs presented in Muzzin et al. (2013), McLeod et al. (2021), and Santini et al. (2022) at a similar redshift. Our results show some sensitivity to the redshift range of the field sample, which has been chosen to correspond well to the redshift distribution of our protocluster sample, as described in Appendix A1.

We make the same comparison for the quiescent population in Fig. 5. While our results are in good agreement with Muzzin et al. (2013) and W23, both McLeod et al. (2021) and Santini et al. (2022) find significantly larger numbers of quiescent galaxies, especially at low stellar masses. Although the redshift ranges do not match exactly, we show in Appendix A2, Fig. A3, that this does not account for the difference. Differences in the definition of quiescent galaxies are also unlikely to be the explanation. While Santini et al. (2022)
<figure><img src="image_6.png" /></figure>

**Figure 5.** We compare the quiescent field galaxy SMF at $2 < z < 2.5$ (solid line) for our selected sample with those from W23, McLeod et al. (2021), Santini et al. (2022), and Muzzin et al. (2013). Note that Muzzin et al. (2013) and Santini et al. (2022) use UVJ colours to define quiescent galaxies, while the others (including our work) use NUVrJ. Also, the McLeod et al. (2021) and Santini et al. (2022) results are for a different redshift range of $2.25 < z < 2.75$. See Appendix A2 for a presentation using different redshift bins and colour selections. The hatched region represents the mass range between our conservative and optimistic mass completeness limits, as discussed in Section 2.2.

and McLeod et al. (2021) use UVJ colours, rather than the NUVrJ selection that we adopt, we show in Figs A3 and A4 that this choice does not make a significant difference to the quiescent SMF that we derive (see also Gould et al. 2023). Furthermore, Muzzin et al. (2013) also use a UVJ definition, and their result is similar to ours. Cosmic variance is estimated to account for an uncertainty of only ~20 per cent for the Santini et al. (2022) and McLeod et al. (2021) samples, which is small relative to the >70 per cent difference between SMFs. We have checked, using three independent samples of ~1000 arcmin² (corresponding to the survey area in Santini et al. 2022) within our sample of COSMOS data, that the cosmic variance for the quiescent population is not significantly different than that of the total over most of the mass range. It is possible that the difference lies in the stellar mass or redshift estimates, though all studies use similar methods (e.g. parametric star formation histories, dust law, etc.). Thus, the larger population of quiescent galaxies in McLeod et al. (2021) and Santini et al. (2022) remains unexplained. If it is due to an incompleteness in COSMOS2020, we would expect that to affect our target protocluster volumes (which are only modestly overdense, see Section 4.1) similarly to the general field, in which case any impact on our conclusions based on the comparison of these samples will be small.

# 4 RESULTS

## 4.1 Protocluster contrast

Due to the large selection volume necessitated by the photometric redshift uncertainties (see Section 2.4), significant field contamination is expected. To quantify this, we measure the contrast of our protocluster sample relative to the field. We calculate the contrast by finding the total number density of galaxies with $log_{10}(M_*/M_\odot) > 10.5$ in the protocluster selection volume, and subtract the field contribution within that volume from the global SMF. We then divide this quantity by the overall field density to get the relative contrast.

<figure><img src="image_7.png" /></figure>

**Figure 6.** The relative contrast is shown as a function of cluster radius $dR$ and redshift selection $dz$, with our selections (see Table 3) indicated by the crosses. Our fiducial sample at $dR = 1$ Mpc and $dz = 0.2$ has a contrast of ~50 per cent, and is indicated by the darker cross. A higher contrast is found at smaller $dR$ and $dz$, at the expense of a less complete sample. Note that $dR$ is in physical units, not comoving.

In Fig. 6 we show how this contrast depends on the choice of $dR$ and $dz$. The contrast of our protocluster sample relative to a random field is low, < 80 per cent. The physical overdensities are likely much higher. For example, if we assume that the protocluster galaxies are contained in a sphere of radius $dR = 1$ Mpc, this contrast corresponds to a physical overdensity of ~1600.

Our fiducial protocluster selection of $dR = 1$ Mpc and $dz = 0.2$ is physically motivated. The radial extent is chosen to correspond approximately to the virial radius of descendent clusters at $z \sim 1.3$, as discussed further in Section 5.3. The $dz = 0.2$ selection is chosen to correspond to the 95th percentile of photometric redshift error for star-forming galaxies, and is still close to the peak contrast shown in Fig. 6. In addition to the fiducial sample, we also consider a ‘Core’ sample restricted to $dR = 0.5$ Mpc. The contrast of this sample is higher, at the cost of a greatly reduced sample size (see Table 3). At the other extreme, we consider a ‘Wide’ sample with $dR = 1.5$ Mpc. While it is known that protocluster structures can extend to even larger distances (Chiang, Overzier & Gebhardt 2013; Muldrew, Hatch & Cooke 2015; Contini et al. 2016), field contamination dominates in such a volume, making a comparable analysis impractical. Finally we consider a ‘Core-complete’ sample with $dR = 0.5$ Mpc and $dz = 0.3$.<fnref n="1" /> The larger redshift selection improves the completeness of the sample for quiescent galaxies. The different samples are listed in Table 3, together with the number of quiescent and star-forming galaxies, as defined in Section 2.1. The NUVrJ colour distributions of each sample are shown in Appendix B, Fig. B1.

## 4.2 Protocluster stellar mass functions

In Fig. 7, we show the SMFs for our fiducial selection. Both the total and star-forming and quiescent populations show a significant overdensity relative to the average field, for $log_{10}(M_*/M_\odot) < 11$. However, while the shape of the star-forming galaxy SMF is similar

<footnote n="1">¹To create this sample, we apply the same cuts described in Section 2.1, but draw from $1.7 < z < 2.8$ instead. We still restrict our analysis to $M_* \ge 10^{9.5}M_\odot$.</footnote>
<figure><img src="image_8.png" /><figcaption>**Figure 7.** The SMFs within the fiducial volume selection A containing the protoclusters, $dR = 1$ Mpc and $dz = 0.2$. The shaded region shows the $1\sigma$ uncertainty about the best-fitting double Schechter function, which is fit to the unbinned data. The contrast relative to the general field is significant, for the total and star-forming population, at $M_* < 10^{11}M_{\odot}$. There is evidence for an excess of quiescent galaxies at low stellar masses in the protocluster volumes, relative to the field. The hatched region represents the mass range between our conservative and optimistic mass completeness limits, as discussed in Section 2.2.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 8.** The intrinsic SMFs for protocluster galaxies in the fiducial sample (A, Table 3) are shown for star-forming (left) and quiescent (right) galaxies. The binned measurements are shown as the blue points, with the blue shaded region representing the difference in the fits to the unbinned data. The SMF of the protocluster is presented in units of dex$^{-1}$cluster$^{-1}$ (blue, left axis) since the physical volume occupied by the protocluster is unknown. For comparison we show the field SMF in red, and the associated y-axis range (red, on the right side of the panel, in units of dex$^{-1}$Mpc$^{-3}$) has been chosen to facilitate comparison of the shapes of the two SMFs such that the field and cluster align near $M^*$. The shape of the SMFs of the star-forming population matches the shape of the field SMF well, given the uncertainties. However, the quiescent galaxy SMF has a qualitatively different shape from the field. While the number of quiescent galaxies in the field decreases monotonically towards lower masses, in the protocluster an upturn is seen, leading to a relative excess of quiescent galaxies at low mass. Similar plots for the other volume selections of Table 3 are shown in Appendix B. The grey hatched region represents the mass range between our conservative and optimistic mass completeness limits, as discussed in Section 2.2.</figcaption></figure>

to that of the field, the SMF for quiescent galaxies in this volume has an overall flatter shape, indicating relatively more low-mass galaxies than observed in the field. Note that here we are showing show the fit for $\phi(M) = \phi_f(M) + \phi_c(M)$, as shown in equation (4) and discussed in Section 3.1.

As discussed in Section 3.1, we also measure the intrinsic protocluster SMF, $\phi_c$. The results for our fiducial sample are shown in Fig. 8, in units of dex$^{-1}$ cluster$^{-1}$ (left, blue axis). Note this

normalization is per cluster rather than per unit volume, since the physical volume occupied by the overdensity within our large cylinders is unknown. This is done by multiplying the SMF by the protocluster selection volume and then dividing by the number of clusters. For comparison we plot the corresponding field SMFs, here in units of dex$^{-1}$ Mpc$^{-3}$ (right, red axis). In this representation, the relative normalization of the cluster and field curves has no meaning since the units are not the same. To facilitate comparison
<figure><img src="image_10.png" /><figcaption>**Figure 9.** The distribution of the log of the ratio $\phi_2^*/\phi_1^*$ and parameter $\log(-\alpha_2 - 1)$ for both the protocluster (blue) and the field (red, dashed) quiescent populations. The curves show the 1, 2, and 3$\sigma$ contour levels. This parameter combination in the protocluster differs from that of the field at the ~2$\sigma$ level.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 11.** The quiescent fraction for galaxies within our fiducial volume selection (A). The field and cluster are statistically indistinguishable for $\log_{10}(M_*/M_{\odot}) > 10.5$. At lower masses, the cluster sample shows a small excess of quiescent galaxies, though the statistical significance is low. Error bars on the binned data represent the 1$\sigma$ binomial confidence interval. The shaded region is derived from the double-Schechter function fits to the unbinned protocluster and field data shown in Fig. 7.</figcaption></figure>

of the shapes, we arbitrarily adjust the axis limits. This illustrates how the shape of the dominant star-forming population in the protocluster sample is indistinguishable from that of the field, with a monotonically increasing number of galaxies towards lower stellar mass. However, the shape of the quiescent SMF in the protocluster is qualitatively different from that in the field. While both the field and protocluster SMF peak at $\log_{10}(M_*/M_{\odot}) \approx 10.75$, the protocluster SMF does not drop off, instead showing signs of an upturn, leading to a relative excess at $\log_{10}(M_*/M_{\odot}) < 10.5$.

To quantify the significance of the difference in the quiescent galaxy SMFs between the protocluster and the field, we show the confidence intervals of the ratio $\phi_2^*/\phi_1^*$ and parameter $\alpha_2$ for both populations in Fig. 9. These parameters characterize the low mass upturn in the SMF, where we observe qualitatively different SMFs in the protocluster and field samples. Each distribution is generated from the MCMC chain, measured as described in Section 3.1. For display clarity we plot $\log(\phi_2^*/\phi_1^*)$ and $\log(-\alpha_2 - 1)$. This shows a ~2$\sigma$ difference in this parameter combination between the protoclusters and the field. We conclude, therefore, that the difference

in shapes at the low mass end is intriguing but not statistically significant.

We show how the intrinsic protocluster SMFs depend on our different selections in Fig. 10. To allow a clear comparison of the relative shapes on a single plot, we do not show the uncertainty ranges, which are especially large for the two Core samples. The best-fitting parameters for all double Schechter function fits are provided in Table B1. See Appendix B for more details and uncertainty ranges for each sample.

# 5 DISCUSSION

## 5.1 Excess of low-mass quiescent galaxies

Figs 7 and 8 show a moderately significant excess of low-mass quiescent galaxies within the protocluster regions. To explore this further, in Fig. 11 we show the quiescent fraction for our fiducial protocluster sample ($dR, dz = 1$ Mpc, 0.2), after field subtraction.

<figure><img src="image_12.png" /><figcaption>**Figure 10.** The intrinsic SMFs for each selection volume (Table 3). To clearly show the qualitative differences between volume selections, we omit the uncertainty ranges on these figures. The hatched region represents the mass range between our conservative and optimistic mass completeness limits, as discussed in Section 2.2. Individual results for each selection are shown in Appendix B.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>**Figure 12.** The QFE (equation 5) for clusters and protoclusters in our work and the literature as a function of redshift. The mass and cluster-centric radius limits for each work are summarized in Table 4. Most are representative of the population with $M_{*} > 10^{10} M_{\odot}$, with major exceptions noted in the legend. The blue and orange crosses represent our measurements, for the Fiducial and Core samples, respectively, for $log_{10}M_{*}/M_{\odot} > 9.5$.</figcaption></figure>

The fraction is generally quite low, with a $1\sigma$ upper limit of ≈0.15 for stellar masses with $log_{10}(M_{*}/M_{\odot}) < 10.75$. For most of the stellar mass range, and certainly for $log_{10}(M_{*}/M_{\odot}) > 10.25$, the field and protocluster population have quiescent fractions that are fully consistent with one another, within the substantial $1\sigma$ uncertainties. At lower masses we find evidence for a small excess in quiescent fraction, though the statistical significance is not high enough to make strong claims, and larger samples will be required to confirm this. Almost all of these low-mass quiescent galaxies are from the ZFOURGE/ZFIRE protocluster (see Fig. B5), which is the most massive protocluster in our sample. We note also that the apparent excess is in the mass regime where the sample may suffer some incompleteness, though we would expect this incompleteness to affect the protocluster and field samples similarly.

## 5.2 Comparison with other protocluster literature

As we discuss further in the following subsection, the lack of a dominant quiescent population in these protoclusters is surprising. This result also contrasts with some recent claims for quiescent populations in protoclusters at a similar redshift. In an analysis of the cluster QO-1000 at $z = 2.77$, Ito et al. (2023) found 14 quiescent galaxies above mass $log_{10}(M_{*}/M_{\odot}) > 10.5$, with a number density excess of $4.2\sigma$ and a quiescent fraction of $0.34 \pm 0.11$. Similarly, Ando, Shimasaku & Momose (2020) analyse 75 protocluster ‘cores’ at $1.5 < z < 3$ in the COSMOS field, using pairs of massive galaxies as tracers. They find a quiescent fraction of $0.17 \pm 0.04$, three times larger than the field, for $log(M_{*}/M_{\odot}) > 9$. We note, though, that this sample is dominated by systems at $z < 2$. At somewhat higher redshift $z \approx 3.4$, McConachie et al. (2022) discovered a couple of protoclusters, one of which has a very high quiescent fraction of $0.73_{-0.17}^{+0.27}$ among the most massive galaxies $log(M_{*}/M_{\odot}) > 11.3$.

To compare with these and other studies, we consider the quenched fraction excess (QFE; van den Bosch et al. 2008; Wetzel, Tinker & Conroy 2012; Bahé et al. 2017; van der Burg et al. 2020). This quantity shows the fraction of galaxies that are quenched in the high-density protocluster region, but would expected to be star-forming in the field. This is given by:

$$QFE = \frac{f_{q,clus} - f_{q,field}}{1 - f_{q,field}}, \qquad (5)$$

where $f_{q,clus}$ and $f_{q,field}$ are the cluster and field quenched fractions, respectively. Since $f_{q,field}$ is quite small at this redshift, $\lesssim 0.2$, in practice this is not very different from $f_{q,clus}$.

We calculate the QFE for our sample and compare it to that of other works in the literature in Fig. 12. We do this for both our fiducial (blue cross) and core (orange cross) samples, considering all galaxies $M_{*} > 10^{9.5} M_{\odot}$. We measure a QFE of $0.03_{-0.03}^{+0.04}$ in the Fiducial sample, and $0.06_{-0.07}^{+0.09}$ for the Core. While low, this is within $\lesssim 2\sigma$ of other high redshift studies such as McConachie et al. (2022), Ito et al. (2023), and Ando, Shimasaku & Momose (2020).

We noted previously that most of the low-mass quiescent galaxies in our sample come from the most massive system, ZFOURGE/ZFIRE. Considering only this protocluster, using our fiducial parameters $dR$, $dz = 1, 0.2$, we measure a QFE of $0.13_{-0.11}^{+0.12}$ over the mass range $9.5 \le log_{10}(M_{*}/M_{\odot}) \le 11.5$. Over the same mass range, we measure a QFE of $0.00_{-0.03}^{+0.04}$ for the rest of our protoclusters. These two measurements are consistent within $1\sigma$, but hint at a halo-mass dependence that will require larger samples to confirm.

Fig. 12 does not present a very clear trend, and the interpretation is complicated by the dependence of QFE on stellar mass, local density, or cluster-centric radius, and possibly halo mass. There is some indication that a modest quiescent excess is already present in some protoclusters at $2 < z < 3$, and that there is significant evolution
<figure><img src="image_14.png" /><figcaption>**Figure 13.** The orange curves show our fiducial protocluster SMF, for the total population (left) and quiescent galaxies (right), as presented in Fig. 8. We project this to $z = 1.3$ by assuming that protoclusters increase in mass by a factor of five through accretion of field galaxies, as described in the text. This projection is shown as the blue curve. This is compared with the observed SMF of Reeves et al. (2021) groups between $1 < z < 1.5$ (green dashed line). We additionally show the observed SMF of the overall GOGREEN cluster sample between $1 < z < 1.5$ (van der Burg et al. 2020) as the red dotted line. Left: Our projected total SMF, while having a similar total integrated mass as that of the groups, has a different shape, with significantly more low mass galaxies. This implies significant merging and/or disruption of galaxies, as has been found in lower redshift studies (Rudnick et al. 2012) and simulations (Bahé et al. 2019). Right: The shape of the projected quiescent SMF agrees reasonably well with that in the descendent clusters, but with fewer high-mass galaxies. A large number of cluster galaxies at masses above $M_* > 10^{10.5} M_{\odot}$ must quench star formation in the 2 Gyr between $z = 2.3$ and $z = 1.3$. However, for galaxies with masses below $M_* < 10^{10.5} M_{\odot}$, no additional quenching upon infall is required.</figcaption></figure>

towards higher QFE in cluster cores already by $z = 1.6$, only ~2 Gyr later. We explore this further in the following section.

## 5.3 Evolution of the quiescent population in clusters

To compare our results with the $z \sim 1$ descendants of these protoclusters, we note that the average halo mass growth between $z = 2.3$ and $z = 1.3$ is about a factor of 5 (Correa et al. 2015). Assuming the total stellar mass grows by the same factor, we predict a total integrated stellar mass, for galaxies with $\log M_*/M_{\odot} > 9.5$, of $\log(M_*/M_{\odot}) \sim 12.1_{-0.06}^{+0.05}$ for the descendent system at $z \sim 1$. From the total stellar mass to halo mass relation at $z \sim 1$ this corresponds to a halo mass of $\log M_{200}/M_{\odot} \approx 13.7$ (Leauthaud et al. 2012; van der Burg et al. 2014), corresponding to group-scale haloes. We therefore compare with the group sample from Reeves et al. (2021) at $1 < z < 1.5$, which have halo masses between $13.65 \le \log_{10}(M_{200}/M_{\odot}) \le 13.93$. These groups are selected from the COSMOS (UltraVISTA; McCracken et al. 2012; Muzzin et al. 2013) and SXDF (SPLASH-SXDF; Mehta et al. 2018) photometric surveys, and have some spectroscopic coverage by GOGREEN. These systems have a total integrated stellar mass of $\sim 12.0 \log(M_*/M_{\odot})$ between $9.5 < \log(M_*/M_{\odot}) < 11.75$, comparable to the projected mass of our protocluster descendants.

We therefore project our protocluster SMFs to $z = 1.3$, assuming they grow by a factor of 5, by adding sufficient field galaxies to increase the total stellar mass by this factor. We assume the field is represented by the $1 < z < 1.5$ field SMF that we measure in COSMOS. This ensures that the accreted population evolves identically to the field, with no additional environmentally driven evolution. Because of this large mass growth, the projected SMF shape is dominated by that of COSMOS $z \sim 1$ field galaxies.

The result is shown in Fig. 13, compared with both the $1 < z < 1.5$ groups described above, and the more massive GOGREEN-only sample from van der Burg et al. (2020). While our projected

SMF has a similar normalization to the groups sample, the shapes of the SMFs are different, as the observed $z = 1$ groups have far fewer low-mass galaxies than the projection. This may indicate that a significant amount of merging and disruption occurs among cluster members during this time, as expected (Bahé et al. 2019). An alternate explanation would be that clusters do not grow through the unbiased accretion of field galaxies (Ahad et al. 2023).

When considering just the quiescent population (right panel, Fig. 13), the predicted number of high-mass ($M_* > 10^{10.5} M_{\odot}$) quiescent galaxies in our projection is about five times lower than what is observed in the $z = 1$ groups. Additional processes are therefore required to build up massive quiescent galaxies in these groups, with mergers a plausible explanation. However, at lower masses ($M_* < 10^{10.5} M_{\odot}$), the observed abundance of quiescent galaxies in $z = 1$ groups is consistent with, or even larger than, our projections. This implies that no additional quenching upon infall is required. This is somewhat surprising, as McNab et al. (2021) found from an analysis of post-starburst galaxies that low mass galaxies in massive $z = 1$ clusters have been only quenched recently, upon infall. This may be evidence for a halo mass dependence on environmental quenching. The GOGREEN clusters studied in McNab et al. (2021) are about ~5 times more massive than the groups in Reeves et al. (2021), on average. Possibly low mass galaxies at this redshift are effectively quenched by environment only when accreted into sufficiently massive structures.

## 5.4 Future work

Most of the protoclusters included in our sample were originally identified based on an overdensity of star-forming galaxies (e.g. Wang et al. 2016). It is possible that this selection is biased against protoclusters with a high fraction of quiescent galaxies. For example, QO-1000 only has a ~1σ excess of star-forming galaxies, and would never have been identified as a protocluster by just looking at the
star-forming population (Ito et al. 2023). Moreover, the COSTCO protoclusters are noted to be mild overdensities at this redshift epoch, and are projected to be relatively low-mass clusters by $z=0$ (Ata et al. 2022). In contrast, we find evidence that the most massive system in our sample, ZFOURGE/ZFIRE, may have a significantly larger quiescent population than the rest of the sample (see Fig. B5), though the uncertainties are too large to be definitive. Our results are limited by statistics, and much larger samples are therefore needed. The Euclid deep fields will cover an area > 20 times larger than the COSMOS subset used in this study (Sartoris et al. 2016) and will reach a similar $5\sigma$ depth (26 mag) as COSMOS for the Y, J, and H filters (Euclid Collaboration 2022). Scaling from our present sample size of 14 protoclusters in a ~2 deg$^2$ region, the resulting uncertainties on the derived protocluster quiescent fraction can be reduced by a factor of ~5. Even better results can be expected from the *Nancy Grace Roman Space Telescope (NGRST)*, which will cover an area that is ~40 times larger than the Euclid Deep Fields, to greater depth.

Additionally, at these high-redshifts where protoclusters lie, better photometric redshift precision is important to reduce line-of-sight uncertainties. The upcoming COSMOS-Web catalogues will help with this, as they are expected to be much deeper, reaching $5\sigma$ depths of 27.5–28.2 magnitudes in the four NIRCam filters used, spanning ~0.54 deg$^2$ (Casey et al. 2023). The Hyperion structure lies in this field, and its fainter members are expected to be mapped for the first time (Casey et al. 2023). Ultimately, however, the largest gains will be with spectroscopy. A deep slitless prism survey with NGRST covering just 10 deg$^2$ could identify ~15 000 protocluster galaxies (Rudnick et al. 2023).

# 6 CONCLUSIONS

We measure the SMFs for the total, star-forming, and quiescent galaxy populations in protoclusters between $2 < z < 2.5$ in the COSMOS field. These are compared with a comparably selected field sample and are estimated to be complete down to a mass limit of $\log(M_{\text{lim}}/M_{\odot}) = 9.5$. We use these SMFs to measure the efficiency of environmental quenching in protocluster regions as opposed to the field. Our main findings are:

(i) On a scale of 1 Mpc we find a significant overdensity of galaxies in fields centred on the protocluster sample. The shape of the protocluster total SMF, and that of the dominant, star-forming population, is consistent with that of the field (Fig. 7).

(ii) The shape of the protocluster quiescent SMF is different from the field. It is flatter than the field at low masses, with a relative excess of galaxies $M_* < 10^{10} M_{\odot}$ (Fig. 8). This difference is only significant at a ~2$\sigma$ level, however (Fig. 9).

(iii) The fraction of quenched galaxies in our fiducial protocluster selection is indistinguishable from the field above $M_* > 10^{10} M_{\odot}$. However, there is a small but significant excess ($0.08^{+0.03}_{-0.02}$) at lower masses (Fig. 11).

(iv) We compare the protoclusters with a sample of groups at $1 < z < 1.5$ from Reeves et al. (2021). The total stellar mass of those groups within 1 Mpc is about a factor ~5 larger than in the protoclusters. This is similar to the expected mass growth over this time (Correa et al. 2015). We calculate a projected descendent SMF by assuming the protoclusters grow via accretion of field galaxies to the mass of the group sample, with no additional quenching. The resulting shape of this projected SMF is significantly different from that of the lower redshift sample, with an excess of galaxies with $M_* \lesssim 10^{11} M_{\odot}$ (Fig. 13, left panel). This may indicate that significant

merging and/or disruption of galaxies takes place between $z=2.3$ and $z=1.3$.

(v) To match the observed quiescent SMF in the group sample, the number of massive quiescent galaxies must increase by about a factor ~5 beyond what is predicted from the accretion of field galaxies, between $1.3 \lesssim z \lesssim 2.3$ (Fig. 13, right panel). However, at low masses ($M_* < 10^{10.5} M_{\odot}$), no additional quenching upon accretion is necessary, and in fact we project even more low-mass galaxies than are observed in Reeves et al. (2021). This is surprising, as McNab et al. (2021) shows that in the GOGREEN clusters, low-mass galaxies are expected to have quenched more recently than high-mass ones. This can plausibly be explained by the much larger halo mass of the GOGREEN clusters having more of an environmental effect on low mass galaxies.

We conclude that the SMF of galaxies within 1 Mpc of these protoclusters is similar to that of the field, with a small fraction of quiescent galaxies. There is some evidence for a small excess of low-mass (log $M/M_{\odot} < 10$) quiescent galaxies relative to the field, though this is of modest significance and may also be impacted by incompleteness. In any case, these are small in number, and most of the quiescent galaxies that dominate rich clusters at $1 < z < 1.5$ must therefore have been quenched later, presumably upon accretion (though see Ahad et al. 2023, for an alternative explanation). The lack of massive, primordially quenched galaxies is a surprise given the results of some other studies (e.g. McConachie et al. 2022). As we rely on photometric redshifts with statistical background subtraction, the uncertainties for this small sample of 14 protoclusters are large, especially for $M_* > 10^{11.0} M_{\odot}$. Future studies based on larger samples (e.g. from the Euclid deep fields) and with more precise and accurate redshifts (e.g. from COSMOS-Web) should significantly improve upon these results.

# ACKNOWLEDGEMENTS

We thank the referees for their useful comments. This work was developed as part of an ISSI workshop on protoclusters, held in Bern, Switzerland in 2022. We are grateful for the support of ISSI and the use of their facilities. MB acknowledges support from the NSERC Discovery Grant program. Based on observations collected at the European Southern Observatory under ESO programme ID 179.A-2005 and on data products produced by CALET and the Cambridge Astronomy Survey Unit on behalf of the UltraVISTA consortium. MCC acknowledges support from the National Science Foundation through grant AST-1815475. RD gratefully acknowledges support by the ANID BASAL project FB210003. GHR acknowledges support from NSF-AST grants 1517815 and 2206473 as well as HST grants AR-14310.001-A GO-15294.012-A and NASA ADAP grant 80NSSC19K0592. BV acknowledges support from the INAF Mini Grant 2022 ‘Tracing filaments through cosmic time’ (PI Vulcani). GC acknowledges the support from the grant ASI n.2018-23-HH.0. GW gratefully acknowledges support from NSF grant AST-2205189 and from HST program number GO-16300. Support for program number GO-16300 was provided by NASA through grants from the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Incorporated, under NASA contract NAS5-26555.

**Software:** ASTROPY (Astropy Collaboration 2022), CORNER (Foreman-Mackey 2016), EMCEE (Foreman-Mackey et al. 2013), MATPLOTLIB (Hunter 2007), NUMPY (van der Walt, Colbert & Varoquaux 2011), SCIPY (Jones et al. 2001), and SEABORN (Waskom 2021).
DATA AVAILABILITY

All data used in this paper are obtained from the published catalogue of Weaver et al. (2022).

REFERENCES

Ahad S. L., Muzzin A., Bahé Y. M., Hoekstra H., 2023, preprint (arXiv:2307.01147)

Ando M., Shimasaku K., Momose R., 2020, MNRAS, 496, 3169

Ando M., Shimasaku K., Momose R., Ito K., Sawicki M., Shimakawa R., 2022, MNRAS, 513, 3252

Arnouts S., Ilbert O., 2011, LePHARE: Photometric Analysis for Redshift Estimate, Astrophysics Source Code Library, record ascl:1108.009

Arnouts S. et al., 2007, A&A, 476, 137

Astropy Collaboration 2022, ApJ, 935, 167

Ata M., Lee K.-G., Vecchia C. D., Kitaura F.-S., Cucciati O., Lemaux B. C., Kashino D., Müller T., 2022, Nat. Astron., 6, 857

Bahé Y. M. et al., 2017, MNRAS, 470, 4186

Bahé Y. M. et al., 2019, MNRAS, 485, 2287

Baldry I. K., Balogh M. L., Bower R. G., Glazebrook K., Nichol R. C., Bamford S. P., Budavari T., 2006, MNRAS, 373, 469

Baldry I. K., Glazebrook K., Driver S. P., 2008, MNRAS, 388, 945

Balogh M. L., Navarro J. F., Morris S. L., 2000, ApJ, 540, 113

Balogh M. L. et al., 2016, MNRAS, 456, 4364

Balogh M. L. et al., 2021, MNRAS, 500, 358

Baxter D. C. et al., 2022, MNRAS, 515, 5479

Baxter D. C. et al., 2023, MNRAS, 526, 3716

Blanton M. R., Eisenstein D., Hogg D. W., Schlegel D. J., Brinkmann J., 2005, ApJ, 629, 143

Casey C. M. et al., 2015, ApJ, 808, L33

Casey C. M. et al., 2023, ApJ, 954, 31

Ceverino D., Klypin A., 2009, ApJ, 695, 292

Chabrier G., 2003, PASP, 115, 763

Chiang Y.-K., Overzier R., Gebhardt K., 2013, ApJ, 779, 127

Chiang Y.-K. et al., 2015, ApJ, 808, 37

Chiang Y.-K., Overzier R. A., Gebhardt K., Henriques B., 2017, ApJ, 844, L23

Continini E., De Lucia G., Hatch N., Borgani S., Kang X., 2016, MNRAS, 456, 1924

Cooke E. A. et al., 2016, ApJ, 816, 83

Correa C. A., Wyithe J. S. B., Schaye J., Duffy A. R., 2015, MNRAS, 450, 1514

Cucciati O. et al., 2018, A&A, 619, A49

Darvish B., Mobasher B., Martin D. C., Sobral D., Scoville N., Stroe A., Hemmati S., Kartaltepe J., 2017, ApJ, 837, 16

Darvish B. et al., 2020, ApJ, 892, 8

Davidzon I. et al., 2017, A&A, 605, A70

Dekel A., Birnboim Y., 2006, MNRAS, 368, 2

Dekel A., Silk J., 1986, ApJ, 303, 39

Dickinson M., Papovich C., Ferguson H. C., Budavári T., 2003, ApJ, 587, 25

Diener C. et al., 2015, ApJ, 802, 31

Dong C., Lee K.-G., Ata M., Horowitz B., Momose R., 2023, ApJ, 945, L28

Euclid Collaboration 2022, A&A, 658, A126

Fontanelli P., 1984, A&A, 138, 85

Foreman-Mackey D., 2016, J. Open Source Softw., 1, 24

Foreman-Mackey D., Hogg D. W., Lang D., Goodman J., 2013, PASP, 125, 306

Gilbank D. G., Balogh M. L., 2008, MNRAS, 385, L116

Gómez P. L. et al., 2003, ApJ, 584, 210

Gould K. M. L. et al., 2023, AJ, 165, 248

Grogin N. A. et al., 2011, ApJS, 197, 35

Gunn J. E., Gott J. R. III, 1972, ApJ, 176, 1

Hopkins P. F., Hernquist L., Cox T. J., Di Matteo T., Robertson B., Springel V., 2006, ApJS, 163, 1

Hunter J. D., 2007, Comput. Sci. Eng., 9, 90

Ilbert O. et al., 2013, A&A, 556, A55

Ito K. et al., 2023, ApJ, 945, L9

Jones E. et al., 2001, SciPy: Open source scientific tools for Python. Available at: http://www.scipy.org/

Kawinwanichakij L. et al., 2017, ApJ, 847, 134

Koekemoer A. M. et al., 2011, ApJS, 197, 36

Lang D., Hogg D. W., Mykytyn D., 2016, The Tractor: Probabilistic astronomical source detection and measurement, Astrophysics Source Code Library, record ascl:1604.008

Larson R. B., Tinsley B. M., Caldwell C. N., 1980, ApJ, 237, 692

Leauthaud A. et al., 2012, ApJ, 744, 159

Lee-Brown D. B. et al., 2017, ApJ, 844, 43

Lee K.-G. et al., 2016, ApJ, 817, 160

Leja J., Speagle J. S., Johnson B. D., Conroy C., van Dokkum P., Franx M., 2020, ApJ, 893, 111

Lewis I. et al., 2002, MNRAS, 334, 673

Malumuth E. M., Kriss G. A., 1986, ApJ, 308, 10

Martin D. C. et al., 2007, ApJS, 173, 342

McConachie I. et al., 2022, ApJ, 926, 37

McCracken H. J. et al., 2012, A&A, 544, A156

McGee S. L., Balogh M. L., Wilman D. J., Bower R. G., Mulchaey J. S., Parker L. C., Oemler A., 2011, MNRAS, 413, 996

McLeod D. J., McLure R. J., Dunlop J. S., Cullen F., Carnall A. C., Duncan K., 2021, MNRAS, 503, 4413

McNab K. et al., 2021, MNRAS, 508, 157

Mehta V. et al., 2018, ApJS, 235, 36

Muldrew S. I., Hatch N. A., Cooke E. A., 2015, MNRAS, 452, 2528

Muzzin A. et al., 2013, ApJ, 777, 18

Muzzin A. et al., 2014, ApJ, 796, 65

Nantais J. B. et al., 2017, MNRAS, 465, L104

Oegerle W. R., Hoessel J. G., Ernst R. M., 1986, AJ, 91, 697

Oke J. B., 1974, ApJS, 27, 21

Peng Y.-j. et al., 2010, ApJ, 721, 193

Pintos-Castro I., Yee H. K. C., Muzzin A., Old L., Wilson G., 2019, ApJ, 876, 40

Poggianti B. M. et al., 2006, ApJ, 642, 188

Poggianti B. M. et al., 2017, ApJ, 844, 48

Pozzetti L. et al., 2010, A&A, 523, A13

Quadri R. F., Williams R. J., Franx M., Hildebrandt H., 2012, ApJ, 744, 88

Reeves A. M. M. et al., 2021, MNRAS, 506, 3364

Rodríguez-Muñoz L. et al., 2019, MNRAS, 485, 586

Rudnick G. et al., 2003, ApJ, 599, 847

Rudnick G. et al., 2006, ApJ, 650, 624

Rudnick G. H., Tran K.-V., Papovich C., Momcheva I., Willmer C., 2012, ApJ, 755, 14

Rudnick G. et al., 2023, preprint (arXiv:2306.15735)

Sandage A., Tammann G. A., Yahil A., 1979, ApJ, 232, 352

Santini P. et al., 2022, ApJ, 940, 135

Sartoris B. et al., 2016, MNRAS, 459, 1764

Schechter P., 1976, ApJ, 203, 297

Scoville N. et al., 2007, ApJS, 172, 1

Silk J., Rees M. J., 1998, A&A, 331, L1

Spitler L. R. et al., 2012, ApJ, 748, L21

Springel V., Di Matteo T., Hernquist L., 2005, ApJ, 620, L79

Taylor E., Almaini O., Merrifield M., Maltby D., Wild V., Hartley W. G., Rowlands K., 2023, MNRAS, 522, 2297

Thomas D., Maraston C., Bender R., Mendes de Oliveira C., 2005, ApJ, 621, 673

van den Bosch F. C., Aquino D., Yang X., Mo H. J., Pasquali A., McIntosh D. H., Weinmann S. M., Kang X., 2008, MNRAS, 387, 79

van der Burg R. F. J. et al., 2020, A&A, 638, A112

van der Burg R. F. J., Muzzin A., Hoekstra H., Wilson G., Lidman C., Yee H. K. C., 2014, A&A, 561, A79

van der Walt S., Colbert S. C., Varoquaux G., 2011, Comput. Sci. Eng., 13, 22

Wang T. et al., 2016, ApJ, 828, 56

Waskom M. L., 2021, J. Open Source Softw., 6, 3021

Weaver J. R. et al., 2022, ApJS, 258, 11 (W22)

Weaver J. R. et al., 2023, A&A, 677, A184 (W23)

Webb K. et al., 2020, MNRAS, 498, 5317
Werner S. V., Hatch N. A., Muzzin A., van der Burg R. F. J., Balogh M. L.,
Rudnick G., Wilson G., 2022, MNRAS, 510, 674

Wetzel A. R., Tinker J. L., Conroy C., 2012, MNRAS, 424, 232

Yuan T. et al., 2014, ApJ, 795, L20

APPENDIX A: FIELD CONSIDERATIONS

A1 Redshift distribution of the field

In this work, we define our ‘field’ to be all objects that match our cut selection (see Section 2.1) between 2 < z < 2.5. This differs from the range in which our candidate protocluster members are selected, 1.8 < z < 2.7. The broader range is necessary to accommodate photometric redshift uncertainties (see Section 2.3). The normalization of the field SMF is somewhat sensitive to the redshift range, and we choose the narrower 2 < z < 2.5 to better match the redshift distribution of the protocluster members. In particular, there are overdense structures at 1.8 < z < 2 that significantly perturb the field SMF when that range is included.

A2 Field comparison

In Fig. 4 we compared the total and quiescent SMFs in our field sample to that measured in previous studies. As noted in the text, there are some differences in how our sample is constructed relative to those comparison studies. Here we show the extent to which these differences affect the SMF measurements. First, in Fig. A2 we show

<figure><img src="image_15.png" /><figcaption>**Figure A1.** The distribution of field and protocluster (fiducial selection, Table 3) photometric redshifts. Arrows indicate individual protocluster candidate redshifts (Table 2). As can be seen, the majority of protocluster galaxies lie in the 2 < z < 2.5 redshift range, as expected. Selecting a field sample in this same range helps us to ensure a similar redshift distribution as the clusters, by avoiding the overdense structures between 1.8 < z < 2.</figcaption></figure>

the total SMF in three different redshift bins, 0.25 < z < 0.75, 1.25
< z < 1.75, and 2.25 < z < 2.75. The latter bin is different from
our default field sample (2 < z < 2.5), chosen here to correspond

<figure><img src="image_16.png" /><figcaption>**Figure A2.** We show the total SMF of our field sample in three redshift bins, chosen to correspond to those of McLeod et al. (2021) and Santini et al. (2022). There is good agreement for M\* < 10\*M⊙.</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>**Figure A3.** Similar to Fig. A2, but for the quiescent population. As with our fiducial sample in Fig. 5, the normalization of our observed SMF is lower than that of McLeod et al. (2021) and Santini et al. (2022), even when using the same redshift bin of 2.25 < z < 2.75. This difference also persists at lower redshift.</figcaption></figure>

to the binning of McLeod et al. (2021) and Santini et al. (2022). In general, the total SMF agrees well with both those works, as well as that of Muzzin et al. (2013), in all three redshift ranges, for $M_* < 10^{11} M_\odot$. We also show the effect of using our default photometric catalogue, Farmer, compared with the Classic. The difference is largely negligible, especially at the $z \approx 2$ epoch that is central to this work.

Fig. A3 is similar, but for just the quiescent population. The lower normalization that we observe relative to McLeod et al. (2021) and Santini et al. (2022) persists even when we use the same redshift interval $2.25 < z < 2.75$, and also extends to the lower redshift interval

$1.25 < z < 1.75$. We also observe that the choice of catalogue (the default, Farmer, compared with the Classic) has a significant impact on the quiescent SMF at low stellar masses, as discussed by W22 and W23.

Finally, we consider the impact of selecting quiescent galaxies from a UVJ colour selection, rather than our default NUVrJ. The result is shown in Fig. A4, for the same redshift bins as the previous two figures. Again, the different choice in definition does not remove the discrepancy with McLeod et al. (2021) and Santini et al. (2022).
<figure><img src="image_18.png" /><figcaption>**Figure A4.** As Fig. A3, but where quiescent galaxies in our sample are defined from their UVJ colours, rather than the default NUVrJ. This does not remove the discrepancy with McLeod et al. (2021) or Santini et al. (2022), who also use a UVJ colour classification.</figcaption></figure>

## APPENDIX B: STELLAR MASS FUNCTIONS IN DIFFERENT VOLUMES

In this section we show how the intrinsic protocluster SMFs depend on the different protocluster volume selections tabulated in Table 3. These can be compared with our fiducial results, in Fig. 8. The NUVrJ colour distributions of each sample are shown in Fig. B1. The morphology of the colour distribution is similar for all samples, with the primary difference being one of sample size.

First, in Fig. B2 we show the Core sample, dR, dz = 0.5 Mpc, 0.2. As with the fiducial sample, we observe a significant excess of low-mass protocluster galaxies, with an SMF that rises even more steeply towards lower masses. In addition, there is a bump at M* ~ 10¹¹.²⁵M☉, corresponding to an excess of very massive galaxies that is not seen in the wider selection. Also different from the fiducial sample is the drop in the number of star-forming (and, hence, total) galaxies at the lowest stellar masses.

Next in Fig. B3 we consider the core-complete selection (dR, dz = 0.5, 0.3). This is similar to the core sample just discussed, but with a higher completeness due to the larger dz range, chosen to include

~ 95 per cent of all quiescent galaxies in the cluster (See Section 2.3). The results are generally indistinguishable from Fig. B2, though the uncertainties on the quiescent SMF are larger due to the increased field contamination. This demonstrates that the narrower dz = 0.2 selection used in our fiducial sample does not significantly bias the results against quiescent galaxies.

Finally, in Fig. B4, we show the Wide selection of dR = 1.5 Mpc and dz = 0.2. For this volume, the SMFs are in general much more similar in shape to that of the field. An excess of low-mass quiescent galaxies is still apparent, though it is not statistically significant.

We also look at the intrinsic protocluster SMF for galaxies with selection parameters dR = 1 Mpc, dz = 0.2 around just the most massive protocluster in our sample, ZFOURGE/ZFIRE. We note that there are no quiescent galaxies above 10¹⁰M☉. However, there is a large low-mass excess in this protocluster; all six quiescent galaxies in this selection have masses below M* < 10¹⁰.⁶ M☉. Notably, this number of quiescent galaxies is about an order of magnitude larger than the average per cluster when considering the full sample.

We present the fit parameters for the intrinsic protocluster SMFs in each selection, as well as the fit parameters for the field in Table B1.
<table><caption>**Table B1.** Summary of best-fitting parameters for the double Schechter functions fit to each selection and population. The field is defined as everything between 2 < z < 2.5, and the fit to the intrinsic protocluster SMF $\phi_c$ are as described in Section 3.1. $M^*$ is in units of $\log(M_*/M_{\odot})$, and $\alpha_1$ and $\alpha_2$ are unitless. $\phi_1^*$ and $\phi_2^*$ are in units of dex$^{-1}$cluster$^{-1}$, except for the field, where it is presented in units of dex$^{-1}$Mpc$^{-3}$.</caption><thead><tr><th>Selection</th><th>Alias</th><th>Population</th><th>M*</th><th>logφ<sub>1</sub><sup>*</sup></th><th>α<sub>1</sub></th><th>logφ<sub>2</sub><sup>*</sup></th><th>α<sub>2</sub></th></tr></thead><tbody><tr><td rowspan="4">A</td><td rowspan="4">Fiducial</td><td>Total</td><td>10.91<sup>+0.19</sup><sub>-0.16</sub></td><td>0.34<sup>+0.20</sup><sub>-0.24</sub></td><td>-0.72<sup>+0.41</sup><sub>-0.63</sub></td><td>-0.34<sup>+0.26</sup><sub>-0.28</sub></td><td>-1.64<sup>+0.36</sup><sub>-0.27</sub></td></tr><tr><td>Quiescent</td><td>10.66<sup>+0.21</sup><sub>-0.18</sub></td><td>-0.15<sup>+0.11</sup><sub>-0.14</sub></td><td>0.16<sup>+0.61</sup><sub>-0.52</sub></td><td>-1.11<sup>+0.32</sup><sub>-0.75</sub></td><td>-1.21<sup>+0.15</sup><sub>-0.43</sub></td></tr><tr><td>Star forming</td><td>10.95<sup>+0.19</sup><sub>-0.16</sub></td><td>0.28<sup>+0.17</sup><sub>-0.28</sub></td><td>-0.93<sup>+0.43</sup><sub>-0.38</sub></td><td>-0.74<sup>+0.47</sup><sub>-1.07</sub></td><td>-1.88<sup>+0.26</sup><sub>-0.75</sub></td></tr><tr><td>Total</td><td>11.20<sup>+0.31</sup><sub>-0.21</sub></td><td>-0.45<sup>+0.27</sup><sub>-0.31</sub></td><td>-1.12<sup>+0.54</sup><sub>-0.30</sub></td><td>-1.10<sup>+0.34</sup><sub>-0.51</sub></td><td>-1.53<sup>+0.26</sup><sub>-0.18</sub></td></tr><tr><td rowspan="4">B</td><td rowspan="4">Core</td><td>Quiescent</td><td>10.95<sup>+0.41</sup><sub>-0.71</sub></td><td>-0.66<sup>+0.17</sup><sub>-0.28</sub></td><td>-0.12<sup>+0.95</sup><sub>-0.71</sub></td><td>-1.54<sup>+0.30</sup><sub>-0.57</sub></td><td>-1.27<sup>+0.18</sup><sub>-0.30</sub></td></tr><tr><td>Star forming</td><td>11.22<sup>+0.37</sup><sub>-0.17</sub></td><td>-0.58<sup>+0.19</sup><sub>-0.30</sub></td><td>-1.24<sup>+0.51</sup><sub>-0.21</sub></td><td>-1.29<sup>+0.34</sup><sub>-0.45</sub></td><td>-1.59<sup>+0.38</sup><sub>-0.19</sub></td></tr><tr><td>Total</td><td>10.91<sup>+0.21</sup><sub>-0.11</sub></td><td>0.47<sup>+0.15</sup><sub>-0.24</sub></td><td>-0.91<sup>+0.30</sup><sub>-0.46</sub></td><td>-0.51<sup>+0.42</sup><sub>-0.58</sub></td><td>-1.90<sup>+0.32</sup><sub>-0.33</sub></td></tr><tr><td>Quiescent</td><td>10.49<sup>+1.32</sup><sub>-0.14</sub></td><td>-0.17<sup>+0.05</sup><sub>-0.39</sub></td><td>0.60<sup>+0.59</sup><sub>-0.66</sub></td><td>-2.05<sup>+0.82</sup><sub>-1.14</sub></td><td>-1.81<sup>+0.65</sup><sub>-0.95</sub></td></tr><tr><td rowspan="4">C</td><td rowspan="4">Wide</td><td>Star forming</td><td>11.00<sup>+0.17</sup><sub>-0.14</sub></td><td>0.24<sup>+0.20</sup><sub>-0.23</sub></td><td>-1.07<sup>+0.35</sup><sub>-0.40</sub></td><td>-0.43<sup>+0.32</sup><sub>-0.56</sub></td><td>-1.79<sup>+0.13</sup><sub>-0.28</sub></td></tr><tr><td>Total</td><td>11.19<sup>+0.24</sup><sub>-0.19</sub></td><td>-0.54<sup>+0.23</sup><sub>-0.27</sub></td><td>-1.27<sup>+0.41</sup><sub>-0.19</sub></td><td>-1.18<sup>+0.33</sup><sub>-0.48</sub></td><td>-1.50<sup>+0.28</sup><sub>-0.19</sub></td></tr><tr><td>Quiescent</td><td>10.77<sup>+0.43</sup><sub>-0.25</sub></td><td>-0.60<sup>+0.12</sup><sub>-0.22</sub></td><td>0.17<sup>+0.82</sup><sub>-0.72</sub></td><td>-1.48<sup>+0.29</sup><sub>-0.79</sub></td><td>-1.27<sup>+0.18</sup><sub>-0.37</sub></td></tr><tr><td>Star forming</td><td>11.25<sup>+0.31</sup><sub>-0.23</sub></td><td>-0.75<sup>+0.27</sup><sub>-0.34</sub></td><td>-1.39<sup>+0.43</sup><sub>-0.19</sub></td><td>-1.39<sup>+0.36</sup><sub>-0.51</sub></td><td>-1.57<sup>+0.37</sup><sub>-0.20</sub></td></tr><tr><td rowspan="4">D</td><td rowspan="4">Core complete</td><td>Quiescent</td><td>10.87<sup>+0.04</sup><sub>-0.03</sub></td><td>-3.31<sup>+0.02</sup><sub>-0.03</sub></td><td>-0.74<sup>+0.12</sup><sub>-0.18</sub></td><td>-4.44<sup>+0.21</sup><sub>-0.56</sub></td><td>-2.05<sup>+0.12</sup><sub>-0.34</sub></td></tr><tr><td>Star forming</td><td>10.54<sup>+0.03</sup><sub>-0.03</sub></td><td>-3.92<sup>+0.002</sup><sub>-0.004</sub></td><td>0.46<sup>+0.09</sup><sub>-0.09</sub></td><td>-6.88<sup>+0.53</sup><sub>-0.74</sub></td><td>-1.30<sup>+0.23</sup><sub>-0.48</sub></td></tr><tr><td>Total</td><td>10.90<sup>+0.05</sup><sub>-0.05</sub></td><td>-3.45<sup>+0.03</sup><sub>-0.04</sub></td><td>-0.80<sup>+0.26</sup><sub>-0.17</sub></td><td>-4.37<sup>+0.31</sup><sub>-0.37</sub></td><td>-1.98<sup>+0.16</sup><sub>-0.20</sub></td></tr></tbody></table>

<figure><img src="image_19.png" /><figcaption>**Figure B1.** The (NUV-r) versus (r-J) colour distribution is shown for each volume selection (Table 3). The division between quiescent and star-forming galaxies is shown as the solid line. We observe a distinct quiescent population in each selection volume.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>**Figure B2.** We show the intrinsic protocluster SMFs for our core selection (dR = 0.5 Mpc, dz = 0.2), to be compared with our fiducial results in Fig. 8. In this sample, the excess of low-mass quiescent galaxies is even more pronounced, with an SMF that increases steeply towards lower masses. There is also an excess of massive, quiescent galaxies, and a deficit of low-mass, star-forming galaxies.</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>**Figure B3.** As Fig. 8, but for the core-complete sample (dR = 0.5 Mpc, dz = 0.3). Results are very similar to the core sample shown in Fig. B2. Uncertainties on the quiescent SMF are larger because the larger dz results in greater field contribution within the volume.</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>**Figure B4.** As Fig. 8, but for the wide selection ($dR = 1.5$ Mpc, $dz = 0.2$). The SMF shapes are generally consistent with the field, though the flatter shape of the quiescent SMF in protoclusters is still present.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Figure B5.** As Fig. 8, but for just galaxies with selection parameters $dR = 1$ Mpc, $dz = 0.2$ around just the most massive protocluster in our sample, ZFOURGE/ZFIRE. The number of low-mass quiescent galaxies here is about a factor ten larger than the average for our full sample.</figcaption></figure>

<footnote n="1">¹Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada</footnote>

<footnote n="2">²Waterloo Centre for Astrophysics, University of Waterloo, Waterloo, Ontario N2L3G1, Canada</footnote>

<footnote n="3">³Institute of Physics, Laboratory of Astrophysics, Ecole Polytechnique Fédérale de Lausanne (EPFL), Observatoire de Sauverny, CH-1290 Versoix, Switzerland</footnote>

<footnote n="4">⁴Leiden Observatory, Leiden University, P.O. Box 9513, NL-2300 RA Leiden, the Netherlands</footnote>

<footnote n="5">⁵Department of Physics and Astronomy, University of California, Irvine, 4129 Frederick Reines Hall, Irvine, CA 92697, USA</footnote>

<footnote n="6">⁶School of Physics and Astronomy, University of Nottingham, Nottingham NG7 2RD, UK</footnote>

<footnote n="7">⁷Department of Physics and Astronomy, York University, 4700, Keele Street, Toronto, Ontario M3J 1P3, Canada</footnote>

<footnote n="8">⁸School of Earth and Space Exploration, Arizona State University, Tempe, AZ 85287, USA</footnote>

<footnote n="9">⁹Department of Physics and Astronomy, University of Kansas, Lawrence, KS 66045, USA</footnote>

<footnote n="10">¹⁰INAF – Osservatorio Astronomico di Padova, Vicolo Osservatorio 5, I-35122 Padova, Italy</footnote>

<footnote n="11">¹¹Department of Physics, University of California Merced 5200 Lake Road, Merced, CA 95343, USA</footnote>

<footnote n="12">¹²INAF – Astronomical Observatory of Trieste, via G.B. Tiepolo 11, I-34143 Trieste, Italy</footnote>

<footnote n="13">¹³Departamento de Astronomía, Facultad de Ciencias Físicas y Matemáticas, Universidad de Concepción, Casilla 160-C, Concepción, Chile</footnote>

<footnote n="14">¹⁴Department of Physics and Astronomy, University of California Davis, One Shields Avenue, Davis, CA 95616, USA</footnote>

<footnote n="15">¹⁵Institute for Physics, Laboratory for Galaxy Evolution and Spectral modelling, Ecole Polytechnique Federale de Lausanne, Observatoire de Sauverny, Chemin Pegasi 51, CH-1290 Versoix, Switzerland</footnote>

<footnote n="16">¹⁶Dipartimento di Fisica e Astronomia 'Augusto Righi, Alma Mater Studio- rum Università di Bologna, Via Gobetti 93/2, I-40129 Bologna, Italy</footnote>
This paper has been typeset from a TEX/LATEX file prepared by the author.

<footnote n="17">¹⁷INAF – Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, via Gobetti 93/3, I-40129 Bologna, Italy</footnote>

<footnote n="18">¹⁸Departamento de Ingeniería Informática y Ciencias de la Computación, Facultad de Ingeniería, Universidad de Concepción, Casilla 160-C, Concepción, Chile</footnote>

<footnote n="19">¹⁹Department of Physics and Astronomy, Siena College, 515 Loudon Rd, Loudonville, NY 12211, USA</footnote>

<footnote n="20">²⁰Laboratoire d'astrophysique, École Polytechnique Fédérale de Lausanne (EPFL), CH-1290 Sauverny, Switzerland</footnote>

<footnote n="21">²¹GEPI, Observatoire de Paris, Université PSL, CNRS, Place Jules Janssen, F-92190 Meudon, France</footnote>

<footnote n="22">²²Astronomical Institute, Tohoku University, 6-3, Aramaki, Aoba, Sendai, Miyagi 980-8578, Japan</footnote>

<footnote n="23">²³Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, Bd de l'Observatoire, Université Côte d'Azur, CS 34229, F-06304 Nice Cedex 4, France</footnote>

<footnote n="24">²⁴Instituto de Astrofísica, Departamento de Ciencias Físicas, Universidad Andres Bello, Fernandez Concha 700, Las Condes 7591538, Santiago, Región Metropolitana, Chile</footnote>

<footnote n="25">²⁵Tianjin Astrophysics Center, Tianjin Normal University, Binshuixidao 393, 300387 Tianjin, People's Republic of China</footnote>