# Glimpsing the imprint of local environment on the galaxy stellar mass function

Adam Tomczak, Brian Lemaux, Lori Lubin, Roy Gal, Po-Feng Wu, Bradford Holden, Dale Kocevski, Simona Mei, Debora Pelliccia, Nicholas Rumbaugh, et al.

► To cite this version:

Adam Tomczak, Brian Lemaux, Lori Lubin, Roy Gal, Po-Feng Wu, et al., Glimpsing the imprint of local environment on the galaxy stellar mass function. Monthly Notices of the Royal Astronomical Society, 2017, 472 (3), pp.3512-3531. 10.1093/mnras/stx2245. hal-02235214

HAL Id: hal-02235214

https://hal.science/hal-02235214v1

Submitted on 7 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Glimpsing the imprint of local environment on the galaxy stellar mass function

Adam R. Tomczak,<fnref n="1" />* Brian C. Lemaux,<fnref n="1" /> Lori M. Lubin,<fnref n="1" /> Roy R. Gal,<fnref n="2" /> Po-Feng Wu,<fnref n="2" />,<fnref n="3" />
Bradford Holden,<fnref n="4" /> Dale D. Kocevski,<fnref n="5" /> Simona Mei,<fnref n="6" />,<fnref n="7" />,<fnref n="8" /> Debora Pelliccia,<fnref n="1" />
Nicholas Rumbaugh<fnref n="9" /> and Lu Shen<fnref n="1" />

Accepted 2017 August 25. Received 2017 August 25; in original form 2017 May 5

## ABSTRACT

We investigate the impact of local environment on the galaxy stellar mass function (SMF) spanning a wide range of galaxy densities from the field up to dense cores of massive galaxy clusters. Data are drawn from a sample of eight fields from the Observations of Redshift Evolution in Large-Scale Environments (ORELSE) survey. Deep photometry allow us to select mass-complete samples of galaxies down to $10^9 M_{\odot}$. Taking advantage of >4000 secure spectroscopic redshifts from ORELSE and precise photometric redshifts, we construct three-dimensional density maps between $0.55 < z < 1.3$ using a Voronoi tessellation approach. We find that the shape of the SMF depends strongly on local environment exhibited by a smooth, continual increase in the relative numbers of high- to low-mass galaxies towards denser environments. A straightforward implication is that local environment proportionally increases the efficiency of (a) destroying lower mass galaxies and/or (b) growth of higher mass galaxies. We also find a presence of this environmental dependence in the SMFs of star-forming and quiescent galaxies, although not quite as strongly for the quiescent subsample. To characterize the connection between the SMF of field galaxies and that of denser environments, we devise a simple semi-empirical model. The model begins with a sample of $\approx 10^6$ galaxies at $z_{start} = 5$ with stellar masses distributed according to the field. Simulated galaxies then evolve down to $z_{final} = 0.8$ following empirical prescriptions for star-formation, quenching and galaxy-galaxy merging. We run the simulation multiple times, testing a variety of scenarios with differing overall amounts of merging. Our model suggests that a large number of mergers are required to reproduce the SMF in dense environments. Additionally, a large majority of these mergers would have to occur in intermediate density environments (e.g. galaxy groups).

**Key words:** techniques: photometric – techniques: spectroscopic – galaxies: clusters: general – galaxies: evolution – galaxies: groups: general.

## 1 INTRODUCTION

It has been broadly understood from early observations that galaxies in clusters evolve along different time-scales and/or pathways

relative to those in more isolated environments (e.g. Gunn & Gott 1972; Oemler 1974; Dressler 1980). A variety of mechanisms either unique to or facilitated in cluster environments have been proposed as a means of contextualizing these observations such as ram-pressure stripping (Gunn & Gott 1972) and gravitational interactions between galaxies (Richstone 1976) and with the cluster potential (Farouki & Shapiro 1981). In more recent years, the Sloan

<footnote n="1">¹Department of Physics, University of California, Davis, One Shields Ave., Davis, CA 95616, USA</footnote>

<footnote n="2">²University of Hawai‘i, Institute for Astronomy, 2680 Woodlawn Drive, HI 96822, USA</footnote>

<footnote n="3">³Max-Planck Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany</footnote>

<footnote n="4">⁴UCO Lick Observatory, Department of Astronomy and Astrophysics, University of California, Santa Cruz, CA 95064, USA</footnote>

<footnote n="5">⁵Department of Physics and Astronomy, Colby College, Waterville, ME 04901, USA</footnote>

<footnote n="6">⁶LERMA, Observatoire de Paris, PSL Research University, CNRS, Sorbonne Universités, UPMC University Paris 06, F-75014 Paris, France</footnote>

<footnote n="7">⁷University of Paris Denis Diderot, University of Paris Sorbonne Cité (PSC), F-75205 Paris Cedex 13, France</footnote>

<footnote n="8">⁸Jet Propulsion Laboratory, Cahill Center for Astronomy & Astrophysics, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA 91109, USA</footnote>

<footnote n="9">⁹National Center for Supercomputing Applications, University of Illinois, 1205 West Clark St., Urbana, IL 61801, USA</footnote>

<footnote>* E-mail: atomczak724@gmail.com</footnote>
Digital Sky Survey (SDSS: York et al. 2000), encompassing a cosmologically significant volume in the local Universe, has confirmed the importance of environment in relation to galaxy evolution including the morphology–density relation (Goto et al. 2003), the colour–density relation (Hogg et al. 2004), and the star formation rate (SFR) versus density relation (Gómez et al. 2003). Nevertheless, despite the modern wealth of data from both observations and simulations, we still lack a fully coherent picture of how galaxies evolve in dense environments.

One of the most key measurements that can be made for a sample of galaxies is their volume-based number density as a function of stellar mass, known simply as the galaxy stellar mass function (SMF). All relevant aspects of galaxy evolution that relate to mass-growth and structure formation will naturally be imprinted in some way in this distribution. Therefore, measuring the SMF provides insight into galaxy evolution as well as being an important diagnostic tool for galaxy simulations.

Over roughly the past decade there has been a multitude of studies dedicated to measuring the galaxy SMF as a function of environment over a broad range of redshifts from $z \sim 1.5$ to the local universe (Bundy et al. 2006; Bolzonella et al. 2010; Peng et al. 2010; Giodini et al. 2012; Vulcani et al. 2012, 2013; Calvi et al. 2013; van der Burg et al. 2013; Hahn et al. 2015; Mortlock et al. 2015; Davidzon et al. 2016). The general picture emerging from many of these studies is that the SMF in denser regions is more heavily weighted towards higher stellar masses, i.e. that the relative number of high- to low-mass galaxies increases with environmental density. A common theme evoked is that this phenomenon is primarily driven by the changing relative contributions of star-forming and quiescent galaxies to the total galaxy population in different environments. Recent studies have explored a formalism of quenching involving two pathways, ‘mass-quenching’ and ‘environment-quenching’, which operate independently of each other (Peng et al. 2010; Muzzin et al. 2012; Quadri et al. 2012).

As mentioned earlier, several physical mechanisms related to galaxy environment are understood as being relevant for galaxy evolution and are pointed to as drivers of environment-quenching. However, the physical origin of mass-quenching is less well-understood. Some suggested mechanisms involve the formation of a stable hot gaseous halo in massive galaxies capable of preventing the accretion of cold gas and effectively quenching a galaxy. Virial shocks (Birnboim & Dekel 2003) and/or active galactic nuclei (Croton et al. 2006) have been proposed as viable energy sources for maintaining such a hot halo.

Combining these two quenching pathways with the observed constancy of the shape of the SMF for star-forming galaxies, Peng et al. (2010) have taken this formalism further to create a model that describes the SMF of quiescent galaxies in terms of two components: mass-quenched galaxies and environment-quenched galaxies. The mass-quenched component is derived from the requirement of maintaining the shape of the star-forming SMF, which when coupled with the SFR-$M_*$ correlation observed for star-forming galaxies (e.g. Noeske et al. 2007) leads to a quenching rate that preferentially affects higher mass galaxies. Under the assumption that the environmental quenching efficiency is independent of stellar mass (for which the previously mentioned works provide observational evidence), it follows that the SMF of environment-quenched galaxies will have the same shape as the SMF of star-forming galaxies. Peng et al. (2010) have shown that this simple, empirically motivated model is able to accurately reproduce the SMF of galaxies in the local universe as measured from the SDSS DR7 (Abazajian et al. 2009). However, recent works have applied this quenching

scheme to measurements of the SMF at $0.5 \lesssim z \lesssim 1$ finding mild to moderate levels of tension (van der Burg et al. 2013; Davidzon et al. 2016). In the latter of these two studies, the authors argue that galaxy–galaxy mergers should also play a significant role in shaping the SMF in high-density environments. In fact, it has been estimated that the merger rate can be $3–4 \times$ greater in high- versus low-density environments (Lin et al. 2010; Kampczyk et al. 2013) and exhibits evolution with redshift (López-Sanjuan et al. 2013). It is important to note that the high-density regime examined in these studies combine both galaxy group and galaxy cluster scales, and are more heavily weighted towards group-like environments. Indeed galaxy groups are believed to be the environment most conducive to galaxy–galaxy merging due to their moderate velocity dispersions, whereas velocities in clusters may act to suppress merging despite bolstering elevated number densities of galaxies (Lin et al. 2010).

In this paper, we present new measurements of the galaxy SMF as a function of local environment from the Observations of Redshift Evolution in Large-Scale Environments (ORELSE) survey (Lubin et al. 2009). ORELSE is a wide-field survey dedicated to studying galaxy evolution across the full range of local environments with extensive photometric and spectroscopic observations of multiple well-known galaxy overdensities at $0.6 < z < 1.3$. In the work presented here, we make use of a subset of eight ORELSE fields from the full sample for which all data have been currently reduced. Nevertheless, this subset covers essentially the entire redshift range of the survey and samples the full range of substructure masses from low-mass groups to rich clusters.

This paper is organized as follows. In Section 2, we discuss the photometric and spectroscopic data as well as SED-fitting techniques used to derive galaxy properties. Section 3 describes the various analysis methodologies employed such as the quantitative definition of local environment and the estimation of mass completeness. In Section 4, we discuss the results from our measurements and introduce a simple semi-empirical model as an aid in understanding the connection between the SMF in different environments. Finally, in Section 5, we summarize our results and consider a future direction for a follow-up study. Throughout this paper, we adopt a standard $\Lambda$ cold dark matter cosmology with $\Omega_M = 0.3$, $\Omega_\Lambda = 0.7$ and $H_0 = 70 \text{ km s}^{-1} \text{ Mpc}^{-1}$.

## 2 DATA

In this study, we make use of several fields taken from the ORELSE survey (Lubin et al. 2009). ORELSE is a large multiwavelength photometric and spectroscopic campaign of $16^1$ fields, each containing massive large-scale structures (LSSs) at $0.6 < z < 1.3$. The goal of this survey is to characterize galaxies over the full range of environments from sparse fields to the dense cores of rich galaxy clusters. In the following sections, we describe in detail the photometric and spectroscopic data and reductions. At the present time, only eight fields from ORESLE have fully reduced spectroscopy and all available imaging data, all of which are used in the work presented here. In Table 1, we list several properties of the LSS fields used in this work. We note here that most of the LSSs in these fields are not individual galaxy clusters but have been revealed as being

<footnote n="1">¹ Note: the original survey design included 20 separate fields. However, since then two fields have been merged into the SC 1324 supercluster and another two into the SC 1604 supercluster (see table 1 of Lubin et al. 2009). Two other fields (Cl 0943+4804 and Cl 1325+3009) have been removed altogether due to incomplete data acquisition.</footnote>
<table><caption>**Table 1.** Large-scale structures.</caption><thead><tr><th>Name</th><th>RA<sup>a</sup><br>(J2000)</th><th>Dec.<sup>a</sup><br>(J2000)</th><th>z<sub>spec</sub><sup>b</sup></th><th>N<sub>spec</sub><sup>c</sup></th><th>log(Σ M<sub>vir</sub>)<sup>d</sup><br>(M<sub>☉</sub>)</th></tr></thead><tbody><tr><td>RXJ1757</td><td>269.3319</td><td>66.5259</td><td>0.693</td><td>374</td><td>14.8</td></tr><tr><td>SC1324</td><td>201.2143</td><td>30.1905</td><td>0.756</td><td>893</td><td>15.3</td></tr><tr><td>RCS0224</td><td>36.14120</td><td>-0.0394</td><td>0.772</td><td>362</td><td>15.0</td></tr><tr><td>RXJ1716</td><td>259.2016</td><td>67.1392</td><td>0.813</td><td>372</td><td>15.3</td></tr><tr><td>RXJ1821</td><td>275.3845</td><td>68.4658</td><td>0.818</td><td>295</td><td>15.2</td></tr><tr><td>SC1604</td><td>241.1409</td><td>43.3539</td><td>0.898</td><td>1150</td><td>15.4</td></tr><tr><td>SC0910</td><td>137.5983</td><td>54.3419</td><td>1.110</td><td>430</td><td>15.0</td></tr><tr><td>SC0849</td><td>132.2333</td><td>44.8711</td><td>1.261</td><td>344</td><td>15.0</td></tr></tbody></table>

Notes. <sup>a</sup>Central coordinates of the main LSS in each field.

<sup>b</sup>Mean redshift of the LSS.

<sup>c</sup>Total number of secure spectroscopic redshifts at 0.55 < z < 1.3.

<sup>d</sup>Note that for many of these LSSs we detect multiple galaxy groups and/or clusters (see Rumbaugh et al. 2017). In these cases, we show the sum of the virial masses of all substructures.

composed of multiple galaxy groups and/or clusters (see section 2 of Rumbaugh et al. 2017).

## 2.1 Photometry

Each LSS field comprises a wealth of photometric data spanning optical to mid-infrared wavelengths. These imaging data were compiled from both proposed observing campaigns part of ORELSE as well as archival data from Suprime-Cam (Miyazaki et al. 2002) on Subaru, the Large Format Camera (LFC; Simcoe et al. 2000) on the Palomar 200-inch Hale telescope, the Wide-field InfraRed Camera (WIRCam: Puget et al. 2004) on the Canada France Hawaii Telescope (CFHT), the Wide Field Camera (WFCAM: Casali et al. 2007) on the United Kingdom InfraRed Telescope (UKIRT), and the InfraRed Array Camera (IRAC: Fazio et al. 2004) on the Spitzer Space Telescope. Table 2 lists the available photometry for each field, which facilities and instruments were used, as well as depth estimates.

Optical and Y-band imaging from Subaru Suprime-Cam was reduced using the SDFRED pipeline<fnref n="2" /> (Ouchi et al. 2004) as well as several TERAPIX<fnref n="3" /> software packages. Basic image processing is first done with SDFRED which we use to perform (1) bias level subtraction based on designated overscan regions on the Suprime-Cam charge coupled devices (CCDs), (2) flat-field correction, (3) optics and atmospheric distortion correction and (4) equalization of the point spread function (PSF). Flat-fields were constructed directly from the science images by masking out objects to create 'super sky flats'. This is possible due to the nature of the observations that include many individual dithered images and because we do not observe large extended objects. Next, we perform astrometric alignment, photometric scaling, and image stacking using the Source Extractor (SEXTRACTOR; Bertin & Arnouts 1996), Software for Calibrating AstroMetric and Photometry (SCAMP; Bertin 2006), and SWARP (Bertin et al. 2002) software packages to create fully-reduced mosaics and weight maps. Photometric calibration is performed from same-night observations of standard star fields from the Landolt (1992) catalogues. Standard fields are reduced identically as described above.

<table><caption>**Table 2.** Photometry.</caption><thead><tr><th>Filter</th><th>Telescope</th><th>Instrument</th><th>Depth<sup>a</sup></th></tr></thead><tbody><tr><td>SC1604</td><td></td><td></td><td></td></tr><tr><td>B</td><td>Subaru</td><td>Suprime-Cam</td><td>26.6</td></tr><tr><td>V</td><td>Subaru</td><td>Suprime-Cam</td><td>26.1</td></tr><tr><td>R<sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>26.0</td></tr><tr><td>I<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.1</td></tr><tr><td>Z<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>24.6</td></tr><tr><td>r'</td><td>Palomar</td><td>LFC</td><td>24.2</td></tr><tr><td>i'</td><td>Palomar</td><td>LFC</td><td>23.6</td></tr><tr><td>z'</td><td>Palomar</td><td>LFC</td><td>23.1</td></tr><tr><td>J</td><td>UKIRT</td><td>WFCAM</td><td>22.1</td></tr><tr><td>K</td><td>UKIRT</td><td>WFCAM</td><td>21.9</td></tr><tr><td>[3.6]</td><td>Spitzer</td><td>IRAC</td><td>24.7</td></tr><tr><td>[4.5]</td><td>Spitzer</td><td>IRAC</td><td>24.3</td></tr><tr><td>[5.8]</td><td>Spitzer</td><td>IRAC</td><td>22.7</td></tr><tr><td>[8.0]</td><td>Spitzer</td><td>IRAC</td><td>22.6</td></tr><tr><td>RXJ1716</td><td></td><td></td><td></td></tr><tr><td>B</td><td>Subaru</td><td>Suprime-Cam</td><td>25.9</td></tr><tr><td>V</td><td>Subaru</td><td>Suprime-Cam</td><td>26.6</td></tr><tr><td>R<sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>26.2</td></tr><tr><td>I<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.4</td></tr><tr><td>Z<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>24.7</td></tr><tr><td>J</td><td>CFHT</td><td>WIRCam</td><td>21.3</td></tr><tr><td>K<sub>s</sub></td><td>CFHT</td><td>WIRCam</td><td>21.7</td></tr><tr><td>[3.6]</td><td>Spitzer</td><td>IRAC</td><td>24.6</td></tr><tr><td>[4.5]</td><td>Spitzer</td><td>IRAC</td><td>24.1</td></tr><tr><td>[5.8]</td><td>Spitzer</td><td>IRAC</td><td>22.4</td></tr><tr><td>[8.0]</td><td>Spitzer</td><td>IRAC</td><td>22.3</td></tr><tr><td>RXJ1757</td><td></td><td></td><td></td></tr><tr><td>B</td><td>Subaru</td><td>Suprime-Cam</td><td>26.4</td></tr><tr><td>V</td><td>Subaru</td><td>Suprime-Cam</td><td>25.9</td></tr><tr><td>R<sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>26.7</td></tr><tr><td>Z<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.6</td></tr><tr><td>r'</td><td>Palomar</td><td>LFC</td><td>25.1</td></tr><tr><td>i'</td><td>Palomar</td><td>LFC</td><td>24.8</td></tr><tr><td>z'</td><td>Palomar</td><td>LFC</td><td>22.9</td></tr><tr><td>Y</td><td>Subaru</td><td>Suprime-Cam</td><td>22.7</td></tr><tr><td>J</td><td>CFHT</td><td>WIRCam</td><td>21.0</td></tr><tr><td>K<sub>s</sub></td><td>CFHT</td><td>WIRCam</td><td>21.8</td></tr><tr><td>[3.6]</td><td>Spitzer</td><td>IRAC</td><td>23.9</td></tr><tr><td>[4.5]</td><td>Spitzer</td><td>IRAC</td><td>23.8</td></tr><tr><td>RCS0224</td><td></td><td></td><td></td></tr><tr><td>B</td><td>Subaru</td><td>Suprime-Cam</td><td>26.2</td></tr><tr><td>V</td><td>Subaru</td><td>Suprime-Cam</td><td>26.0</td></tr><tr><td>R<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.9</td></tr><tr><td>I<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.5</td></tr><tr><td>Z<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>24.9</td></tr><tr><td>J</td><td>UKIRT</td><td>WFCAM</td><td>21.2</td></tr><tr><td>K</td><td>UKIRT</td><td>WFCAM</td><td>21.4</td></tr><tr><td>[3.6]</td><td>Spitzer</td><td>IRAC</td><td>24.0</td></tr><tr><td>[4.5]</td><td>Spitzer</td><td>IRAC</td><td>23.6</td></tr><tr><td>SC0849</td><td></td><td></td><td></td></tr><tr><td>B</td><td>Subaru</td><td>Suprime-Cam</td><td>26.4</td></tr><tr><td>V</td><td>Subaru</td><td>Suprime-Cam</td><td>26.5</td></tr><tr><td>R<sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>26.2</td></tr><tr><td>I<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.5</td></tr><tr><td>Z<sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.1</td></tr><tr><td>Z<sub>R</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>23.5</td></tr><tr><td>r'</td><td>Palomar</td><td>LFC</td><td>24.7</td></tr><tr><td>i'</td><td>Palomar</td><td>LFC</td><td>24.4</td></tr><tr><td>z'</td><td>Palomar</td><td>LFC</td><td>23.3</td></tr><tr><td>NB711</td><td>Subaru</td><td>Suprime-Cam</td><td>23.7</td></tr><tr><td>NB816</td><td>Subaru</td><td>Suprime-Cam</td><td>25.9</td></tr><tr><td>J</td><td>UKIRT</td><td>WFCAM</td><td>21.8</td></tr><tr><td>K</td><td>UKIRT</td><td>WFCAM</td><td>21.6</td></tr></tbody></table>

<footnote n="2">² Note that there are two versions of this software: SDFRED1 and SDFRED2 for data taken before and after July 2008, respectively. See http://subarutelescope.org/Observing/Instruments/SCam/sdfred/index.html.en.</footnote>

<footnote n="3">³ http://terapix.iap.fr</footnote>
<table><caption>**Table 2** – continued</caption><thead><tr><th>Filter</th><th>Telescope</th><th>Instrument</th><th>Depth<sup>a</sup></th></tr></thead><tbody><tr><td>[3.6]</td><td><i>Spitzer</i></td><td>IRAC</td><td>24.8</td></tr><tr><td>[4.5]</td><td><i>Spitzer</i></td><td>IRAC</td><td>24.3</td></tr><tr><td>SC1324</td><td></td><td></td><td></td></tr><tr><td><i>B</i></td><td>Subaru</td><td>Suprime-Cam</td><td>26.6</td></tr><tr><td><i>V</i></td><td>Subaru</td><td>Suprime-Cam</td><td>25.7</td></tr><tr><td><i>R</i><sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.7</td></tr><tr><td><i>I</i><sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.2</td></tr><tr><td><i>Z</i><sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>22.6</td></tr><tr><td><i>r'</i></td><td>Palomar</td><td>LFC</td><td>24.9</td></tr><tr><td><i>i'</i></td><td>Palomar</td><td>LFC</td><td>24.3</td></tr><tr><td><i>z'</i></td><td>Palomar</td><td>LFC</td><td>22.6</td></tr><tr><td><i>J</i></td><td>UKIRT</td><td>WFCAM</td><td>22.4</td></tr><tr><td><i>K</i></td><td>UKIRT</td><td>WFCAM</td><td>21.7</td></tr><tr><td>[3.6]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.9</td></tr><tr><td>[4.5]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.8</td></tr><tr><td>RXJ1821</td><td></td><td></td><td></td></tr><tr><td><i>B</i></td><td>Subaru</td><td>Suprime-Cam</td><td>26.0</td></tr><tr><td><i>V</i></td><td>Subaru</td><td>Suprime-Cam</td><td>26.0</td></tr><tr><td><i>r'</i></td><td>Palomar</td><td>LFC</td><td>24.4</td></tr><tr><td><i>i'</i></td><td>Palomar</td><td>LFC</td><td>24.3</td></tr><tr><td><i>z'</i></td><td>Palomar</td><td>LFC</td><td>23.3</td></tr><tr><td><i>Y</i></td><td>Subaru</td><td>Suprime-Cam</td><td>23.4</td></tr><tr><td><i>J</i></td><td>CFHT</td><td>WIRCam</td><td>21.4</td></tr><tr><td><i>K</i><sub>s</sub></td><td>CFHT</td><td>WIRCam</td><td>21.7</td></tr><tr><td>[3.6]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.9</td></tr><tr><td>[4.5]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.8</td></tr><tr><td>SC0910</td><td></td><td></td><td></td></tr><tr><td><i>B</i></td><td>Subaru</td><td>Suprime-Cam</td><td>24.4</td></tr><tr><td><i>V</i></td><td>Subaru</td><td>Suprime-Cam</td><td>25.6</td></tr><tr><td><i>R</i><sub>C</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>26.4</td></tr><tr><td><i>I</i><sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>25.8</td></tr><tr><td><i>Z</i><sub>+</sub></td><td>Subaru</td><td>Suprime-Cam</td><td>24.8</td></tr><tr><td><i>J</i></td><td>UKIRT</td><td>WFCAM</td><td>22.1</td></tr><tr><td><i>K</i></td><td>UKIRT</td><td>WFCAM</td><td>21.7</td></tr><tr><td>[3.6]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.2</td></tr><tr><td>[4.5]</td><td><i>Spitzer</i></td><td>IRAC</td><td>23.2</td></tr></tbody></table>

Note. $^{a}$80 per cent completeness limits derived from the recovery rate of artificial sources inserted at empty sky regions.

Optical imaging from the LFC mounted on the 200-inch Hale Telescope at Palomar Observatory is reduced using a suite of scripts<fnref n="4" /> written in Image Reduction and Analysis Facility (IRAF). These scripts perform standard procedures for image processing (e.g. bias subtraction and flat-field correction) as well as a few routines written specifically for handling LFC data. We refer the reader to Gal, Lubin & Squires (2005) for a more detailed discussion of the processing of LFC images.

Near-infrared imaging in the *J* and *K*/*K*<sub>s</sub> bands were taken using UKIRT/WFCAM and CFHT/WIRCam. Both facilities implement automated data reduction pipelines that output fully-reduced mosaics and weight maps. We photometrically calibrate these mosaics using bright (<15 mag), non-saturated objects with existing photometry from 2MASS in each field. It is important to note that the 2MASS *K*<sub>s</sub> filter is significantly different than the WFCAM *K* filter. In order to account for this we make use of infrared spectral observations of stars from the Infrared Telescope Facility (IRTF) spectral library (Rayner, Cushing & Vacca 2009) to calculate a

transformation between these two passbands ($K_{WFCAM} - K_{s,2MASS}$) based on their ($J - K_s$) colours.

Once all ground-based optical and near-infrared images (*B* − *K* bands) have been assembled for a given field we register them to a common pixel grid (0.2 arcsec pixel<sup>−1</sup>). We account for small astrometric distortions, such as offsets and rotations, amongst the final mosaics using the SCAMP software. Due to differences in the imaging for each LSS field, we carefully choose detection images on an ad hoc basis. In some cases, we use an inverse-variance weighted stack of two or more images for source detection. For each field, we require that the detection image be predominantly redward of the 4000 Å break at the redshift of the LSS contained in it, have sufficiently good seeing (<1 arcsec), and be sufficiently deep to detect low-mass galaxies (completeness limits will be discussed in Section 3.1). We make an exception for the SC1604 field for which only the $R_C$ image covered the full extent of the spectroscopic footprint. Table 3 lists images used for source detection as well as their seeing.

PSFs are created for each image by stacking a selection of bright, non-saturated point sources. We then use the Richardson & Lucy algorithm in SCIKIT-IMAGE (van der Walt et al. 2014) to construct convolution kernels that are used to smooth all optical through near-infrared images to the largest PSF. Photometry is extracted from PSF-matched images in fixed circular apertures by running S<sub>EXTRACTOR</sub> in dual-image mode using the aforementioned detection images. Aperture diameters are chosen to be 1.3 × the full-width half-maximum (FWHM) of the largest PSF, where the signal-to-noise ratio (S/N) of the extracted flux is near its maximum (see section 4.1 of Whitaker et al. 2011). Fluxes are corrected for attenuation from dust in the Milky Way based on the reddening maps derived by Schlafly & Finkbeiner (2011) as provided by the NASA/IPAC Infrared Science Archive<fnref n="5" /> (IRSA). We define an aperture correction as the ratio of an object’s total flux to its aperture flux in the PSF-matched detection image, where we use S<sub>EXTRACTOR</sub>’s auto flux as the ‘total’ flux.

All of our cluster fields have non-cryogenic *Spitzer*/IRAC imaging ([3.6] and [4.5] µm). However, two fields (RXJ1716 and SC1604) were observed during the cryogenic mission and thus have [5.8] and [8.0] µm imaging. We process the individual corrected basic calibrated data (cBCD) images using the MOPEX software package (Makovoz et al. 2006) along with several custom scripts. We first preprocess the cBCD frames using a custom IDL code (J. Surace, private communication) that performs and improved correction for column pull-down and muxbleed artefacts. The images are next background-matched using the overlap.pl routine from MOPEX prior to mosaicking with the mosaic.pl routine. All *Spitzer* imaging is flux-calibrated and provided in units of MJy sr<sup>−1</sup>.

Due to the significantly larger PSFs of *Spitzer*/IRAC images (~2 arcsec), we take an alternate approach for measuring photometry. We use *T-PHOT* (Merlin et al. 2015), a software package designed to extract photometry in crowded images where blending from neighbours is significant. In brief, the methodology of *T-PHOT* is as follows: first, the positions and morphologies of objects are obtained for use as priors based on a segmentation map (produced by S<sub>EXTRACTOR</sub>) of a higher resolution image, in our case the detection image mentioned above. Cutouts are taken from this higher resolution image that are then used to create low-resolution models of each object by smoothing with a provided convolution kernel. These models are then simultaneously fit to the IRAC image until optimal

<footnote n="4">⁴ http://www.ifa.hawaii.edu/~rgal/science/lfcred/lfc_red.html</footnote>

<footnote n="5">⁵ http://irsa.ipac.caltech.edu/applications/DUST/</footnote>
<table><caption>**Table 3.** Image properties and completeness limits.</caption><thead><tr><th>Name</th><th>Detection image</th><th>FWHM (arcsec)</th><th>Area<sup>a</sup> (arcmin<sup>2</sup>)</th><th>M<sub>*</sub>, lim<sup>b</sup> (M<sub>☉</sub>)</th><th>M<sub>*</sub>, ssp<sup>c</sup> (M<sub>☉</sub>)</th><th>σ<sub>Δz/(1+z<sub>spec</sub>)</sub> (%)</th><th>f<sub>outlier</sub> (%)</th></tr></thead><tbody><tr><td>RXJ1757</td><td>r', i'</td><td>0.79</td><td>203</td><td>9.21</td><td>9.72</td><td>3.0</td><td>8.7</td></tr><tr><td>SC1324</td><td>i'</td><td>0.97</td><td>465</td><td>9.46</td><td>10.02</td><td>3.2</td><td>9.2</td></tr><tr><td>RCS0224</td><td>I<sub>+</sub></td><td>0.67</td><td>172</td><td>8.73</td><td>9.74</td><td>2.8</td><td>3.1</td></tr><tr><td>RXJ1716</td><td>R<sub>C</sub>, I<sub>+</sub>, Z<sub>+</sub></td><td>0.69</td><td>194</td><td>9.05</td><td>9.70</td><td>2.2</td><td>5.4</td></tr><tr><td>RXJ1821</td><td>Y</td><td>0.83</td><td>179</td><td>9.73</td><td>10.01</td><td>3.1</td><td>4.8</td></tr><tr><td>SC1604</td><td>R<sub>C</sub></td><td>1.00</td><td>270</td><td>9.28</td><td>10.05</td><td>2.9</td><td>9.5</td></tr><tr><td>SC0910</td><td>R<sub>C</sub>, I<sub>+</sub>, Z<sub>+</sub></td><td>0.65</td><td>185</td><td>9.52</td><td>10.13</td><td>3.2</td><td>8.1</td></tr><tr><td>SC0849</td><td>Z<sub>+</sub></td><td>0.77</td><td>147</td><td>9.46</td><td>10.19</td><td>2.7</td><td>6.0</td></tr></tbody></table>

Notes. $^a$Total projected area covered by spectroscopic observations.

$^b$80 per cent mass completeness limit from artificial sources.

$^c$Mass limit from a dust-free SSP with $z_{form} = 5$.

scale factors, assessed through a global $\chi^2$ minimization, for each object are obtained. We run T-PHOT in cells-on-objects mode where when fitting a model for a given object only neighbours within a 1 × 1 arcmin box centred on the object are considered in the fitting. After this initial sequence, T-PHOT is then rerun in a second pass in which registered kernels generated during the first pass are utilized to account for mild astrometric differences between the input images. Output fluxes are the total model flux from the best fit. Therefore, in order to make these fluxes consistent with the fixed-aperture photometry of the optical through near-infrared imaging, we scale the IRAC flux of a given object by the ratio of the aperture to total flux from the PSF-matched detection image. We use the S-EXTRACTOR/AUTO aperture flux for the ‘total’ flux.

## 2.2 Spectroscopy

The bulk of the spectroscopic data used in this study were obtained as part of a massive 300 h Keck 2/DEep Imaging Multi-Object Spectrograph (DEIMOS; Faber et al. 2003) campaign targeting all fields of the ORELSE survey listed in table 1 of Lubin et al. (2009). In this section, we briefly describe the targeting, acquisition and reduction of those data pertaining to the eight fields presented in this study. A total of 67 slitmasks were observed over the eight fields, with the number of slitmasks per field ranging from four (RCS0224) to 18 (SC1604). Slitmasks were observed with the 1200 line mm$^{-1}$ grating with 1 arcsec slits resulting in a plate scale of 0.33 Å pix$^{-1}$, an $R \sim 5000$ ($\lambda/\theta_{FWHM}$, where $\theta_{FWHM}$ is the full-width half-maximum resolution), and a wavelength coverage of $\Delta\lambda \sim 2600$. Central wavelengths ranged from 7200 Å $\le \lambda_c \le 8700$ Å depending on the redshift of the field and exposure times varied from 3600 to 10 800 s, scaled to roughly achieve an identical distribution in continuum S/N across all masks independent of conditions and the median faintness of the target population. For more details on the range of conditions, set-ups and exposure times for each individual field, see Rumbaugh et al. (2017). In addition to the DEIMOS campaign, a small fraction ($\le 3$ per cent) of the spectroscopic redshifts used in this study were drawn from a variety of other studies (Oke, Postman & Lubin 1998; Gal & Lubin 2004; Tanaka et al. 2008; Mei et al. 2012) which utilized a variety of different telescopes and instruments. For the spectroscopic redshifts drawn from these studies, we imposed, to the best of our ability, similarly stringent criteria as those imposed on the DEIMOS data (see below) in order to consider a redshift secure.

Targets for DEIMOS slitmasks were selected primarily based on the observed-frame colour-colour cuts presented in Lubin et al. (2009). Objects that satisfied the observed-frame $r' - i'$ and $i' - z'$

colour range for the redshift of the targeted LSS (see table 2 of Lubin et al. 2009), a colour range designed to select galaxies on or near the red sequence at that redshift, were given the highest targeting priority (priority 1). Objects of progressively bluer colours were assigned progressively lower targeting priorities. It is important to note that while redder objects were heavily favoured in our selection scheme, because of the strictness of our cuts and the relative rarity of objects at these colours, the majority of spectroscopic targets had colours bluer than the cuts used to define priority 1 objects. The average fraction of priority 1 objects per mask varied considerably from field to field, from 10.0 per cent in RX J0910 to 45.4 per cent in RCS0224, with more well-sampled fields generally having a smaller average priority 1 fraction. This colour-colour selection scheme was augmented by additional prioritization designed to select targets of particular interest that included multiwavelength objects (i.e. radio or X-ray) and known or possible member galaxies targeted by previous surveys. Targeted objects were generally restricted to $i' < 24.5$, though this was not a hard limit. While this colour-colour preselection was designed to target members of the main LSS in each field, a significant number of priority 2+ objects were also observed, helping to sample the full colour space of galaxies (see Lubin et al. 2009, for a full description). Indeed, as was shown by Shen et al. (2017), the ORELSE spectroscopic sample is broadly representative of the underlying galaxy population over a wide range of stellar masses and colours.

Data were reduced using a version of the Deep Evolutionary Extragalactic Probe 2 (DEEP2; Davis et al. 2003; Newman et al. 2013) spec2d pipeline modified to improve the precision of the response correction, perform absolute spectrophotometric flux calibration, and improve the handling of the joining of the blue and red ends of the spectra over the ~5 Å gap that separates the two CCD arrays. All objects, those targeted and those which serendipitously fell in a slit, were visually inspected and assigned a spectroscopic redshift (hereafter $z_{spec}$) and a redshift quality code ($Q$) in the $z_{spec}$ environment following the DEEP2 scheme (see Gal et al. 2008; Newman et al. 2013). The process for finding and extracting serendipitous detections and assigning photometric counterparts is detailed in Lemaux et al. (2009). Spectroscopic redshifts were assigned a quality code of $Q = -1$ (stars) or $Q = 3, 4$ (galaxies) were considered secure. To be identified securely as a star, we required either the presence of multiple significant narrow photospheric absorption features (typically $H\alpha$ and the Ca II triplet) or obvious broad continuum features indicative of a late-type star (primarily TiO). An extragalactic spectroscopic redshifts was considered secure only in the presence of two or more emission or absorption features, where $Q = 3$ indicates that one or more of the features was slightly questionable in S/N or for some other reason. The two
<figure><img src="image_2.png" /><figcaption>**Figure 1.** A plot of spectroscopic versus photometric redshifts for the LSS fields of ORELSE used in this work. Indicated in the top-left corner are the total number of galaxies, the 1σ scatter derived from fitting a Gaussian to the residual $\Delta z/(1+z_{\text{spec}})$, and the catastrophic outlier fraction ($|\Delta z|/(1+z_{\text{spec}}) \ge 0.15$). Subpanels on the right show the same distribution and statistics for each field individually.</figcaption></figure>

components of the $\lambda$3726 Å, $\lambda$3729 Å [O II] feature were considered sufficient to assign a spectrum $Q=4$ when unblended as long as both components were significantly detected. A quality code of $Q=3$ was assigned when the two components moderately blended by velocity effects in the absence of additional corroborating features. It has been shown that adopting this scheme and these quality code cuts results in extragalactic redshift measurements that are reliable at the $\gtrsim 99$ per cent level (Newman et al. 2013). For more details on the meaning of the quality codes and the likelihood for a redshift of a given quality code to be accurate see Newman et al. (2013).

## 2.3 Spectral energy distribution fitting

We perform spectral energy distribution (SED) fitting on the observed optical through mid-IR photometry in order to derive photometric redshifts as well as physical properties (e.g. stellar mass, $A_V$). In estimating photometric redshifts, we employ the code Easy and Accurate Redshifts from Yale (EAZY; Brammer, van Dokkum & Coppi 2008). For this we use fixed-aperture fluxes measured from

PSF-matched images. Briefly, EAZY utilizes a set of six basis template SEDs derived from a non-negative matrix factorization decomposition of the Projet d'Etude des Galaxies par Synthèse Evolutive template library (PÉGASE; Fioc & Rocca-Volmerange 1997), as well as one additional template representing an old stellar population from the Maraston (2005) library (this last template was added in order to help accommodate evolved galaxies at $z < 1$ that are not well fit by the six basis templates). EAZY performs $\chi^2$ minimization at points along a user-defined redshift grid using linear combinations of this default template set. A probability density function (PDF) is then calculated from the minimized $\chi^2$ values: $P(z) \propto e^{-\chi^2/2}$. Finally, this PDF is modulated by a magnitude prior (R-band for this work) that is designed to mimic the intrinsic redshift distribution for galaxies of given apparent magnitude. EAZY is capable of reporting multiple different types of photometric redshifts that are derived from different manipulations of the final redshift PDF. Throughout this paper, we adopt EAZY's 'zpeak' for the photometric redshifts of galaxies. zpeak is obtained by marginalizing over the final PDF. Additionally, if an object has multiple peaks in its PDF, EAZY will only marginalize over the peak with the largest integrated probability.
To assess the precision and accuracy of these photometric redshifts, we compare to our spectroscopic redshift measurements. In Fig. 1, we plot $z_{phot}$ versus $z_{spec}$ for the sample of galaxies with high-quality spectroscopic redshifts. For each field, we fit a Gaussian to the distribution of the residual $(z_{phot} - z_{spec})/(1+z_{spec})$ for all objects and take the best-fitting $\sigma_{\Delta z/(1+z_{spec})}$ as the $z_{phot}$ uncertainty. Values of $\sigma_{\Delta z/(1+z_{spec})}$, ranging between 2.2 and 3.2 per cent, are shown in Fig. 1 and Table 3.

We next utilize the code Fitting and Assessment of Synthetic Templates (FAST; Kriek et al. 2009) for deriving stellar masses and other physical properties. Here, we use aperture-corrected fluxes and fix galaxies to their derived photometric redshift (or secure $z_{spec}$ when available). In brief, FAST creates a multidimensional cube of model fluxes from a provided stellar population synthesis (SPS) library. Each object in the photometric catalogue is fit by every model in this cube and the minimum $\chi^2$ for each is recorded. The model with the lowest minimum $\chi^2$ is adopted as the best fit. In this work, we use the SPS library of Bruzual & Charlot (2003; BC03 hereafter) assuming a Chabrier (2003) stellar initial mass function (IMF) and solar metallicity. For dust extinction, we adopt the Calzetti et al. (2000) attenuation curve. We adopt delayed exponentially declining star-formation histories (SFH $\propto t \times e^{-t/\tau}$), allowing $\log(\tau \text{ yr}^{-1})$ to range between 7 and 10 in steps of 0.2, $\log(\text{age yr}^{-1})$ to range between 7.5 and 10.1 in steps of 0.1, and $A_V$ to range between 0 and 4 in steps of 0.1. These assumptions are broadly consistent with other contemporary multiwavelength extragalactic surveys.

To aid in the selection of real galaxies within our catalogues, we define a 'use' flag for objects in similar fashion to recent multiwavelength photometric surveys (Whitaker et al. 2011; Muzzin et al. 2013; Skelton et al. 2014; Straatman et al. 2016). This useflag is designed to reject objects that are either poorly detected (S/N < 3), saturated, have catastrophic SED fits, or are likely foreground stars. Catastrophic SED fits are defined as objects with reduced $\chi^2_{galaxy} > 10$ from fitting with EAZY. This threshold was chosen arbitrarily based on visual inspection of individual SED fits as well as distributions of derived rest-frame colours of objects. For identifying foreground stars, we employ a separate set of criteria. First, we perform another round of SED fitting with EAZY using the stellar template library of Pickles (1998). These stellar templates are fit in single-template mode with no redshifting applied. An object is flagged as a star in our catalogues if it:

(1) is detected at S/N > 3,

(2) has a diameter that is <1.3× the FWHM of the PSF,

(3) has a major- to minor-axis ratio <1.1,

(4) has a reduced $\chi^2_{stellar} < \chi^2_{galaxy}$ and

(5) is not flagged as a galaxy based on its observed spectrum.

Furthermore, any object which happens to not meet all of these criteria but has been flagged as a star based on the spectroscopic observations discussed in Section 2.2 will automatically be rejected as a star.

# 3 METHODS

## 3.1 Completeness limits

In this section, we describe how photometric and stellar mass completeness limits are derived. For each image, we estimate the photometric depth by performing simulations. Artificial sources with a range of input magnitudes are added at empty sky positions. We then run SEXTRACTOR with the same configuration file as for the catalogues and calculate the fraction of artificial sources recovered

as a function of input magnitude. This recovery fraction provides a sense as to the depth of each image. Magnitudes corresponding to the 80 per cent recovery limit are listed in Table 2.

We then use the 80 per cent completeness magnitude of the detection image for each LSS field to estimate stellar mass completeness limits. In order to convert magnitude limits into stellar mass limits, we make use of the FourStar Galaxy Evolution survey (ZFOURGE; Straatman et al. 2016). For a given LSS field, we select all galaxies in ZFOURGE that have a magnitude that is within ±0.1 mag of the 80 per cent limiting magnitude in the corresponding detection bandpass (for cases where the detection image is a stack of multiple images we use inverse-variance weighted mean magnitudes from ZFOURGE). This is a representative sample of galaxies that are at the threshold of detection in our data, or to put it more precisely, these represent the galaxies that would be detected at a rate of 80 per cent. Plotting stellar mass versus redshift for this sample we adopt the upper 95th percentile as a function of redshift as the limiting stellar mass $M_{*, lim}$. Despite the fact that ZFOURGE is a $K_s$-band selected survey, it is complete to lower stellar masses. We estimate the ZFOURGE to be between 0.2 and 1.5 dex deeper in terms of stellar mass completeness at the redshifts of each respective ORELSE LSS.

We also consider mass completeness limits more appropriate for galaxies dominated by old stellar populations that have elevated mass-to-light ratios. These limits are derived from a single-burst BC03 stellar population with $z_{form} = 5$ with no dust attenuation. At all $z < z_{form}$, the model is scaled to the limiting magnitude of the detection bandpass and its stellar mass is recorded as $M_{*, ssp}$. Throughout this analysis, we adopt the former ($M_{*, lim}$) as the completeness limit for the total as well as star-forming galaxy populations and the latter ($M_{*, ssp}$) as the completeness limit for the quiescent population. All stellar mass completeness limits are listed in Table 3.

## 3.2 Classifying star-forming and quiescent galaxies

It has been shown that a simple combination of rest-frame broad-band colours in the *U*, *V* and *J* filters can be an effective way of classifying galaxies as actively star-forming versus quenched (e.g. Wuyts et al. 2007; Williams et al. 2009). The main advantage of this method is in its ability to decouple dust-reddened star-forming galaxies from quiescent galaxies in the *U* − *V* versus *V* − *J* colour space, thus allowing for a less biased selection of active and passive galaxies.

We make use of EAZY in estimating rest-frame fluxes of galaxies. For each galaxy, the best-fitting $z_{phot}$ from the initial run of EAZY (or $z_{spec}$ when available) is used to identify which observed-frame photometry is nearby a given rest-frame filter. EAZY then fits a SED to these identified photometry and interpolates accordingly to extract a flux for the rest-frame filter. This procedure is performed separately for each of the rest-frame *U*, *V* and *J* bands.

Fig. 2 shows the distribution of ORELSE galaxies in the UVJ diagram for our sample. The boundaries delineating star forming and quiescent are taken from Whitaker et al. (2011); quiescent galaxies are selected with $(U - V) > 1.3 \cap (V - J) < 1.6 \cap (U - V) > 0.88(V - J) + 0.59$.

## 3.3 Defining local environment

There exist a variety of quantitative metrics for defining the environment that a galaxy resides in. In a qualitative sense, these metrics can be viewed as falling into one of two broad categories: 'local environment' that relates to small physical scales internal to the
<figure><img src="image_3.png" /></figure>

**Figure 2.** Rest-frame *U* − *V* versus *V* − *J* colour diagram used to classify galaxies as star-forming or quiescent. Only galaxies at 0.55 < *z* < 1.3 above the estimated stellar mass completeness limits with good use-flags are shown. The left and right panels show the photometric and spectroscopic samples respectively, with total numbers of galaxies indicated in the top-left corner. Spectroscopic targeting was designed to prioritize galaxies that are probable members of the LSSs in each field (see Lubin et al. 2009) which explains the elevated quiescent fraction relative to the photometric sample. The main advantage of this diagram over using a single rest-frame colour is that it is robust against misclassifying star-forming galaxies reddened by dust as quiescent, illustrated by the arrow which shows the reddening vector caused by Δ*A*<sub>*V*</sub> = 1 following the Calzetti et al. (2000) extinction law.

<figure><img src="image_4.png" /></figure>

**Figure 3.** A brief illustration of how the Voronoi Monte Carlo overdensity maps are constructed. For each LSS field, galaxies are separated into narrow redshift bins (Δv = ±1500 km s⁻¹) across the broad redshift range of 0.55 < z < 1.3. Photometric redshifts are used only when a spectroscopic redshift is not available. Here, we show one such redshift slice bracketing the central redshift of the main overdensity in the RXJ 1716 field of view. A Voronoi tessellation is generated, separating galaxies into polygonal cells that represent the projected area that is nearest to the galaxy (left-hand panel). Densities are calculated as the inverse of the cell area multiplied by the square of the angular diameter distance at the corresponding redshift. These densities are then projected on to a two-dimensional pixel grid with a pixel scale of 75 × 75 proper kpc. Finally, overdensities are calculated as log(1 + δgal) ≡ log(1 + (Σi,j − ȳ)/ȳ), where Σi,j is the density of pixel (i, j) and ȳ is the median density (centre panel). This procedure is repeated 100 times where for each iteration photometric redshifts are randomly sampled based on the estimated zphot uncertainties. The final overdensity map is computed by median-stacking the maps generated from all 100 iterations.

halo of a given galaxy and ‘large-scale environment’ that relates to
large physical scales external to the halo of a given galaxy (Muldrew
et al. 2012). For the purposes of this work, we focus on a galaxy’s
local environment and its impact on the galaxy SMF.

We measure local environment for galaxies in our sample us-
ing a Voronoi Monte Carlo algorithm that will be described in
full detail in a future paper (Lemaux et al. in preparation). An

illustration of the approach described below is shown in Fig. 3. The
Voronoi tessellation has been shown to be one of the most accurate
techniques for estimating the local density field in simulations
(Darvish et al. 2015). For each MC realization, all objects that
lack a high-quality zspec are assigned a perturbed zphot. A perturbed
zphot for a given galaxy is calculated by randomly sampling a Gaus-
sian with a mean and dispersion set to the original zphot and 1σ
uncertainty, respectively. All galaxies with a good use-flag are then divided into thin redshift slices and a Voronoi tessellation is calculated on each slice. These redshift slices span $0.55 < z < 1.3$ and have widths set such that they encompass $\pm 1500$ km s$^{-1}$ in velocity space from their central redshift. The areas of the Voronoi cells are then projected on to a two-dimensional (2D) grid of $75 \times 75$ proper kpc pixels. Local density is defined as the inverse of the cell area multiplied by the square of the angular diameter distance. A final density map is computed by median combining the density maps from 100 Monte Carlo realizations. The local overdensity at pixel $(i, j)$ is calculated as $\log(1 + \delta_{\text{gal}}) \equiv \log(1 + (\Sigma_{i,j} - \bar{\Sigma})/\bar{\Sigma})$, where $\bar{\Sigma}$ is the median density of all pixels where the map is reliable (i.e. coverage in nearly all images and not near the edge of the detection image). For reference, the largest (projected) environmental densities probed by our data reach as high as $\sim 100$ galaxies per Mpc$^2$. Note that this is based on the smallest Voronoi cells in the creation of the density maps and does not mean that we observe a single contiguous $1$ Mpc$^2$ area containing 100 galaxies.

Note that true $z_{\text{phot}}$ PDF of a galaxy is not always well represented by a Gaussian distribution that could introduce a bias in the calculations described above. To test this, we have remeasured the density maps for one of our fields (RCS0224) using the $z_{\text{phot}}$ PDFs output by EAZY and directly compared the output overdensity values, pixel by pixel, to the fiducial case where we assume a Gaussian. From this comparison, we calculate that the median offset and NMAD scatter in $\log(1 + \delta_{\text{gal}})$, between the Gaussian and formal PDF cases, is 0.005 and 0.18 dex, respectively. These values do not vary significantly with redshift. Furthermore, these values are comparable to the level of variation seen when the density maps are recreated from a new 100-iteration MC calculation using the same (Gaussian) formalism. Therefore, we conclude that (1) the assumption of Gaussianity of the $z_{\text{phot}}$ PDF does not bias the construction of the Voronoi MC (over)density maps, and (2) the offset and scatter quoted above are representative of the natural level of deviation rooted in the Monte Carlo algorithm itself.

In order to test our ability to recover the true underlying density field, and thus the accuracy with which the galaxies are assigned densities from the Voronoi MC maps, we perform a series of tests on mock galaxy catalogues. These mock catalogues were created to simulate three galaxy groups and one galaxy cluster at $z \sim 0.8$, similar to the SG0023 system studied in Lemaux et al. (2016) and ostensibly spanning the range of environments probed by ORELSE. First, a sample of field galaxies were generated, randomly dispersed within a $\sim 15 \times 15'$ region between $z = 0.7$ and $0.9$. Although this redshift range seems relatively narrow, it represents roughly $\pm 2\sigma_{\Delta z/(1+z_{\text{spec}})} \times (1+z)$ from the central redshift and thus includes essentially all galaxies that would meaningfully contribute to the density map at $z \sim 0.8$. Members of a galaxy group are inserted by randomly sampling from a three-dimensional (3D) Gaussian of $\sigma = 0.33$ Mpc (proper) in all spatial dimensions with an additional scatter placed on the line-of-sight direction designed to mimic a velocity dispersion of 500 km s$^{-1}$. The galaxy cluster is populated in the same way but instead sampled with $\sigma = 0.5$ Mpc and perturbed to mimic a velocity dispersion of 1000 km s$^{-1}$. Luminosities are assigned to simulated galaxies by sampling from the rest-frame B-band luminosity function of Giallongo et al. (2005) at the appropriate redshifts. For group and cluster galaxies, the value of $M^*$ was set to be 0.25 and 0.5 mag brighter, respectively, in accordance with recent findings on the luminosity function in these types of environments (e.g. De Propris, Phillipps & Bremer 2013).

After creating mock catalogues from this process, we generate density and overdensity maps following the procedures described earlier, applying identical magnitude cuts. No $k$-correction was

applied to the rest-frame B-band magnitudes when translating them to observed-frame $i'$ magnitudes as this correction is only $\sim 0.1$ mag at these redshifts and varies weakly with galaxy type. We test a set of scenarios with differing rates of spectroscopic coverage (fraction of all galaxies assumed to have spectroscopic redshifts) ranging from 3 to 78 per cent. Galaxies that do not have a spectroscopic redshift are assumed to have a photometric redshift with a precision of $0.03 \times (1+z)$ and a catastrophic outlier rate of 6 per cent; this is accomplished by perturbing their initial, true (redshift space) redshifts by a values consistent with these statistics. We find that in a case with $\approx 20$ per cent spectroscopic coverage, roughly 60 per cent and 80 per cent of galaxies lie within $\pm 0.25$ and $\pm 0.5$ dex of their true overdensity, respectively. This level of spectroscopic completeness is roughly equal to that of all ORELSE fields considered in this study over the spectroscopic footprint and subject to the magnitude ranges used to generate density and overdensity maps. Further, these recovery rates increase rapidly as the spectroscopic coverage increases, an increase which is observed in ORELSE with increasing $\log(1 + \delta_{\text{gal}})$ (see Lemaux et al. 2016; Shen et al. 2017). Therefore, we conclude that over the broad range of overdensities examined in this study (2.5 orders of magnitude, as discussed in the following section), we are able to quantify the environments of galaxies with sufficient accuracy. A more detailed description and analysis of these tests will be presented in a future publication (Lemaux et al. in preparation).

## 3.4 Measuring the SMF in different environments

To create samples of galaxies occupying different local environments, we define five bins of local overdensity between $-0.5 \le \log(1+\delta_{\text{gal}}) \le 2$ each of width 0.5 dex. For each of these overdensity bins, we construct the galaxy SMF by counting numbers of galaxies in stellar mass bins of width 0.25 dex. Fig. 4 shows an illustration of this binning scheme. Because of the heterogeneity of our catalogues and the subsequent variety of mass completeness limits, we employ a methodology that effectively mimics the $1/V_{\text{max}}$ technique (Avni & Bahcall 1980). In brief, the concept of this approach is to evaluate the galaxy number density separately for each stellar mass bin, only considering the volume for which the catalogues are mass-complete.

We perform this by stepping through each of the redshift slices used to construct the overdensity maps discussed in Section 3.3 and identify the spatial area and corresponding volume associated with each overdensity bin. If a given stellar mass bin is partially below the limiting stellar mass at the central redshift of the slice then the volume is ignored and galaxies are not counted. Otherwise all galaxies that fall within the redshift bounds of the slice ($z_{\text{spec}}$ when available) are counted towards the SMF of the corresponding mass-overdenisty bin. As a result of this approach, some stellar mass bins will count galaxies from smaller volumes than others based on whether they lie above/below the mass-completeness limit in a given redshift slice for different numbers of fields across the full redshift range of $0.55 < z < 1.3$. Each stellar mass bin is normalized by the total comoving volume probed for that bin. In Figs 5 and 6, we show our measurements of the SMF of all galaxies as well as the star-forming and quiescent components, respectively.

To test the stability of the SMFs measured in this way, we perform a jackknife analysis on the eight ORELSE fields used in this study. We do this by remeasuring all SMFs using every permutation of eight, seven, six, ..., one field(s), amounting to $2^8 - 1 = 255$ realizations in total. With these measurements, we calculate the $1\sigma$ scatter for each data point of each SMF in Figs 5 and 6, respectively.
<figure><img src="image_5.png" /><figcaption>**Figure 4.** The distribution of galaxies in the overdensity versus stellar mass plane. The grey-scale shows a logarithmic scaling of a 2D histogram of all galaxies at $0.55 < z < 1.3$ that are above our estimated stellar mass completeness limits (see Section 3.1). Vertical and horizontal lines trace out the binning scheme used to measure the galaxy SMFs in this analysis. The total number of galaxies in each bin is indicated in the top-left corner of each box. We refer the reader to Section 3.4 for a full description of how the SMF is constructed from this distribution.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 5.** In the left-hand panel are shown the galaxy SMFs (normalized by comoving volume) for all galaxies at $0.55 < z < 1.3$ in five bins of local environmental overdensity. Curves showing the best-fitting Schechter function (single or double) that yields the lower BIC as shown in Table 4. These measurements span a broad dynamic range of environments from the average densities hosting field galaxies to the central cores of massive galaxy clusters. For reference, we plot the field galaxy SMF at $0.5 < z < 1.25$ as measured by ZFOURGE (Tomczak et al. 2014) that shows remarkable consistency with a combination of our two lowest density SMFs as expected. It can be clearly seen that as environmental density increases there is a smooth, continuous increase in the relative numbers of higher to lower mass galaxies. This behaviour is elucidated in the right-hand panel that plots the ratio of each SMF relative to that of the lowest density bin. This type of trend is consistent with the picture that dense environments preferentially destroy lower mass galaxies and/or promote the growth of higher mass galaxies, though it is likely that these mechanisms are interdependent.</figcaption></figure>

We find that the median $1\sigma$ scatter per data point for the total mass functions from the lowest to the highest overdensity bins is 0.05, 0.04, 0.07, 0.12 and 0.12 dex, respectively. For the star-forming SMFs, this scatter is 0.06, 0.04, 0.08, 0.14 and 0.18 dex, respectively, and for the Quiescent SMFs, it is 0.09, 0.07, 0.09, 0.12 and 0.13 dex, respectively. Next, we refit Schechter functions to each jackknife realization of each SMF. In general, the $1\sigma$ scatter for each best-fit Schechter parameter across all jackknife realizations is on average only $1.37 \times$ larger than the formal uncertainties quoted in Table 4. Therefore, we conclude that the overall measurements of

the SMFs are largely stable, despite some minor variation amongst the eight ORELSE fields used in this study.

# 4 RESULTS

## 4.1 Dependence of the SMF on local environment

We find that the shape of the galaxy SMF at $0.55 < z < 1.3$ is a strong function of local environment, echoing similar works at these redshifts (Bolzonella et al. 2010; Vulcani et al. 2012; van
<figure><img src="image_7.png" /><figcaption>**Figure 6.** Galaxy SMFs split into star-forming (top panels) and quiescent (bottom panels) subpopulations based on rest-frame broad-band colours. Symbols and curves are the same as described in Fig. 5. Similar to the case for all galaxies, both the star-forming and quiescent subsamples exhibit an environmental dependence on the shape of their respective SMFs. Although there is still general consistency between ZFOURGE and our measurements in field environments, there are some differences that jump out. First, for quiescent galaxies the linear combination of the two lowest density SMFs overestimates the number densities of ~$10^{10}$ M$_{\odot}$ galaxies relative to ZFOURGE by about a factor of 2×. Second, for star-forming galaxies there is similarly a factor of ~2× excess in our measured number densities of ~$10^{11.2}$ M$_{\odot}$ which may be the result of Eddington bias (Eddington 1913).</figcaption></figure>

der Burg et al. 2013; Mortlock et al. 2015; Davidzon et al. 2016). In particular, there is a continual and gradual increase in the relative number of high-mass to low-mass galaxies as environmental density increases. This is most clearly illustrated in the right-hand panel of Fig. 5 that plots the ratio of each SMF relative to that of the lowest density bin. This ratio tends to follow an uninterrupted log-linear relationship with stellar mass becoming gradually steeper towards higher overdensities. For every order of magnitude increase in stellar mass, there is roughly a factor of 1.2, 3.0, 5.3 and 6.2 increase in the number of galaxies relative to the SMF of the lowest density bin in each of the higher overdensity bins, respectively.

Two possible explanations of this behaviour are that regions of higher local overdensity (a) destroy lower mass galaxies at a quicker rate than they are added and/or (b) promote the growth of higher mass galaxies. The former point is consistent with past studies that have indeed found that lower mass galaxies are more likely to be ‘destroyed’ via merging. Leja et al. (2015) show that merging leads

to a destruction rate of galaxies that is inversely proportional to stellar mass and a growth rate that is proportional with stellar mass within the semi-analytic model (SAM) of Guo et al. (2013). Using a combined morphological and galaxy-pair counting analysis of the COSMOS legacy survey, Lotz et al. (2011) infer that the volume-averaged minor merger rate is 3× that of the major merger rate. Furthermore, merger rates in galaxy group/cluster environments have been estimated to be 3–4× greater relative to lower density environments (Lin et al. 2010; Kampczyk et al. 2013)<fnref n="6" /> that bolsters this picture.

Lastly, our measurements probe to low enough stellar masses to reveal, in some cases, a clear departure from standard single-Schechter behaviour (Schechter 1976). However, it is not immediately clear that a more complex model is necessary in all cases, particularly for the higher density SMFs where uncertainties can be

<footnote n="6">⁶This result is likely driven by group-like environments that dominate the high-density sample investigated in these studies.</footnote>
<table><caption>**Table 4.** Schechter parameters.</caption><thead><tr><th>Overdensity bin</th><th>log(M*)<br>(M<sub>☉</sub>)</th><th>α<sub>1</sub></th><th>Φ<sup>*</sup><sub>1</sub><br>(10<sup>-3</sup> Mpc<sup>-3</sup>)</th><th>α<sub>2</sub></th><th>Φ<sup>*</sup><sub>2</sub><br>(10<sup>-3</sup> Mpc<sup>-3</sup>)</th><th>BIC</th></tr></thead><tbody><tr><td colspan="7" style="text-align: center; font-weight: bold;">Total</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>11.17 ± 0.07</td><td>-1.46 ± 0.03</td><td>0.11 ± 0.02</td><td>...</td><td>...</td><td>44.3</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>10.77 ± 0.11</td><td>-0.14 ± 0.47</td><td>0.21 ± 0.04</td><td>-1.52 ± 0.06</td><td>0.14 ± 0.05</td><td>27.4</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>11.20 ± 0.08</td><td>-1.38 ± 0.04</td><td>0.21 ± 0.04</td><td>...</td><td>...</td><td>161.8</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>10.87 ± 0.05</td><td>-0.59 ± 0.20</td><td>0.45 ± 0.04</td><td>-1.60 ± 0.07</td><td>0.12 ± 0.04</td><td>25.6</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>11.07 ± 0.08</td><td>-0.85 ± 0.07</td><td>2.99 ± 0.51</td><td>...</td><td>...</td><td>36.6</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>10.87 ± 0.08</td><td>-0.36 ± 0.27</td><td>4.34 ± 0.40</td><td>-1.52 ± 0.32</td><td>0.25 ± 0.37</td><td>19.0</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>11.04 ± 0.10</td><td>-0.65 ± 0.10</td><td>11.16 ± 2.03</td><td>...</td><td>...</td><td>21.1</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>10.93 ± 0.16</td><td>-0.43 ± 0.45</td><td>13.47 ± 2.70</td><td>-1.66 ± 1.58</td><td>0.17 ± 1.23</td><td>22.8</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>11.03 ± 0.09</td><td>-0.58 ± 0.10</td><td>30.35 ± 4.98</td><td>...</td><td>...</td><td>11.0</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>10.98 ± 0.15</td><td>-0.49 ± 0.31</td><td>33.01 ± 7.78</td><td>-2.00 ± 3.54</td><td>0.05 ± 0.76</td><td>15.8</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">Star forming</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>11.03 ± 0.05</td><td>-1.54 ± 0.02</td><td>0.08 ± 0.01</td><td>...</td><td>...</td><td>19.0</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>10.68 ± 0.12</td><td>0.49 ± 0.52</td><td>0.06 ± 0.02</td><td>-1.50 ± 0.04</td><td>0.15 ± 0.04</td><td>22.5</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>11.06 ± 0.04</td><td>-1.48 ± 0.02</td><td>0.13 ± 0.02</td><td>...</td><td>...</td><td>34.7</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>10.76 ± 0.08</td><td>-0.21 ± 0.37</td><td>0.17 ± 0.03</td><td>-1.52 ± 0.04</td><td>0.17 ± 0.04</td><td>22.9</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>11.09 ± 0.08</td><td>-1.09 ± 0.05</td><td>1.04 ± 0.18</td><td>...</td><td>...</td><td>16.2</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>10.98 ± 0.11</td><td>-0.93 ± 0.21</td><td>1.40 ± 0.35</td><td>-2.13 ± 1.11</td><td>0.01 ± 0.04</td><td>17.6</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>10.91 ± 0.12</td><td>-0.92 ± 0.12</td><td>3.36 ± 0.94</td><td>...</td><td>...</td><td>14.8</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>10.70 ± 0.27</td><td>-0.09 ± 1.44</td><td>3.98 ± 2.48</td><td>-1.26 ± 0.63</td><td>1.23 ± 3.01</td><td>17.7</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>10.83 ± 0.15</td><td>-0.82 ± 0.16</td><td>8.57 ± 2.70</td><td>...</td><td>...</td><td>11.3</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>10.81 ± 0.18</td><td>-0.81 ± 0.20</td><td>8.80 ± 3.36</td><td>-15.45 ± 11.64</td><td>2e-26 ± 6e-22</td><td>16.1</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">Quiescent</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>10.84 ± 0.03</td><td>-0.71 ± 0.05</td><td>0.18 ± 0.01</td><td>...</td><td>...</td><td>10.3</td></tr><tr><td>-0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 0.0</td><td>10.84 ± 0.04</td><td>-0.68 ± 0.08</td><td>0.18 ± 0.02</td><td>-4.20 ± 5.22</td><td>4e-7 ± 8e-6</td><td>14.4</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>10.93 ± 0.04</td><td>-0.78 ± 0.06</td><td>0.29 ± 0.03</td><td>...</td><td>...</td><td>22.2</td></tr><tr><td>0.0 &lt; log(1+δ<sub>gal</sub>) &lt; 0.5</td><td>10.91 ± 0.05</td><td>-0.71 ± 0.13</td><td>0.31 ± 0.04</td><td>-2.78 ± 2.14</td><td>0.0001 ± 0.001</td><td>21.7</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>10.89 ± 0.05</td><td>-0.39 ± 0.10</td><td>2.82 ± 0.23</td><td>...</td><td>...</td><td>8.4</td></tr><tr><td>0.5 &lt; log(1+δ<sub>gal</sub>) &lt; 1.0</td><td>10.88 ± 0.08</td><td>-0.38 ± 0.23</td><td>2.83 ± 0.35</td><td>-3.61 ± 54.15</td><td>0.0006 ± 0.01</td><td>12.6</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>11.07 ± 0.13</td><td>-0.63 ± 0.20</td><td>7.97 ± 2.01</td><td>...</td><td>...</td><td>11.3</td></tr><tr><td>1.0 &lt; log(1+δ<sub>gal</sub>) &lt; 1.5</td><td>10.69 ± 0.18</td><td>-0.11 ± 0.35</td><td>11.59 ± 1.38</td><td>4.65 ± 1.23</td><td>0.02 ± 0.04</td><td>12.4</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>11.04 ± 0.09</td><td>-0.52 ± 0.15</td><td>24.18 ± 3.89</td><td>...</td><td>...</td><td>7.3</td></tr><tr><td>1.5 &lt; log(1+δ<sub>gal</sub>) &lt; 2.0</td><td>10.56 ± 0.09</td><td>0.25 ± 0.26</td><td>31.44 ± 1.53</td><td>4.87 ± 1.09</td><td>0.05 ± 0.09</td><td>11.0</td></tr></tbody></table>

relatively large. To test this, we fit all SMFs with both single- and double-Schechter functions, respectively, defined as

$$ \Phi_{\text{single}}(M) = \ln(10) \Phi^* \left(\frac{M}{M^*}\right)^{\alpha+1} \exp\left(-\frac{M}{M^*}\right) \quad (1) $$

$$ \Phi_{\text{double}}(M) = \ln(10) \exp\left(-\frac{M}{M^*}\right) \times \left[ \Phi_1^* \left(\frac{M}{M^*}\right)^{\alpha_1+1} + \Phi_2^* \left(\frac{M}{M^*}\right)^{\alpha_2+1} \right], \quad (2) $$

where $M^*$ is the characteristic turnover mass, $(\alpha, \alpha_1, \alpha_2)$ are the slopes and $(\Phi^*, \Phi_1^*, \Phi_2^*)$ are the normalizations. Next, for each fit we calculate the Bayesian Information Criterion (BIC: Schwarz 1978) defined as

$$ \text{BIC} = -2 \ln(L) + k \ln(n), \quad (3) $$

where $L$ is the maximum likelihood of the best-fitting model, $k$ is the number of free parameters in the model and $n$ is the number of measurements. This statistic is designed to compare models with different numbers of free parameters and appropriately assess whether the improvement to the goodness of fit justifies an increase in the dimensionality of the model. The model yielding the lower BIC is generally the favourable choice. Best-fitting Schechter parameterizations and their corresponding BIC values are shown in Table 4.

## 4.2 Star-forming and quiescent SMFs

We further split the total galaxy SMF of each overdensity bin into its star-forming and quiescent components in Fig. 6. Several recent studies have found that, similar to the total galaxy population, the SMFs of star-forming and quiescent galaxies also exhibit an upturn at $\lesssim 10^{9.5} M_{\odot}$ at the redshifts examined here (e.g. Drory et al. 2009; Muzzin et al. 2013; Tomczak et al. 2014; Mortlock et al. 2015). As described in the previous section, we test for the presence of this feature in our measurements by comparing the Bayesian Information Criteria between the best-fitting single- versus double-Schechter function. From these comparisons we find that only the low-density $0 < \log(1+\delta_{gal}) < 0.5$ bin appears to favour the double-Schechter model for both star-forming and quiescent SMFs. Parameters of the best-fitting single- and double-Schechter functions and corresponding BIC are shown in Table 4. An important subtlety, however, is that to robustly discriminate the necessity of a double-Schechter parametrization requires having measurements below the threshold at which the upturn begins to dominate ($\lesssim 10^{9.5} M_{\odot}$) which, for quiescent galaxies in our data set, lies near and/or below the stellar mass completeness limit. Therefore, a direct comparison of the BIC for quiescent SMFs does not carry the same weight as for the total or star-forming SMFs, and more-so does not rule out the potential existence of such an upturn in all cases.

What else is immediately noticeable is that, similar to the full galaxy sample, the shape of both the star-forming and quiescent SMFs are strongly dependent on local environment, in agreement
<figure><img src="image_8.png" /><figcaption>**Figure 7.** Likelihood contours for the single-Schechter parameters of the star-forming (left) and quiescent (right) SMFs. Despite the fact that some SMFs favour the double-Schechter function (e.g. Drory et al. 2009; Muzzin et al. 2013; Tomczak et al. 2014; Mortlock et al. 2015), we exclusively use the single-Schechter function here in order to perform a self-consistent comparison. Colours correspond to the same overdensity bins as in previous figures. Solid and dot-dashed lines indicate the $1\sigma$ and $2\sigma$ confidence levels respectively. For both the star-forming and quiescent subsamples we see a clear environmental dependence in the Schechter parameters. This feature persists even if the fitting is restricted to stellar masses >$10^{10}$ M$_\odot$ as shown in the bottom panels. In particular the difference is roughly bimodal separating into loci for low- and high-density SMFs. This suggests that the transformation in the SMF begins to occur in group-scale environments where the local galaxy density is $\gtrsim 3$× that of the field.</figcaption></figure>

with some recent studies (Vulcani et al. 2012; Davidzon et al. 2016). In Fig. 7, we examine the shapes of the star-forming and quiescent SMFs by plotting likelihood contours of the single-Schechter parameters $M^*$ and $\alpha$.<fnref n="7" /> These contours are derived from a series of Monte Carlo simulations where for each iteration we resample the number counts of each SMF from a Poisson distribution and re-fit a single-Schechter function. What we see is that the Schechter parameters appear to separate into two general loci for low- and high-density SMFs, where ‘low’ and ‘high’ in this context are defined as being below and above $log(1+\delta_{gal}) = 0.5$, respectively. This observation goes to suggest that the environmental footprint in the SMF is established in the intermediate densities of galaxy groups.

Nevertheless, many past studies have found no significant environmental dependence in the SMF shape (Peng et al. 2010;

Bolzonella et al. 2010; Giodini et al. 2012; Vulcani et al. 2013; Calvi et al. 2013; van der Burg et al. 2013), and in fact, an environment-independent SMF for star-forming galaxies is a key assumption in the quenching model introduced by Peng et al. (2010). While it is difficult to robustly identify the cause of this discrepancy several important caveats are worth mentioning. First, at these redshifts ($0.7 < z < 1.2$) the aforementioned studies have been limited to stellar masses >$10^{10}$ M$_\odot$. However, if we restrict our comparison to >$10^{10}$ M$_\odot$ we see only a mild reduction in the significance of this result (see the bottom panels of Fig. 7). It is important to note that for the star-forming SMF this observation is almost exclusively driven by a difference in the low-mass slope $\alpha$ (whereas for the quiescent SMF it is a combination of $M^*$ and $\alpha$).

The next set of caveats regard definitions, interpretations, and range of galaxy environment. In general, environment can act on ‘local’ and ‘global’ scales and effects on observed galaxy properties can be sensitive to the scale probed by the chosen environmental metric (Muldrew et al. 2012). The importance of the distinction

<footnote n="7">⁷ Note that in order to perform a self-consistent comparison we exclusively use the single-Schechter function.</footnote>
between these environmental definitions in relation to the galaxy SMF are discussed in Vulcani et al. (2012) and Vulcani et al. (2013). As discussed in Section 3.3, the essence of the environmental metric used in this work is based on a Voronoi tessellation which has been shown to be one of the most effective methods for recovering the local density field (Darvish et al. 2015). However, this metric has a complex interrelation with global environment (e.g. Weinmann et al. 2006; De Lucia et al. 2012; Hearin et al. 2016) which we do not attempt to reconcile here. Therefore, caution should be exercised when comparing the results of this work to studies which employ global density metrics. In addition, the range of environments probed is not always consistent as some studies utilize cosmological galaxy surveys whereas others utilize dedicated surveys of LSSs. The scarcity of massive LSSs in cosmological surveys, such as zCOSMOS and VIPERS (Bolzonella et al. 2010; Davidzon et al. 2016), limits their ability to place constraints for the highest density environments. The typical density contrast between low- and high-density environments in these studies is ~10×, roughly an order of magnitude less than the range probed in ORELSE.

Another distinction is that many of these past studies use only a single rest-frame colour to define star-forming subsamples. This is important because star-forming galaxies enshrouded by dust can be easily misclassified as quiescent, and in particular, the fraction of star-forming galaxies that have a significant level of dust attenuation correlates with stellar mass (Wild et al. 2014; Martis et al. 2016). Thus using a single rest-frame colour introduces a mass-dependent bias in the classification of star-forming and quiescent galaxies.

## 4.3 A semi-empirical model of the galaxy SMF

In recent years there have been many studies devoted to modelling the evolution of the galaxy SMF. A large subset of these take the approach of using the observed SMF at high redshift as an initial condition and allowing it to evolve forward in time after incorporating a variety of known aspects of galaxy evolution such as growth due to star formation and quenching (e.g. Leja et al. 2015; Tomczak et al. 2016; Contini et al. 2017; Steinhardt, Yurk & Cappak 2017). Despite various differences in the exact approach taken, one common result from these studies is that there exists some tension between the observed evolution of the galaxy SMF and the well-known correlation between SFR and stellar mass (e.g. Noeske et al. 2007). One emerging picture from many of these past studies is that galaxy–galaxy merging appears to be necessary at some level in order to help relieve this disagreement. Furthermore, these past studies have been limited to modelling the SMF of field galaxies. Here, we present a simple semi-empirical model in a similar context as these past works but with the aim of explaining the SMF of dense environments.

Galaxy merging is well-recognized as a crucial mechanism in the mass assembly of galaxies (e.g. Le Fèvre et al. 2000; Guo & White 2008; Lin et al. 2010; Rodriguez-Gomez et al. 2016) as well as a mechanism for building the diffuse stellar component of the intra-cluster medium known as the intra-cluster light, or ICL (e.g. Willman et al. 2004; Rudick, Mihos & McBride 2006; Murante et al. 2007; Contini et al. 2014). Furthermore, the galaxy–galaxy merger rate has been shown to be elevated in overdense relative to underdense regions by factors of 3–4× (Lin et al. 2010; Kampczyk et al. 2013). This increase is generally believed to be driven by galaxy group-like environments where moderate velocity dispersions (a few ×100 km s⁻¹) would not suppress the likelihood of galaxy mergers as could be the case for galaxy clusters with velocity dispersions ≳1000 km s⁻¹. In a case study of a galaxy

proto-cluster at z = 1.62, Lotz et al. (2013) estimate a merger rate that is 3–10× greater than a sample of field galaxies matched in redshift and stellar mass. These results provide strong evidence that galaxy–galaxy mergers must play a crucial in shaping the galaxy SMF of LSSs.

Given that the evolution of the galaxy SMF at z < 5 is predominantly in its normalization, whereas the shape remains roughly constant (e.g. Ilbert et al. 2013; Tomczak et al. 2014; Grazian et al. 2015; Mortlock et al. 2015), it is reasonable to utilize the shape of the field SMF at z ~ 0.8 as measured in our data as a starting point for understanding the SMF in dense environments. We begin by generating a sample of ≈10⁶ galaxies with stellar masses down to 10⁶ M☉ distributed according to the best-fitting double-Schechter function of the SMF in our lowest density environments (−0.5 < log(1+δgal) < 0.5). Assuming this sample is representative of the galaxy population at zstart = 5, the simulation progresses forward incrementally in 100 Myr intervals to zfinal = 0.8. At each time-step (1) galaxies grow via star formation, provided they are not quenched, (2) a number of galaxy pairs are selected to be merged and (3) a number of galaxies are selected to be quenched. The only property of this model that we allow to vary is the total number of mergers that occur between the zstart and zfinal, all other prescriptions/assumptions are fixed as described below.

To track the stellar mass evolution of a simulated galaxy, we first define individual ‘stellar mass particles’. Each stellar mass particle is treated as a single stellar population (SSP) and is subject to mass-loss over time due to stellar evolution. For this, we adopt the mass-loss formula presented in equation (16) of Moster, Naab & White (2013) that is based on the mass evolution of a model SSP from BC03 with an assumed Chabrier (2003) IMF.

A simulated galaxy is defined as a collection of stellar mass particles that effectively represent its constituent stellar populations, either formed *in situ* or *ex situ*. At the onset of the simulation, galaxies are assigned a single ‘seed’ stellar mass particle, representing the galaxy’s total stellar mass at zstart, with an age randomly selected between 0 and 1 Gyr. This constraint ensures that no stellar mass particle is older than the universe at zstart.

### 4.3.1 Prescription for star formation

SFRs of galaxies are assigned and evolve based on the SFR–*M*~*s*~ relations presented in equations (2) and (4) of Tomczak et al. (2016). The left-hand panel of Fig. 8 shows the range of the SFR–*M*~*s*~ relation between the redshift limits of the model. In general, for galaxies below 10¹⁰.⁵ M☉, this prescription is very close to a power-law relation of SFR ∝ *M*~*s*~¹⁰.⁹.

First, galaxies are assigned an SFR based on their stellar mass at zstart. With each time-step of the simulation, galaxies that are not quenched generate a new associated stellar mass particle with a formation mass equal to the galaxy’s instantaneous SFR multiplied by the time interval of the simulation (100 Myr). At the end of each time-step, a galaxy’s SFR is set to the value of the SFR–*M*~*s*~ relation at the corresponding redshift and stellar mass.

### 4.3.2 Prescription for quenching

Quenching in our model simply entails setting a galaxy’s SFR to zero at a given time-step. Once a galaxy is quenched, it will remain quenched for the remainder of the simulation (i.e. rejuvenated star-formation is not incorporated). To quench galaxies, we implement
<figure><img src="image_9.png" /><figcaption>**Figure 8.** Left: Star-formation versus stellar mass relations as a function of redshift used to construct star-formation histories for galaxies in the model described in Section 4.3. These relations are taken from the analytical parameterizations presented in equations (2) and (4) of Tomczak et al. (2016). At each 100 Myr time-step between $z_{\text{start}} = 5$ and $z_{\text{final}} = 0.8$, non-quenched galaxies grow based on their instantaneous SFR and are reassigned a new SFR based on where they fall on the SFR-$M_*$ relation of the subsequent time-step. Right: An illustration of the prescribed evolution for the quiescent fraction versus stellar mass of simulated galaxies as a function of redshift. For this prescription, we adopt two boundary conditions: (1) all galaxies begin as star forming at the onset of the simulation $z_{\text{start}} = 5$ and (2) the quiescent fraction at fixed stellar mass at the end of the simulation ($z_{\text{final}} = 0.8$) must match the observed quiescent fraction of the densest environment in ORELSE. At each time-step of the simulation, a certain fraction of galaxies at every stellar mass are quenched at random in order to gradually transition the quiescent fraction between these boundary conditions, as shown.</figcaption></figure>

the following procedure. We enforce that the quiescent fraction versus stellar mass (i.e. the relative number of quenched galaxies to total at fixed $M_*$) at the end of the simulation match that of the measured quiescent fraction as dictated by the best-fitting Schechter functions to the SMFs in the highest density bin ($1.5 < \log(1+\delta_{\text{gal}}) < 2$) shown in Fig. 6 and Table 4. In essence, this treats the quiescent fraction versus stellar mass measured from the ORELSE data as a boundary condition for the endpoint of the simulation ($z_{\text{final}} = 0.8$). In practice, at each time-step of the simulation, we examine the quiescent fraction in bins of stellar mass, which at the start of the simulation is 0 per cent as all galaxies begin as star forming. We then randomly select galaxies to be quenched in order to match the quenching rate defined by these boundary conditions. In this way, the quiescent fraction of each stellar mass bin gradually increases linearly with time from 0 per cent at $z_{\text{start}}$ to its final value at $z_{\text{final}}$. An illustration of this is shown in the right-hand panel of Fig. 8.

We note that this procedure involves extrapolating the Schechter fits to the ORESLE mass functions far below their stellar mass completeness limits which may not accurately represent galaxies in the low-mass regime. At first glance, it may seem that this approach considers only stellar mass as relevant towards determining when a galaxy will quench (a.k.a. ‘mass-quenching’). However, by having the quiescent fraction measured in the highest density bin be the endpoint, ‘environmental quenching’ is effectively encoded into the quenching scheme, albeit in an indirect way. Nevertheless, this will only be the case (for certain) above the limiting stellar mass of our data ($\gtrsim 10^{9.7} M_{\odot}$), whereas the true evolution of quiescent fractions at lower stellar masses remains unknown. Recent studies in the Local Group have suggested that the environmental quenching efficiency dramatically increases by $\gtrsim 5 \times$ for satellites at $\lesssim 10^8 M_{\odot}$, likely tied to ram-pressure stripping (Fillingham et al. 2015, 2016).

However, it is unclear if this trend extends to higher redshifts where hot gaseous haloes become more rare.

### 4.3.3 Prescription for merging

The final major prescription in this model is the implementation of galaxy–galaxy merging. As mentioned earlier, the only feature of this model that is varied systematically is the total number of galaxies that merge between $z_{\text{start}}$ and $z_{\text{final}}$. This is denoted as the fraction of galaxies merged ($f_{\text{merged}}$) over the full duration of the simulation relative to the total number from the start. We allow $f_{\text{merged}}$ to vary between 0 per cent (i.e. no merging) and 95 per cent (i.e. only 5 per cent of galaxies remaining at $z_{\text{final}}$). In Fig. 9, we show the resulting SMFs at $z_{\text{final}}$ of the simulation for a range of values of $f_{\text{merged}}$.

We impose a redshift dependence wherein the merger rate (number of mergers per unit time) evolves as $\sim (1+z)^{2.7}$, in accordance with recent work by Rodriguez-Gomez et al. (2015) based on the Illustris hydrodynamic simulation (Genel et al. 2014). It is important to note, however, that at present there is no broadly accepted consensus regarding redshift evolution of the galaxy–galaxy merger rate where some studies find that it increases with redshift (e.g. Hopkins et al. 2010a,b; Man et al. 2012), while others find it to be roughly constant with redshift (e.g. Guo & White 2008; Williams, Quadri & Franx 2011).

Additionally, we enforce that minor mergers occur at $3 \times$ the frequency of major mergers (Lotz et al. 2011). Major and minor mergers are defined as having stellar mass ratios ($\mu_* \equiv M_{\text{satellite}} / M_{\text{primary}}$) between the limits of $[1:4-1:1]$ and $[\le 1:4]$, respectively. For each merger event, the less massive galaxy is ‘destroyed’, having its stellar mass particles transferred to the more massive galaxy. During
<figure><img src="image_10.png" /><figcaption>**Figure 9.** Simulated SMFs from the semi-empirical model described in Section 4.3. Briefly, the simulation begins at $z_{\text{start}} = 5$ with $\approx 10^6$ galaxies having stellar masses down to $10^6 \text{ M}_\odot$ distributed in accordance with the measured SMF of the field. The simulation then proceeds in 100 Myr time increments down to $z_{\text{final}} = 0.8$ allowing galaxies to grow via star-formation, quench, and merge. The grey dot-dashed line shows the assumed initial state at $z_{\text{start}}$ and coloured solid lines show the resultant SMF at $z_{\text{final}}$ for different values of $f_{\text{merged}}$ (fraction of galaxies merged between $z_{\text{start}}$ and $z_{\text{final}}$). The colour-scale corresponds to the fraction of stellar mass in the ICL at $z_{\text{final}}$ relative to total ($M_{\star, \text{ICL}}/M_{\star, \text{total}}$) estimated from the model based on assuming 30 per cent mass loss per merger.</figcaption></figure>

this transfer, we assume that 30 per cent of the stellar mass of the less massive galaxy is stripped and becomes part of the ICL. While this implementation is overly simplistic as the actual amount of stellar mass stripped through the relaxation process will depend on, e.g. merging time-scales and orbital velocities, this average value has been argued to be a reasonable first-order assumption from numerical simulations (Somerville et al. 2008; Contini et al. 2014, 2017, and references therein). We keep track of this fraction of the total stellar mass ($M_{\star, \text{ICL}}/M_{\star, \text{total}}$).

### 4.3.4 Comparing the model to observations

We compare the model to the measured SMFs from Fig. 5 and identify which value of $f_{\text{merged}}$ best reproduces the data. This is done by scaling the simulated to the measured SMFs and minimizing $\chi^2$ for each realization of $f_{\text{merged}}$. We then marginalize over all realizations of $f_{\text{merged}}$ based on their minimized $\chi^2$. Despite its simplicity, the model is able to reproduce the shape of the measured SMFs reasonably well as shown in Fig. 10.

We find that the measured SMFs in the three highest overdensity bins strongly favour scenarios where large majorities of galaxies merge in the simulation. This implies that galaxy–galaxy mergers are vitally important in shaping the galaxy SMF of dense environments, consistent with other recent works (Davidzon et al. 2016; Steinhardt et al. 2017). Furthermore, we observe that the best-fitting value of $f_{\text{merged}}$ jumps up rapidly to 79 per cent for the intermediate density bin, $0.5 < \log(1+\delta_{\text{gal}}) < 1.0$, and increases more modestly

to the two higher density bins (Fig. 10). The implication from this is that intermediate environments (e.g. galaxy groups) are likely where most mergers would be occurring. This is qualitatively consistent with expectations as galaxy groups are believed to be the environment most conducive to galaxy–galaxy merging due to their moderate velocity dispersions, whereas velocities of typical cluster galaxies may act to suppress merging (Lin et al. 2010).

Our model yields values of $M_{\star, \text{ICL}}/M_{\star, \text{total}}$ of 2–5 per cent. These numbers are lower than recent estimates of the luminosity and stellar mass fractions of the ICL at low redshifts from the literature that tend to range between 10 and 40 per cent (e.g. Willman et al. 2004; Rudick et al. 2006; Gonzalez, Zaritsky & Zabludoff 2007; Murante et al. 2007; Sand et al. 2011; Contini et al. 2014; Mihos et al. 2017). Although these values are in tension with each other, a few caveats bear mentioning. First, these studies use a variety of approaches making it difficult to fully understand the underlying systematics. Some examples of these methods for obtaining ICL fractions include inferences from N-body and/or hydrodynamical simulations, direct measurements of emission from the ICL, and inferences from the rate of hostless Type Ia supernovae in the ICM. Furthermore, these studies pertain to low redshifts ($z < 0.15$; roughly 5 Gyr later in cosmic time than this work), which could in part explain why our values are lower if the ICL is still building at $z \lesssim 0.8$. This is in fact consistent with simulations that show that ICL fractions can roughly double at $z \le 1$ (Willman et al. 2004; Rudick et al. 2006). Finally, it is important to point out that our model does not include tidal stripping of stellar mass not associated with mergers as a separate channel for contributing to the ICL. This would indicate that the ICL fractions from our model are underpredictions given that Contini et al. (2014) argue that most of a galaxy cluster’s ICL component is formed through the stripping/disruption of satellites as opposed to mass-loss from mergers.

However, we would like to remind the reader that this model presents a simplified view of galaxy evolution and relies on various assumptions and extrapolations of empirical relations well below regimes where current observational data are able to probe. For example, some features of the model where systematic uncertainties may lurk include

(i) using empirical SFR-$M_*$ relations to construct SFHs,

(ii) choice of quenching mechanism(s),

(iii) adopting a redshift-dependent merger rate,

(iv) lack of infall of new galaxies/groups and

(v) lack of tidal stripping not associated with mergers.

It may also be the case that the assumption of 30 per cent satellite stellar mass-loss per merger is wrong, or that this quantity is dependent on the merger stellar mass ratio $\mu_*$. Furthermore, galaxies with higher stellar masses will generally also reside in more massive dark matter haloes with greater gravitational potentials. This will inevitably influence merger rates in a way that gives preference to more massive central galaxies as shown in fig. 6 of Rodriguez-Gomez et al. (2015). In a follow-up paper, we plan to develop this model further and examine the results in more detail, as well as add the remaining seven survey fields of ORELSE, virtually doubling our sample size.

## 5 SUMMARY AND CONCLUSIONS

In this work, we explore and compare the galaxy SMF across a wide range of local environments as probed by the ORELSE survey (Lubin et al. 2009). Leveraging extensive photometric and spectroscopic coverage from eight ORELSE fields hosting massive LSSs at
<figure><img src="image_11.png" /><figcaption>**Figure 10.** Thick solid lines show best-fits of our semi-empirical model to each of the measured SMFs in our three highest overdensity bins, with residuals plotted in the bottom panels. The grey dot-dashed curves show the version of the model in which no galaxy-galaxy mergers are included ($f_{\text{merged}} = 0$ per cent). Despite the simplicity of the model it is able to reproduce the shape of each measured SMF reasonably well. In the bottom-left corners of the top panels we indicate the reduced $\chi^2$, the value of $f_{\text{merged}}$, and the inferred stellar mass fractions of the ICL from the best-fitting model. In general, our model shows that (1) galaxy mergers are necessarily important in shaping the SMF and (2) most of these mergers are likely to have occurred in environments of intermediate density such as galaxy groups.</figcaption></figure>

0.6 < z < 1.3, we measure the SMF down to $10^9 M_{\odot}$. Utilizing the large number of spectroscopically confirmed redshifts (>4000) we employ a Monte Carlo Voronoi tessellation algorithm (described in Section 3.3) to estimate environmental density. By discretizing in narrow slices of redshift, we construct 3D environmental overdensity maps between 0.55 < z < 1.3 for each of our eight fields. These overdensity maps detail a wide range of environments from those occupied by field galaxies, $\log(1+\delta_{\text{gal}}) \sim 0$, to the central cores of massive galaxy clusters, $\log(1+\delta_{\text{gal}}) > 1.5$.

We define five bins of overdensity of width 0.5 dex between -0.5 ≤ $\log(1+\delta_{\text{gal}}) \le 2$ and construct the galaxy SMF from galaxies that fall into these corresponding environments. The resulting SMFs show a strong dependence on local environment. More specifically this dependence is manifested as a smooth and continuous enhancement in the numbers of higher to lower mass galaxies towards denser regions. This finding echoes results from several recent works which found, at varying levels of significance, similar behaviour in the SMF at these redshifts (Bolzonella et al. 2010; Vulcani et al. 2012; van der Burg et al. 2013; Mortlock et al. 2015; Davidzon et al. 2016). We next repeat this analysis for galaxies separated into star-forming and quiescent subpopulations based on rest-frame *U* − *V* and *V* − *J* colours. Interestingly, we find the same relative enhancement of high- to low-mass galaxies as with the total galaxy population for both star-forming and quiescent galaxies, albeit at lower significance for the latter subsample. While this effect has been observed in a couple of recent studies (Vulcani et al. 2012; Davidzon et al. 2016), it is at odds with several other studies that found no statistically significant environmental dependence in the star-forming and/or quiescent SMFs (Peng et al. 2010; Bolzonella

et al. 2010; Giodini et al. 2012; Vulcani et al. 2013; van der Burg et al. 2013).

To help draw a link between the SMFs of low- and high-density environments, we devise a simple semi-empirical model to test the importance of galaxy–galaxy merging. We generate a large sample of simulated galaxies ($\approx 10^6$) having stellar masses down to $10^6 M_{\odot}$ distributed according the best-fitting double-Schechter function of our two lowest-density bins (Table 4). Assuming this sample represents the universe at $z = 5$, we evolve it forward in time, stopping at $z = 0.8$ (the median redshift of ORELSE). Within this model, we allow galaxies to (1) grow via star formation in accordance with the observed SFR–$M_*$ relation, (2) quench at a rate that reproduces the observed quiescent fraction at $z = 0.8$ and (3) merge at a rate that is proportional to $(1+z)^{2.7}$ (see Section 4.3 for a full description). Merger pairs are selected randomly, although we enforce that minor mergers occur at $3 \times$ the rate of major mergers (Lotz et al. 2011). Furthermore, with each merger event, we assume that 30 per cent of the stellar material of the lower mass galaxy is stripped and becomes part of the intra-cluster light (Somerville et al. 2008; Contini et al. 2014).

The only property of the simulation that we systematically vary is the overall number of galaxies that merge between the beginning and the end (0.8 < z < 5), which we quantify as a fraction relative to the initial number of galaxies ($f_{\text{merged}}$). In general, we find that the observed SMFs in dense environments ($\log(1+\delta_{\text{gal}}) > 0.5$) are strongly favoured by versions of the model where large numbers of galaxies merge ($f_{\text{merged}} \ge 79$ per cent). Because a large value of $f_{\text{merged}}$ is found for the intermediate density bin, $0.5 < \log(1+\delta_{\text{gal}}) < 1.0$, we argue that most of the mergers necessary to transform the SMF
would have to occur in intermediate density environments such as galaxy groups. This is consistent with expectations as galaxy groups with moderate velocity dispersions are generally thought to be the most ideal environment for galaxy–galaxy mergers as opposed to galaxy clusters (Lin et al. 2010). Nevertheless, the overall implication from our model is that galaxy–galaxy merging is a very relevant process for shaping the galaxy SMF of dense environments, consistent with other recent studies (Davidzon et al. 2016; Steinhardt et al. 2017).

Another deduction that can be made from our model is the fraction of stellar mass locked in the ICL of galaxy clusters ($M_{*,ICL}/M_{*,total}$). Because the only way to increase $M_{*,ICL}$ in the model is through mass-loss from mergers, this fraction will, in general, monotonically increase with the total number of mergers ($f_{merged}$). The best-fitting models yield ICL fractions around 2–5 per cent. In general, this is less than low-$z$ estimates from past literature that find a wide range of values mostly between 10 and 40 per cent (e.g. Willman et al. 2004; Rudick et al. 2006; Gonzalez et al. 2007; Murante et al. 2007; Sand et al. 2011; Contini et al. 2014; Mihos et al. 2017). An important caveat is that our model does not include tidal stripping of stellar mass *not associated with mergers*, which Contini et al. (2014) argue contributes more to the ICL than mass-loss from mergers. This suggests that our derived ICL fractions are underestimated. However, because many of these studies pertain to low redshifts ($z < 0.15$; roughly 5 Gyr later in cosmic time than this work) the low values from the model presented in this paper could suggest that the ICL fraction continues to grow at $z \lesssim 0.8$. In fact, simulations show that the ICL fraction can increase by as much as $2 \times$ between $z = 1$ and $z = 0$ (Willman et al. 2004; Rudick et al. 2006) which, if true, would place our estimates at the lower end of measured values at $z \sim 0$ even without the additive effects of tidal stripping. Nevertheless, it is also important to note that estimates of ICL fractions come from a variety of different methodologies including direct measurements of the low surface brightness emission of the ICL, measurements of the rate of hostless Type Ia supernovae in the ICM, and various types of simulations (N-body, semi-analytic, hydrodynamical, etc.). Each of these approaches have their own inherent sources of systematic uncertainty that can cloud comparisons between them.

It is important to note that this semi-empirical model presents a simplified picture of galaxy evolution and is subject to various systematic uncertainties. Nevertheless, the main conclusions regarding the necessity of prevalent merging to reproduce the stellar mass distribution of cluster and group galaxies, and the preference of such merging to occur in group-scale environments, is almost certainly not sensitive to such uncertainties. In a follow-up analysis, we plan to incorporate the remaining ORESLE survey fields (for which data are still being processed presently), which will effectively double the number of galaxies in our sample. This will help to further improve constraints on the SMF shape as a function of environment, as well as open up the possibility of subsampling by redshift. Furthermore, we plan to develop/test the semi-empirical model further and analyse results from it in more detail.

## ACKNOWLEDGEMENTS

This material is based upon work supported by the National Aeronautics and Space Administration under NASA Grant Number NNX15AK92G. Part of the work presented herein is supported by the National Science Foundation under Grant No. 1411943. This research made use of ASTROPY, a community-developed core

PYTHON package for Astronomy (Astropy Collaboration, 2013). ART would also like to thank Chris Fassnacht for aiding in the reduction of photometric data through helpful discussions and provision of well-documented code. SM acknowledges financial support from the Institut Universitaire de France (IUF), of which she is senior member. A portion of this work made use of the Peloton computing cluster operated by the Division of Mathematical and Physical Sciences at the University of California, Davis. Work presented here is based in part on data collected at Subaru Telescope as well as archival data obtained from the SMOKA, which is operated by the Astronomy Data Center, National Astronomical Observatory of Japan. This work is based in part on observations made with the LFC mounted on the 200-inch Hale Telescope at Palomar Observatory, owned and operated by the California Institute of Technology. A subset of observations were obtained with WIRCam, a joint project of CFHT, Taiwan, Korea, Canada, France, at the Canada-France-Hawaii Telescope (CFHT) which is operated by the National Research Council (NRC) of Canada, the Institut National des Sciences de l'Université of the Centre National de la Recherche Scientifique of France, and the University of Hawaii. UKIRT is supported by NASA and operated under an agreement amongst the University of Hawaii, the University of Arizona, and Lockheed Martin Advanced Technology Center; operations are enabled through the cooperation of the East Asian Observatory. When the data reported here were acquired, UKIRT was operated by the Joint Astronomy Centre on behalf of the Science and Technology Facilities Council of the UK. This work is based in part on observations made with the Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory, California Institute of Technology under a contract with NASA. Spectroscopic observations used in the work presented here were obtained at the W. M. Keck Observatory, which is operated as a scientific partnership amongst the California Institute of Technology, the University of California and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W. M. Keck Foundation. We wish to recognize and acknowledge the very significant cultural role and reverence that the summit of Mauna Kea has always had within the indigenous Hawaiian community. We are most fortunate to have the opportunity to conduct observations from this mountain.

## REFERENCES

* Abazajian K. N. et al., 2009, ApJS, 182, 543

* Avni Y., Bahcall J. N., 1980, ApJ, 235, 694

* Bertin E., 2006, in Gabriel C., Arviset C., Ponz D., Enrique S., eds, ASP Conf. Ser. Vol. 351, Astronomical Data Analysis Software and Systems XV. Astron. Soc. Pac., San Francisco, p. 112

* Bertin E., Arnouts S., 1996, A&AS, 117, 393

* Bertin E., Mellier Y., Radovich M., Missonnier G., Didelon P., Morin B., 2002, in Bohlender D. A., Durand D., Handley T. H., eds, ASP Conf. Ser. Vol. 281, Astronomical Data Analysis Software and Systems XI. Astron. Soc. Pac., San Francisco, p. 228

* Birnboim Y., Dekel A., 2003, MNRAS, 345, 349

* Bolzonella M. et al., 2010, A&A, 524, A76

* Brammer G. B., van Dokkum P. G., Coppi P., 2008, ApJ, 686, 1503

* Bruzual G., Charlot S., 2003, MNRAS, 344, 1000 (BC03)

* Bundy K. et al., 2006, ApJ, 651, 120

* Calvi R., Poggianti B. M., Vulcani B., Fasano G., 2013, MNRAS, 432, 3141

* Calzetti D., Armus L., Bohlin R. C., Kinney A. L., Koornneef J., Storchibergmann T., 2000, ApJ, 533, 682

* Casali M. et al., 2007, A&A, 467, 777

* Chabrier G., 2003, Publ. Astron. Soc. Pac., 115, 763
Contini E., De Lucia G., Villalobos Á., Borgani S., 2014, MNRAS, 437, 3787

Contini E., Kang X., Romeo A. D., Xia Q., 2017, ApJ, 837, 27

Croton D. J. et al., 2006, MNRAS, 365, 11

Darvish B., Mobasher B., Sobral D., Scoville N., Aragon-Calvo M., 2015, ApJ, 805, 121

Davidzon I. et al., 2016, A&A, 586, A23

Davis M. et al., 2003, in Guhathakurta P., ed., Proc. SPIE Conf. Ser. Vol. 4834, Discoveries and Research Prospects from 6- to 10-Meter-Class Telescopes II. SPIE, Bellingham, p. 161

De Lucia G., Weinmann S., Poggianti B. M., Aragón-Salamanca A., Zaritsky D., 2012, MNRAS, 423, 1277

De Propris R., Phillipps S., Bremer M. N., 2013, MNRAS, 434, 3469

Dressler A., 1980, ApJ, 236, 351

Drory N. et al., 2009, ApJ, 707, 1595

Eddington A. S., 1913, MNRAS, 73, 359

Faber S. M. et al., 2003, in Iye M., Moorwood A. F. M., eds, Proc. SPIE Conf. Ser. Vol. 4841, Instrument Design and Performance for Optical/Infrared Ground-based Telescopes. SPIE, Bellingham, p. 1657

Farouki R., Shapiro S. L., 1981, ApJ, 243, 32

Fazio G. G. et al., 2004, ApJS, 154, 10

Fillingham S. P., Cooper M. C., Wheeler C., Garrison-Kimmel S., Boylan-Kolchin M., Bullock J. S., 2015, MNRAS, 454, 2039

Fillingham S. P., Cooper M. C., Pace A. B., Boylan-Kolchin M., Bullock J. S., Garrison-Kimmel S., Wheeler C., 2016, MNRAS, 463, 1916

Fioc M., Rocca-Volmerange B., 1997, A&A, 326, 950

Gal R. R., Lubin L. M., 2004, ApJ, 607, L1

Gal R. R., Lubin L. M., Squires G. K., 2005, AJ, 129, 1827

Gal R. R., Lemaux B. C., Lubin L. M., Kocevski D., Squires G. K., 2008, ApJ, 684, 933

Genel S. et al., 2014, MNRAS, 445, 175

Giallongo E., Salimbeni S., Menci N., Zamorani G., Fontana A., Dickinson M., Cristiani S., Pozzetti L., 2005, ApJ, 622, 116

Giodini S. et al., 2012, A&A, 538, A104

Gómez P. L. et al., 2003, ApJ, 584, 210

Gonzalez A. H., Zaritsky D., Zabludoff A. I., 2007, ApJ, 666, 147

Goto T., Yamauchi C., Fujita Y., Okamura S., Sekiguchi M., Smail I., Bernardi M., Gomez P. L., 2003, MNRAS, 346, 601

Grazian A. et al., 2015, A&A, 575, A96

Gunn J. E., Gott J. R., III, 1972, ApJ, 176, 1

Guo Q., White S. D. M., 2008, MNRAS, 384, 2

Guo Q., White S., Angulo R. E., Henriques B., Lemson G., Boylan-Kolchin M., Thomas P., Short C., 2013, MNRAS, 428, 1351

Hahn C. et al., 2015, ApJ, 806, 162

Hearin A. P., Zentner A. R., van den Bosch F. C., Campbell D., Tollerud E., 2016, MNRAS, 460, 2552

Hogg D. W. et al., 2004, ApJ, 601, L29

Hopkins P. F. et al., 2010a, ApJ, 715, 202

Hopkins P. F. et al., 2010b, ApJ, 724, 915

Ilbert O. et al., 2013, A&A, 556, A55

Kampczyk P. et al., 2013, ApJ, 762, 43

Kriek M., van Dokkum P. G., Labbé I., Franx M., Illingworth G. D., Marchesini D., Quadri R. F., 2009, ApJ, 700, 221

Landolt A. U., 1992, AJ, 104, 340

Le Fèvre O. et al., 2000, MNRAS, 311, 565

Leja J., van Dokkum P. G., Franx M., Whitaker K. E., 2015, ApJ, 798, 115

Lemaux B. C. et al., 2009, ApJ, 700, 20

Lemaux B. C., Tomczak A. R., Lubin L. M., Wu P.-F., Gal R. R., Rumbaugh N., Kocevski D. D., Squires a. G. K., 2016, MNRAS, preprint (arXiv:1608.00973)

Lin L. et al., 2010, ApJ, 718, 1158

López-Sanjuan C. et al., 2013, A&A, 553, A78

Lotz J. M., Jonsson P., Cox T. J., Croton D., Primack J. R., Somerville R. S., Stewart K., 2011, ApJ, 742, 103

Lotz J. M. et al., 2013, ApJ, 773, 154

Lubin L. M., Gal R. R., Lemaux B. C., Kocevski D. D., Squires G. K., 2009, AJ, 137, 4867

Makovoz D., Roby T., Khan I., Booth H., 2006, in Lewis H., Bridger A., eds, Proc. SPIE Conf. Ser. Vol. 6274, Advanced Software and Control for Astronomy. SPIE, Bellingham, p. 62740C

Man A. W. S., Toft S., Zirm A. W., Wuyts S., van der Wel A., 2012, ApJ, 744, 85

Maraston C., 2005, MNRAS, 362, 799

Martis N. S. et al., 2016, ApJ, 827, L25

Mei S. et al., 2012, ApJ, 754, 141

Merlin E. et al., 2015, A&A, 582, A15

Mihos J. C., Harding P., Feldmeier J. J., Rudick C., Janowiecki S., Morrison H., Slater C., Watkins A., 2017, ApJ, 834, 16

Miyazaki S. et al., 2002, PASJ, 54, 833

Mortlock A. et al., 2015, MNRAS, 447, 2

Moster B. P., Naab T., White S. D. M., 2013, MNRAS, 428, 3121

Muldrew S. I. et al., 2012, MNRAS, 419, 2670

Murante G., Giovalli M., Gerhard O., Arnaboldi M., Borgani S., Dolag K., 2007, MNRAS, 377, 2

Muzzin A. et al., 2012, ApJ, 746, 188

Muzzin A. et al., 2013, ApJS, 206, 8

Newman J. A. et al., 2013, ApJS, 208, 5

Noeske K. G. et al., 2007, ApJ, 660, L430

Oemler A., Jr, 1974, ApJ, 194, 1

Oke J. B., Postman M., Lubin L. M., 1998, AJ, 116, 549

Ouchi M. et al., 2004, ApJ, 611, 660

Peng Y.-j. et al., 2010, ApJ, 721, 193

Pickles A. J., 1998, PASP, 110, 863

Puget P. et al., 2004, in Moorwood A. F. M., Iye M., eds, Proc. SPIEConf. Ser. Vol. 5492, Ground-based Instrumentation for Astronomy. SPIE, Bellingham, p. 978

Quadri R. F., Williams R. J., Franx M., Hildebrandt H., 2012, ApJ, 744, 88

Rayner J. T., Cushing M. C., Vacca W. D., 2009, ApJS, 185, 289

Richstone D. O., 1976, ApJ, 204, 642

Rodriguez-Gomez V. et al., 2015, MNRAS, 449, 49

Rodriguez-Gomez V. et al., 2016, MNRAS, 458, 2371

Rudick C. S., Mihos J. C., McBride C., 2006, ApJ, 648, 936

Rumbaugh N. et al., 2017, MNRAS, 466, 496

Sand D. J. et al., 2011, ApJ, 729, 142

Schechter P., 1976, ApJ, 203, 297

Schlafly E. F., Finkbeiner D. P., 2011, ApJ, 737, 103

Schwarz G., 1978, Ann. Statist., 6, 461

Shen L. et al., 2017, MNRAS, 472, 998

Simcoe R. A., Metzger M. R., Small T. A., Araya G., 2000, in American Astronomical Society Meeting Abstracts #196. p. 758

Skelton R. E. et al., 2014, ApJS, 214, 24

Somerville R. S., Hopkins P. F., Cox T. J., Robertson B. E., Hernquist L., 2008, MNRAS, 391, 481

Steinhardt C. L., Yurk D., Capak P., 2017, MNRAS, 468, 849

Straatman C. M. S. et al., 2016, ApJ, 830, 51

Tanaka M. et al., 2008, A&A, 489, 571

Tomczak A. R. et al., 2014, ApJ, 783, 85

Tomczak A. R. et al., 2016, ApJ, 817, 118

van der Burg R. F. J. et al., 2013, A&A, 557, A15

van der Walt S. et al., 2014, PeerJ, 2, e453

Vulcani B. et al., 2012, MNRAS, 420, 1481

Vulcani B. et al., 2013, A&A, 550, A58

Weinmann S. M., van den Bosch F. C., Yang X., Mo H. J., 2006, MNRAS, 366, 2

Whitaker K. E. et al., 2011, ApJ, 735, 86

Wild V. et al., 2014, MNRAS, 440, 1880

Williams R. J., Quadri R. F., Franx M., van Dokkum P., Labbé I., 2009, ApJ, 691, 1879

Williams R. J., Quadri R. F., Franx M., 2011, ApJ, 738, L25

Willman B., Governato F., Wadsley J., Quinn T., 2004, MNRAS, 355, 159

Wuyts S. et al., 2007, ApJ, 655, 51

York D. G. et al., 2000, AJ, 120, 1579
## SUPPORTING INFORMATION

Supplementary data are available at MNRAS online.

**Figure 5.**
**Figure 6.**
*massFunctions_TOT_Tomczak+2017.dat*
*massFunctions_SF_Tomczak+2017.dat*

*massFunctions_QU_Tomczak+2017.dat*

Please note: Oxford University Press is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

This paper has been typeset from a LaTeX/LATEX file prepared by the author.