# Reconstructing the galaxy density field with photometric redshifts – II. Environment-dependent galaxy evolution since z 3

Nicola Malavasi, Lucia Pozzetti, Olga Cucciati, Sandro Bardelli, Olivier Ilbert, Andrea Cimatti

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Nicola Malavasi, Lucia Pozzetti, Olga Cucciati, Sandro Bardelli, Olivier Ilbert, et al.. Reconstructing the galaxy density field with photometric redshifts – II. Environment-dependent galaxy evolution since z 3. Monthly Notices of the Royal Astronomical Society, 2017, 470 (2), pp.1274-1290. 10.1093/mn-ras/stx1323. hal-01582413

HAL Id: hal-01582413

https://hal.science/hal-01582413v1

Submitted on 23 Aug 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Reconstructing the galaxy density field with photometric redshifts – II. Environment-dependent galaxy evolution since $z \simeq 3$

Nicola Malavasi,<fnref n="1" />,<fnref n="2" />* Lucia Pozzetti,<fnref n="3" /> Olga Cucciati,<fnref n="1" />,<fnref n="3" /> Sandro Bardelli,<fnref n="3" /> Olivier Ilbert<fnref n="4" /> and Andrea Cimatti<fnref n="1" />,<fnref n="5" />

Accepted 2017 May 25. Received 2017 May 23; in original form 2016 August 31

## ABSTRACT

Although extensively investigated, the role of the environment in galaxy formation is still not well understood. In this context, the galaxy stellar mass function (GSMF) is a powerful tool to understand how environment relates to galaxy mass assembly and the quenching of star formation. In this work, we make use of the high-precision photometric redshifts of the UltraVISTA Survey to study the GSMF in different environments up to $z \sim 3$, on physical scales from 0.3 to 2 Mpc, down to masses of $M \sim 10^{10} M_{\odot}$. We witness the appearance of environmental signatures for both quiescent and star-forming galaxies. We find that the shape of the GSMF of quiescent galaxies is different in high- and low-density environments up to $z \sim 2$ with the high-mass end ($M \gtrsim 10^{11} M_{\odot}$) being enhanced in high-density environments. On the contrary, for star-forming galaxies, a difference between the GSMF in high- and low-density environments is present for masses $M \lesssim 10^{11} M_{\odot}$. Star-forming galaxies in this mass range appear to be more frequent in low-density environments up to $z < 1.5$. Differences in the shape of the GSMF are not visible anymore at $z > 2$. Our results, in terms of general trends in the shape of the GSMF, are in agreement with a scenario in which galaxies are quenched when they enter hot gas-dominated massive haloes that are preferentially in high-density environments.

**Key words:** galaxies: distances and redshifts – galaxies: evolution – galaxies: formation – galaxies: high-redshift – galaxies: luminosity function, mass function – galaxies: statistics.

## 1 INTRODUCTION

The current understanding of the galaxy formation and evolution paradigm strongly relies on observational evidence of a correlation between galaxy properties and the environment in which galaxies reside. Although the ways for defining galaxy environment are numerous, starting from early works on the morphology–density relation (e.g. Dressler 1980), evidence has been gathered on the existence of a relation between local density and galaxy properties, such as colour, star formation, stellar mass and size (see e.g. Balogh et al. 1998, 2004; Lewis et al. 2002; Gómez et al. 2003; Kauffmann et al. 2004; Blanton et al. 2005; Cooper et al. 2006, 2012; Cucciati et al. 2006; Elbaz et al. 2007). There is general agreement over galaxies in high-density environments being more massive,

less star-forming and generally more evolved in comparison to low-density environments (see e.g. Blanton & Moustakas 2009, for a review on environmental properties of nearby galaxies).

The role played by environment, defined in terms of both the local density field and global large-scale structure (LSS) features (such as clusters, filaments and voids); is still poorly understood. High-density regions are characterized by specific processes (such as interactions of galaxies with the hot intracluster medium, interactions of galaxies with a cluster potential well or interactions of galaxies with other galaxies) that can easily interrupt the star formation. Moreover, as galaxies are biased tracers of the underlying dark matter distribution, different galaxy samples and/or different environment parametrizations may probe different kinds of local and global environment (i.e. DM haloes with different mass, see e.g. Haas, Schaye & Jeeson-Daniel 2012; Muldrew et al. 2012; Fossati et al. 2015). On the other hand, the galaxy stellar mass is related to the halo mass. It is also for this reason that it has not yet been

<footnote n="1">¹Dipartimento di Fisica e Astronomia (DIFA), Università di Bologna, v.le Berti Pichat 6/2, I-40127 Bologna, Italia</footnote>

<footnote n="2">²Department of Physics and Astronomy, Purdue University, 525 Northwestern Avenue, West Lafayette, IN 47907, USA</footnote>

<footnote n="3">³INAF – Osservatorio Astronomico di Bologna, via Ranzani 1, I-40127 Bologna, Italia</footnote>

<footnote n="4">⁴Aix Marseille Université, CNRS, LAM (Laboratoire d'Astrophysique de Marseille) UMR 7326, F-13388 Marseille, France</footnote>

<footnote n="5">⁵INAF – Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, I-50125 Firenze, Italia</footnote>

<footnote>* E-mail: nmalavas@purdue.edu</footnote>
determined whether star formation quenching can be separated in two distinct processes (one depending only on environment and one on galaxy stellar mass, as proposed e.g. by Peng et al. 2010) or whether mass and environment are just two aspects of the same underlying physical mechanisms (as proposed e.g. by Gabor & Davé 2015; see also Section 5).

One of the best ways to study galaxy evolution in different environments is to compare the galaxy stellar mass function (GSMF) as a function of redshift, galaxy type (star-forming or quiescent) and environmental density. GSMFs are a powerful tool, as they allow us to summarize in a single distribution function the galaxy number density as a function of mass and to study its evolution with redshift or its dependence on other galaxy properties such as colour and star formation activity. The study of the shape of the GSMF is a powerful indicator of how the build-up of galaxy mass happens throughout cosmic history. Moreover, theory and numerical simulations can make predictions for the shape of the GSMF to be compared with observations and therefore understand the physical processes responsible for galaxy evolution. Many works have already studied the comparison of the predicted GSMF from semi-analytical models and simulations and the observed GSMF (especially in different environments, see e.g. Fontana et al. 2006; Drory et al. 2009; Fontanot et al. 2009; Lo Faro et al. 2009; Marchesini et al. 2009; Bolzonella et al. 2010; Cirasuolo et al. 2010; Pozzetti et al. 2010; Guo et al. 2011; Bower, Benson & Crain 2012; Vulcani et al. 2014). These works find that in the case of galaxies in the general field, the number of low-mass galaxies with old stellar populations is overpredicted at intermediate redshifts ($z > 0.5$), while the number of high-mass galaxies ($M \gtrsim 10^{11} M_{\odot}$) is underpredicted at high redshift ($z > 2$). In particular, Vulcani et al. (2014) performed a comparison between model and observed GSMFs in different global environments, finding that the discrepancies at low masses are present also for the cluster GSMF. Moreover, the models fail to reproduce the observed evolution for the high-mass end of both the cluster and the field GSMF. A detailed comparison of the GSMF presented in this work with GSMFs in different environments derived from semi-analytical models of galaxy formation (included, but not limited to the one used in Malavasi et al. 2016) will be the subject of a future paper.

Several studies have addressed the investigation of the GSMF using spectroscopic redshift surveys, from the local Universe (see e.g. Baldry et al. 2004; Baldry, Glazebrook & Driver 2008; Baldry et al. 2012), using surveys such as the Sloan Digital Sky Survey (SDSS; York et al. 2000) or the Galaxy and Mass Assembly survey (GAMA; Driver et al. 2011), up to $z \sim 1$ (see e.g. Fontana et al. 2004; Pozzetti et al. 2007, 2010; Davidzon et al. 2013; Moustakas et al. 2013), relying on data from surveys such as the K20 survey (Cimatti et al. 2002), the VIMOS VLT Deep Survey (VVDS; Le Fèvre et al. 2005), the zCOSMOS survey (Lilly et al. 2007), the PRIsM Multi-object Survey (PRIMUS; Coil et al. 2011) and the VIMOS Public Extragalactic Redshift Survey (VIPERS; Guzzo et al. 2014). Photometric redshift surveys such as the COSMOS-UltraVISTA (Scoville et al. 2007; McCracken et al. 2012) and the VIPERS Multi-λ Survey (VIPERS-MLS; Moutard et al. 2016a) have instead been intensively used to explore the GSMF up to $z \sim 3$ (see e.g. Ilbert et al. 2010, 2013; Muzzin et al. 2013; Moutard et al. 2016b).

The GSMF in different environments has been thoroughly investigated in several works, again relying on both spectroscopic surveys of local galaxies (see e.g. Bundy et al. 2006, who used SDSS data; McNaught-Roberts et al. 2014, who used GAMA data; and Balogh et al. 2001, who relied on the Two Micron All Sky Survey, 2MASS; Jarrett et al. 2000; Las Campanas Redshift Survey, LCRS; Shectman et al. 1996). Spectroscopic surveys allowed the study of the GSMF also up to $z \sim 1-1.5$ (see e.g. Kodama & Bower 2003; Bolzonella et al. 2010; Cooper et al. 2010; Giodini et al. 2012; Vulcani et al. 2012; Hahn et al. 2015; Annunziatella et al. 2016; Davidzon et al. 2016), while photometric redshift surveys have been used up to $z \sim 3$ (see e.g. Scoville et al. 2013; Darvish et al. 2015; Mortlock et al. 2015). For example, Bolzonella et al. (2010) studied the GSMF in different environments in the COSMOS field (using the zCOSMOS survey, see Lilly et al. 2007, 2009) up to $z = 1$, finding a difference between the GSMF of high- and low-density environments, with the massive end of the GSMF being more enhanced in high-density environments. This result has been confirmed also by Davidzon et al. (2016), by means of the VIPERS survey (Guzzo et al. 2014; Garilli et al. 2014). Both these works relied on a local measurement of the environment, which is the same strategy adopted in this work. A complementary approach often used at $z \le 1-1.5$ is the study of the GSMF in different global environments, e.g. by comparing the GSMF in clusters and in the field. This approach is substantially different from ours and for this reason it can yield very different results (see e.g. Calvi et al. 2013; van der Burg et al. 2013; Vulcani et al. 2013, 2011; Nantais et al. 2016). In all these works, the GSMF does not seem to depend on global environment, the variations being small (Calvi et al. 2013).

The difficulty in performing environmental studies at high redshift relies mainly in the scarce availability of spectroscopic surveys of all galaxy types that sample a large enough volume (wide area and deep limiting magnitude) with large enough statistical samples. Using photometric redshift surveys performed in the COSMOS field (McCracken et al. 2012; Ilbert et al. 2013), Darvish et al. (2015) found a strong evidence for massive ($M > 10^{11} M_{\odot}$), quiescent galaxies showing an increasingly important difference between high- and low-density environments at $z \lesssim 1.5$. Also using photometric redshifts (the UKIRT Infrared Deep Sky Survey-Ultra Deep Survey, UKIDSS-UDS, and the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey, CANDELS, see Galametz et al. 2013; Guo et al. 2013), Mortlock et al. (2015) found that the GSMF is different in high- and low-density environments up to $z \sim 1.5$.

Although photometric redshifts allow us to study the galaxy population at higher redshifts and on larger areas than spectroscopic redshifts (which are usually available on large areas up to $z \sim 1-1.5$, or at higher redshifts but on much smaller sky fields and are generally characterized by a low sampling rate), their use is limited by an uncertainty much larger than that of spectroscopic redshifts. Several works have investigated the effect of photometric redshifts on the measurement of the environment (see e.g. Cooper et al. 2005; Muldrew et al. 2012; Etherington & Thomas 2015; Fossati et al. 2015; Cucciati et al. 2016; Lai et al. 2016). We base this study on our previous work (Malavasi et al. 2016), in which we extensively tested our methods on mock galaxy catalogues to investigate whether it is still possible to study the GSMF in extreme environments if the density field is measured with photometric redshifts.

In this work, we exploit the large statistical sample of the UltraVISTA Survey (McCracken et al. 2012), a deep photometric survey performed in the near-infrared in the COSMOS field. In particular, we make use of the high-precision photometric redshift sample of Ilbert et al. (2013), which allows us to reach high redshifts with a large enough statistical sample to study the GSMF in different environments for both quiescent and star-forming galaxies. This large statistical data set allows us to obtain a unique picture of the appearance of environmental signatures in the galaxy mass distribution. In fact, we are able to measure the GSMF in a self-consistent manner, exploring the redshift range of $0.2 < z < 3$, analysing different environments.
environments and both quiescent and star-forming galaxy populations with a sufficiently faint K-band limiting magnitude to reach a rather low-mass completeness limit ($M \sim 10^{10} M_{\odot}$). Moreover, the high precision of the photometric redshifts we used has allowed us to obtain a robust measurement of the local density field, clearly distinguishing the most extreme environments (i.e. high- and low-density regions) out to high-redshift. This, coupled with the preparatory work described in Malavasi et al. (2016), allows us to achieve robust results and to confidently track environmental effects on the GSMF over a large redshift range.

We briefly describe the data set that we used in Section 2. We present our main results in Section 3. We compare our findings with previous works in the literature in Section 4 and discuss our results in Section 5. We summarize our conclusions in Section 6. A standard cosmology with $\Omega_{\Lambda} = 0.7$, $\Omega_{m} = 0.3$ and $H_{0} = 70 \text{ km s}^{-1} \text{ Mpc}^{-1}$ is adopted throughout, together with a Chabrier (2003) IMF.

## 2 DATA AND METHOD

We will briefly review the data set that we used to perform the analysis, together with the methods used to estimate the environment and to calculate the GSMF for the various samples.

### 2.1 Sample

The sample that we used is composed of galaxies from the Ultra VISTA Survey (McCracken et al. 2012), with photometric redshifts and physical parameters (stellar masses, absolute magnitudes and restframe colours) derived by Ilbert et al. (2013). In particular, photometric redshifts and stellar masses have been measured by fitting to the multiband photometry synthetic spectra generated using stellar population models (Bruzual & Charlot 2003) and galaxy templates (Polletta et al. 2007) with the Le Phare code (Arnouts et al. 2002; Ilbert et al. 2006). A Calzetti et al. (2000) extinction law has been assumed, while emission line contributions have been modelled after Ilbert et al. (2009). Ilbert et al. (2013) assumed three metallicity values ($Z = 0.004, 0.008, 0.02$) and exponentially declining star formation histories in the form of $\tau^{-1}e^{-t/\tau}$ (with $\tau$ values in the range of 0.1–30 Gyr). Moreover, Ilbert et al. (2009) imposed a low extinction prior on galaxies with low SFR (in the form of $E(B-V) < 0.15$ if age/τ > 4).

The total sample has been selected in the $K_S$ band and is composed of 339 384 objects. After the removal of X-ray sources, stars and objects in masked areas, we are left with 209 758 galaxies with photometric redshift between $0.2 \le z \le 4$, $K_S \le 24$ and measured stellar mass. These objects constitute the final sample on which we performed our analysis. The $K_S$-band and redshift cuts have been performed to be consistent with Ilbert et al. (2013) and to be able to compare the GSMF for the total, quiescent and star-forming populations with what derived by Ilbert et al. (2013). We divided the galaxies of the final sample in quiescent and star-forming following the colour–colour diagram ($NUV - r^+ \text{ versus } r^+ - J$) as in Ilbert et al. (2013). Compared to a selection based on a UVJ diagram, Ilbert et al. (2010, 2013) argued that the $NUV - r^+ - J$ colour–colour diagram allows us to obtain a better distinction between star-forming and quiescent galaxies, as the $NUV - r^+$ colour is a better indicator of the current (compared to past) star formation activity (see e.g. Arnouts et al. 2007; Martin et al. 2007). Moreover, the $NUV$ rest-frame band is still sampled by optical data at $z > 2$ that does not happen for the rest-frame U band. According to the $NUV - r^+ \text{ versus } r^+ - J$ selection, ~10 per cent of the galaxies

between $0.2 \le z \le 4$ are quiescent and the remaining fraction of ~90 per cent are star forming.

We performed our analysis in eight redshift bins from $z = 0.2$ to $z = 4$. Following Ilbert et al. (2013), we assumed the photometric redshift uncertainty to be $\sigma_{\Delta z/(1+z)} = 0.01$ for all the galaxies in the sample. We chose a value of $\sigma_{\Delta z/(1+z)} = 0.01$ to be consistent with fig. 1 of Ilbert et al. (2013), which shows a comparison between the photometric redshifts for the UltraVISTA Survey and spectroscopic redshifts from a set of various samples up to $K_S \le 24$. Moreover, a value of $\sigma_{\Delta z/(1+z)} = 0.01$ is in agreement with the average of the error reported in table 1 of Ilbert et al. (2013), weighted by the number of galaxies in each spectroscopic sample used to determine the error. These samples are rather small, sometimes only tens of galaxies, and may therefore overestimate the photometric redshift uncertainty if used independently. Nevertheless, we know that the value we assumed may underestimate the photometric redshift uncertainty at $z \ge 1.5$ and for faint galaxies. For this reason, we have performed a test using a larger photometric redshift error for galaxies at $z > 1.5$. As briefly discussed in the Appendix we found that our results are not significantly affected by larger photometric redshift uncertainties.

### 2.2 Method for the estimation of the environment and the galaxy stellar mass function

The environment has been determined using a fixed aperture method (similar to what done in Gallazzi et al. 2009). The performance of this method with photometric redshifts has been extensively tested using mock galaxy catalogues in Malavasi et al. (2016). Following what we have found in our previous work, we used a cylinder, centred on each galaxy, with radius $R = 0.3, 0.6, 1$ and $2$ Mpc and with height $h$ equal to the $3\sigma$ photometric redshift error:

$$h = \pm 1.5 \times \sigma_{\Delta z/(1+z)} \times (1+z) \quad (1)$$

$$\text{with } \sigma_{\Delta z/(1+z)} = 0.01.$$

All the galaxies in the sample were used as both targets and tracers for the density field estimation. The measurement of the environment around each galaxy (hereafter defined as target galaxy) was performed by counting how many other galaxies were present inside the cylinder (in the following referred to as tracer galaxies) and then dividing by the cylinder volume. We decided to use volume densities instead of surface densities because they allow us to take the variations of the cylinder volume (due to the variation of the volume height inside the same redshift bin) into account on a galaxy by galaxy basis. In fact, as we chose a cylinder length in the radial direction proportional to the photometric redshift error of each galaxy, galaxies at different redshifts, even inside the same redshift bin, will have different volume sizes. This can create differences in their environment if not properly accounted for. By using volume densities, the problem is solved in a self-consistent fashion and environmental densities can be better compared (see Malavasi et al. 2016, for details).

The UltraVISTA-COSMOS field has a complicated shape, due to many holes left in the field by saturated stars. Galaxies close to edges or holes in the field can have their environmental measurement biased. In order to limit this effect, we applied a correction to the measured environments for galaxies too close to the edges. We rejected all galaxies for which the fraction of the area outside the survey edges (including holes in the field) was greater than 50 per cent and we corrected the measurement of the density field for all other galaxies by dividing for the fraction of the aperture area inside the edges. Moreover, galaxies with $R.A.(^\circ) > 150.55$
and Dec. (°) < 1.8 were not used in the measurement of the environment, as they lie in a small sky area far from the main field and they would have been too dominated by edge effects. The sample sizes after the correction for the edge effects are reduced to 208 624, 208 446, 208 138 and 207 183 in the $R = 0.3, 0.6, 1$ and 2 Mpc case, respectively. Mass completeness limits for these samples have been calculated as in Pozzetti et al. (2010) and are in very good agreement with those of Ilbert et al. (2013). In particular, we select the 20 per cent faintest galaxies in each redshift bin, separately in the case of the total, quiescent and star-forming galaxy populations. For these galaxies, we measure $M_{min}$, the mass that they would have if their apparent magnitude ($m_K$) were equal to the limiting magnitude of the UltraVISTA survey in the $K_S$ band ($K_S = 24$), through the formula $\log(M_{min}) = \log(M) + 0.4(m_K - 24)$. The mass completeness limit $M_{lim}$ is then the value of $M_{min}$ corresponding to the 90th percentile of the $M_{min}$ distribution. As an example, the mass completeness limits derived with this procedure for the total, passive and star-forming populations range from ($M_{lim,tot}$, $M_{lim,q}$, $M_{lim,sf}$) = ($10^{8.5}, 10^9, 10^{8.5}$) at $z \sim 0.5$ to ($10^{9.2}, 10^{9.5}, 10^{9.2}$) at $z \sim 1$ and ($10^{9.7}, 10^{10.2}, 10^{9.7}$) at $z \sim 2$.

High and low-density environments were selected as those above the 75th percentile or below the 25th percentile of the volume density distribution of galaxies with $M^* \ge 10^{10} M_{\odot}$, with the quartiles of the distribution computed at each redshift bin. We chose this mass threshold because the increase in the mass completeness limit of our sample with redshift can influence the density value of the percentiles used to define high- and low-density environments. In fact, at low redshifts, our sample is complete at lower masses, the dynamic range of the density measurement is large and the environmental density thresholds used to define high- and low-density environments are lower. Conversely at high redshifts, the dynamic range is reduced, because the sample is complete only at higher masses compared to the low-redshift case, and the threshold for the definition of high- and low-density environments is higher. The volume density distribution is sensitive to the mass completeness limit of the sample, because of the mass-density relation. In this way, we would not be able to compare the same kind of environments at low and high redshifts. By choosing a mass cut close to the mass completeness limit of the highest redshift bin, we are then able to compare the galaxy population at low and high redshift in a consistent way. High- and low-density environments have been defined for both quiescent and star-forming galaxies using the quartiles of the total galaxy population. In the following, we will use the notation $D_{75}$ and $D_{25}$ to refer to high- and low-density environments, respectively.

GSMFs have been calculated with the non-parametric $1/V_{max}$ estimator (Avni & Bahcall 1980). They have been calculated separately for quiescent and star-forming galaxies, both in high-density and in low-density environments. A comparison of our GSMF and those of Ilbert et al. (2013) shows perfect agreement. As the thresholds for defining high- and low-density environments have been calculated using only galaxies more massive than $10^{10} M_{\odot}$, the GSMF result roughly normalized at high masses, due to the fact that

$$ \int_{M \ge 10^{10} M_{\odot}} \Phi_{D75}(M) dM = \int_{M \ge 10^{10} M_{\odot}} \Phi_{D25}(M) dM. \quad (2) $$

When calculating mass functions, if the number of galaxies in a given mass bin is lower than two, we applied the prescription for small counts Poisson statistics of Gehrels (1986), in the form of $1\sigma$ upper and lower limits of tables 1 and 2. In particular, these tables present the number of true Poisson events corresponding to a rate

of one or zero observed events. In the case of one observed event (mass bins containing only one galaxy), the number of real Poisson events can be used to correct the error measurement. In the case of empty mass bins (zero observed events), the number of real Poisson events divided by the volume corresponding to the selected redshift bin can be used to set an upper limit for the GSMF in a larger mass range.

### 3 RESULTS

Although the analysis presented in this work is based on photometric redshifts, the method that we used to reconstruct local density and that we tested on mock galaxy catalogues in our previous work (Malavasi et al. 2016) is able to provide us with a robust measurement of galaxy environments up to $z = 3$ and on various physical scales. The high-precision photometric redshifts of the UltraVISTA sample allow us to trace environmental effects on galaxy properties throughout cosmic history, contributing in the creation of a consistent picture of galaxy evolution. In the following, only the cases with $R = 0.3$ and $R = 2$ Mpc will be discussed at length. The other values of $R$, which constitute intermediate cases between those reported here, have been analysed, but will not be reported for the sake of clarity and conciseness.

Fig. 1 shows an example of the performance of the fixed aperture method in estimating the density field. This figure shows the Ultra VISTA sky field in a representative high-redshift bin ($1.5 \le z \le 2.0$) for the total UltraVISTA sample and only high- and low-density environments separately. Red and blue dots refer to quiescent and star-forming galaxies, respectively. Only the fixed aperture radius $R = 0.3$ Mpc is represented, as an example. It can be seen that the fixed aperture method that we implemented is able to identify galaxies in different environments. Galaxies belonging to high-density environments tend to be more clustered, while low-density galaxies appear spatially distributed in a more uniform fashion.

Interestingly, it can be seen how, although rare at this redshift, quiescent galaxies tend to be slightly more visible in the high-density regions compared to the low-density ones. This trend can be expressed quantitatively by looking at the fraction of quiescent galaxies as a function of environment, redshift and mass (shown in Fig. 2). As expected the fraction of quiescent galaxies increases with cosmic time in both environments. Nevertheless, these fractions show how quiescent galaxies are more numerous in high-density environments compared to low-density ones as a function of mass. Although the difference is a function of mass and redshift, it remains well visible up to $z \sim 2$ for both the $R = 0.3$ Mpc and the $R = 2$ Mpc case. For masses $\sim 10^{11} M_{\odot}$, at $z \sim 0.5$ 60 per cent of the galaxies in high-density environments are quiescent, while only 40 per cent in low-density environments. At $z \sim 1$, the difference is reduced to $\lesssim 10$ per cent, but it is still visible. In the $R = 2$ Mpc case, differences of $\sim 10$ per cent at $z \sim 0.5$ are reduced to $\sim 5$ per cent at $z \sim 1$.

The trend visible in our data is in agreement also with what found in other works, using both global environment (see e.g. van der Burg et al. 2013; Nantais et al. 2016; Muzzin et al. 2012, who performed analysis using various samples of clusters at $z \sim 1-1.5$) and local environment definitions. In particular, with respect to local environment, Baldry et al. (2006) found a fraction of quiescent galaxies $\sim 20$ per cent higher in high-density environments for masses of $\sim 10^{11} M_{\odot}$ at $z \lesssim 0.1$, compared to the lowest densities they explored. Results in agreement with our fractions of quiescent galaxies are found also by Darvish et al. (2016), who reported a fraction of quiescent galaxies higher by $\sim 20-40$ per cent for masses of $\sim 10^{11} M_{\odot}$ at $z \sim 0.5$ and by $\lesssim 20$ per cent at $z \sim 1$ in high-density
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Sky maps of the UltraVISTA field. Only the case for *R* = 0.3 Mpc, *z* ∈ [1.5, 2.0] is shown. Top panel: black dots represent the total Ultra VISTA sample in the considered redshift bin. Middle and bottom panels are only high- and low-density environments, respectively. Red triangles and blue dots represent quiescent and star-forming galaxies. The fixed aperture method is able to identify galaxies in different environments at high redshift. Quiescent galaxies are located preferentially in high-density environments.</figcaption></figure>

environments compared to low-density ones in the COSMOS field (see also Scoville et al. 2013; Lin et al. 2016). For direct comparison, in Fig. 2 also the fractions of quiescent galaxies in different environments for the zCOSMOS Survey (Bolzonella et al. 2010, see their fig. 8) are reported (for a more detailed comparison between our work and Bolzonella et al. 2010, see Section 4).

## 3.1 The GSMF of the UltraVISTA sample

Fig. 3 shows a first example of the GSMF for all UltraVISTA galaxies. In the same figure also the GSMF for high- and low-density environments are shown, in the case of *R* = 0.3 Mpc. We do not report the GSMF for other radii for the sake of clarity, although when performing the analysis on the shape of the GSMF in different environments all apertures will be considered for completeness. It can be seen how the GSMFs of high- and low-density environments are different. The high-mass end of the GSMF (above *M* = 10$^{10.5–11}$ M$_{⊙}$) is enhanced in the case of high-density environments in comparison to low-density ones, while the low-mass end is depleted. This difference can be appreciated up to *z* ~ 2, where no more differences can be seen between high- and low-density environments.

If we divide the galaxy population into quiescent and star-forming galaxies, we see how the difference between high- and low-density environments affects different parts of the GSMF in the case of the quiescent galaxy population (Fig. 4) and in the case of the star-forming galaxy population (Fig. 5). These figures show the GSMF for the quiescent and star-forming components of the total GSMF in high- and low-density environments. For quiescent galaxies, the enhancement of the high-mass end in high-density environments is visible in comparison to low-density environments up to *z* ~ 2. Moreover, a steep decline at low masses is visible. This decline is visible in the high- and low-density GSMFs and is found also by Ilbert et al. (2013), Muzzin et al. (2013) and Mortlock et al. (2015). For the star-forming population, instead, the difference is mainly present at low masses (below 10$^{11}$ M$_{⊙}$) and at lower redshifts (below *z* ~ 1.5). In these figures, the GSMFs from Davidzon et al. (2016), Bolzonella et al. (2010) and Mortlock et al. (2015) are reported for reference and a more detailed comparison between these works and our results will be carried out in Section 4.

A more quantitative analysis of the differences between high- and low-density GSMF for the different galaxy populations can be performed by taking the ratios of the high-density to the low-density GSMF for the total, the quiescent and the star-forming galaxy populations as a function of mass and redshift (Fig. 6). In the quiescent and star-forming case, the ratios are calculated using the quiescent and star-forming component of the total GSMF in high- and low-density environments. For this reason, the ratio can be greater than 1 (logarithm of the ratio greater than 0, in the figure). It can be seen how the ratio of the high-density to the low-density GSMF is typically higher in the case of quiescent galaxies compared to star-forming ones, at least up to *z* ~ 2 for both the *R* = 0.3 Mpc and the *R* = 2 Mpc cases. The ratio of the high-density to the low-density GSMF is generally ≳1 for quiescent galaxies (logarithm of the ratio ≳0) and it is generally ≲1 for star-forming galaxies (logarithm of the ratio ≲0, in the figure). This can be interpreted as quiescent galaxies being more represented in high-density environments and star-forming galaxies being more present in low-density environments. These ratios also show a trend with mass both for quiescent and star-forming galaxies. High-density environments are dominated by a more massive galaxy population, and this is generally true for both quiescent and star-forming galaxies. Instead the ratio of the high-density to the low-density GSMF for the total
<figure><img src="image_5.png" /><figcaption>**Figure 2.** Fraction of quiescent galaxies. The solid magenta line refers to high-density environments, the dashed cyan line to low-density environments. The shaded regions correspond to the propagated errors on the fraction. For each redshift bin, the number of galaxies above the mass limit in the two environments (magenta for high-density environments and cyan for low-density environments) is reported. The vertical black dashed line corresponds to the mass completeness limit. Top panel refers to a fixed aperture radius of R = 0.3 Mpc, bottom panel to R = 2 Mpc. In the first three redshift bins, the fractions of quiescent galaxies in high- and low-density environments from the work by Bolzonella et al. (2010, see their fig. 8) are reported for comparison above the mass completeness limit. Red circles and solid lines refer to high-density environments, blue squares and dashed lines to low-density environments. The redshift bins in which the fractions of Bolzonella et al. (2010) have been calculated are reported in parentheses in the bottom right corners of the plots. The fraction of quiescent galaxies is larger in high-density environments up to z ~ 2.</figcaption></figure>

galaxy population follows the same ratio of star-forming galaxies at low masses and the one of quiescent galaxies at high masses, as expected.

## 3.2 The shape of the GSMF in different environments

Differences between the shape of the GSMF in high- and low-density environments can be better seen by taking the ratio of the

high-mass to the intermediate-mass end of the GSMF. In particular, we calculated the quantity

$$ \log \frac{\Phi(HM)}{\Phi(\overline{IM})} = \log \frac{\int_{\log(M) \in [11, 11.5]} \Phi(M) dM}{\int_{\log(M) \in [10, 10.5]} \Phi(M) dM} \quad (3) $$

for both quiescent and star-forming galaxies in both high- and low-density environments (shown in Fig. 7). To calculate the ratio, we did not include upper limits due to mass bins with zero galaxies, but we
<figure><img src="image_6.png" /><figcaption>**Figure 3.** GSMF of UltraVISTA galaxies – all galaxies. The solid black curve refers to the total GSMF for all UltraVISTA galaxies, the long-dashed magenta curve refers to high-density environments and the short-dashed cyan curve refers to low-density environments. Vertical lines are the mass completeness limits, style- and colour-coded as the corresponding GSMF. Diamonds represent mass bins with only one galaxy, downwards triangles are upper limits for mass bins with zero galaxies. Shaded areas represent Poissonian errors. In the bins at redshift 1.1 < z < 1.5, 1.5 < z < 2.0, and 2.0 < z < 2.5 green stars and orange squares correspond to the UKIDSS-CANDELS GSMF (Mortlock et al. 2015, see their fig. 8, green stars refer to high-density environments, orange squares to low-density environments). For the Mortlock et al. (2015) GSMF, only points above their mass completeness limit have been considered. The redshift bins in which the Mortlock et al. (2015) GSMFs have been calculated are 1.0 < z < 1.5, 1.5 < z < 2.0 and 2.0 < z < 2.5. The high-mass end of the UltraVISTA GSMF is enhanced in high-density environments, while the low-mass end is depleted with respect to low-density environments.</figcaption></figure>

did include mass bins with only one count. Moreover, the analysis has been performed only up to $z \sim 2.5$ as the sample of quiescent galaxies begins to be incomplete at $10 \le \log(M/M_{\odot}) \le 10.5$ in the last redshift bin.

These ratios show clearly how the difference between high-density and low-density environments is present mainly for quiescent galaxies, rather than for star-forming galaxies. For quiescent galaxies, the ratio of the high-mass to the intermediate-mass end of the GSMF is higher in high-density environments compared to low-density ones. This ratio also shows a trend with redshift, monotonically

increasing up to $z \sim 2$ (for high-density, quiescent galaxies the trend with redshift is more evident for the $R = 2$ Mpc case). This reflects the gradual build-up of the intermediate mass part of the GSMF with cosmic time for the quenched galaxy population, and is in agreement with a scenario in which massive galaxies became passive at earlier times than lower mass galaxies (downsizing). The difference between high- and low-density environments seems to be present for both small and large radii, with no significant differences amongst them. In the bottom panel of the same figure, we report the difference between the high-density and the
<figure><img src="image_7.png" /><figcaption>**Figure 4.** GSMF of UltraVISTA galaxies – quiescent galaxies. The solid red curve refers to the total GSMF for quiescent galaxies only, the long-dashed magenta curve refers to high-density environments and the short-dashed cyan curve refers to low-density environments. Vertical lines are the mass completeness limits, style- and colour-coded as the corresponding GSMF. Diamonds represent mass bins with only one galaxy, downwards triangles are upper limits for mass bins with zero galaxies. Shaded areas represent Poissonian errors. In the bins at redshift 0.5 < z < 0.8 and 0.8 < z < 1.1 red circles and blue upwards triangles correspond to the zCOSMOS GSMF (Bolzonella et al. 2010, see their Fig. 5, red circles represent high-density environments and blue upwards triangles represent low-density environments), for quiescent galaxies. Purple stars and orange squares correspond to the VIPERS GSMF (Davidzon et al. 2016, see their fig. 4, purple stars represent high-density environments, orange squares represent low-density environments), for quiescent galaxies. For zCOSMOS and VIPERS GSMF only points above the respective mass completeness limits are shown. The redshift bins in which zCOSMOS GSMFs have been calculated are 0.5 < z < 0.7 and 0.7 < z < 1.0, the redshift bins in which VIPERS GSMFs have been calculated are 0.65 < z < 0.8 and 0.8 < z < 0.9. The high-mass end of the UltraVISTA quiescent GSMF is enhanced in high-density environments up to z ~ 2, while no difference seems to be present at the intermediate- and low-mass end.</figcaption></figure>

low-density curves from the top panel, normalized to the sum in quadrature of their errors (in the case where the errors on the ratios are asymmetric, the largest of the two has been considered). These plots show how the difference between high- and low-density environments reaches and largely exceeds 3σ (being as large as 5σ in the R = 0.3 Mpc case) for quiescent galaxies, while it is lower for star-forming galaxies (mostly of the order of 2σ and even lower in the R = 2 Mpc case).

Conversely, if we take the ratio of the intermediate-mass to the low-mass end of the GSMF, differences emerge only for star-forming galaxies at $z \le 1.5$. In particular, we calculated the quantity

$$ \log \frac{\Phi(\mathrm{IM})}{\Phi(\mathrm{LM})} = \log \frac{\int_{\log(M) \in [10.5, 11]} \frac{\Phi(M) \mathrm{d}M}{\int_{\log(M) \in [9.5, 10]} \frac{\Phi(M) \mathrm{d}M}} \quad (4) $$
<figure><img src="image_8.png" /><figcaption>**Figure 5.** GSMF of UltraVISTA galaxies – star-forming galaxies. The solid blue curve refers to the total GSMF for star-forming galaxies only, the long-dashed magenta curve refers to high-density environments and the short-dashed cyan curve refers to low-density environments. Vertical lines are the mass completeness limits, style- and colour-coded as the corresponding GSMF. Diamonds represent mass bins with only one galaxy, downwards triangles are upper limits for mass bins with zero galaxies. Shaded areas represent Poissonian errors. In the bins at redshift 0.5 < z < 0.8 and 0.8 < z < 1.1 red circles and blue upwards triangles correspond to the zCOSMOS GSMF (Bolzonella et al. 2010, see their fig. 5, red circles represent high-density environments and blue upwards triangles represent low-density environments), for star-forming galaxies. Purple stars and orange squares correspond to the VIPERS GSMF (Davidzon et al. 2016, see their Fig. 4, purple stars represent high-density environments, orange squares represent low-density environments), for star-forming galaxies. For zCOSMOS and VIPERS GSMF only points above the respective mass completeness limits are shown. The redshift bins in which zCOSMOS GSMFs have been calculated are 0.5 < z < 0.7 and 0.7 < z < 1.0, the redshift bins in which VIPERS GSMFs have been calculated are 0.65 < z < 0.8 and 0.8 < z < 0.9. The UltraVISTA star-forming GSMF shows an excess of low-mass galaxies in low-density environments up to z ~ 1.5, while no difference seems to be present in the high-mass end.</figcaption></figure>

for both quiescent and star-forming galaxies in both high- and low-density environments (shown in Fig. 8). Again, to calculate the ratio we did not include upper limits due to mass bins with zero galaxies, but we did include mass bins with only one count. The quantity described in equation (4) has been computed only up to z ~ 1.5 as for higher redshifts the sample of quiescent galaxies becomes incomplete in the range of 9.5 ≤ log (M/M$_{⊙}$) ≤ 10. This figure

clearly shows how a difference between high- and low-density environments is present mainly for star-forming galaxies (except in the first redshift bin) up to z ~ 1–1.5 and generally not for quiescent galaxies. The ratio is <1 for star-forming galaxies, and it is smaller in low-density environments by ~0.2 dex for environments measured with a fixed aperture radius of R = 0.3 Mpc. The difference between high- and low-density environments seems to get smaller
<figure><img src="image_9.png" /><figcaption>**Figure 6.** Ratio of the high- to the low-density GSMF. Ratio of the high-density ($\Phi_{D75}$) to the low-density ($\Phi_{D25}$) GSMF as a function of mass and redshift. The solid black line refers to all galaxies, the long-dashed red line to quiescent galaxies and the short-dashed blue line to star-forming galaxies. The shaded regions correspond to the propagated errors on the ratio. The vertical black dashed line corresponds to the mass completeness limit. Top panel refers to a fixed aperture radius of $R = 0.3$ Mpc and bottom panel to $R = 2$ Mpc. The fact that the logarithm of the ratio of the high-density to the low-density GSMF is generally $\gtrsim 0$ for quiescent galaxies and generally $\lesssim 0$ for star-forming galaxies can be interpreted as quiescent galaxies being more represented in high-density environments and star-forming galaxies being more present in low-density environments.</figcaption></figure>

increasing the fixed aperture radius. Nevertheless, an indication of low-mass star-forming galaxies being more present in low-density environments is visible in the data. Indeed, from the difference between the high- and low-density curves from the top panels, normalized to the sum in quadrature of the errors, it is possible to see how the difference between high- and low-density environments in the case of star-forming galaxies reaches values as high as $5\sigma$ in the $R = 0.3$ Mpc case, while being systematically larger than $3\sigma$.

On the other hand, the difference between high- and low-density environments for quiescent galaxies is always in the $\pm 2\sigma$ range, being close to $3\sigma$ only in the $R = 2$ Mpc case. The monotonic trend with redshift of the intermediate- to low-mass ratio for the quiescent galaxies is an indication of a progressive steepening of the low-mass end of the quiescent GSMF with redshift (this seems to be at variance with what hinted by Davidzon et al. 2016, although the different redshift range explored, mass completeness limit and
<figure><img src="image_10.png" /><figcaption>**Figure 7.** Shape of the GSMF. Top panel shows the ratio of the high-mass to the intermediate-mass end of the GSMF (see equation 3) as a function of redshift. Solid lines refer to high-density environments and dashed lines to low-density ones. Red lines represent quiescent galaxies and blue lines star-forming galaxies. In black we report the ratio for the total galaxy population. Bottom panel shows the difference between the high-density and the low-density cases of the curves reported in the top panel, normalized to the sum of the errors (long-dashed red, short-dashed blue and solid black curves refer to the quiescent, star-forming and total galaxy populations, respectively). For reference, values corresponding to a difference of 0, ±1σ and ±3σ are reported as thin solid, dashed and dot-dashed horizontal lines. These ratios quantitatively indicate that the high-mass end of the GSMF is enhanced in high-density environments for quiescent galaxies and not for star-forming ones up to $z \sim 2$.</figcaption></figure>

environmental definition prevent us from drawing any firm conclusion from the comparison). Nevertheless, this monotonic trend with cosmic time is not observed for the star-forming galaxy population, and this can again be related to the gradual build-up of galaxy mass with cosmic time, in a complementary way than what found before with the high- to intermediate-mass end ratios. The low-mass end of the quiescent GSMF is gradually enhanced as more low-mass galaxies are quenched with cosmic time, while the low-mass end of the star-forming GSMF is continuously replenished by galaxies that increase their stellar mass through ongoing star formation activity. This result is in agreement also with what found by Pozzetti et al. (2010, see their fig. 14) and Ilbert et al. (2013, see their Fig. 6) as well as by many other works (see e.g. Muzzin et al. 2013; Tomczak et al. 2014; Mortlock et al. 2015; Moutard et al. 2016b, and references therein).

### 3.3 The relative importance of quiescent and star-forming GSMF in different environments

Fig. 9 shows the mass at which the quiescent and the star-forming GSMF intersect ($M_{cross}$) as a function of redshift and environment (i.e. the mass above which the GSMF is dominated by the quiescent population). As for Fig. 7, the analysis has been limited at redshift $z \sim 2.5$ as at higher redshift the size of the quiescent galaxy sample becomes too limited. It can be seen how $M_{cross}$ is higher in low-density environments compared to high-density ones

up to redshift $z \sim 1.5$, where the two curves become indistinguishable. This is in agreement with the current paradigm of galaxy evolution, which predicts that massive galaxies became quiescent at earlier times compared to less massive galaxies. Therefore, as redshift increases, the mass at which the quiescent GSMF starts to dominate over the star-forming GSMF increases as well. The fact that $M_{cross}$ is higher in low-density environments compared to high-density ones is an evidence of the fact that the processes that lead to the quenching of the star formation and to the transformation of star-forming galaxies into quiescent galaxies are more efficient in high-density environments, leading to less massive galaxies being already quenched, while at the same redshift, in low-density environments, they will still be star forming. In high-density environments, $M_{cross}$ is a monotonically increasing function of redshift, increasing from $\sim 10^{10} M_{\odot}$ at $z \sim 0.5$ to $10^{1.5}$ at $z \sim 2$. If we consider also upper limits to the value of $M_{cross}$ derived when GSMF do not intersect, then an increase of $M_{cross}$ as a function of redshift is roughly true also for low-density environments for redshifts $z \gtrsim 1$, while at lower redshifts Fig. 9 shows an upturn in the value of $M_{cross}$. This upturn seems to become less evident going from $R = 0.3$ Mpc to $R = 2$ Mpc. The upturn at low redshifts of the $M_{cross}$ in low-density environments is probably due to the lowest density environments probed by the fixed aperture method, especially on small scales (e.g. $R = 0.3$ Mpc). In such underdense environments quiescent galaxies are rare, as the fraction of quiescent galaxies is higher in high-density environments (see e.g. Fig. 2).
<figure><img src="image_11.png" /><figcaption>**Figure 8.** Shape of the GSMF. Top panel shows the ratio of the intermediate-mass to the low-mass end of the GSMF (see equation 4) as a function of redshift. Solid lines refer to high-density environments and dashed lines to low-density ones. Red lines represent quiescent galaxies and blue lines star-forming galaxies. In black we report the ratio for the total galaxy population. Bottom panel shows the difference between the high-density and the low-density cases of the curves reported in the top panel, normalized to the sum of the errors (long-dashed red, short-dashed blue and solid black curves refer to the quiescent, star-forming and total galaxy populations, respectively). For reference, values corresponding to a difference of 0, $\pm 1\sigma$ and $\pm 3\sigma$ are reported as thin solid, dashed and dot-dashed horizontal lines. These ratios quantitatively indicate that the low-mass end of the star-forming GSMF is depleted in high-density environments up to $z \sim 1.5$, while no difference is present for quiescent galaxies.</figcaption></figure>

Therefore, the quiescent and star-forming GSMF will be more separated, especially at high masses, and the $M_{\text{cross}}$ results higher.

## 4 COMPARISON WITH PREVIOUS STUDIES

Thanks to its excellent combination of multiwavelength coverage, fairly large area and presence of data sets with a high statistical power, the UltraVISTA-COSMOS is a perfect field where to perform studies of galaxy evolution. For this reason, it has been deeply exploited in several works. As the GSMF is a very powerful tool to study the galaxy formation from a statistical point of view, many studies have investigated its relation to galaxy environment. In this section, we review some of the main works that studied the GSMF in different environments from low redshifts, using spectroscopic samples, up to high redshift, using photometric redshift surveys.

The main spectroscopic survey that has been performed in the COSMOS field is the zCOSMOS Survey (see Lilly et al. 2007). Using the 10k spectroscopic sample of Lilly et al. (2009) in synergy with the COSMOS photometric sample (Capak et al. 2007; McCracken et al. 2010) and the environmental estimate by Kovač et al. (2010), Bolzonella et al. (2010) performed a thorough study of the GSMF in different environments up to $z = 1$. In their work, they found a difference between the GSMF of high- and low-density environments, with the massive end of the GSMF being more enhanced in high-density environments. This is qualitatively in agreement

with our results in the common redshift range. In Figs 4 and 5, we show a comparison between our GSMF and those of Bolzonella et al. (2010, see their fig. 5) for two common redshift bins. We compared quiescent and star-forming GSMF in both high- and low-density environments. GSMF have been normalized so to be equal at a given mass (that of the lowest mass bins considered for VIPERS GSMF, see below), which allows us to compare their shape. With the exception of quiescent galaxies in the range $0.8 \le z \le 1.1$ (where the GSMF of both high- and low-density environments are in good agreement), the GSMF of Bolzonella et al. (2010) and those of our work show a slightly different shape, with the GSMF of Bolzonella et al. (2010) displaying a steeper slope in the high- and/or low-mass end(s).

The steeper slope of the zCOSMOS mass functions could be due to the different environment estimator used in our work (fixed aperture with $R = 0.3–2$ Mpc) and in Bolzonella et al. (2010) (distance to the fifth nearest neighbour) to the different definition of quiescent and star-forming galaxies [photometric type coming from SED fitting estimate for Bolzonella et al. (2010) versus restframe colour-colour classification for our work] or to the fact that Bolzonella et al. (2010) use spectroscopic redshift while we use photometric redshifts. Thanks to the higher statistics and lower mass limit compared to Bolzonella et al. (2010), we see a difference between high- and low-density environments for both passive and star-forming galaxies.
<figure><img src="image_12.png" /><figcaption>**Figure 9.** *M*<sub>cross</sub> is the Mass at which the star-forming and the quiescent GSMF intersect. The solid magenta line refers to high-density environments, while the dashed cyan line refers to low-density environments. Triangles (upwards for low-density environments and downwards for high-density environments) are upper limits to the value of *M*<sub>cross</sub> for when the two GSMFs do not intersect, defined as the mass at which the two GSMF are closer to each other. Galaxies in high-density environments undergo an accelerated evolutionary process with respect to galaxies in low-density environments.</figcaption></figure>

A difference between our work and Bolzonella et al. (2010) is also confirmed by the fraction of quiescent galaxies in different environments, as shown in Fig. 2. In particular, while the fraction of quiescent galaxies in both environments is comparable to the one derived in our work for the lowest mass bins considered by Bolzonella et al. (2010), their fractions become quickly higher than ours with increasing mass and especially in low-density environments. This is likely due to the different environmental estimator used in our work and in Bolzonella et al. (2010), and it is likely the origin of the discrepancy observed in the values of *M*<sub>cross</sub> between our work and Bolzonella et al. (2010, see their fig. 7). In fact, while the value of *M*<sub>cross</sub> in high-density environments is in agreement between our work and Bolzonella et al. (2010), we find an upturn in the values of *M*<sub>cross</sub> in low-density environments that is totally absent in the work of Bolzonella et al. (2010).

Although a quantitative comparison is difficult, the range of densities corresponding to low-density environments explored by our work is much lower than that in Bolzonella et al. (2010). Therefore, in such environments the quiescent galaxy population will be more under-represented and the corresponding quiescent GSMF

will result depressed with respect to the star-forming GSMF, with the corresponding value of *M*<sub>cross</sub> increased, as confirmed by the different fraction of quiescent galaxies.

In Figs 4 and 5, we also report a comparison between our mass functions and those of Davidzon et al. (2016, see their fig. 4). Although with a different spectroscopic data set (VIPERS Survey, Garilli et al. 2014; Guzzo et al. 2014), Davidzon et al. (2016) performed a thorough study of the GSMF in different environments at *z* ≲ 1, finding consistent results with Bolzonella et al. (2010). Although their mass completeness limit allowed only a characterization of the high-mass end of the GSMF, still their GSMFs for quiescent and star-forming galaxies in different environments are consistent within errorbars with those of our work in the overlapping redshift bins.

As the UltraVISTA sample offers high-quality photometric redshifts for a large statistical sample, some works have been performed at redshift *z* > 1 in the COSMOS field. A couple of recent works have explored the dependence of the GSMF on the environment using the same sample as we did in this work. Both Scoville et al. (2013) and Darvish et al. (2015) used a 2D Voronoi tessellation performed in subsequent redshift slices to study the environmental effects on the galaxy population and the GSMF. In particular, Darvish et al. (2015) found a strong evidence for massive (*M* > 10<sup>11</sup> M<sub>☉</sub>), quiescent galaxies showing an increasingly important difference between high- and low-density environments at *z* ≲ 1.5. They found that the number density of massive quiescent galaxies in high-density environments is ~10 times higher than in low-density environments at redshift *z* ≲ 0.5 (see their fig. 10). This is in agreement with the results of this work, which see environmental effects disappear for quiescent galaxies at *z* ~ 2, with the ratio of the high- end to the intermediate-mass end of the GSMF for quiescent galaxies being different by more than 3σ in high-density environments with respect to low-density environments at *z* ≲ 0.5 (see Fig. 7). Both the work by Darvish et al. (2015) and this work found no environmental effect for massive star-forming galaxies at any redshift. However, with our work we have been able also to extend the analysis to low-mass star-forming galaxies, finding an environmental effect up to *z* ≲ 1.5.

It is also important to mention that our work is in agreement with what found in the UKIDSS UDS field (see Mortlock et al. 2015). By using the UKIDSS UDS data set and the CANDELS Survey (photometric redshifts, Galametz et al. 2013; Guo et al. 2013), Mortlock et al. (2015) found that the GSMF is different in high- and low-density environments up to *z* ~ 1.5. We show a comparison between our work and the work by Mortlock et al. (2015) in three high-redshift bins in Fig. 3. GSMFs are again normalized to be equal at *M* = 10<sup>11</sup> M<sub>☉</sub>, so to be able to compare their shape in a consistent way. It can be seen how our GSMF and those of Mortlock et al. (2015) are in good agreement except for the last redshift bin (2.0 < *z* < 2.5), where they show a different shape, with the GSMF of Mortlock et al. (2015) characterized by a steeper slope. This difference in shape at high redshift could be due to the different high- and low-density environment definition (25th and 75th percentile of the volume density distribution in our work, 1σ deviation from the mean of the density distribution in Mortlock et al. 2015).

These comparisons show how we have been able to exploit the excellent UltraVISTA data set to extend previous works done at low redshift with spectroscopic surveys (e.g. Bolzonella et al. 2010; Davidzon et al. 2016) and to complement other works performed at high redshift with photometric redshift surveys (e.g. Darvish et al. 2015; Mortlock et al. 2015).
# 5 DISCUSSION

Galaxies evolve in parallel with cosmic structures. As galaxies form, so do galaxy clusters, groups and the LSS and transformations in galaxy properties happen at the same time as changes in their local and global environment. It is therefore expected some correlation between galaxy environment and galaxy properties as a function of redshift. The current understanding of the effect of the environment on galaxy evolution is that environment plays a role in determining the cease of star formation in galaxies and in causing their transformation from blue, actively star-forming, disc-like objects to red, quiescent spheroidal systems.

This picture of galaxy formation in relation to environmental effects is supported by evidence on both the theoretical and observational sides. For example, many mechanisms correlated to galaxy environment have been proposed to end the star formation in a galaxy (see e.g. fig. 10 of Treu et al. 2003) and many correlations have been found between the main observables and the density field (e.g. galaxy colour, mass, AGN activity, star formation and morphology). Moreover, some works (e.g. Peng et al. 2010) have proposed a separability of the processes that lead a galaxy to quiescence on the basis of mass and environment. In this work, we present observational evidence of the presence of a complex interplay amongst galaxy mass, star formation activity (or lack thereof) and local environment.

Although relying only on photometric redshifts, this work is able to recover with good accuracy the environmental trends of the GSMF by making use of a method that has been fully tested on mock galaxy catalogues. In fact, in Malavasi et al. (2016), we carefully tested the effect that a density field measured using photometric redshifts has on the analysis of the GSMF in different environments, finding that no spurious effects are introduced and that the differences that are found would be greater if more precise redshifts were used. This is indeed an important argument in the present analysis: we can expect all the results that are described in this work to be more evident if spectroscopic redshifts were used. Of course, larger statistical samples at higher redshifts with precise redshift measurement are vital to perform this kind of studies. Here, we discuss the results of Section 3 in a theoretical framework of galaxy evolution after having decisively ruled out possible effects due to photometric redshift uncertainties (see Malavasi et al. 2016 and Section A of this work).

We find that the galaxy population is different in different environments. High-density environments are populated by a higher fraction of quiescent galaxies and this distinction is particularly visible at high masses ($M \gtrsim 10^{11} M_{\odot}$) up to redshift $z \sim 2$. The lack of differences between the fraction of quiescent galaxies in different environments at higher redshifts is probably due to the fact that at higher redshifts structures are at an earlier stage of formation (see e.g. Chiang, Overzier & Gebhardt 2013) and quiescent galaxies, even the massive ones, are rarer (while star-forming galaxies still dominate). This evidence is complemented by the total GSMF divided according to local environment, which shows how in high-density environments massive galaxies are more represented up to $z \sim 2$. Therefore, peaks in the density field seem to constitute a particular kind of environment where galaxies are more massive and more quiescent. Environment, therefore, plays a role in shaping the galaxy population and is connected to the build-up of galaxy mass and to the end of the star formation. Environmental effects are visible since $z \sim 2$ and on scales of $R \approx 2$ Mpc, being effective for a long period of galaxy formation in a strong way.

A particularly interesting scenario for galaxy evolution is the one proposed by Gabor & Davé (2015). In their work, the authors used numerical simulations to investigate the new unified quenching model that they propose. In this model, a galaxy is quenched once the gas in its host halo becomes hot ($T \gtrsim 10^{5.4} K$) and this happens when the host halo reaches a mass of $10^{12} M_{\odot}$ (roughly corresponding to a stellar mass of $10^{10.5} M_{\odot}$). In this scenario, both ‘mass quenching’ and ‘environmental quenching’ (Peng et al. 2010) are seen as separate evidences of the same underlying quenching mechanism due to the presence of hot gas. This theoretical model can be used to give an interpretation of our results. In particular, according to this model, the galaxies populating the high-mass end of the GSMF ($M \gtrsim 10^{11} M_{\odot}$) are being quenched because they live in hot gas dominated haloes. Although they are found also in low-density environments (see fig. 6 of Gabor & Davé 2015), massive haloes ($M \gtrsim 10^{12} M_{\odot}$) are found preferentially in high-density environments. This is the cause of the difference between high- and low-density environments seen in the GSMF of quiescent galaxies at masses $M \gtrsim 10^{11} M_{\odot}$.

This difference is not seen in the GSMF of star-forming galaxies because massive galaxies in high-density environments are quenched; therefore, they are not included in the star-forming GSMF. This goes in the direction of diluting the signal of potential differences in the high-mass end of the GSMF of star-forming galaxies as a function of environment. Instead, a difference is visible at low masses, with low-mass star-forming galaxies being more present in low-density environments. This is due to the fact that these galaxies live in too low-mass haloes to develop a hot gas environment and be quenched. Nevertheless, those living in high-density environments can still be quenched as satellites of more massive galaxies that live in hot gas dominated haloes. Therefore, the low-mass end of the star-forming GSMF is depleted in high-density environments compared to low-density ones. Interestingly, this trend should reflect in a difference in the low-mass end of the GSMF of quiescent galaxies in high-density environments, which seem to be absent in our data. This lack of a difference between the high- and low-density, low-mass end of the quiescent GSMF could be due to uncertainties in the photometric redshift calculation or in the distinction between quiescent and star-forming galaxies using the colour-colour diagram. Star-forming galaxies being the majority of the sample, a difference in the low-mass end of the GSMF can be recovered for them, but not for quiescent galaxies, which may suffer from residual contamination from star-forming galaxies at low masses. Nevertheless, a more accurate analysis, with more precise redshifts and a larger data set has to be performed to solve the problem.

# 6 CONCLUSIONS AND SUMMARY

In this work, we have used the GSMF and the high-precision photometric redshifts of the UltraVISTA Survey (McCracken et al. 2012; Ilbert et al. 2013) to outline a history of the role of the environment in galaxy evolution from $z=3$ to $z=0$. Thanks to the high precision of the photometric redshift data set used for this work, together with the accurate preparatory analysis performed in Malavasi et al. (2016), the study described in this paper presents an additional perspective, both extending and complementing known results in the literature in a consistent way over a broader redshift range. Although derived with photometric redshifts, the results presented in this work are robust and provide a reliable observational evidence to support theoretical scenarios of galaxy formation. Our main findings can be summarized as follows:
(i) The fraction of massive quiescent galaxies is higher in high-density environments compared to low-density ones. The difference is visible up to a scale of R = 2 Mpc, and it is present up to redshift z ~ 2. The fraction of quiescent galaxies increases with mass and decreases with redshift.

(ii) The shape of the galaxy stellar mass function is different in high- and low-density environments for the total galaxy population. The high-mass end of the mass function (log(*M*\*/M⊙) ∈ [11, 11.5]) is enhanced with respect to the intermediate-mass end (log(*M*\*/M⊙) ∈ [10, 10.5] in high-density environments) up to z ~ 2.

(iii) The difference in the shape of the GSMF between high- and low-density environments is visible for quiescent galaxies up to z ~ 2 and at masses *M* > 10¹¹ M⊙.

(iv) The difference in the shape of the GSMF between high- and low-density environments is visible for star-forming galaxies up to z ~ 1.5 and at masses *M* < 10¹¹ M⊙.

(v) No environmental effects seem to be visible in our data at *z* ≳ 2.

(vi) The mass at which galaxies become quiescent at a given redshift is lower in high-density environments compared to low-density ones. This effect is visible up to redshift *z* ~ 1.5. In high-density environments, the mass at which the quiescent GSMF starts to dominate over the star-forming GSMF is a monotonically increasing function of redshift.

We have shown that local environment plays indeed a role in shaping galaxy evolution, in the redshift range 0 ≤ *z* ≤ 2. High-density environments show an enhanced fraction of massive (~10¹¹ M⊙) quiescent galaxies, compared to low-density ones. At high redshift, many structures may be at an earlier stage of formation, with clear environmental dependences not yet in place. Although present at *z* > 2, structures with a clear segregation of quiescent galaxies may be rare, requiring a larger area than the COSMOS field to be detected in a sufficient number. As at these redshifts also lower number statistics and larger uncertainties in the photometric redshift determination may start to affect the sample, these results would benefit from a confirmation by the means of wide-area spectroscopic redshift surveys such as Euclid and WFIRST.

## ACKNOWLEDGEMENTS

NM wishes to thank Micol Bolzonella, Iary Davidzon and Alice Mortlock for providing data points and suggestions that helped to significantly improve the paper. NM also wishes to thank Stéphane Arnouts for the many useful discussions regarding this work. This work is based on data products from observations made with ESO Telescopes at the La Silla Paranal Observatory under ESO programme ID 179.A-2005 and on data products produced by TERAPIX and the Cambridge Astronomy Survey Unit on behalf of the UltraVISTA consortium. We acknowledge the financial contributions by grants ASI/INAF I/023/12/0, PRIN MIUR 2010-2011 'The dark Universe and the cosmic evolution of baryons: from current surveys to Euclid', and PRIN MIUR 2015 'Cosmology and Fundamental Physics: illuminating the Dark Universe with Euclid'.

## REFERENCES

* Annunziatella M. et al., 2016, A&A, 585, A160

* Arnouts S. et al., 2002, MNRAS, 329, 355

* Arnouts S. et al., 2007, A&A, 476, 137

* Avni Y., Bahcall J. N., 1980, ApJ, 235, 694

* Baldry I. K., Glazebrook K., Brinkmann J., Ivezić Ž., Lupton R. H., Nichol R. C., Szalay A. S., 2004, ApJ, 600, 681

* Baldry I. K., Balogh M. L., Bower R. G., Glazebrook K., Nichol R. C., Bamford S. P., Budavari T., 2006, MNRAS, 373, 469

* Baldry I. K., Glazebrook K., Driver S. P., 2008, MNRAS, 388, 945

* Baldry I. K. et al., 2012, MNRAS, 421, 621

* Balogh M. L., Schade D., Morris S. L., Yee H. K. C., Carlberg R. G., Ellingson E., 1998, ApJ, 504, L75

* Balogh M. L., Christlein D., Zabludoff A. I., Zaritsky D., 2001, ApJ, 557, 117

* Balogh M. L., Baldry I. K., Nichol R., Miller C., Bower R., Glazebrook K., 2004, ApJ, 615, L101

* Blanton M. R., Moustakas J., 2009, ARA&A, 47, 159

* Blanton M. R., Eisenstein D., Hogg D. W., Schlegel D. J., Brinkmann J., 2005, ApJ, 629, 143

* Bolzonella M. et al., 2010, A&A, 524, A76

* Bower R. G., Benson A. J., Crain R. A., 2012, MNRAS, 422, 2816

* Bruzual G., Charlot S., 2003, MNRAS, 344, 1000

* Bundy K. et al., 2006, ApJ, 651, 120

* Calvi R., Poggianti B. M., Vulcani B., Fasano G., 2013, MNRAS, 432, 3141

* Calzetti D., Armus L., Bohlin R. C., Kinney A. L., Koornneef J., Storch-Bergmann T., 2000, ApJ, 533, 682

* Capak P. et al., 2007, ApJS, 172, 99

* Chabrier G., 2003, PASP, 115, 763

* Chiang Y.-K., Overzier R., Gebhardt K., 2013, ApJ, 779, 127

* Cimatti A. et al., 2002, A&A, 392, 395

* Cirasuolo M., McLure R. J., Dunlop J. S., Almaini O., Foucaud S., Simpson C., 2010, MNRAS, 401, 1166

* Coil A. L. et al., 2011, ApJ, 741, 8

* Cooper M. C., Newman J. A., Madgwick D. S., Gerke B. F., Yan R., Davis M., 2005, ApJ, 634, 833

* Cooper M. C. et al., 2006, MNRAS, 370, 198

* Cooper M. C. et al., 2010, MNRAS, 409, 337

* Cooper M. C. et al., 2012, MNRAS, 419, 3018

* Cucciati O. et al., 2006, A&A, 458, 39

* Cucciati O., Marulli F., Cimatti A., Merson A. I., Norberg P., Pozzetti L., Baugh C. M., Branchini E., 2016, MNRAS, 462, 1786

* Darvish B., Mobasher B., Sobral D., Scoville N., Aragon-Calvo M., 2015, ApJ, 805, 121

* Darvish B., Mobasher B., Sobral D., Rettura A., Scoville N., Faisst A., Capak P., 2016, ApJ, 825, 113

* Davidzon I. et al., 2013, A&A, 558, A23

* Davidzon I. et al., 2016, A&A, 586, A23

* Dressler A., 1980, ApJ, 236, 351

* Driver S. P. et al., 2011, MNRAS, 413, 971

* Drory N. et al., 2009, ApJ, 707, 1595

* Elbaz D. et al., 2007, A&A, 468, 33

* Etherington J., Thomas D., 2015, MNRAS, 451, 660

* Fontana A. et al., 2004, A&A, 424, 23

* Fontana A. et al., 2006, A&A, 459, 745

* Fontanot F., De Lucia G., Monaco P., Somerville R. S., Santini P., 2009, MNRAS, 397, 1776

* Fossati M. et al., 2015, MNRAS, 446, 2582

* Gabor J. M., Davé R., 2015, MNRAS, 447, 374

* Galametz A. et al., 2013, ApJS, 206, 10

* Gallazzi A. et al., 2009, ApJ, 690, 1883

* Garilli B. et al., 2014, A&A, 562, A23

* Gehrels N., 1986, ApJ, 303, 336

* Giodini S. et al., 2012, A&A, 538, A104

* Gómez P. L. et al., 2003, ApJ, 584, 210

* Guo Q. et al., 2011, MNRAS, 413, 101

* Guo Y. et al., 2013, ApJS, 207, 24

* Guzzo L. et al., 2014, A&A, 566, A108

* Haas M. R., Schaye J., Jeeson-Daniel A., 2012, MNRAS, 419, 2133

* Hahn C. et al., 2015, ApJ, 806, 162
Ilbert O. et al., 2006, A&A, 457, 841

Ilbert O. et al., 2009, ApJ, 690, 1236

Ilbert O. et al., 2010, ApJ, 709, 644

Ilbert O. et al., 2013, A&A, 556, A55

Jarrett T. H., Chester T., Cutri R., Schneider S., Skrutskie M., Huchra J. P., 2000, AJ, 119, 2498

Kauffmann G., White S. D. M., Heckman T. M., Ménard B., Brinchmann J., Charlot S., Tremonti C., Brinkmann J., 2004, MNRAS, 353, 713

Kodama T., Bower R., 2003, MNRAS, 346, 1

Kovač K. et al., 2010, ApJ, 708, 505

Lai C.-C. et al., 2016, ApJ, 825, 40

Le Fèvre O. et al., 2005, A&A, 439, 845

Lewis I. et al., 2002, MNRAS, 334, 673

Lilly S. J. et al., 2007, ApJS, 172, 70

Lilly S. J. et al., 2009, ApJS, 184, 218

Lin L. et al., 2016, ApJ, 817, 97

Lo Faro B., Monaco P., Vanzella E., Fontanot F., Silva L., Cristiani S., 2009, MNRAS, 399, 827

McCracken H. J. et al., 2010, ApJ, 708, 202

McCracken H. J. et al., 2012, A&A, 544, A156

McNaught-Roberts T. et al., 2014, MNRAS, 445, 2125

Malavasi N., Pozzetti L., Cucciati O., Bardelli S., Cimatti A., 2016, A&A, 585, A116

Marchesini D., van Dokkum P. G., Förster Schreiber N. M., Franx M., Labbé I., Wuyts S., 2009, ApJ, 701, 1765

Martin D. C. et al., 2007, ApJS, 173, 342

Mortlock A. et al., 2015, MNRAS, 447, 2

Moustakas J. et al., 2013, ApJ, 767, 50

Moutard T. et al., 2016a, A&A, 590, A102

Moutard T. et al., 2016b, A&A, 590, A103

Muldrew S. I. et al., 2012, MNRAS, 419, 2670

Muzzin A. et al., 2012, ApJ, 746, 188

Muzzin A. et al., 2013, ApJ, 777, 18

Nantais J. B. et al., 2016, A&A, 592, A161

Peng Y.-j. et al., 2010, ApJ, 721, 193

Polletta M. et al., 2007, ApJ, 663, 81

Pozzetti L. et al., 2007, A&A, 474, 443

Pozzetti L. et al., 2010, A&A, 523, A13

Scoville N. et al., 2007, ApJS, 172, 1

Scoville N. et al., 2013, ApJS, 206, 3

Shectman S. A., Landy S. D., Oemler A., Tucker D. L., Lin H., Kirshner R. P., Schechter P. L., 1996, ApJ, 470, 172

Tomczak A. R. et al., 2014, ApJ, 783, 85

Treu T., Ellis R. S., Kneib J.-P., Dressler A., Smail I., Czoske O., Oemler A., Natarajan P., 2003, ApJ, 591, 53

van der Burg R. F. J. et al., 2013, A&A, 557, A15

Vulcani B. et al., 2011, MNRAS, 412, 246

Vulcani B. et al., 2012, MNRAS, 420, 1481

Vulcani B. et al., 2013, A&A, 550, A58

Vulcani B., De Lucia G., Poggianti B. M., Bundy K., More S., Calvi R., 2014, ApJ, 788, 57

York D. G. et al., 2000, AJ, 120, 1579

## APPENDIX: A TEST ON THE EFFECT OF PHOTOMETRIC REDSHIFT UNCERTAINTIES

As discussed in Malavasi et al. (2016), photometric redshift uncertainties are a major limitation in the reconstruction of the density field. Nevertheless, in Malavasi et al. (2016), we demonstrated that it is still possible to perform a study of the GSMF in different environments using photometric redshifts provided that their uncertainty is small ($\sigma_{\Delta z/(1+z)} \lesssim 0.01$). In this case, differences between high- and low-density environments that are present in the GSMF calculated using each galaxy’s true redshift up to $z \sim 2.5$ result damped when using photometric redshifts, but they will still be recovered. Following what shown in Malavasi et al. (2016) and as explained in Section 2.1, we have chosen for this work an uncertainty value for the photometric redshifts of $\sigma_{\Delta z/(1+z)} = 0.01$ that may be, nevertheless, underestimated at high redshift. Moreover, the photometric redshift uncertainty depends on $K_S$-band magnitude, as shown, for example, in fig. 2 of Scoville et al. (2013). This figure shows the photometric redshift uncertainty as a function of $K_S$-band magnitude and redshift, together with the median $K_S$-band magnitude of a sample of galaxies extracted from the UltraVISTA Survey and similar to the one used in this work. For this reason, we have tested also the effect of a larger uncertainty. Following fig. 2 of Scoville et al. (2013), we have redone our work assuming a photometric redshift uncertainty of

$$ \sigma_{\Delta z/(1+z)} = \begin{cases} 0.01 & \text{for } z \le 1.5 \\ 0.03 & \text{for } z > 1.5 \end{cases} \qquad (A1) $$

As the main purpose of this paper is to study the shape of the GSMF in different environments, we tested whether the differences that we see between the GSMF in high- and low-density environments (Figs 7 and 8) are maintained when considering larger errors at higher redshift. Because in the case of the ratio between the intermediate- and low-mass ends of the GSMF, we limit the analysis at $z \le 1.5$, we only report for comparison in Fig. A1 the ratio of the high-mass to the intermediate-mass end of the GSMF (see Fig. 7), performed with the higher photometric redshift uncertainty value at high redshift. It can be seen how, even with larger photometric redshift errors, the trends are maintained without significant differences. The increase in the redshift error affects only the analysis at $z \ge 1.5$ and in a negligible way.
<figure><img src="image_13.png" /><figcaption>**Figure A1.** Shape of the GSMF – increased errors. As Fig. 7, but with larger photometric redshift errors at high redshift (see the text). Increased errors for photometric redshifts at high redshift do not affect the results of this work.</figcaption></figure>

This paper has been typeset from a TeX/LATEX file prepared by the author.