# Comparing pymorph and SDSS photometry – II. The differences are more than semantics and are not dominated by intracluster light

M. Bernardi, J. -L. Fischer, R.K. Sheth, A. Meert, M. Huertas-Company, F. Shankar, V. Vikram

<figure><img src="image_1.png" /></figure>

## To cite this version:

M. Bernardi, J. -L. Fischer, R.K. Sheth, A. Meert, M. Huertas-Company, et al.. Comparing pymorph and SDSS photometry – II. The differences are more than semantics and are not dominated by intracluster light. Monthly Notices of the Royal Astronomical Society, 2017, 468 (3), pp.2569-2581. 10.1093/mnras/stx677. hal-01582392

HAL Id: hal-01582392

https://hal.science/hal-01582392v1

Submitted on 26 Aug 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# Comparing PYMORPH and SDSS photometry – II. The differences are more than semantics and are not dominated by intracluster light

M. Bernardi,<fnref n="1" />★ J.-L. Fischer,<fnref n="1" /> R. K. Sheth,<fnref n="1" /> A. Meert,<fnref n="1" /> M. Huertas-Company,<fnref n="2" />
F. Shankar<fnref n="3" /> and V. Vikram<fnref n="1" />

Accepted 2017 March 15. Received 2017 March 2; in original form 2017 January 13

## ABSTRACT

The Sloan Digital Sky Survey (SDSS) pipeline photometry underestimates the brightnesses of the most luminous galaxies. This is mainly because (i) the SDSS overestimates the sky background, and (ii) single-component or two-component Sérsic-based models better fit the surface brightness profile of galaxies, especially at high luminosities, than the de Vaucouleurs model used by the SDSS pipeline. We use the PYMORPH photometric reductions to isolate effect (ii) and show that it is the same in the full sample as in small group environments, and for satellites in the most massive clusters as well. None of these are expected to be significantly affected by intracluster light (ICL). We only see an additional effect for centrals in the most massive haloes, but we argue that even this is not dominated by ICL. Hence, for the vast majority of galaxies, the differences between PYMORPH and SDSS pipeline photometry cannot be ascribed to the semantics of whether or not one includes the ICL when describing the stellar mass of massive galaxies. Rather, they likely reflect differences in star formation or assembly histories. Failure to account for the SDSS underestimate has significantly biased most previous estimates of the SDSS luminosity and stellar mass functions, and therefore halo model estimates of the $z \sim 0.1$ relation between the mass of a halo and that of the galaxy at its centre. We also show that when one studies correlations, at fixed group mass, with a quantity that was not used to define the groups, then selection effects appear. We show why such effects arise and should not be mistaken for physical effects.

**Key words:** galaxies: clusters: intracluster medium – galaxies: formation – galaxies: fundamental parameters – galaxies: photometry.

## 1 INTRODUCTION

The observed magnitudes reported by the Sloan Digital Sky Survey (SDSS) are underestimated, especially at the highest luminosities (e.g. Bernardi et al. 2007; von der Linden et al. 2007; Bernardi et al. 2013; D’Souza, Vegetti & Kauffmann 2015; Meert et al. 2015; Bernardi et al. 2017). In a companion paper (Fischer, Bernardi & Meert 2017), we show that this is due to a combination of sky background and model fitting effects: (i) The SDSS overestimates the sky background (Blanton et al. 2005; Bernardi et al. 2007; Hyde & Bernardi 2009; Blanton et al. 2011; Meert et al. 2015), and (ii) single- or two-component Sérsic model based estimates of galaxy luminosities are more reliable than estimates based on

single-exponential, single de Vaucouleurs, or a linear combination of the best-fitting exponential and de Vaucouleurs models (the so-called *cModel* magnitudes) used by the SDSS pipeline (e.g. Bernardi et al. 2010; Bernardi et al. 2014). Fischer et al. show that PYMORPH sky estimates are fainter than those of the SDSS DR7 or DR9 pipelines, but are in excellent agreement with the estimates of Blanton et al. (2011). The PYMORPH Sérsic-based estimates (Meert, Vikram & Bernardi 2013, 2015, 2016) are more reliable and return more light than SDSS estimates, and the difference is most pronounced at the highest luminosities. This can have a dramatic impact on the estimated $z \sim 0.1$ luminosity and stellar mass functions (Bernardi et al. 2013, 2016, 2017), and hence on halo model- (Cooray & Sheth 2002) based estimates of the relation between stellar and halo mass (Kravtsov, Vikhlinin & Meshscheryakov 2014; Shankar et al. 2014). They also affect models for the formation of the massive galaxies because they affect estimates of the mass

<footnote n="1">¹Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA 19104, USA</footnote>

<footnote n="2">²Observatoire de Paris (GEPI), CNRS and Université Paris Diderot, 4 Rue Thomas Mann, F-75013 Paris, France</footnote>

<footnote n="3">³School of Physics and Astronomy, University of Southampton, Southampton SO17 1BJ, UK</footnote>

<footnote>* E-mail: bernardm@sas.upenn.edu</footnote>
scale on which galaxy scaling relations show curvature (Bernardi et al. 2011).

The most luminous galaxies reside in or at the centres of clusters. Since clusters are known to possess intercluster light (hereafter ICL), it is natural to ask if the PYMORPH Sérsic-based photometry, with its improved sky estimates, is brighter than the SDSS measurements (i.e. Model or cModel magnitudes) because it includes more of this ICL component. Previous work, based on stacked images of central galaxies of massive clusters, suggests that the ICL contributes only substantially to the surface brightness profile on scales larger than 50 kpc (Zibetti et al. 2005), where the surface brightness has dropped below about 27 mag arcsec⁻². A stacking analysis of LRGs suggests that this scale may be even larger for lower mass groups (Tal & van Dokkum 2011). For the vast majority of central galaxies in the SDSS, this corresponds to a scale where the surface brightness has dropped to less than 1 per cent of the sky brightness, making it extremely difficult to detect in individual images. While this means that it is very unlikely that the ICL contributes substantially to the PYMORPH reductions, in what follows, we provide additional evidence that the PYMORPH–SDSS difference is unlikely to be due to ICL.

Our logic is simple: Since the ICL should be centred on the cluster centre, and is expected to be fainter at larger cluster-centric distances, it is reasonable to suppose that it will affect the photometry of the central galaxy more than the satellites. The ICL is also expected to be fainter around central galaxies in less massive groups. Therefore, we study whether the difference between PYMORPH and SDSS magnitudes depends on whether the galaxy is a central or a satellite, as well as whether it depends on cluster mass. Section 2 describes our galaxy sample and shows why the ICL is unlikely to play a major role in the images of individual galaxies. Section 3 describes the group catalogues we use for separating centrals from satellites. Section 4 presents the main results of our analysis and Section 5 summarizes our findings. While the main text presents the results using the Sérsic-exponential photometry, Appendix A shows a similar analysis for the single-component Sérsic fits since these are much more common in the literature. Finally, Appendix B discusses a pernicious selection effect that arises when working with group catalogues, and which we were careful to avoid.

When necessary, we assume a spatially flat background cosmology with parameters ($\Omega_m, \Omega_\Lambda$) = (0.3, 0.7), and a Hubble constant at the present time of $H_0 = 70 \text{ km s}^{-1} \text{ Mpc}^{-1}$.

# 2 COMPARISON OF PYMORPH AND SDSS LUMINOSITIES: THE MAIN GALAXY SAMPLE

## 2.1 The parent galaxy sample

The analysis that follows is based on the main galaxy sample of the Ninth Data Release of the Sloan Digital Sky Survey (hereafter SDSS DR9; Aihara et al. 2011); the sample is limited to a Petrosian r-band apparent magnitude of $m_{rPet} \le 17.7$. (Our conclusions are unchanged if we use DR7 values instead of DR9.) To about ~660 000 of these (those at $z \le 0.25$), Huertas-Company et al. (2011) have assigned Bayesian automated morphological classification weights that represent the probabilities that the galaxy is elliptical, S0, Sab or Scd. The vast majority of the most luminous galaxies are Es or S0s, so, in what follows, we define the E+S0 weight to equal $p(E+S0) \equiv p(E) + p(S0)$.

For every DR9 galaxy, the SDSS pipeline provides Model magnitudes, which are based on separately fitting an exponential and a de Vaucouleurs form to the surface brightness profile and choosing

the value returned by the model that fits best. Hence, for essentially all E+S0s, the Model magnitude is that from the de Vaucouleurs fit. (Our conclusions are unchanged if we use cModel magnitudes.)

We will compare these Model magnitudes with single-component Sérsic and two-component Sérsic-exponential fits (hereafter Ser and SerExp, respectively) to these same objects provided by Meert et al. (2015). These Ser and SerExp fits are returned by the PYMORPH algorithm, which is described and tested in Meert et al. (2013, 2015, 2016) and used by Bernardi et al. (2013, 2014, 2016, Bernardi et al. 2017). Although those tests were based on DR7 objects, Fischer et al. (2017) show that the difference between DR7 and DR9 is negligible for PYMORPH, i.e. the parameters provided by Meert et al. (2015) for DR7 can also be used for DR9. The two-component SerExp fits are the most accurate (i.e. least biased) of the PYMORPH outputs (Meert et al. 2013; Bernardi et al. 2014). For this reason, we use these in the main text. However, single-component fits are much more common in the literature; we show our analysis of Ser photometry in Appendix A.

We also present our results as a function of stellar mass $M_*$. For a given luminosity (e.g. Model, or SerExp), we obtain $M_*$ by multiplying L by the dust-free $M_*/L$ estimates of Mendel et al. (2014) assuming a Chabrier (2003) initial mass function. Bernardi et al. (2017) describe a number of other reasonable choices of $M_*/L$. The results that follow are robust to changes in this choice.

## 2.2 Example profiles and fits

Before showing results of a statistical analysis, it is useful to look at a few representative images. This helps build intuition for what it is that we will be averaging. Our aim here is to directly address what previous work based on stacked images implies for our analysis of individual images. We are particularly interested in the effects of the ICL. If ICL is defined as being an excess above the best-fitting dev profile, then it is expected to contribute significantly only on scales larger than 50 kpc (Zibetti et al. 2005). On the other hand, if ICL is a departure from the best-fitting Ser profile, then the relevant scale may be about a factor of 2 larger (Tal & van Dokkum 2011).

Figs 1–3 show the surface brightness profiles of three luminous ($M_r \le -23$) galaxies in the SDSS main galaxy sample, which are at a range of distances ($z = 0.06, 0.15$ and $0.25$). As our primary interest here is in the role of the ICL, all three objects are central galaxies in the group catalogues we use extensively later in this paper. In all cases, the top panels show the image, the middle panels show the best 2D PYMORPH SerExp fit and residuals from it, and the bottom panels show 1D angular averages of the profile, the fit and residuals (the latter are computed as fit-data). The 2D residuals show the image including sky, whereas in those at the bottom, the sky has been subtracted. These bottom panels are for illustration only, as the fitting was done in 2D with more care taken to remove light associated with the other objects in the field. The legends on either side provide a wealth of information about the parameters of the fit (see Meert et al. 2015 for details). In all cases, the image cutouts are labelled in units of arcseconds, and the colour scale is in mag arcsec⁻².

Fig. 1 shows a galaxy that was selected because of its large apparent brightness ($m_r \sim 14.3$), so that the dynamic range between the half-light radius $R_{hl}$ and the scale $R_{1\text{per cent sky}}$ on which the profile has dropped to 1 per cent of sky is large. The SerExp absolute magnitude of this object is ~−23 and $R_{hl} \approx 13$ kpc. For this object, $R_{1\text{per cent sky}} \approx 50$ kpc; note that there is no obvious feature in the profile shape on scales smaller than this. Indeed, a single Ser fit to this object has $n_{\text{Ser}} = 5$ and is not very different
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Image (top panels) and 2D SerExp fit and residuals (middle panels) of an object for which the SerExp fit indicates a half-light radius $R_{hl} = 12.68$ kpc. The axis labels are in arcseconds and the colour scale is in mag arcsec$^{-2}$ (the residuals are computed as fit-data). The legends on either side provide a wealth of information about the parameters of the fit. The 2D data images are shown with background sky included, while the 1D angular average profile shown in the bottom panel (black points) is computed using the background subtracted data. The dot-dashed vertical lines in the bottom panels show the half-light radius ($R_{hl}$ in arcseconds) and its multiples. Dashed and dotted horizontal lines show the measured sky level and 1 per cent of its value, respectively. The surface brightness profile of this galaxy drops to 1 per cent of sky at a scale that is about 4 × $R_{hl}$. Whereas the fitting is done in 2D, and accounts for the profiles of the other objects in the field, the residuals – and the 1D angular averages shown in the bottom panel – do not. Hence, one should resist the temptation to associate the fact that the data in the bottom panels are slightly brighter than the SerExp fit with ICL; some of the apparent excess is due to the extended profiles of the other objects in the group, rather than ICL. A single Ser fit to this object has $n_{Ser} = 5$. The bulge component of the SerExp fit is very similar to a single deV fit. The departure from a deV profile is observed at ~1$R_{hl}$ ~ 13 kpc, which is much smaller than the 50–100 kpc scale expected for the ICL.</figcaption></figure>

from the best-fitting SerExp profile shown by the solid curve in the bottom left-hand panel. The dashed curve in the same panel shows the bulge component of the SerExp fit (a single Sérsic profile with $n \approx 4$, meaning that this bulge component has a deV profile). Beyond about 10 arcsec from the centre – i.e. beyond about $R_{hl}$ – the second component (dotted curve) is clearly necessary. Note, in particular, that this second component, which describes light in excess of a deV profile, is necessary on scales that are much smaller than the 50 kpc associated with the ICL.

Fig. 2 shows a more luminous ($M_r = -23.65$) and distant ($z = 0.15$) central galaxy with $m_r \approx 16$. For this object too, the bulge component is not too different from a deV profile, but the second component is necessary even on scales as small as 15 kpc. A single Ser profile with $n = 5.2$ also provides a good fit on all scales smaller than $R_1$ per cent sky.

Fig. 3 shows another object of similar luminosity, but at even higher redshift, so its apparent magnitude is significantly fainter ($m_r \sim 17.4$). Comparison with the bottom left-hand panels of the previous figures shows the reduced dynamic range, which makes

it more difficult to detect departures from a single Ser fit, which, in this case, has $n_{Ser} = 7.9$. As for the previous two examples, the bulge is closer to deV, and the need for a second component is already evident on scales of the order of 20 kpc.

Thus, to the extent that these galaxies are representative of all central galaxies, these figures make two points. First, departures from a deV profile are almost always detected with high significance (see also e.g. Gonzalez, Zabludoff & Zaritsky 2005; Bernardi et al. 2007). Moreover, a pure deV profile becomes a poor fit on much smaller scales than is reasonable to associate with the ICL. Secondly, if the ICL is associated with departures from a single Ser (rather than deV) fit, then if this occurs, it is at surface brightnesses that are too faint to be seen in individual images. Therefore, the departures from a pure deV profile, which PYMORPH detects in its Ser or SerExp fits and which are the subject of this paper, likely reflect differences in star formation or assembly histories, rather than ICL.

Having illustrated that differences from deV photometry are common, we now turn to a statistical analysis of these differences.
<figure><img src="image_5.png" /><figcaption>**Figure 2.** Same as the previous figure, but now for a more luminous object that is more distant. In this case, the *SerExp* fit has $R_{hl} = 13.37$ kpc; $R_{1\,\text{per cent sky}}$ is four times larger. Inside $R_{1\,\text{per cent sky}}$, a single *Ser* profile with $n_{\text{Ser}} = 5.2$ also provides a good fit (not shown).</figcaption></figure>

## 2.3 Comparison of SDSS and PYMORPH PIPELINES

Fig. 4 shows a comparison of SDSS Model and PYMORPH SerExp magnitudes; a similar analysis using Ser magnitudes is shown in Appendix A. Here, as in most of the figures that follow, objects have been weighted by $p(E+S0)$. This ensures that we are working with a sample for which Model = de Vaucouleurs, and removes the question of how the morphological mix affects the PYMORPH–SDSS comparison.

Symbols with error bars show the median and the error on it (only bins with more than 50 objects are shown), and dashed curves show the region that encloses 68 per cent of the sample in each absolute magnitude bin. The SDSS Model magnitudes are increasingly fainter as luminosity increases. Some of the differences in Fig. 4 are simply due to the fact that SDSS magnitudes are based on integrating the fitted profile to approximately 7.5× the semimajor axis $a_e$, whereas PYMORPH does not truncate. For this reason, we have shown the result of truncating the PYMORPH fits as well: This makes them fainter (by less than 0.1 mag on average), but other than this shift, the overall trends with luminosity are unchanged. Therefore, truncation is not the primary reason why SDSS is fainter (see sections 2.3 and 2.4 in Fischer et al. 2017 for more discussion on truncation).

Fischer et al. (2017) show that once truncation has been accounted for, there are two effects that contribute to biasing SDSS low (rather than biasing PYMORPH high). These are due to differences in how the background sky is estimated, and what model is fitted to the surface

brightness profile. Regarding the first effect, there is now general consensus that the SDSS treatment of the sky is flawed. This affects not only nearby galaxies (as emphasized by Blanton et al. 2011) but also high-luminosity galaxies that have relatively large angular sizes and tend to be in crowded fields (Fischer et al. 2017, and references therein). PYMORPH attributes less light to the sky than the SDSS; as a result, it assigns more light to the galaxy than the SDSS. Fischer et al. (2017) also show that PYMORPH sky estimates are in excellent agreement with those of Blanton et al. (2011). About half of the bias at high luminosities arises from fitting different models (the second effect). At the high-luminosity end, most galaxies are E+S0s, for which the SDSS Model magnitudes are essentially *de V* magnitudes, and *de V* fits return less light than SerExp or Ser fits.

## 3 GALAXY GROUPS IN THE SDSS

Our goal is to check if the differences between PYMORPH and Model photometry shown in Fig. 4 depend on whether a galaxy is a central or a satellite. To achieve our goal, we use two group catalogues in which centrals and satellites have been identified. One of these is from Yang et al. (2007) (the DR7 version, hereafter Yang+) and the other is the redMapper sample of Rykoff et al. (2014). Whereas the former identifies groups spanning a wide range of masses, the latter identifies only the very most massive clusters (the redMapper algorithm is not well suited for identifying lower mass/richness groups).
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Same as the previous figure, but now for an object of similar luminosity that is even more distant. In this case, $R_{hl} = 18.07$ kpc and $R_{1\ per\ cent\ sky}$ is three times larger. A single *Ser* profile with $n_{Ser} = 7.9$ provides a good fit inside $R_{1\ per\ cent\ sky}$ (not shown).</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 4.** At high luminosities, the mean magnitude difference between SDSS *Model* and PYMORPH *SerExp* magnitudes depends strongly on which quantity is used as reference. Truncating the PYMORPH magnitudes similarly to what is done for the SDSS makes them fainter by ≤0.1 mag on average; it is not the dominant effect.</figcaption></figure>

## 3.1 Description of group catalogues

The *redMapper* sample is drawn from the SDSS DR8 footprint, which covers ~10 000 deg². Groups are identified on the basis of angular positions and colour. Of the 1.7 × 10⁶ objects in 2.6 × 10⁴ groups in the *redMapper* sample, Meert et al. provide only

PYMORPH reductions for 1.8 × 10⁴: These are the subsets that have spectroscopic information and were in the SDSS DR7 ~7700 deg² footprint. These objects are in about 3400 clusters, of which about 2400 are at $z \le 0.25$ and 1200 are at $z \le 0.2$.

In contrast, Yang+ identify 6 × 10⁵ galaxies in about 4.7 × 10⁵ groups at $z \le 0.2$ in the SDSS DR7 footprint. Most of these are much less massive groups, of course. Yang+ defined their catalogue using a complex iterative procedure that makes use of observed angular positions, redshifts and *Model* photometry. They estimate the halo masses $M_{Halo}$ of their groups using two simple proxies for halo mass. One is based on summing up the *Model* luminosities of the central+satellite galaxies. As a result, there is a tight correlation between $M_{Halo}$ and *Model*, which will be important below (see also Appendix B). The other is based on summing their stellar masses, which they estimate by multiplying the *Model* luminosities by a simple colour-based estimate of $M_*/L$ (see their equation 2). We use their luminosity-based estimate, but the results that follow are robust to changes in this choice.

The Yang+ catalogue extends only out to $z = 0.2$, so our first step was to identify objects that appear in the *redMapper* catalogue as well. There are 13 253 such objects, of which 8081 have $p(E+S0) \ge 0.7$. Fig. 5 shows the distribution of these objects in the $M_{Halo}-Model$ magnitude plane.

Red and green symbols show objects that Yang+ classify as centrals and satellites, respectively. Note that the centrals define a rather tight $M_{Halo}-Model$ magnitude correlation, which has a
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Distribution of E+S0s that are in both the Yang+ and the redMaPPer catalogues. The Yang+ definition of $M_{\text{Halo}}$ means that there is a tight correlation between $M_{\text{Halo}}$ and Model magnitudes, especially at smaller masses. Many of the objects that Yang+ classify as being centrals in groups less massive than $10^{14} M_{\odot}$ are called satellites by redMaPPer (the redMaPPer satellites are all the points that are not brown). Only above $10^{14} M_{\odot}$ do the two groups agree on the central-satellite classification.</figcaption></figure>

sharp boundary. This is because of the Yang+ definition of $M_{\text{Halo}}$, and will be important later (see discussion related to Figs 7 and 8, and Appendix B). More important in the present context are the brown symbols, which show the centrals in redMaPPer (the redMaPPer satellites are all the points that are not brown). Below $M_{\text{Halo}} \sim 10^{14} M_{\odot}$, only a small fraction of the Yang+ centrals are also centrals in redMaPPer (i.e. redMaPPer labels most Yang+ centrals as satellites – remember that this is the subsample of galaxies that is common to both group catalogues). If Yang+ are correct, then redMaPPer is wrongly linking together objects that are really in separate haloes – this will compromise efforts to use redMaPPer to address assembly bias like effects. On the other hand, if redMaPPer is correct, then the Yang+ misclassifications will make centrals and satellites seem more similar than they really are. This disagreement between the two groups complicates any attempt to draw unambiguous conclusions about the central-satellite difference in lower mass haloes.

In view of the central-satellite disagreement, in what follows, we work primarily with the Yang+ objects that Yang+ classified as being in haloes more massive than $10^{14} M_{\odot}$. While Fig. 5 suggests that this is a reasonable choice, things are not completely straightforward, since only about one in three of the Yang+ centrals is in the redMaPPer catalogue. We show this in Fig. 6, where $N$ is the total number of objects in each catalogue (i.e. not the small subsets that were common to both catalogues) weighted by $p(E+S0)$. [Since we have only BAC weights at $z \le 0.25$, we set $p(E+S0) = 1$ for all objects at $z > 0.25$, since this is very likely to be realistic. We have checked that making cuts on concentration index or colour smooths out the small plateau at $z = 0.25$ in the grey histogram, but makes essentially no difference to the others.] The bottom panel includes only galaxies at $z \le 0.25$. Remember that the Yang+ catalogue includes only objects at $z < 0.2$. Restricting the full sample (grey histogram) to galaxies with $z \le 0.2$ gives the same distribution as for the full Yang+ catalogue (black histogram). For completeness, the bottom panel also shows the subsample of the centrals in the redMaPPer catalogue at $z < 0.2$ (brown dotted line). (We show the number of objects $N$, rather than the comoving number density, since our goal is to show how many objects contribute to each bin in the figures that follow.)

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 6.** Redshift and luminosity distributions of objects in the group catalogues we use. Grey histograms in the two panels show the distribution in the full sample. Other linestyles are as indicated in the legend in the bottom panel. At $z \le 0.25$, objects were weighted by the probability of being an E+S0. Only $z \le 0.25$ objects were used to make the histograms in the bottom panel. Note that the Yang+ catalogue includes only galaxies at $z \le 0.2$; restricting the full sample (grey histogram) to galaxies with $z \le 0.2$ gives the same distribution as for the full Yang+ catalogue (black histogram). Note that the vast majority of objects having $-23.4 \le M_r \le -22.4$ are centrals in groups less massive than $10^{14} M_{\odot}$. Using the Yang+SerExp catalogue (described in Section 3.2) and plotting versus SerExp magnitudes instead mainly shifts the counts to brighter magnitudes, but the differences between the samples remain.</figcaption></figure>

Note that, in the bottom panel, there are three times more centrals in Yang+ than in redMaPPer, even when restricting Yang+ to $M_{\text{Halo}} \ge 10^{14} M_{\odot}$. This remains true if we use the Yang+SerExp catalogue (described in Section 3.2) and show the counts as a function of SerExp magnitudes instead (the main difference is a shift to brighter magnitudes). We assume that the reason for this difference (i.e. the factor of 3×) is not the central-satellite designation, but the group richness: Presumably, many of the Yang+ objects are centrals of lower richness groups that failed the redMaPPer richness cut. In partial support of this, we note that the number of redMaPPer satellites is nearly the same as Yang+, despite having three times fewer centrals; this is consistent with them being more massive. In what follows, we use all the redMaPPer objects within $z \le 0.25$. We do not include higher $z$ objects because we do not have E+S0 classifications above $z = 0.25$. (We have checked, but do not show here, that including the red and/or high-concentration
objects from $z \ge 0.25$ makes no difference to our results, other than to improve the statistical significance.) Moreover, Yang+ extends only to $z = 0.2$, and we did not want questions of evolution being different in the two samples to complicate our results.

Finally, the magenta curves show the distribution of ~5500 E+S0 central galaxies in the MaxBCG catalogue of Rykoff et al. (2012) restricted to $z \le 0.25$ and that have PYMORPH reductions from Meert et al. (2015). At the highest luminosities, their comoving density is greater than redMaPPer but smaller than the $\ge 10^{14} M_{\odot}$ Yang+ centrals, so we expect them to be intermediate in mass as well. This is our primary reason for including this sample.

## 3.2 Combining PYMORPH with Yang+

We remarked in the previous subsection that Model photometry plays an important role in the Yang+ catalogue. This raises the question of how the catalogue is modified if we use PYMORPH photometry instead. Although the appropriate thing to do is to re-run the algorithm, this is well beyond the scope of this study. Instead, we have performed the following simple but reasonable procedure.

For each Yang+ group, we assume that although changing the photometry may change the central-satellite designation within a group, it will not change the group membership. We then replace the Model photometry with PYMORPH values for each group member, and sum these new values to obtain a new estimate of the total group luminosity. We rank-order this quantity. Since this new ordering is different from that based on Model photometry, we reassign halo masses based on this new rank ordering. We also define the most luminous galaxy in a group as the central. Doing this separately for SerExp and Ser photometry gives us the Yang+SerExp and Yang+Ser catalogues. For both catalogues, the set of halo masses associated with the original Yang+ catalogue is unchanged (by construction), but the mapping between $M_{Halo}$ and central galaxy luminosity is modified.

The main effect to a plot like Fig. 5 is to shift the points slightly to brighter magnitudes – we explore the consequences of this for halo model like analyses elsewhere – but the motivation for splitting the sample at $M_{Halo} \sim 10^{14} M_{\odot}$ remains. Since the shifts are small, we have not included a plot showing, for example, $M_{Halo}$ versus SerExp magnitude in the Yang+SerExp catalogue. However, defining the Yang+SerExp and Yang+Ser catalogues is important for what follows (see also Appendix B).

# 4 COMPARISON OF CENTRALS AND SATELLITES OVER A RANGE OF GALAXY AND HALO MASSES

## 4.1 Centrals and satellites in poor groups

Fig. 7 shows the difference between SerExp and Model photometry, as a function of Model magnitude. (The SerExp magnitudes are based on truncating the fits at $7.5a_e$, as described previously.) Grey symbols show the full E+S0 sample; red and green curves show the subset of galaxies identified as being centrals (red curve) and satellites (green curve) in haloes having masses between $10^{13}$ and $10^{14} M_{\odot}$ in the original Yang+ catalogue. They are both very similar to the grey curve, which shows the median trend for all E+S0 galaxies (from the top panel of Fig. 4). Hence, either the Yang+ central-satellite designations are completely random, or the difference between Model and SerExp magnitudes does not depend on whether a galaxy is a central or a satellite. If the latter, then either these groups are too low mass to have a significant ICL

<figure><img src="image_11.png" /><figcaption>**Figure 7.** Difference between Model and SerExp (truncated) magnitudes for galaxies in Yang+ having group masses between $10^{13}$ and $10^{14} M_{\odot}$, as a function of Model magnitudes. The median difference defined by all the E+S0 galaxies (grey curve; same as corresponding curve in top panel of Fig. 4) is significantly different from zero and is almost exactly the same for centrals (red curve) as for satellites (green curve).</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Figure 8.** Same as the previous figure, but now shown as a function of SerExp magnitude in the Yang+SerExp catalogue. As in the previous figure, centrals and satellites are similar to the bulk of the population, even though the median difference is larger than in Fig. 7.</figcaption></figure>

component or what ICL is present does not play an important role in the PYMORPH–SDSS difference. In either case, the PYMORPH–SDSS difference, at least for galaxies in groups less massive than $10^{14} M_{\odot}$, is real – it is not just semantics.

Although Fig. 4 indicates that the difference depends strongly on which quantity is used as reference, we show in Appendix B that simply plotting versus SerExp magnitudes (instead of Model) is biased by a selection effect that arises because $M_{Halo}$ in the Yang+ group catalogue is tightly correlated with Model magnitude (cf. Fig. 5), but less so with SerExp. For this reason, Fig. 8 shows the difference in the Yang+SerExp catalogue, where $M_{Halo}$ is correlated with SerExp rather than Model. (For a given group, the central-satellite designation in Yang+SerExp may be different from that in Yang+. Defining the ‘central’ as the ‘brightest’ is important: Using the Yang+ designation here produces a noticeable bias, see Fig. B1.) Note, in particular, that although the median difference (grey curve) is larger than in Fig. 7, centrals and satellites are still remarkably similar. Hence, the agreement of the red and
<figure><img src="image_13.png" /><figcaption>**Figure 9.** Difference between (truncated) SerExp and Model photometry, as a function of Model (left-hand panel) and SerExp magnitude (right-hand panel). Grey symbols show the full E+S0 sample; red and green symbols show the subset of these E+S0s that are centrals and satellites in groups whose halo mass is greater than $10^{14}$ M$_\odot$ in the Yang+ (left-hand panel) and Yang+SerExp (right-hand panel) catalogues; brown and blue symbols show corresponding measurements in the redMaPPer catalogue; and magenta symbols show the MaxBCG centrals. In the left-hand panel, satellites are similar to the full sample, whereas centrals tend to be brighter. However, differences between centrals and satellites are substantially smaller than the differences between SerExp and Model photometry except for the most massive (i.e. redMaPPer) groups. In the right-hand panel, centrals and satellites are more similar even though the median offset from zero is larger.</figcaption></figure>

green lines with the grey one in Figs 7 and 8 strongly suggests that centrals and satellites are similar, and that the PYMORPH-SDSS difference is not just semantics.

In this context, it is interesting to note that luminous red galaxies (LRGs) are believed to populate groups having masses of a few times $10^{13}$ M$_\odot$. This is similar to the mass scale we are considering here. For LRGs at $z \sim 0.34$, a stacking analysis shows that the characteristic scale where the ICL becomes apparent is $\sim 100$ kpc (Tal & van Dokkum 2011), where the surface brightness is 28 mag arcsec$^{-2}$. Accounting for surface brightness dimming between $z \sim 0.15$ and $z = 0.34$ would make this 27.4 mag arcsec$^{-2}$. This is substantially fainter than $\sim 26$ mag arcsec$^{-2}$, which corresponds to $\sim 1$ per cent of the sky value in the individual r-band SDSS images of the objects that contribute to Figs 7 and 8, i.e. the individual images will show no sign of the ICL, so it is very unlikely that PYMORPH's Sérsic-based fits are sensitive to it. And, indeed, Figs 1 and 2 show that the PYMORPH fits are rather good within $R_1$ per cent sky. Stated differently, the ICL component in the LRG stacks shows up as excess light compared to a Sérsic profile (with $n = 5.5$) on scales larger than $\sim 8R_\odot$, but Figs 1 and 2 show that the individual images typically have $R_1$ per cent sky $\approx 4R_\odot$, which is why they show no sign of the ICL.

Finally, note that a de Vaucouleurs profile is not a good fit to the regions within $\sim 8R_\odot$, neither for LRGs nor for the galaxies we are considering here. Thus, if one thinks of a Sérsic profile as describing light in excess of a de Vaucouleurs profile, then this excess light is not ICL. The ICL shows up as an additional departure from a Sérsic profile beyond $8R_\odot$. This is consistent with our assertion above: At least for galaxies in groups less massive than $10^{14}$ M$_\odot$, the PYMORPH-SDSS difference is not due to the ICL.

Now, the PYMORPH-SDSS difference for the full sample of galaxies (grey curve) is similar to that for these centrals and satellites. Since the vast majority of these are in lower mass haloes, ICL effects do not play a role for the vast majority of objects that contribute to the grey curve. These are the objects that Bernardi et al. (2013, 2016, Bernardi et al. 2017) used to estimate the $z \sim 0.1$ luminosity and stellar mass functions. Hence, the differences between these mass functions and those based on SDSS pipeline photometry are real:

It is incorrect to attribute it to the semantics of whether or not one includes the ICL when estimating the light from a galaxy.

## 4.2 Centrals and satellites in massive clusters

We now consider galaxies in more massive groups. Fig. 9 shows the difference between SerExp and Model photometry, as a function of Model (left-hand panel) and SerExp magnitudes (right-hand panel). The grey curves in the two panels are for the full sample and are taken from Fig. 4, as before. Red and green symbols and curves show the subset of galaxies that Yang+ identify as being centrals and satellites in haloes more massive than $10^{14}$ M$_\odot$ – these are shown only on the left-hand panel due to the selection effect just mentioned above (and discussed in Appendix B). In the right-hand panel, the red and green symbols and curves show the corresponding subset of galaxies in our Yang+SerExp sample. (That is, they are in haloes more massive than $10^{14}$ M$_\odot$, but because the assignment of halo masses – and in some cases the central-satellite designation – has changed, they are not exactly the same objects as in the panel on the left-hand side.) Brown and blue symbols and curves in both panels show centrals and satellites in the redMaPPer catalogue, and magenta shows the MaxBCG centrals.

In both panels, the satellites are in good agreement with the grey curve: Satellites are similar to the average over the full population, even at $M_r < -23$. In the panel on the right-hand side, centrals and satellites are also in good agreement (only the redMapper centrals are slightly offset), even though the median differences between SDSS and SerExp are larger compared to the panel on the left-hand side. Only in the panel on the left-hand side do the centrals tend to be slightly more luminous, with the difference increasing with group mass (recall that the redMaPPer groups are more massive than Yang+). At $M_r < -23$, where the grey curve indicates that the average difference is 0.15 mag, the redMaPPer centrals show an additional 0.08 mag difference. Clearly, the centrals of the most massive clusters, which are less than 30 per cent of the rarest most luminous objects (Fig. 6), are different from the vast majority of the galaxy population.
<figure><img src="image_14.png" /><figcaption>**Figure 10.** Comparison of sky estimates; colours are the same as Fig. 9. The top set of curves, showing a larger offset and an asymmetric scatter, show the difference between PYMORPH SerExp and SDSS DR7 sky estimates [ΔSky = Sky (PYMORPH SerExp) − Sky (SDSS)]. The SDSS sky estimate is brighter; this overestimate is largest for the most luminous objects, and is even larger for centrals than for satellites. The bottom set of curves, showing no offset and a smaller scatter, show the difference between PYMORPH SerExp and PYMORPH deV sky estimates [i.e. ΔSky = Sky (PYMORPH SerExp) − Sky (PYMORPH deV)]. The difference in these sky estimates is insignificant. This conclusion is similar if we compare PYMORPH Ser and PYMORPH SerExp sky estimates.</figcaption></figure>

However, even for these centrals, it is not obvious that one can attribute the additional offset entirely to ICL-related effects for the reasons given in Section 2.2. As Figs 1–3 illustrate (Fig. 3 is a redMaPPer central), the PYMORPH fits are usually accurate out to ~1 per cent of sky. If there is a difference beyond this scale, it is in the sense that the ICL will be yet another addition to the SerExp estimate.

## 4.3 Differences in sky and profile shape

When discussing Fig. 4, we noted that the median trend for the full E+S0 sample (labelled ‘All’) is due to two effects: (1) differences in the PYMORPH and SDSS sky estimates, and (2) the increased freedom that the two-component SerExp profile has compared to Model, which, for E+S0s, is basically a single de Vaucouleurs profile. This raises the question: Which effect dominates the central–satellite differences we see in Fig. 9?

To address this, the lower set of curves in Fig. 10 compare the sky estimates in the PYMORPH deV, Ser and SerExp fits. These show no significant differences from zero. This is in marked contrast to the top set of curves showing a large asymmetric scatter. These indicate that the SDSS DR7 sky estimate is brighter than PYMORPH’s for the most luminous objects. Moreover, the SDSS overestimate is even brighter for centrals than for satellites. The former trend was reported by Fischer et al. (2017, and references therein), who show that the SDSS sky estimate is brighter than both PYMORPH and Blanton et al. (2011), while the latter two are in excellent agreement with one another. Fischer et al. argued that it was a consequence of the fact that the SDSS does not use a large enough region from which to estimate the background sky. The issue with the SDSS sky will only be exacerbated in clusters, so the fact that it is worse for centrals than satellites is not unexpected. (The comparison here uses the SDSS DR7 rather than DR9 sky values, since the PYMORPH photometry is based on the DR7 rather than DR9 flux-calibrated

images. However, section 3 and figs 9 and 10 in Fischer et al. (2017) show that despite the changes from DR7 to DR9, the bias in the SDSS DR7 sky estimates is also present for the DR9 reductions.)

To see if this can account for all the differences shown in Fig. 9, it is useful to write

$$ \text{Model} - \text{SerExp} = (\text{deV} - \text{SerExp}) - (\text{deV} - \text{Model}), \quad (1) $$

where deV is the result of using PYMORPH to fit a (truncated) de Vaucouleurs profile to the image. The first term (i.e. deV-SerExp) isolates the effect of fitting different models, whereas the second term (i.e. deV-Model) is entirely due to the difference between the PYMORPH and SDSS sky estimates.

Fig. 11 shows these two contributions for the various samples shown in Fig. 9. Comparison with the left-hand panel of Fig. 9 shows that the sky accounts for a little less than half the total difference, and it affects the redMaPPer centrals the most. At $M_r \sim -23$, the choice of profile shape alone accounts for about 0.08 mag when averaged over all E+S0s, with the average for redMaPPer centrals being larger by an additional 0.08 mag. However, in the panel on the right-hand side, the central–satellite difference is entirely due to differences in sky.

Fig. 11 makes one more point. The grey curve in the left-hand panel shows that the deV-SerExp difference is not monotonic with luminosity. Section 4 in Fischer et al. (2017) discusses this in more detail and argues that it suggests that there are two populations at high luminosities. However, their analysis did not provide a physically motivated model for the two populations. Our analysis suggests that the two populations may correspond to centrals and satellites.

Fig. 12 shows that none of our conclusions are changed if we plot the magnitude difference versus the corresponding stellar mass estimate rather than luminosity. And Figs A1 and A2 show that they are unchanged if we replace SerExp magnitudes with Ser. (For all these figures, we use Yang+ for the panels on the left-hand side, and Yang+Ser for the panels on the right-hand side.) In all cases, once the bias in the SDSS sky has been accounted for, the luminosity or stellar mass dependence of the SDSS–PYMORPH difference is the same for all satellites, whatever the mass of the cluster they inhabit. Although for centrals of the most massive clusters there is an additional effect that increases as cluster mass increases, this additional effect is substantially smaller when shown as a function of Sérsic-based rather than SDSS Model magnitudes.

To explore the mass dependence further, we divided the Yang+ and Yang+SerExp samples into objects having $14 \le \log_{10}(M_{\text{halo}}/M_{\odot}) \le 14.3$ and $\log_{10}(M_{\text{halo}}/M_{\odot}) \ge 14.3$. We saw a small difference between the two, which is consistent with the mass trend we infer from the comparison with redMaPPer. Similarly, when redMaPPer is split into two subsamples at richness $\lambda = 32$, the one with higher richness shows a slightly larger difference. In both cases, the smaller sample size means that the statistical significance of the finer mass trend is smaller, so we have not shown these additional trends here.

We conclude that the analysis in this section has shown that, for the vast majority of luminous galaxies, the difference between PYMORPH and SDSS photometry is not dominated by ICL-like effects. Rather, it reflects real structural differences, and is not just a matter of semantics.

## 5 DISCUSSION

The SDSS and PYMORPH luminosity estimates differ (Fig. 4) because the SDSS sky estimate is biased and because the fitted models differ
<figure><img src="image_15.png" /><figcaption>**Figure 11.** Same as Fig. 9, but now the difference between *SerExp* and *Model* photometry is broken up into two terms: one isolates sky subtraction effects and the other is due to true differences in the shape of the light profile.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Figure 12.** Same as Fig. 11, but now versus $M_*$ rather than absolute magnitude.</figcaption></figure>

(Fischer et al. 2017). Biases in the SDSS sky estimate are worst for central galaxies in the most massive haloes (brown curve in Fig. 10). In general, sky-related biases account for a little less than half the total difference between the SDSS and PYMORPH luminosities (lower set of curves in Figs 11, 12, A1 and A2). The remaining difference (upper set of curves in Figs 11, 12, A1 and A2) is due to fitting different models to the surface brightness profile (i.e. de Vaucouleurs profile).

For the vast majority of galaxies, once biases in the SDSS sky estimate have been accounted for, the SDSS-PYMORPH difference does not depend strongly on whether a galaxy is a central or a satellite. The difference, averaged for the full sample of galaxies (grey curve), is the same as that of central or satellite galaxies of less massive groups (red and green curves in Figs 7 and 8), and of satellites in more massive groups (blue curves in Figs 11 and 12). We see only an additional effect for centrals in the most massive haloes (which Fig. 6 shows to be rare), and then only when shown as a function of SDSS rather than Sérsic-based photometry. A similar conclusion is reached when one uses either two-component *SerExp* photometry or single-component *Ser* fits (compare Figs 11 and 12 with Figs A1 and A2).

The ICL is expected to be fainter at larger clustercentric distances or around central galaxies in smaller groups (e.g. Tal &

van Dokkum 2011). Now, the difference between Sérsic-based and SDSS Model magnitudes, when averaged over the full population, is the same for centrals or satellites in smaller groups (Figs 7 and 8). In fact, Figs 11 and 12 show that it is also the same for satellites in more massive groups. Therefore, we conclude that the vast majority of massive galaxies are not well fitted by a simple de Vaucouleurs profile: The difference from Model magnitudes is not due to the ICL, but indicates real structural differences. Hence, the difference between PYMORPH Sérsic-based luminosity and stellar mass functions in Bernardi et al. (2013, 2016 and 2017) with respect to estimates based on SDSS Model magnitudes is real – it is not just semantics.

The issue is slightly more complicated for the small number of galaxies that are centrals of the most massive clusters. Even though the top set of curves in the right-hand panels of Figs 11, 12, A1 and A2 are so similar, they are offset brightwards in the left-hand panel (compare brown and grey curves). Hence, there must be differences between these objects and the vast majority of the galaxy population. However, we believe that even these differences – which affect less than 30 per cent of the rarest most luminous objects (Fig. 6) – should not be attributed to the ICL entirely. This is because the surface brightness where the ICL becomes apparent is fainter than ~27 mag arcsec⁻² (Zibetti et al. 2005;
based not on individual galaxies, but a stacking analysis of ~600 brightest cluster galaxies). This is fainter than ~26 mag arcsec⁻² associated with ~1 percent of the background sky in individual r-band SDSS images, so it is very unlikely that the ICL plays a role when fitting to individual images (Figs 1–3 and related discussion).

Moreover, we note that the right-hand panels of Fig. 12 and especially Fig. A2 show a dramatic change in slope around 2 × 10¹¹ M☉. This is the same mass scale where a number of other scaling relations also change (Bernardi et al. 2011, 2014), and where the galaxy population becomes dominated by slow rotators (Cappellari et al. 2013). These other features are thought to indicate a change in the typical assembly mechanism of the population. While this change may also be related to the build-up of the ICL, the features are not caused by it.

For all these reasons, we believe that PYMORPH SerExp photometry represents a significant improvement to SDSS-Model photometry. For the vast majority of massive galaxies, the differences between PYMORPH and SDSS are real. In particular, our work shows that previous halo model analyses that used SDSS pipeline photometry when relating the stellar mass of the central galaxy to the dark matter mass of the halo that surrounds it should be redone. Ascribing the difference compared to analyses based on Sérsic magnitudes to the semantics of whether or not one includes the ICL when describing the stellar mass of the central galaxy is incorrect.

We also showed that when studying correlations at fixed group mass, care must be taken to ensure that one does not mistake selection effects for physical effects. The SerExp and Model magnitude difference, when plotted versus SerExp magnitude, shows strong trends even though no such trends were apparent when plotting versus Model magnitude (compare red and green curves in Figs 7 and B1). We argued that this is because the group masses in the Yang+ catalogue are tightly correlated with Model magnitudes. A simple model illustrated why selection effects appear when one studies correlations at fixed group mass with a quantity that was not used to define the groups (Appendix B). Indeed, when we reassign halo masses to the Yang+ groups on the basis of SerExp magnitudes, then the selection effect appears when plotting versus Model rather than SerExp magnitudes (compare Figs 8 and B3). This conclusion is general. For example, we have checked (but do not show here) that the selection effect is even stronger when Ser rather than SerExp magnitudes are used. We hope that our demonstration of the existence of this sort of pernicious selection effect, our explanation of its cause (Appendix B) and how one should correct for it will prevent future confusion.

## ACKNOWLEDGEMENTS

We would like to thank the referee for a helpful report. MB and RKS are grateful to the ICTP for its hospitality during the summer of 2016.

## REFERENCES

* Aihara H. et al., 2011, ApJS, 193, 29

* Bernardi M., Hyde J. B., Sheth R. K., Miller C. J., Nichol R. C., 2007, AJ, 133, 1741

* Bernardi M., Shankar F., Hyde J. B., Mei S., Marulli F., Sheth R. K., 2010, MNRAS, 404, 2087

* Bernardi M., Roche N., Shankar F., Sheth R. K., 2011, MNRAS, 412, L6

* Bernardi M., Meert A., Sheth R. K., Vikram, Huertas-Company M., Mei S., Shankar F., 2013, MNRAS, 436, 697

* Bernardi M., Meert A., Vikram V., Huertas-Company M., Mei S., Shankar F., Sheth R. K., 2014, MNRAS, 443, 874

* Bernardi M., Meert A., Sheth R. K., Huertas-Company M., Maraston C., Shankar F., Vikram V., 2016, MNRAS, 455, 4122

* Bernardi M., Meert A., Sheth R. K., Fischer J.-L., Huertas-Company M., Maraston C., Shankar F., Vikram V., 2017, MNRAS, 467, 2217

* Blanton M. R., Eisenstein D., Hogg D. W., Schlegel D. J., Brinkmann J., 2005, ApJ, 629, 143

* Blanton M. R., Kazin E., Muna D., Weaver B. A., Price-Whelan A., 2011, AJ, 142, 31

* Cappellari M. et al., 2013, MNRAS, 432, 1862

* Chabrier G., 2003, ApJ, 586, 133

* Cooray A., Sheth R. K., 2002, Phys. Rep., 372, 1

* D'Souza R., Vegetti S., Kauffmann G. A. M., 2015, MNRAS, 454, 4027

* Fischer J.-L., Bernardi M., Meert A., 2017, MNRAS, 467, 490

* Gonzalez A. H., Zabludoff A. I., Zaritsky D., 2005, ApJ, 618, 195

* Huertas-Company M., Aguerri J. A. L., Bernardi M., Mei S., Sánchez Almeida J., 2011, A&A, 525, 157

* Hyde J. B., Bernardi M., 2009, MNRAS, 394, 1978

* Kravtsov A., Vikhlinin A., Meshscheryakov A., 2014, ApJ, preprint (arXiv:1401.7329)

* Meert A., Vikram V., Bernardi M., 2013, MNRAS, 433, 1344

* Meert A., Vikram V., Bernardi M., 2015, MNRAS, 446, 3943

* Meert A., Vikram V., Bernardi M., 2016, MNRAS, 455, 2440

* Mendel J. T., Simard L., Palmer M., Ellison S. L., Patton D. R., 2014, ApJS, 210, 3

* Rykoff E. S. et al., 2012, ApJ, 746, 178

* Rykoff E. S. et al., 2014, ApJ, 785, 104

* Shankar F. et al., 2014, ApJ, 797, L27

* Tal T., van Dokkum P. G., 2011, ApJ, 73, 89

* von der Linden A., Best P. N., Kauffmann G., White S. D. M., 2007, MNRAS, 379, 867

* Yang X., Mo H. J., van den Bosch F. C., Pasquali A., Li C., Barden M., 2007, ApJ, 671, 153

* Zibetti S., White S. D. M., Schneider D. P., Brinkmann J., 2005, MNRAS, 358, 949

## APPENDIX A: COMPARISON WITH SINGLE SER PHOTOMETRY

The main text studied differences between PYMORPH SerExp and SDSS pipeline photometry. Here we show the corresponding results if we use (truncated) PYMORPH Ser instead of (truncated) SerExp values. Meert et al. (2013) and Bernardi et al. (2014) have argued that the SerExp photometry we present in the main text is more unbiased than the Ser photometry here. However, single Ser fits are more common in the literature, which is why we show them here.

Fig. A1 shows that the differences with respect to SDSS sky are similar to when fitting SerExp profiles (compare with Fig. 11), but the remaining difference is substantially larger than before. In addition, even though the differences are larger, centrals and satellites are similar when shown as a function of PYMORPH photometry but different when shown as a function of Model magnitudes (left-hand and right-hand panels, respectively), as was the case in Fig. 11. Fig. A2 shows that this is also true if we use Ser photometry to determine the stellar masses.
<figure><img src="image_17.png" /><figcaption>Figure A1. Same as Fig. 11 but with (truncated) Ser magnitudes in place of SerExp, and Yang+Ser in place of Yang+SerExp.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure A2. Same as Fig. A1 but now versus $M_*$ rather than absolute magnitude.</figcaption></figure>

## APPENDIX B: SELECTION EFFECTS FOR ANALYSES AT FIXED HALO MASS

We noted in the main text that it was important to present results using the same photometry as used to define the centrals, satellites and halo masses in the Yang+ group catalogue. Fig. 7 shows the result of plotting the PYMORPH–SDSS difference versus Model magnitude. Fig. B1 shows that the scalings change dramatically when we plot versus SerExp instead. The faintest satellites (green curve) and brightest centrals (red curve) are remarkably similar to the average (grey curve), but the brightest satellites and faintest centrals curve away. We see similar effects if we replace the two-component SerExp with single-component Ser photometry, and if we use $M_* = (M_*/L)L_{Ser}$ or $(M_*/L)L_{SerExp}$ instead of $(M_*/L)L_{Model}$. We now show that these offsets are selection effects that result from the fact that the group catalogue was constructed using Model magnitudes. For brevity, we consider only the trends for centrals.

Let $m_i$ denote the model magnitude of galaxy $i$, $s_i$ is its SerExp magnitude and $h_i$ is the mass of the halo to which it belongs. Let $n(m, H)$ denote the number density of objects that have Model magnitude $m$ and reside in haloes with mass $H \equiv (h_{min} \le h \le h_{max})$. Then,

$$n(m, H) = n(m) \int_H dh \, p(h|m), \qquad (B1)$$

<figure><img src="image_19.png" /><figcaption>Figure B1. Same as Fig. 7, except that the difference between Model and SerExp (truncated) magnitudes for galaxies in Yang+ having group masses between $10^{13}$ and $10^{14} M_{\odot}$ is shown as a function of SerExp magnitudes. The median difference defined by all the E+S0 galaxies (grey curve; same as the corresponding curve in Fig. 7) is significantly different from zero. The curvature away from this median relation, for the brightest satellites (green curve) and faintest centrals (red curve), is a selection effect arising from the fact that $M_{Halo}$ is strongly correlated with Model magnitudes, but there is scatter in the Model–SerExp relation.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>**Figure B2.** Monte Carlo demonstration of the selection effect associated with plotting versus the variable on which the group catalogue was not defined. The offsets from the straight line, which are qualitatively similar to those in Figs B1 and B3, are selection effects that our simple model (equation B6) describes well. The offsets depend on halo mass as well as on the range of halo masses included: The bin width is widest for the lowest mass bin and narrowest for the bin in the middle.</figcaption></figure>

where *H* indicates the limited range in halo masses. The measurement in Fig. 7 is

$$ \langle s - m | m, H \rangle = \frac{n(m)}{n(m, H)} \int_H dh \, p(h|m) \times \int ds \, p(s|m, h) (s - m). \quad (B2) $$

If the scatter between *s* and *m* does not depend on halo mass, then we can set $p(s|m, h) = p(s|m)$, and so

$$ \langle s - m | m, H \rangle = \int ds \, p(s|m) (s - m). \quad (B3) $$

This is the same for all choices of *H*, and is consistent with the fact that all the curves in Fig. 7 are the same.

In contrast, the measurement in Fig. B1 is

$$ \begin{aligned} \langle s - m | s, H \rangle &= \int_H dh \int dm \frac{n(m, s, h)}{n(s, H)} (s - m) \\ &= \frac{n(s)}{n(s, H)} \int_H dh \, p(h|s) \int dm \, p(m|s, h) (s - m), \end{aligned} \quad (B4) $$

where

$$ n(s, H) = \int_H dh \int dm \, n(m, s, h) = n(s) \int_H dh \, p(h|s). \quad (B5) $$

To gain insight, suppose that $p(m|s, h)$ is Gaussian. Then, the mean $\langle m|s, h \rangle$ can always be written as $\langle m|s \rangle + C[h - \langle h|s \rangle]$, where $C$ does not depend on either $s$ or $h$. Then,

$$ \langle s - m | s, H \rangle = s - \langle m | s \rangle - \frac{n(s) C}{n(s, H)} \int_H dh \, p(h|s) [h - \langle h | s \rangle]. \quad (B6) $$

The second term is zero if $C = 0$; this happens only if the $mh$ correlation is entirely due to the $ms$ and $sm$ correlations. If $C \neq 0$, then the second term equals zero only if *H* allows the full range

<figure><img src="image_21.png" /><figcaption>**Figure B3.** Same as Fig. 8, but now showing the magnitude difference as a function of Model magnitude in the Yang+SerExp catalogue, in which $M_{Halo}$ is strongly correlated with SerExp. As for Fig. B1, the curvature away from the median relation shown by the grey curve is a selection effect.</figcaption></figure>

of halo masses. However, if *H* selects only a subset of haloes, then the integral is non-vanishing, yielding a correction factor that depends on *H*. If $p(s|h)$ were also Gaussian, then the integral above would equal $\sigma_{h|s} [\exp(-y_{min}^2/2) - \exp(-y_{max}^2/2)]/\sqrt{2\pi}$ and the denominator would be $[\text{erf}(y_{max}/\sqrt{2}) - \text{erf}(y_{min}/\sqrt{2})]/2$, where $y_{max} = (h_{max} - \langle h|s \rangle)/\sigma_{h|s}$ and similarly for $y_{min}$.

To test this, we assumed that the distribution of *m* is Gaussian with unit variance, that $p(s|m)$ is Gaussian with mean 1.025 *m* and rms 0.2, and $p(h|m)$ is Gaussian with mean -0.4 *m* and rms 0.1. While these scalings are not exactly the same as those in the data, they are similar. Note, in particular, that *h* is directly related to *m*; it only correlates with *s* because *s* too is directly related to *m*. (That is, the factor *C* is proportional to the $hm$ correlation.)

We first made Monte Carlo realizations of this joint distribution of *m*, *s* and *h*, and checked that measurements of $\langle s - m | m, H \rangle$ do not depend on *H*, in good agreement with equation (B3). Fig. B2 shows a similar analysis of $\langle s - m | s, H \rangle$, i.e. when we plot the difference between *s* and *m* versus *s* rather than *m*. The straight black line shows $s - \langle m|s \rangle$ as a function of *s*, and the other three curves show equation (B6) with $(h_{min}, h_{max}) = (-5, -0.5)$, $(-0.25, 0.25)$ and $(0.75, 1.75)$. We have chosen the different bin centres to illustrate the mass dependence of the offsets, and the different bin widths to show that these offsets appear primarily at the edges of the bins. The error bars show the corresponding measurements of $\langle s - m|s, H \rangle$ in the Monte Carlo realizations in which $\langle s - m | m, H \rangle$ showed no offsets. The agreement between the measurements and equation (B6) indicates that we understand the origin of the offsets from the straight line.

In particular, the fact that we see no offsets when we plot versus *m* indicates that the offsets here, and in Fig. B1, are selection effects – they are not physical. Additional evidence that they are selection effects comes from the comparison of Figs 8 and B3. Both use the Yang+SerExp catalogue defined in Section 3.2, for which halo masses were assigned to the Yang+ groups on the basis of SerExp rather than Model magnitudes. In this case, plots as a function of SerExp magnitudes were well behaved (Fig. 8), whereas those for Model showed offsets (Fig. B3), i.e. the selection effect is reversed, as expected, based on the analysis above.

This paper has been typeset from a TeX/LATEX file prepared by the author.