SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meena Jagadeesan
mjagadeesan@berkeley.edu
Jacob Steinhardt*
jsteinhardt@berkeley.edu
jordan@cs.berkeley.edu
Nika Haghtalab*
nika@berkeley.edu
As the scale of machine learning models increases, trends such as scaling laws anticipate consistent downstream improvements in predictive accuracy. However, these trends take the perspective of a single model-provider in isolation, while in reality providers often compete with each other for users. In this work, we demonstrate that competition can fundamentally alter the behavior of these scaling trends, even causing overall predictive accuracy across users to be non-monotonic or decreasing with scale. We define a model of competition for classification tasks, and use data representations as a lens for studying the impact of increases in scale. We find many settings where improving data representation quality (as measured by Bayes risk) decreases the overall predictive accuracy across users (i.e., social welfare) for a marketplace of competing model-providers. Our examples range from closed-form formulas in simple settings to simulations with pretrained representations on CIFAR-10. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare in marketplaces with multiple model providers.
Scaling trends in machine learning suggest that increasing the scale of a system consistently improves predictive accuracy. For example, scaling laws illustrate that increasing the number of model parameters [Kaplan et al., 2020, Sharma and Kaplan, 2020, Bahri et al., 2021] and amount of data [Hoffmann et al., 2022] can reliably improve model performance, leading to better representations and thus better predictions for downstream tasks [Hernandez et al., 2021].
However, these scaling laws typically take the perspective of a single model-provider in isolation, when in reality, model-providers often compete with each other for users. For example, in digital marketplaces, multiple online platforms may provide similar services (e.g., Google search vs. Bing, Spotify vs. Pandora, Apple Maps vs. Google) and thus compete for users on the basis of prediction quality. A distinguishing feature of competing platforms is that users can switch between platforms and select a platform that offers them the highest predictive accuracy for their specific requests. This breaks the direct connection between predictive accuracy of a single platform in isolation and social welfare across competing platforms, and raises the question: what happens to scaling laws when model-providers compete with each other?
Figure 1: Comparison of equilibrium loss on two data distributions, one with high Bayes risk (left) and one with lower Bayes risk (right). Each plot shows the linear predictors chosen at equilibrium under competition between three model-providers (solid lines), along with two approximately Bayes-optimal predictors (dashed lines). The equilibrium social loss is lower in the left plot than the right plot, even though the Bayes risk is much higher. The intuition is that approximate Bayes optima disagree on more data points in the left plot than in the right plot; thus, users have a greater likelihood of at least one predictor offering them a correct prediction, which increases the overall predictive accuracy for users (i.e., the social welfare).
We show that the typical intuition about scaling laws can fundamentally break down under competition. Surprisingly, even monotonicity can be violated: increasing scale can decrease the overall predictive accuracy (social welfare) for users. More specifically, we study increases to scale through the lens of data representations (i.e., learned features), motivated by how increasing scale generally improves representation quality [Bengio et al. 2013]. We exhibit several multi-class classification tasks where better data representations (as measured by Bayes risk) decrease the overall predictive accuracy (social welfare) for users, when varying data representations along several different axes.
The basic intuition for this non-monotonicity is illustrated in Figure 1. When data representations are low quality, any predictor will be incorrect on a large fraction of users, and near-optimal predictors may disagree on large subpopulations of users. Model providers are thus incentivized to choose complementary predictors that cater to different subpopulations (market segments), thus improving the overall predictive accuracy for users. In contrast, when representations are high quality, each optimal predictor is incorrect on only a small fraction of users, and near-optimal predictors likely agree with each other on most data points. As a result, model-providers are incentivized to select similar predictors, which decreases the overall predictive accuracy for users.
To study when non-monotonicity can occur, we first focus on a stylized setup that permits closed-form calculations of the social welfare at equilibrium (Section 3). Using this characterization, in three concrete binary classification setups, we show that the equilibrium social welfare can be non-monotonic in Bayes risk. In particular, we vary representations along three axes—the per-representation Bayes risks, the noise level of representations, and the dimension of the data representations—and exhibit non-monotonicity in each case (Figure 2).
Going beyond the stylized setup of Section 3, in Section 4 we consider linear function classes and demonstrate empirically that the social welfare can be non-monotonic in the data representation
2We are motivated by emerging marketplaces where different model-providers utilize the same pretrained model, but finetune the model in different ways. To simplify this complex training process, we conceptualize pretraining as learning data representations (e.g., features) and fine-tuning as learning a predictor from these representations. In this formalization, increasing the scale of the pretrained model leads to improvements in data representations accessible to the model-providers during "fine-tuning".
quality. We consider binary and 10-class image classification tasks on CIFAR-10 where data representations are obtained from the last-layer representations of AlexNet, VGG16, ResNet18, ResNet34, and ResNet50, pretrained on ImageNet. Better representations (as measured by Bayes risk) can again perform worse under competition (Figure 3). We also consider synthetic data where we can vary representation quality more systematically, again finding ubiquitous non-monotonicities.
Altogether, our results demonstrate that the classical setting of a single model-provider can be a poor proxy for understanding multiple competing model-providers. This suggests that caution is needed when inferring that increased social welfare necessarily follows from the continuing trend towards improvements in predictive accuracy in machine learning models. Machine learning researchers and regulators should evaluate methods in environments with competing model-providers in order to reasonably assess the implications of raw performance improvements for social welfare.
Our work connects to research threads on the welfare implications of algorithmic decisions and competition between data-driven platforms.
Welfare implications of algorithmic decisions. Recent work investigates algorithmic monoculture [Kleinberg and Raghavan 2021, Bommasani et al. 2022], a setting in which multiple model-providers use the same predictor. In these works, monoculture is intrinsic to the decision-making pipeline: model-providers are given access to a shared algorithmic ranking [Kleinberg and Raghavan 2021] or shared components in the training pipeline [Bommasani et al. 2022]. In contrast, in our work, monoculture may arise endogenously from competition, as a result of scaling trends. Model-providers are always given access to the same function classes and data, but whether or not monoculture arises depends on the quality of data representations and its impact on the incentives of model-providers. Our work thus offers a new perspective on algorithmic monoculture, suggesting that it may arise naturally in competitive settings as a side effect of improvements in data representation quality.
More broadly, researchers have identified several sources of mismatch between predictive accuracy and downstream welfare metrics. This includes narrowing of a classifier under repeated interactions with users [Hashimoto et al., 2018], preference shaping of users induced by a recommendation algorithm [Carroll et al., 2022; Dean and Morgenstern, 2022; Curmei et al., 2022], strategic adaptation by users under a classifier [Brückner et al., 2012; Hardt et al., 2016], and the long-term impact of algorithmic decisions [Liu et al., 2018, 2020].
Competition between data-driven platforms. Our work is also related to the literature on competing predictors. The model in our paper shares similarities with Ben-Porat and Tennenholtz [2017, 2019], who studied equilibria between competing predictors. Ben-Porat and Tennenholtz [2017, 2019] show that empirical risk minimization is not an optimal strategy for a model-provider under competition and design algorithms that compute the best-responses; in contrast, our focus is on the equilibrium social welfare and how it changes with data representation quality. The specifics of our model also slightly differ from the specifics of Ben-Porat and Tennenholtz [2017, 2019]. In their model, each user has an accuracy target that they wish to achieve and randomly chooses between model-providers meeting that target; in contrast, in our model, each user noisily chooses the model-provider that minimizes their loss and model-providers can have asymmetric market reputations.
Our work also relates to bias-variance games [Feng et al. 2019] between competing model-providers. However, Feng et al. [2019] focus on the the equilibrium strategies for the model-provider, but do not consider equilibrium social welfare for users; in contrast, our work focuses on the equilibrium social welfare. The model of Feng et al. [2019] also differs from the model in our work. In Feng et al. [2019], a model-provider action is modeled as choosing an error distribution for each user, and the action set includes error distributions with a range of different variances. In contrast, in our setup, the error distribution for every user is always a point mass (variance 0). Thus, the equilibrium characterization of Feng et al. [2019] does not translate to our model. The specifics of the model-provider utility in the work of Feng et al. [2019] differs slightly from our model as well.
Other aspects studied in this research thread include competition between model-providers using out-of-box learning algorithms that do not directly optimize for market share [Ginart et al., 2021; Kwon et al., 2022; Dean et al., 2022], competition between model-providers selecting regularization parameters that tune model complexity [Iyer and Ke, 2022], competition between bandit algorithms where data directly comes from users [Aridor et al., 2020; Jagadeesan et al., 2022], and competition
between algorithms dueling for a user [Immorlica et al. 2011]. Our work also relates to classical economic models of product differentiation such as Hotelling's model [Hotelling, 1981, d'Aspremont et al., 1979] (see Anderson et al. [1992] for a textbook treatment), as well as the emerging area of platform competition [see, e.g., Jullien and Sand-Zantman 2021, Calvano and Polo 2021].
We focus on a multi-class classification setup with input space  $X \subseteq \mathbb{R}^d$  and output space  $Y = \{0,1,2,\ldots,K-1\}$ . Each user has an input  $x$  and a corresponding true output  $y$ , drawn from a distribution  $\mathcal{D}$  over  $X \times Y$ . Model providers choose predictors  $f$  from some model family  $\mathcal{F} \subseteq (\Delta(Y))^X$  where  $\Delta(Y)$  is the set of distributions over  $Y$ . A user's loss given predictor  $f$  is  $\ell(f(x),y) = \mathbb{P}[y \neq f(x)]$ . In Section 3, we take  $\mathcal{F} = \{0,1,2,\ldots,K-1\}^X$  to be all deterministic functions mapping inputs to classes, while in Section 4 we consider linear predictors of the form  $f(x) = \text{softmax}(Wx + b)$ .
We study competition between  $m \geq 2$  model-providers for users, building on the model of Ben-Porat and Tennenholtz [2017, 2019]. We index the model-providers by  $[m] := \{1, 2, \dots, m\}$ , and let  $f_j$  denote the predictor chosen by model provider  $j$ . After the model-providers choose predictors  $f_1, \dots, f_m$ , each user then chooses one of the  $m$  model-providers to link to, based on prediction accuracy. Model-providers aim to optimize the number of users that they win. We describe the model in detail below. (We note that this model is stylized and will make several simplifying assumptions; we defer a detailed discussion of the implications of these assumptions to Appendix A.)
User decisions. Users noisily pick the model-provider offering the best predictions for them. That is, a user with representation  $x$  and true label  $y$  chooses a model-provider  $j^{*}(x,y)$  such that the loss  $\ell(f_{j^{*}(x,y)}(x),y)$  is the smallest across all model-providers  $j\in [m]$ , subject to noise in user decisions. More formally, we model user noise with the logit model [Train, 2002], also known as the Boltzmann rationality model:
$$
\mathbb {P} [ j ^ {*} (x, y) = j ] = \frac {e ^ {- \ell \left(f _ {j} (x) , y\right) / c}}{\sum_ {j ^ {\prime} = 1} ^ {m} e ^ {- \ell \left(f _ {j ^ {\prime}} (x) , y\right) / c}}, \tag {1}
$$
where  $c > 0$  denotes a noise parameter. We extend this model to account for uneven market reputations across decisions in Section B.1.
Model provider incentives. A model-provider's utility is captured by the market share that they win. That is, model-provider  $j$ 's utility is
$$
u (f _ {j}; \mathbf {f} _ {- j}) := \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \mathbb {P} \left[ j ^ {*} (x, y) = j \right] \right],
$$
where  $\mathbf{f}_{-j}$  denotes the predictors chosen by the other model-providers and where the expectation is over  $(x,y)$  drawn from  $\mathcal{D}$ . Since the market shares always sum to one, this is a constant-sum game.
Each model-provider chooses a best response to the predictors of other model-providers. That is, model-provider  $j$  chooses a predictor  $f_{j}$  such that
$$
f_{j}\in \operatorname *{arg  max}_{f\in \mathcal{F}}u(f_{j};\mathbf{f}_{-j}).
$$
The best-response captures that model-providers optimize for market share. In practice, model-providers may do so via A/B testing to steer towards predictors that maximize profit, or by actively collecting data on market segments where competitors are performing poorly.
We study market outcomes  $\mathbf{f} = (f_1, f_2, \dots, f_m)$  that form a Nash equilibrium. Recall that  $\mathbf{f}$  is a pure strategy Nash equilibrium if for every  $j \in [m]$ , model-provider  $j$ 's predictor is a best-response to  $\mathbf{f}_{-j}$ : that is,  $f_j \in \arg \max_{f \in \mathcal{F}} u(f_j; \mathbf{f}_{-j})$ . In well-behaved instances, pure-strategy equilibria exist (see Proposition and simulation results in Section 4). However, for uneven market reputations (Appendix B.1), we must turn to mixed strategy equilibria where model-providers choose distributions  $\mu$  over  $\mathcal{F}$ .
Quality of market outcome for users. We are interested in studying the quality of a market outcome  $\mathbf{f} = (f_1, f_2, \dots, f_m)$  in terms of user utility. The quality of  $\mathbf{f}$  is determined by the overall social loss that it induces on the user population, after users choose between model-providers:
$$
\mathrm {S L} \left(f _ {1}, \dots f _ {m}\right) := \mathbb {E} \left[ \ell \left(f _ {j ^ {*} (x, y)} (x), y\right) \right]. \tag {2}
$$
When  $f_{1}, \ldots, f_{m}$  is a Nash equilibrium, we refer to  $\mathrm{SL}(f_{1}, \ldots, f_{m})$  as the equilibrium social loss.
Our goal is to study how the equilibrium social loss changes when the representation quality (i.e., the quality of the input representations  $X$ ) improves. We formalize representation quality as the minimum risk  $\mathsf{OPT}_{\mathrm{single}}$  that a single model-provider could have achieved on the distribution  $\mathcal{D}$  with the model family  $\mathcal{F}$ . This means that  $\mathsf{OPT}_{\mathrm{single}}$  is equal to the Bayes risk:
$$
\mathsf {O P T} _ {\text {s i n g l e}} := \min  _ {f \in \mathcal {F}} \mathbb {E} \left[ \ell (f (x), y) \right].
$$
We show that the equilibrium social loss  $\mathrm{SL}(f_1^*,\dots f_m^*)$  can be non-monotonic in the representation quality (as measured by  $\mathrm{OPT}_{\mathrm{single}}$ ), when representations are varied along a variety of axes.
3 Non-monotonicity of Equilibrium Social Loss in a Stylized Setup
To understand when non-monotonicity can occur, we first consider a stylized setup (described below) that permits closed-form calculations of the social loss. We characterize the equilibrium social loss in this setup for binary classification (Proposition 2), and apply this characterization to three concrete setups that vary representation quality along different axes (Section 3.2): we show that the equilibrium social loss can be non-monotonic in Bayes risk in all of these setups (Figures 2b, 2c). Finally, we extend our theoretical characterization in Proposition 2 to setups with more than 2 classes (Section 3.3), and we extend to model-providers with unequal market reputations (Appendix B.1).
Specification of stylized setup. Assume the input space  $X$  is finite and let  $\mathcal{F} = \mathcal{F}_{\mathrm{all}}^{\mathrm{multi-class}}$  contain all deterministic functions from  $X$  to  $\{0,1,\dots ,K - 1\}$ . For simplicity, we assume that users make noiseless decisions (i.e.,  $c\rightarrow 0$ ), so a user's choice of model-provider  $j^{*}(x,y)$  is specified as follows:
$$
\mathbb {P} [ j ^ {*} (x, y) = j ] = \left\{ \begin{array}{l l} 0 & \text {i f} j \notin \arg \min  _ {j ^ {\prime} \in [ m ]} \mathbb {1} [ y \neq f _ {j ^ {\prime}} (x) ] \\ \frac {1}{\left| \arg \min  _ {j ^ {\prime} \in [ m ]} \mathbb {1} [ y \neq f _ {j ^ {\prime}} (x) ] \right|} & \text {i f} j \in \arg \min  _ {j ^ {\prime} \in [ m ]} \mathbb {1} [ y \neq f _ {j ^ {\prime}} (x) ]. \end{array} \right. \tag {3}
$$
In other words, users pick the model-provider with minimum loss, choosing randomly in case of ties. We show that pure strategy equilibria are guaranteed to exist in this setup.
Proposition 1. Let  $X$  be a finite set of representations, let there be  $K \geq 2$  classes, let  $\mathcal{F} = \mathcal{F}_{all}^{multi-class}$ , and let  $\mathcal{D}$  be the distribution over  $(X,Y)$ . Suppose that user decisions are noiseless (i.e., user decisions are given by (3)). For any  $m \geq 2$ , there exists a pure strategy equilibrium.
3.1 Characterization of the equilibrium social loss for binary classification
We first focus on binary classification. Let  $\mathcal{F}_{\mathrm{all}}^{\mathrm{binary}}$  denote the function class  $\mathcal{F}_{\mathrm{all}}^{\mathrm{multi-class}}$  in the special case of  $K = 2$  classes. Since  $\mathcal{F}_{\mathrm{all}}^{\mathrm{binary}}$  lets model-providers make independent predictions about each representation  $x$ , the only source of error is noise in individual data points. To capture this, we define the per-representation Bayes risk  $\alpha(x)$  to be  $\alpha(x) \coloneqq \min(\mathbb{P}(y = 1 \mid x), \mathbb{P}(y = 0 \mid x))$ . The value  $\alpha(x)$  measures how random the label  $y$  is for a given representation  $x$ . As a result,  $\alpha(x)$  is the minimum error that a model-provider can hope to achieve on the given representation  $x$ . Increasing  $\alpha(x)$  increases the Bayes risk  $\mathsf{OPT}_{\mathrm{single}}$ : in particular,  $\mathsf{OPT}_{\mathrm{single}}$  is equal to the average value  $\mathbb{E}[\alpha(x)]$  across the population. The equilibrium social loss, however, depends on other aspects of  $\alpha(x)$ .
We characterize the equilibrium social loss in terms of the per-representation Bayes risks in the following proposition. Our characterization focuses on pure-strategy equilibria, which are guaranteed to exist in this setup (see Proposition [1]).
Proposition 2. Let  $X$  be a finite set, let  $K = 2$ , and let  $\mathcal{F} = \mathcal{F}_{\text{all}}^{\text{binary}}$ . Suppose that user decisions are noiseless (i.e., user decisions are given by [3]). Suppose also that  $\alpha(x) \neq 1/m$  for all  $x \in X$ . At any pure strategy Nash equilibrium  $f_1^*, \ldots, f_m^*$ , the social loss  $SL(f_1^*, \ldots, f_m^*)$  is equal to:
$$
\mathcal {S L} \left(f _ {1} ^ {*}, \dots , f _ {m} ^ {*}\right) = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \alpha (x) \cdot \mathbb {1} [ \alpha (x) <   1 / m ] ]. \tag {4}
$$
The primary driver of Proposition 2 is that as the per-representation Bayes risk  $\alpha(x)$  decreases, the equilibrium predictions for  $x$  go from heterogeneous (different model-providers offer different
3When  $\alpha (x) = 1 / m$  , there turn out to be multiple pure-strategy equilibria with different social losses.
(f)
Figure 2: Equilibrium social loss (y-axis) versus data representation quality (x-axis) given  $m$  model-providers, for different function classes  $\mathcal{F}$  (rows) and when representations are varied along different aspects (columns). Top row:  $\mathcal{F} = \mathcal{F}_{\mathrm{all}}^{\mathrm{binary}}$ , with closed-form formula from Proposition 2. Bottom row: linear functions, computed via simulation (Section 4). We vary representations with respect to per-representation Bayes risk (a,d), noise level (b,e), and dimension (c,f). The dashed line indicates the Bayes risk (omitted if it is too high to fit on the axis). The Bayes risk is monotone, but the equilibrium social loss is non-monotone.
predictions for  $x$ ) to homogenous (all model-providers offer the same prediction for  $x$ ). In particular, if  $\alpha(x)$  is below  $1/m$ , then all model-providers choose the Bayes optimal label  $y^* = \arg \max_{y'} \mathbb{P}[y' \mid x]$ , so predictions are homogeneous; on the other hand, if  $\alpha(x)$  is above  $1/m$ , then at least one model-provider will choose  $1 - y^*$ , so predictions are heterogeneous. When predictions are heterogeneous, each user is offered perfect predictive accuracy by some model-provider, which results in zero social loss. On the other hand, if predictions are homogeneous and all model-providers choose the Bayes optimal label, the social loss on  $x$  is the per-representation Bayes risk  $\alpha(x)$ . Putting this all together, the equilibrium social loss takes the value in (4). We defer a proof of Proposition 2 to Appendix D.
3.2 Non-monotonicity along several axes of varying representations
Using Proposition 2 we next vary representations along several axes and compute the equilibrium social loss, observing non-monotonicity in each case.
Setting 1: Varying the per-representation Bayes risks. Consider a population with a single value of  $x$  with Bayes risk  $\alpha(x) = \alpha$ . We vary representation quality by varying  $\alpha$  from 0 to 0.5. Figure 2a depicts the result: by Proposition 2, the equilibrium social loss is zero if  $\alpha > 1/m$  and is  $\alpha$  if  $\alpha < 1/m$ , leading to non-monotonicity at  $\alpha = 1/m$ . When there are  $m \geq 3$  model-providers, the equilibrium social loss is thus non-monotonic in  $\alpha$ . As  $m$  increases, the non-monotonicity occurs at a higher data representation quality (a lower Bayes risk).