Title: No Free Delivery Service: Epistemic limits of passive data collection in complex social systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: validity

model validation

complex systems

epistemology

scaling

large language models

recommender systems
Abstract: Rapid model validation via the train-test paradigm has been a key driver for the breathtaking progress in machine learning and AI. However, modern AI systems often depend on a combination of tasks and data collection practices that violate all assumptions ensuring test validity. Yet, without rigorous model validation we cannot ensure the intended outcomes of deployed AI systems, including positive social impact, nor continue to advance AI research in a scientiﬁcally sound way. In this paper, I will show that for widely considered inference settings in complex social systems the train-test paradigm does not only lack a justiﬁcation but is indeed invalid for any risk estimator, including counterfactual and causal estimators, with high probability. These formal impossibility results highlight a fundamental epistemic issue, i.e., that for key tasks in modern AI we cannot know whether models are valid under current data collection practices. Importantly, this includes variants of both recommender systems and reasoning via large language models, and neither naïve scaling nor limited benchmarks are suited to address this issue. I am illustrating these results via the widely used MovieLens benchmark and conclude by discussing the implications of these results for AI in social systems, including possible remedies such as participatory data curation and open science.

No free delivery service

Abstract

Rapid model validation via the train-test paradigm has been a key driver for the breathtaking progress in machine learning and AI. However, modern AI systems often depend on a combination of tasks and data collection practices that violate all assumptions ensuring test validity. Yet, without rigorous model validation we cannot ensure the intended outcomes of deployed AI systems, including positive social impact, nor continue to advance AI research in a scientifically sound way. In this paper, I will show that for widely considered inference settings in complex social systems the train-test paradigm does not only lack a justification but is indeed invalid for any risk estimator, including counterfactual and causal estimators, with high probability. These formal impossibility results highlight a fundamental epistemic issue, i.e., that for key tasks in modern AI we cannot know whether models are valid under current data collection practices. Importantly, this includes variants of both recommender systems and reasoning via large language models, and neither naïve scaling nor limited benchmarks are suited to address this issue. I am illustrating these results via the widely used MOVIELENS benchmark and conclude by discussing the implications of these results for AI in social systems, including possible remedies such as participatory data curation and open science.

Possible world

𝑓

(

𝑋

)

Possible world

𝑓

(

𝑋

)

Possible world

𝑓

:

X

1

×

X

2

→

R

Sample locations

S

:

X

1

×

X

2

→

R

Assumptions

A

samples

determine

determine

Model validation possible?

P

𝑓

∼

F

(|

𝜃

-

𝐿

T

𝑓 ℎ

|

< 𝜖

)

based on

based on

1 Introduction

Model validation, long taken to be 'solved' via the train-test paradigm, has become one of the central challenges in modern machine learning and artificial intelligence. In unison with the dramatic increase of their capabilities, AI systems are now supposed to solve tasks of vastly expanded scope, including potentially AI-complete tasks such as open domain question answering, autonomous decision making, and ultimately, artificial general intelligence. Even before the recent triumphs of large language models and deep learning, Anderson [3] proclaimed 'the end of theory' and the scientific method being obsolete due to the wonders of big data, large-scale computing, and data mining. At the same time, it is entirely unclear how to rigorously evaluate the quality of models for these ambitious tasks. This lack of proper evaluation can then materialize in persistent issues of deployed systems related to generalization, e.g., hallucination [30], out-of-distribution generalization [39], fairness [6], and generalization to the long-tail [23, 24]. Importantly, these issues do not only affect the accuracy of models in a vacuum, but can also affect their social impact if they are deployed in consequential social contexts [16]. In this paper, I aim to connect the former developments with the latter issues through the lens of epistemology. More concretely, I ask:

Research Question 1. Given the ambitious tasks that we ask AI systems to solve and given how we currently collect data, can we know whether a model performs well for these tasks?

Answering RQ 1 positively is central not only for the deployment of machine learning systems, but also for scientific progress within artificial intelligence itself. After all, knowledge of a model's quality is a prerequisite to detect generalization issues and develop improved models. In deployed systems, a model's predictions are useless - as good as they might be - without knowing that they are, in fact, reliable. In social systems, where the consequences of model errors can be severe, having this knowledge is of even greater importance. Hence, the epistemic question of this work gets to the heart of various debates surrounding AI and its capabilities: How can we understand and measure the true capabilities of modern AI systems, which are so very impressive and yet lacking in fundamental ways at the same time [11]? What can we know about the quality of our models? Are our benchmarks suited to give insights into the intended tasks or do they project a false image of quality? How can we develop systems such that they work for everyone? Will naïve scaling solve all these problems or do we need to invest into entirely new approaches for evaluation within the scope of modern AI?

A prerequisite to answering RQ 1 positively is the validity of model validation: Without model validation we can not know wether a model is good or bad and without a valid model validation procedure we can not attain this knowledge. The almost exclusively used method for model validation in machine learning and AI is the ubiquitous train-test paradigm, i.e., the practice of estimating the generalization performance of a model on a test set distinct from the training set. Arguably, much of the breathtaking progress in machine learning has been driven by the success of this single experimental paradigm as it allows for the rapid validation and, therefore, improvement of models [10]. However, it is crucial to note that the train-test paradigm is inherently an inductive method that aims to infer, not measure , the generalization error of a model from its error on a test set. It is well known -dating back at least to Hume [28, 29] and formalized in the context of machine learning by Wolpert [68] - that it is not possible to justify the validity of such inductive inferences without further assumptions. This raises the question: is the train-test paradigm still valid for the combination of tasks and data sets considered in modern AI and under what assumptions is this the case?

Importantly, such assumptions should be minimal in terms of ontological commitments, i.e., meet ontological parsimony (or minimality), since (a) model validation results can not provide insights about validity in the real world if they are contingent on strong ontological assumptions (b) any assumptions that are required to ensure the validity of model validation can not be validated through the same method without circular reasoning. In traditional machine learning settings , these ontological commitments are placed entirely on the data collection process and, as such, the train-test paradigm is indeed suitable to validate any model assumption outside the data collection process. More concretely, under active data collection , i.e., when we actively control the data collection process, we can create large enough test sets that are (approximately) sampled i.i.d. from the target distribution. Under these conditions, it is well known that the train-test paradigm allows us to validate models simply via their performance on this test set without making any further ontological commitments . This property is the beauty of the train-test paradigm and what makes it so valuable and successful.

However, domains in modern machine learning have become far too large to be covered via data sets in this active and controlled manner - the required effort would be prohibitively difficult and costly.

In lieu, passive data collection has become the predominant way to create data sets for modern AI systems. Here, data is collected without intervention from some social system that generates data within the domain of interest. For instance, rather than meticulously collecting independent samples from all possible facts in a domain, training and validation corpora for QA models are gathered from what has been published on the internet. Similarly, preferences of users are collected over items that a recommender system has pre-selected, rather than sampling them i.i.d. over all possible user-item pairs. Importantly, these sample generating systems need not correspond to the target data generating process, have their own internal dynamics, and are driven by complex interactions of their parts and social processes, e.g., well-known phenomena such as popularity bias [1], homophily [22, 37], or feedback loops [14].

Hence, I will ground RQ 1 in these conditions of current machine learning practice: Under passive data collection from a social system, can model validation be valid or not? To formalize the social systems with which an AI system interacts, I am taking a complex systems perspective and describe them as networks with well-established sampling biases and degree distributions. For these properties, I will show how they affect necessary conditions of test validity. These results can also be understood as a strengthening of the seminal No Free Lunch (NFL) theorems for supervised learning [68] in the context of social systems. While the NFL theorems show the impossibility of an assumption-free general purpose learning algorithm, a common criticism is that they need to assume an inductionhostile universe, i.e., full ontological neutrality [60]. In practice, where assuming a reasonably induction-friendly universe is common, the NFL theorems have had therefore limited impact. In contrast, the results of this work are grounded in current machine learning practice and considerably stronger: Even for non-trivial assumptions of an induction-friendly universe, model validation can be shown to be invalid when data is collected passively in social systems. In other words, there is no free delivery service of data for model validation in complex social systems. To discuss the above results, I will provide a synthesis of results from learning theory, social science, and complex systems -and combine them with new theoretical and empirical results on the validity of model validation.

In particular, the main contributions of this paper are as follows:

Theorem 1 (Informal) . For passively collected data in complex social systems the train-test paradigm cannot be valid under ontological parsimony for the vast majority of the system. This includes widely considered variants of recommender systems and question answering.

Corollary 2 (Informal) . Naïve scaling and limited benchmarks are prohibitively inefficient to address theorem 1 and therefore not suited to attain test validity in these scenarios.

Supporting evidence. Theoretical results are supported via experiments on the popular MOVIELENS benchmark where widely considered recommendation tasks are shown to be test-invalid.

The remainder of this paper proceeds as follows: Sections 2 and 3 formalize passive data collection in social systems and connect it to test validity. Section 4 develops theorem 1, corollary 2, and supporting evidence. Sections 5 and 6 discuss related work and implications for AI in social systems.

2 Passive data collection and inference tasks in social systems

To construct validation data sets for large-scale domains, there exist currently two main practical approaches: (i) 'scaling', i.e., indiscriminately collecting as much data as possible from some domain and (ii) manually constructing benchmarks of limited size that probe certain subareas of the domain. In the following, I will focus on formalizing (i) as passive data collection from social systems. Section 4 will then show that neither (i) nor (ii) can be solutions to the issues of this paper.

In sociology, a social system is often considered a pattern of networked interactions that exists between individuals, groups, or institutions [44]. For the purposes of this paper, I will consider a social system to be a pair ( 𝑓 , S ) where 𝑓 : X → Y is a possible world of interactions such that X = X 1 × · · · × X 𝑛 denotes the domain of interactions, Y denotes the set of outcomes (or labels) of an interaction, and S : X → [ 0 , 1 ] denotes the sampling distribution of the system over interactions. Within this framework, passive data collection refers to sampling directly from S . This is in contrast to active data collection where we would aim to sample directly from the target distribution T : X → [ 0 , 1 ] for an inference task, e.g., via simple random sampling, stratified sampling, etc.

In complex social systems, S is driven by social processes that lead to two characteristic properties of samples: (i) they are biased and (ii) they follow heavy-tailed or power-law distributions. The earliest

Domain

X

Possible world

𝑓

Sample distribution

S

Target distribution

T

Recommender

systems

U

×

I

User preferences

Probability of user interacting with

item, heavy-tailed in

U

and

I

Uniform,

𝑝

𝑇

(

𝑢, 𝑖

)

=

1

/|

U

×

I

|

reasoning

S

×

P

×

O

Truth value

of factoids

Probability

of

observing

factoid,

heavy-tailed in

S

,

P

, and

O

Uniform,

𝑝

𝑇

(

𝑠, 𝑝, 𝑜

)

=

1

/|

S

×

P

×

O

|

(a) Heavy-tailed samples

'Humans are mortal and

Socrates is a human"

(Socrates, isA, human)

(human, hasProperty, mortal)

(Socrates, hasProperty, mortal)

'Socrates is mortal"

Observed

Inferred

Natural

Language

knowledge representation

(decode)

reasoning

question answering

(encode)

(b) Reasoning

Recommender-MovieLens1o0k

Reasoning-FB15k-237

Users

Subject

scale)

10-1

Items

Pareto α = 1.85 (rescaled)

scale)

10-1

Object

Proportion(logs

601)

102

102

Proportion

103

103

104

Q

200

400

600

0

200

400

600

800

1000

Observations

Observations (aggregated over all predicates)

Figure 2: (a) Heavy-tailed samples in recommender and reasoning datasets. (b) Symbolic reasoning via LLMs. To validate reasoning capabilities of LLMs, natural language has to be mapped to logical knowledge representations. This shows that validation of reasoning in LLMs is subject to the results of this paper. See also fig. 6b and supp. G.1.

work on (ii) is due to Simon [56], and has independently been discovered in multiple contexts. In fact, (ii) can often be understood as a consequence of (i), e.g., popularity bias leading to power-law distributions in social networks [5, 48]. See also supp. C for further discussion of these properties.

In the remainder, I will therefore focus on the presence of heavy-tailed distributions in S to understand how this ubiquitous property of social systems affects test validity. For this purpose, I will first introduce the concept of a sample graph, i.e., the observed interactions that we receive from S :

Definition 1 (Sample graph) . A data set S ∼ S 𝑚 ⊂ X 1 × X 2 of observed interactions induces a bipartite sample graph 𝐺 = ( X 1 , X 2 , S ) between entities of X 1 and X 2 where an edge indicates that the corresponding interaction has been observed. In the following, I will use S and 𝐺 interchangeably.

For higher arity relations, definition 1 can easily be generalized to hypergraphs. For simplicity, I will focus on bipartite graphs in the following. In sample graphs, the heavy-tailed property of complex systems materializes then through their degree distribution. While the exact nature of these distributions is disputed [13], I will follow V oitalov et al. [64] and assume that node degrees in S follow a regularly-varying power-law distribution . Based on this observation, passive data in complex social systems will then refer to the following:

Definition 2 (Passive data in complex social systems) . Let S ∼ S 𝑚 be a sample graph drawn from sampling distribution S . Let 𝐾 1 , 𝐾 2 denote random variables that model the degree distribution in S of nodes in X 1 and X , respectively. For passively collected data from complex social systems, I will then assume that 𝐾 1 , 𝐾 2 follow regularly-varying power-law distributions, i.e.,

where 𝛼 𝑖 > 0 are the tail indices and 𝑢 𝑖 are slowly varying functions such that lim 𝑥 →∞ 𝑢 ( 𝑟𝑥 )/ 𝑢 ( 𝑥 ) = 1 for any 𝑟 > 0 . Higher arity relations are defined analogously. Next, I will show how passive data in social systems materializes in key inference settings (see also table 1).

Example 1 (Recommender Systems) . Recommender systems are concerned with inferring the true preferences of a user over all items from a set of revealed preferences sampled from S . As such they are a typical example for ( 𝑓 , S ) where the target distribution T corresponds to the uniform distribution over all possible interactions. Importantly, S is typically influenced by social processes and sampling bias as well as heavy-tailed distributions are well documented in recommender systems. For instance, an important factor for sampling biases are feedback loops, e.g., that past recommendations influence which recommendations are shown in the future [34, 14]. Another source of sampling bias is user feedback, which is often biased towards items with high ratings [59], as well as popularity bias [1, 48]. Popularity bias leads directly to heavy-tailed distributions in the degree distribution of the sample graph [48, 5]. See also fig. 2a for evidence of this property on MOVIELENS.

Example 2 (Symbolic reasoning and QA) . Reasoning and question answering over symbolic knowledge representations are another key example for ( 𝑓 , S ) . In this setting, factoids are represented in form of (subject, predicate, object) triples and the task is to infer the truth value for any unknown factoid, i.e., for a uniform target distribution T . Importantly, while facts about the world itself do not need to be influenced by social processes, our available knowledge about them, i.e., S , often is. In addition to aspects such as popularity bias, causes for this can range from which questions are studied in science [35, 36], over how data is collected [31], to who has access to the internet and the ability to contribute to knowledge [67]. Consequently, heavy-tailed distributions are also well-documented in this setting. For instance, Steyvers et al. [61] showed that semantic networks typically follow heavy-tailed degree distributions. Similar distributions have been observed in large-scale knowledge graphs such as DBPEDIA [4], YAGO [62], FREEBASE [9], and WIKIDATA [65]. See also fig. 2a for evidence of this property on FB15K. Importantly, this setting applies to any reasoning task over factoids in general - irrespective of the data representation. For instance, the validation of reasoning capabilities for general purpose question answering in systems such as LLAMA [63, 21] and CHATGPT [47] needs to follow this blueprint. See also fig. 2b for an illustration.

3 Test validity

To answer RQ 1, I will focus on the test validity of inference settings , i.e., whether task, assumptions, and data allow for any valid validations at all. For this purpose, I will use a deductive approach: model validation is valid if it is a logical consequence of its assumptions that the difference between its estimate and the true generalization error is bounded with high probability. To formalize this, let ℎ, 𝑓 : X → Y denote functions that map from sample domain X to target domain Y . For clarity, I will assume noise-free 𝑓 and ℎ . Furthermore, let S ∼ S 𝑚 = { 𝑥 𝑖 } 𝑚 𝑖 = 1 denote a data set of 𝑚 samples drawn from a sampling distribution S : X → [ 0 , 1 ] and let D = {( 𝑥, 𝑓 ( 𝑥 )) : 𝑥 ∈ S } denote its supervised extension. For notational convenience, I will also write D ∼ S 𝑚 when 𝑓 is clear from context. In addition, let A ⊆ { 𝑓 | 𝑓 : X → Y } be the set of all functions from X to Y that are consistent with some set of assumptions on 𝑓 such as being low-rank. Next, note that A and D then induce a set of possible worlds as follows:

Definition 3 (Possible worlds) . Let A be a set of assumptions, D ⊂ X × Y a set of observations, and 𝑓 : X → Y . The set of possible worlds F is then the set of functions consistent with A and D , i.e.,

Furthermore, I will consider an inference setting ( A , D , T , F ) to be a set of assumptions A , a fixed dataset D ∼ S 𝑚 , a target distribution T : X → [ 0 , 1 ] for which we want to make inferences, and an assumed distribution over possible worlds F . Note that if S ≠ T , D can not be an i.i.d. sample from T . For further details and notation see supps. A and B.

Next, let 𝑋 be a random variable over X and let ℓ : Y × Y → R + be a positive loss function. The risk of hypothesis ℎ with respect to a single world 𝑓 is then denoted by

Furthermore, let 𝜃 denote any risk measure of a hypothesis ℎ on some test set T . For instance, 𝜃 could denote the empirical risk or a re-weighted estimator such as the Horvitz-Thompson adjusted empirical risk (see also table 4 in the supp. material). Hence, 𝜃 does not only cover the standard Monte-Carlo estimator for the i.i.d. setting, but also estimators used in counterfactual and causal settings. To determine the test-validity of an inference setting, I am then interested in bounding the difference between the estimated risk ( 𝜃 ) and the true risk of h ( 𝐿 T 𝑓 ℎ ). Importantly, it is necessary to consider the risk of ℎ relative to the distribution F over all possible worlds since no world 𝑓 ∈ F can be excluded based on D and A . Hence, test validity is defined as follows:

Definition 4 (Test validity) . Let 𝑓 ∼ F denote a distribution over possible worlds F and let H denote a hypothesis class. Furthermore, let 𝐿 T 𝑓 ℎ denote the risk of hypothesis ℎ for target distribution T and possible world 𝑓 . Let 𝜃 ∈ R + denote any empirical risk measure of ℎ on a test set. Then, ( A , D , T , F ) is ( 𝜖, 𝛿 ) -test-valid ( test-invalid ) if 𝜃 's difference to 𝐿 T 𝑓 ℎ can (cannot) be bounded accordingly, i.e.,

The conditions in definition 4 for a valid validation setting are very mild since it requires only a single hypothesis class in which 𝜃 for a single hypothesis has bounded difference to the true risk with high probability. Since invalidity follows directly from validity via complement rule and negation, the conditions for a validation setting to be invalid are strong: For any possible hypothesis class it has to hold that the difference between 𝜃 and the true risk of all hypotheses can not be bounded with sufficient probability. Importantly, both are statements about an inference setting, i.e., the combination of assumptions, observed data, and target distribution, and not about a specific hypothesis (class). Furthermore, note that definition 4 implies realizability with regard to the assumptions: if { 𝑓 | ∀( 𝑥, 𝑦 ) ∈ D : 𝑓 ( 𝑥 ) = 𝑦 } ∩ A = ∅ , an inference setting is test-invalid since P (∅) = 0 . However, definition 4 imposes no realizability or any other constraints on H .

Next, note that definition 4 implies straightforward necessary conditions for test validity:

Corollary 1 (Necessary condition for test validity) . Let ( A , D , T , F ) be an inference setting, let ℓ : Y × Y → R + be a positive loss function, and let H be a hypothesis class. Furthermore, let 𝜃 ∈ R + be any risk estimate for ℎ . Then, if ( A , D , T , F ) is ( 𝜖, 𝛿 ) -test-valid, it must hold that

Proof sketch. Corollary 1 follows simply via the monotonicity of probability, i.e., it holds that 1 -𝛿 ≤ P 𝑓 ∼ F (| 𝜃 -𝐿 T 𝑓 ℎ | ≤ 𝜖 ) ≤ P 𝑓 ∼ F ( 𝐿 T 𝑓 ℎ ≤ 𝜖 + 𝜃 ) . This holds for any risk measure 𝜃 , loss ℓ ∈ R + and hypothesis ℎ . See supp. D for proof details. □

4 Test validity under passive data collection in complex systems

In the following, I will provide an overview of the main results as well as high-level proof sketches. For clarity, I will consider only binary relations X = X 1 × X 2 and possible worlds over unbounded output domains 𝑓 : X → R . For detailed proofs and discussion, as well as extensions to ternary relations and bounded domains, see supps. E to G. To meet ontological parsimony 1 and get insights into the validity of the train-test paradigm, I will focus on F being the uniform distribution U and A imposing only minimal assumptions on 𝑓 .

Next, to derive bounds on the validity of inference settings in complex social systems, I will represent possible worlds 𝑓 as partially observed matrices which are constructed as follows:

Definition 5 (Matrix representation) . For a function 𝑓 : X 1 × X 2 → Y over finite sets of size | X 1 | = 𝑛 1 and | X 2 | = 𝑛 2 , its matrix representation F ∈ R 𝑛 1 × 𝑛 2 is given via F 𝑖 𝑗 = 𝑓 ( 𝑥 𝑖 , 𝑥 𝑗 ) for all ( 𝑥 𝑖 , 𝑥 𝑗 ) ∈ X 1 × X 2 . 2 In the following, I will use 𝑓 and F interchangeably.

Using this matrix representation of a system, I will show in lemma 2 that the train-test paradigm is invalid if the rank of 𝑓 , i.e., the complexity of the system, exceeds the 𝑘 -connectivity of the sample graph S and if 𝑓 is chosen uniformly from F . Here, 𝑘 -connectivity is defined as follows:

Definition 6 ( 𝑘 -core and 𝑘 -connectivity) . The 𝑘 -core (or core of order 𝑘 ) of a graph is its maximal subgraph such that all vertices are at least of degree 𝑘 . 3 A graph is 𝑘 -connected if and only if every vertex is in a core of order at least 𝑘 .

Lemma 1 (Rank𝑘 underdetermination) . Let A = { 𝑓 | rank ( 𝑓 ) ≤ 𝑘 } . Then, if S is not 𝑘 -connected, the set of possible worlds F forms a non-empty vector space.

Proof sketch. Since S is not 𝑘 -connected, any 𝑓 with rank ( 𝑓 ) = 𝑘 can not be S -isomeric. It then holds via [38, Lemma 5.1] that F , i.e., the set of matrices of rank 𝑘 or less that are consistent with D , form a non-empty vector space. See supp. E for proof details. □

In the spirit of Occam's razor, higher ranks of 𝑓 correspond to more complex possible worlds. Lemma 1 establishes then that if the 𝑘 -connectivity of S does not match the complexity of the system 𝑓 , the observations S do not constrain F sufficiently and a randomly chosen possible world can be arbitrarily different on the non-observed entries. Via corollary 1, lemma 1 implies then

that 𝑘 -connectivity is necessary for test validity if ℓ belongs to the broad class of scalar Bregman divergences, i.e., widely used loss functions such as the square loss, the log loss, or the KL-divergence (see also table 5 in the supplementary material).

Lemma 2 (Rank𝑘 test-invalidity) . Let A be identical to lemma 1, let ℓ be a scalar Bregman divergence, let F be the uniform distribution over F , and let T be the uniform distribution over X . Furthermore, let 𝜃 ∈ R + be any risk estimator on a test set. Then, if S is not 𝑘 -connected, ( A , D , T , F ) is test-invalid, i.e., it holds for any 𝜖 > 0 that

Proof sketch. If S is not 𝑘 -connected, F is a vector space according to lemma 1. Lemma 2 follows then from corollary 1 for uniformly sampled 𝑓 ∈ F and ℎ ∈ F via a simple volume argument. For ℎ ∉ F , the result follows again from F being a vector space via the generalized Pythagorean theorem for Bregman divergences [20, Eq. 2.3]. See supp. B.2 for proof details. □

The consequences of lemma 2 are non-trivial. Under ontological parsimony, it shows that passive data from complex social systems, i.e., the foundation of basically all large-scale AI tasks, can not be used to validate the quality of models if S ≠ T . Clearly, no subset of S , e.g., cross-validation, can fulfill this task either. Importantly, lemma 2 holds not only for empirical risk, but for any estimator on D , including counterfactual estimators, i.e., methods which are exactly meant to address S ≠ T . This illustrates that lemma 2 is not simply an out-of-distribution or counterfactual estimation problem. Rather, it is caused by a combination of out-of-distribution ( S ≠ T ) and insufficient data ( 𝑘 -connectivity < rank ( 𝑓 ) ). Next, I will connect these results to the main result of this work.

Theorem 1 (Test validity in complex social systems) . Let ( A , D , T , F ) be identical to lemma 2. Furthermore, let S ∼ S 𝑚 where S follows power-law distributions such that the degrees of 𝑥 ∈ X 𝑖 in the sample graph S are drawn i.i.d. from a regularly-varying power-law distribution P ( deg ( 𝑥 ) > 𝑘 ) = 𝑢 ( 𝑘 ) 𝑘 -𝛼 𝑖 . Furthermore, let 𝑛 𝑖 = | X 𝑖 | be the size of domain X 𝑖 . Then, the number 𝑉 𝑖 of nodes in X 𝑖 for which test validity holds decreases with a power-law decay in rank ( 𝑓 ) = 𝑘 , i.e,

Proof sketch. Test validity requires the 𝑘 -connectivity of S to be greater or equal to rank ( 𝑓 ) via lemmas 1 and 2. Hence, only subgraphs where all vertices are at least of degree 𝑘 can be valid. Theorem 1 follows then via the expected number of nodes with degree at least 𝑘 in X 𝑖 , i.e, E [ 𝑉 𝑖 ] = ˝ 𝑥 ∈ X 𝑖 P ( deg ( 𝑥 ) ≥ 𝑘 ) . □

For heavy-tailed distributions, most nodes will be outside the required k-core for even moderately complex worlds. Hence, theorem 1 shows that the train-test paradigm cannot be valid under ontological parsimony for the vast majority of nodes in realistic social systems. Table 2 illustrates this using parameters that match the well-known Book Crossing dataset.

An immediate next question is then if the issues raised by theorem 1 can simply be solved by scaling, i.e., by collecting more data from S -or via manually constructed benchmarks such as BigBench [58] to extrapolate from their results to the risk on T . Corollary 2 answers both questions via lemma 1 (see supp. H for a detailed discussion and proof): (i) For scaling , we can ask how many draws from S would be necessary such that all nodes are within the 𝑘 -core of S with high probability, i.e., how many samples are needed until arriving at a valid test setting. While there exists no easily computable solution to this problem, we can compute a (weak) lower bound by asking how many samples from S are needed to sample a random node in X 𝑖 once . (ii) For benchmarks , we can ask how many nodes would need at least one additional data point to arrive at a valid test setting, i.e., how much manual data collection is at least needed to create a benchmark that extrapolates to T .

(a) eCDF of Maximum NAE

(b) eCDF of Pairwise NAE

(c)

𝑘

-core per occupation

1.0

1.0

1.0

Technician

0.8

80th Percentil.

0.8

80thPercentle

0.8

Other

Writer

Homemaker

Artist

Executive

Engineer

.5oth Percentile.

Proportion

0.5

50thPercentile.

Administrator

Lawyer

Student

Marketing

None

Healthcare

Educator

Retired

Scientist

Salesman

0.2

.2othPercentle.

0.2

20thPercentile.

0.2

Programmer

Entertainment

Doctor

0.0

0

max｜Xi-X1/(Xmax-Xmin)

X,X

0.34

0.77 0.82

0.0

0.00

0.23

[Xi-Xj1 / (Xmax-Xmin)

0.49

1.00

0.0

20

40

k-core

60

80

Bio.Sex

Occupation

1.0

Complexity(rank)

20

0.92

0.96

1

1

0.94

0.920.910.93

0.89

0.97

0.95

0.94

0.96

0.94

0.86

0.97

1

1

0.97

0.97

0.89

Proportion of users

40

0.56

0.66

0.43

0.42

0.55

0.43

0.38

0.57

0.64

0.64

0.44

0.69

0.62

0.67

0.69

0.58

0.59

0.64

0.5

0.75

0.78

0.73

0.74

0.5

60

0.4

0.47

0.14

0.25

0.29

0.29

0.35

0.4

0.32

0.21

0.44

0.44

0.45

0.44

0.51

0.35

0.47

0.47

0.33

0.56

0.58

0.55

0.67

80

0.23

0.31

0

0.083

0.13

0.14

0.19

0.21

0.21

0.21

0.22

0.25

0.26

0.28

0.29

0.29

0.29

0.32

0.33

0.38

0.42

0.42

0.48

0.0

Corollary 2 (Inefficiency of scaling and benchmarks) . Let ( A , D , T , F ) and S be identical to theorem 1. Furthermore, let (i) 𝑇 𝑖 denote the expected number of samples from S until node 𝑥 𝑖 ∈ X is sampled, and let (ii) 𝑁 𝑗 denote the number of nodes in X 𝑗 with less then 𝑘 samples. Then, 𝑇 𝑖 scales at least polynomially and 𝑁 𝑗 scales lineraly in the size of the domain | X | . Specifically,

Clearly, sampling from S is highly inefficient to overcome the issues raised by theorem 1 since (i) it is extremely difficult to get successful samples from the heavy tail (rare events) and (ii) covering all nodes outside sufficiently large k-cores in selective benchmarks is prohibitively expensive. See also table 2 for examples of these aspects for typical distributions in complex social systems.

Experimental evidence To illustrate the real consequences of the previous theoretical results, I will now provide experimental evidence based on the MOVIELENS 100k dataset [27], a critical benchmark that has, for years, been widely-used in recommender systems research. As predicted by lemma 2, I will show that there exist possible worlds of low complexity that all explain the observed data equally well but are widely different on the unobserved data. Hence, any quality metric that is inferred on this benchmark, or subsets of it, can not be informative about the true generalization error. For this purpose, I fit 𝑝 = 100 matrices of rank 𝑘 = 50 to the observed data D . All matrices, or possible worlds, fit the observed data and rank constraint with error below 10 -3 and 10 -2 , respectively. See supp. I.1 for details. For a pair of possible worlds ( 𝑓 , 𝑓 ′ ) , I compute then the normalized absolute error (NAE) for each unobserved entry ( 𝑖, 𝑗 ) ∉ S via NAE ( 𝑓 𝑖 𝑗 , 𝑓 ′ 𝑖 𝑗 ) = | 𝑓 𝑖 𝑗 -𝑓 ′ 𝑖 𝑗 |/( 𝑓 max -𝑓 min ) . This informs us about how different pairs of possible worlds can be on the unobserved data. Figures 3a and 3b shows the empirical CDF (eCDF) of the NAE over unobserved entries for such pairwise comparisons of possible worlds as well as the worst-case over all worlds per entry. From fig. 3a, it can be seen that the worst case error across possible worlds per entry is substantial for the vast majority of unobserved entries. For instance, for 50% of entries the NAE is above 77% of the worst case error. For arbitrary pairs of possible worlds, the situation is similar, where, depending on the particular pair of worlds, the NAE is between 23% to 49% for 50% of entries. Furthermore, the area over the eCDF curves in fig. 3b corresponds directly to the risk for a pair of possible worlds and is again substantial for all pairs (see supp. I.2 for details). Since any possible world can be the 'true' world this shows again that the test error for any subset of this benchmark can not be informative for the true generalization error of this task.

In addition to the NAE, fig. 3c shows the cumulative distribution of users within cores of order 𝑘 per demographic group for MOVIELENS 100k. It can be seen that the cumulative distribution can vary significantly between different demographics. For instance, while only 25% of 'homemakers' are in

a 𝑘 -core larger than 50, 40% of 'technicians' are in a 𝑘 -core larger than 80. It follows from lemma 2, that test-validity will therefore also vary significantly between demographic groups (if we assume that there are no significant differences in the complexity of preferences between groups). Figure 3d illustrates this point by showing the proportion of users for which test-validity holds relative to the rank of a model. It can be seen that there exist clear differences already for moderately complex worlds. For instance, for a model of rank 60, test-validity would hold for 67% of 'technicians' while it would only hold for 14% of 'homemakers'. Clearly, this has important implications for fairness, bias, and whether recommender systems work for everyone .

5 Related work

The no-free-lunch theorems for machine learning [68, 60] share important similarities to this work as both consider the expected risk over possible worlds. However, the results in this paper are stronger and directly applicable to current machine learning practice. While the NFL theorems consider the performance over all possible worlds without any restrictions - an assumption that is too restrictive in most instances - the results of this paper show that even for relatively strong assumptions about the set of possible worlds, e.g., low-rank structures, valid model validation is not generally possible for passive data collection in complex social systems. In motivation, this paper is also related to the works [19, 7, 25, 54, 41, 66] which study outcomes of underspecification in ML pipelines, model multiplicity and Rashomon sets. In the restricted context of personalized prediction, Monteiro Paes et al. [46], discusses related limits to testing and estimation. Schaeffer et al. [52] discuss whether seemingly emergent capabilities of LLMs are rather a result of insufficient metrics. In statistics, Meng [43] analyzed a scaling-related question similar to this paper: Given a carefully collected survey with low response rate (small data) or a large, self-reported dataset without data curation (big data), which dataset should one trust more to estimate population averages? Outside machine learning, validity theory has a long history in fields such as psychology and sociology. Here, test validity is considered a measure of the degree to which a test measures what it is intended to measure [18] and has been studied extensively in the context of psychological tests [45] and educational testing [32]. Increasingly, these notions of validity, have also been considered in machine learning [17, 51, 50, 2].

With regard to technical tools, this paper is also closely related to prior work in matrix completion. For instance, [33] studied the problem of unique and finite completability of matrices and derived similar 𝑘 -core related bounds using determinantal varieties and algebraic geometry. Srebro et al. [57] studied the problem of matrix completion based on non-uniform samples such as power-laws but assume that S = T . Meka et al. [42] focused on power-law samples for S ≠ T and, consistent with this work, require at least 𝑘 samples per row and column to guarantee completability of a rank𝑘 matrix. Cheng et al. [15] derive similar results based on graph 𝑘 -connectivity. Related to non-i.i.d. observations, [38] developed a framework to provide necessary conditions for matrix completion under deterministic sampling. Lemma 1 is based on these results. Different to these prior works, I provide formal impossibility results for test validity based on passive data in complex social systems. This allows to gain rigorous insights into the epistemic limits of what we can know based on this form of data collection. See also supp. J for further related work.

6 Discussion

The results in this paper provide new insights into the validity of the train-test paradigm when data is passively collected from complex social systems. In particular, I have shown that there exists no free delivery service of data that allows for test validity on a global scale in this setting. While valid inferences are possible with respect to the sampling distribution S and within high 𝑘 -cores, they are unlikely if T extends to the entirety of the system. Hence, test validity depends on the interplay between task ( T ), the complexity of the system ( A ), and the 𝑘 -connectivity of the sample graph ( S ) underlying the observed data ( D ), what is a combinatorial property of the data. These results are attained by establishing novel necessary conditions for which validation is possible. As AI systems are increasingly applied in conditions for which sufficient conditions of validity are difficult to guarantee, understanding such minimal conditions can provide guidelines into developing better and more robust systems. Importantly, it can help to demarcate inference goals that are not meaningful from ones that are attainable. It helps to understand the limits of what we can know and

which questions are futile to ask. This work provides a first step in this direction by establishing such epistemic limits of AI in complex social systems.

Furthermore, I have shown that the sub-system for which valid inferences are possible shrinks rapidly with the complexity of the system and that a naïve application of the scaling paradigm is prohibitively inefficient to overcome these validity issues. As a consequence, solving many complex AI tasks are unlikely to come for free through scaling or for cheap through extrapolating from limited smallscale benchmarks. Instead, there exists an inherent trade-off between data quality, quantity, and task complexity. If we want to avoid asking AI systems to solve simpler tasks (e.g., non-out-of-distribution or smaller scope), new data curation efforts are likely needed. Due to the substantial amount of data that would have to be collected, centralized data collection is often infeasible to overcome the validity issues of this paper. Instead, decentralized methods such as participatory data curation could provide a way forward. This aligns with insights from fairness which also highlight the need for participatory methods in data collection [31]. Similar arguments apply to the importance of open science and open-source models in this context.

Importantly, the theoretical results of this paper also provide direct insights into how to improve data collection for model validation via its 𝑘 -core conditions. In particular, lemma 1 and corollary 2 imply two clear objectives for targeted data collection: (a) collecting data points that increase the 𝑘 -connectivity of the sample graph and (b) collecting data points that increase the size of the rank ( 𝑓 ) -core of the sample graph, where rank ( 𝑓 ) is the complexity of the world that we want to assume. Pursuing (a) would increase the complexity of the world that can be assumed such that model validation is still valid for the entire sample graph, while pursuing (b) would increase the size of the subgraph for which a rank ( 𝑓 ) = 𝑘 assumption would still yield valid model validation. Hence, both objectives are based on the k-core conditions of this work and can be computed from a given sample graph. Creating new mechanisms for efficient data collection based on these insights is therefore a very promising avenue for future work.

Acknowledgments

I gratefully acknowledge the valuable feedback from Léon Bottou, Smitha Milli, Tina Eliassi-Rad, Mark Tygert, and anonymous reviewers which all helped to improve various versions of this paper.

References

Himan Abdollahpouri. 'Popularity bias in ranking and recommendation'. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society . 2019, pp. 529-530.

Rediet Abebe. Algorithms on the Bench: Examining Validity of ML Systems in the Public Sphere . 2022.

Chris Anderson. The End of Theory: The Data Deluge Makes the Scientific Method Obsolete . en. June 23, 2008. (Visited on 02/27/2024).

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 'DBpedia: A Nucleus for a Web of Open Data'. In: The Semantic Web . Springer Berlin Heidelberg, 2007, pp. 722-735.

Albert-László Barabási and Réka Albert. 'Emergence of scaling in random networks'. In: Science 286 (5439 1999), pp. 509-512.

Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities . en. MIT Press, Dec. 19, 2023. 340 pp.

Emily Black, Manish Raghavan, and Solon Barocas. 'Model Multiplicity: Opportunities, Concerns, and Solutions'. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea). FAccT '22. New York, NY, USA: Association for Computing Machinery, June 20, 2022, pp. 850-863. (Visited on 02/27/2024).

Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert. 'Efficient and Modular Implicit Differentiation'. In: arXiv preprint arXiv:2105. 15183 (2021).

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 'Freebase: a collaboratively created graph database for structuring human knowledge'. In: Proceedings of the 2008 ACM SIGMOD international conference on Management of data (Vancouver, Canada). SIGMOD '08. New York, NY, USA: Association for Computing Machinery, June 9, 2008, pp. 1247-1250. (Visited on 12/31/2023).

Léon Bottou. Two big challenges in machine learning . 2015.

Léon Bottou and Bernhard Schölkopf. 'Borges and AI'. In: arXiv [cs.CL] (Sept. 27, 2023).

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs . Comp. software. Version 0.3.13. 2018.

Anna D Broido and Aaron Clauset. 'Scale-free networks are rare'. In: Nat. Commun. 10 (1 2019), pp. 1-10.

Allison J B Chaney, Brandon M Stewart, and Barbara E Engelhardt. 'How algorithmic confounding in recommendation systems increases homogeneity and decreases utility'. In: Proceedings of the 12th ACM conference on recommender systems . 2018, pp. 224-232.

Dehua Cheng, Natali Ruchansky, and Yan Liu. 'Matrix completability analysis via graph k-connectivity'. In: Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics . Ed. by Amos Storkey and Fernando Perez-Cruz. Vol. 84. Proceedings of Machine Learning Research. PMLR, 2018, pp. 395-403.

Lu Cheng, Kush R Varshney, and Huan Liu. 'Socially Responsible AI Algorithms: Issues, Purposes, and Challenges'. en. In: jair 71 (Aug. 28, 2021), pp. 1137-1181. (Visited on 12/25/2023).

Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. 'A validity perspective on evaluating the justified use of data-driven decision-making algorithms'. In: arXiv [cs.LG] (June 29, 2022). (Visited on 11/01/2024).

Lee J Cronbach and Paul E Meehl. 'Construct validity in psychological tests'. In: Psychol. Bull. 52 (4 1955), p. 281.

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D Sculley. 'Underspecification presents challenges for credibility in modern machine learning'. In: J. Mach. Learn. Res. 23 (1 Jan. 1, 2022), pp. 10237-10297.

Inderjit S Dhillon and Joel A Tropp. 'Matrix nearness problems with Bregman divergences'. In: SIAM J. Matrix Anal. Appl. 29 (4 Jan. 2008), pp. 1120-1146.

Abhimanyu Dubey et al. 'The Llama 3 herd of models'. In: arXiv [cs.AI] (July 31, 2024). (Visited on 10/30/2024).

Francesco Fabbri, Francesco Bonchi, Ludovico Boratto, and Carlos Castillo. 'The Effect of Homophily on Disparate Visibility of Minorities in People Recommender Systems'. en. In: ICWSM 14 (May 26, 2020), pp. 165-175. (Visited on 12/25/2023).

Vitaly Feldman. 'Does learning require memorization? a short tale about a long tail'. en. In: Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing . STOC '20: 52nd Annual ACM SIGACT Symposium on Theory of Computing (Chicago IL USA). New York, NY, USA: ACM, June 22, 2020. (Visited on 04/28/2024).

Vitaly Feldman and Chiyuan Zhang. 'What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation'. In: Advances in Neural Information Processing Systems . Ed. by H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 2881-2891.

Aaron Fisher, C Rudin, and F Dominici. 'All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously'. In: J. Mach. Learn. Res. 20 (177 Jan. 4, 2018), pp. 1-81. (Visited on 04/28/2024).

Philippe Flajolet, Danièle Gardy, and Loÿs Thimonier. 'Birthday paradox, coupon collectors, caching algorithms and self-organizing search'. en. In: Discrete Appl. Math. 39 (3 Nov. 11, 1992), pp. 207-229.

F Maxwell Harper and Joseph A Konstan. 'The MovieLens Datasets: History and Context'. In: ACM Trans. Interact. Intell. Syst. 5 (4 Dec. 22, 2015), pp. 1-19.

David Hume. A Treatise of Human Nature (I) of the understanding . Vol. 1. 1739. (Visited on 12/21/2023).

David Hume. An Enquiry Concerning Human Understanding . 1748.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, and Pascale Fung. 'Survey of Hallucination in Natural Language Generation'. In: arXiv [cs.CL] (Feb. 8, 2022).

Eun Seo Jo and Timnit Gebru. 'Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning'. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain). FAT* '20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 306-316.

Michael T Kane. 'Validating the interpretations and uses of test scores'. en. In: J. Educ. Meas. 50 (1 Mar. 2013), pp. 1-73.

Franz J Király, Louis Theran, and Ryota Tomioka. 'The algebraic combinatorial approach for low-rank matrix completion'. In: J. Mach. Learn. Res. 16 (1 2015), pp. 1391-1436.

Karl Krauth, Yixin Wang, and Michael I Jordan. 'Breaking Feedback Loops in Recommender Systems with Causal Inference'. In: arXiv preprint arXiv:2207. 01616 (2022).

Thomas S Kuhn. The structure of scientific revolutions . en. 2nd Edition. Chicago, IL: University of Chicago Press, Apr. 1, 1970. 210 pp.

Hugh Lacey. Is Science Value Free?: Values and Scientific Understanding . Taylor Francis, 2005. (Visited on 01/24/2024).

David Liu, Virginie Do, Nicolas Usunier, and Maximilian Nickel. 'Group fairness without demographics using social networks'. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency . ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA, 2023). FAccT '23. New York, NY, USA: Association for Computing Machinery, June 12, 2023, pp. 14321449. (Visited on 04/28/2024).

Guangcan Liu, Qingshan Liu, Xiao-Tong Yuan, and Meng Wang. 'Matrix completion with deterministic sampling: Theories and methods'. In: IEEE Trans. Pattern Anal. Mach. Intell. 43 (2 2019), pp. 549-566.

Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 'Towards Out-Of-Distribution Generalization: A Survey'. In: arXiv [cs.LG] (Aug. 31, 2021).

Benjamin M Marlin and Richard S Zemel. 'Collaborative prediction and ranking with non-random missing data'. In: Proceedings of the third ACM conference on Recommender systems . 2009, pp. 5-12.

Charles T Marx, Flavio du Pin Calmon, and Berk Ustun. 'Predictive Multiplicity in Classification'. In: arXiv [cs.LG] (Sept. 14, 2019).

Raghu Meka, Prateek Jain, and Inderjit Dhillon. 'Matrix Completion from Power-Law Distributed Samples'. In: Advances in Neural Information Processing Systems . Ed. by Y Bengio, D Schuurmans, J Lafferty, C Williams, and A Culotta. Vol. 22. Curran Associates, Inc., 2009.

Xiao-Li Meng. 'Statistical Paradises And Paradoxes In Big Data (I) Law Of Large Populations, Big Data Paradox, And The 2016 US Presidential Election'. In: Ann. Appl. Stat. 12 (2 2018), pp. 685-726.

Merriam Webster Dictionary. Social system . en. (Visited on 10/30/2024).

Samuel Messick. 'Meaning and Values in Test Validation: The Science and Ethics of Assessment'. In: Educ. Res. 18 (2 1989), pp. 5-11.

Lucas Monteiro Paes, Carol Long, Berk Ustun, and Flavio Calmon. 'On the Epistemic Limits of Personalized Prediction'. In: Advances in Neural Information Processing Systems . Ed. by S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 19791991.

OpenAI et al. 'GPT-4 Technical Report'. In: arXiv [cs.CL] (Mar. 15, 2023).

Fragkiskos Papadopoulos, Maksim Kitsak, M Ángeles Serrano, Marián Boguná, and Dmitri Krioukov. 'Popularity versus similarity in growing networks'. In: Nature 489 (7417 2012), p. 537.

Daniel L Pimentel-Alarcón, Nigel Boston, and Robert D Nowak. 'A characterization of deterministic sampling patterns for low-rank matrix completion'. In: IEEE J. Sel. Top. Signal Process. 10 (4 2016), pp. 623-636.

Deborah Raji. There's more to data than distributions . Mar. 31, 2022. (Visited on 03/31/2022).

Benjamin Recht. Machine Learning has a validity problem . 2022. (Visited on 03/15/2022).

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 'Are Emergent Abilities of Large Language Models a Mirage?' In: Thirty-seventh Conference on Neural Information Processing Systems . Nov. 2, 2023. (Visited on 05/13/2024).

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 'Recommendations as Treatments: Debiasing Learning and Evaluation'. In: Proceedings of The 33rd International Conference on Machine Learning . Ed. by Maria Florina Balcan and Kilian Q Weinberger. Vol. 48. Proceedings of Machine Learning Research. New York, New York, USA: PMLR, 2016, pp. 1670-1679.

Lesia Semenova, Cynthia Rudin, and Ronald Parr. 'On the existence of simpler machine learning models'. en. In: 2022 ACM Conference on Fairness, Accountability, and Transparency . FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul Republic of Korea). New York, NY, USA: ACM, June 21, 2022. (Visited on 04/28/2024).

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . en. Cambridge University Press, May 19, 2014. 415 pp.

Herbert A Simon. 'On a class of skew distribution functions'. In: Biometrika 42 (3/4 1955), pp. 425-440.

Nathan Srebro and Russ R Salakhutdinov. 'Collaborative Filtering in a Non-Uniform World: Learning with the Weighted Trace Norm'. In: Advances in Neural Information Processing Systems . Ed. by J Lafferty, C Williams, J Shawe-Taylor, R Zemel, and A Culotta. Vol. 23. Curran Associates, Inc., 2010.

Aarohi Srivastava et al. 'Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models'. In: Transactions on Machine Learning Research (Jan. 19, 2023). (Visited on 01/06/2024).

Harald Steck. 'Training and testing of recommender systems on data missing not at random'. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining . 2010, pp. 713-722.

Tom F Sterkenburg and Peter D Grünwald. 'The no-free-lunch theorems of supervised learning'. In: Synthese 199 (3-4 2021), pp. 9979-10015.

Mark Steyvers and Joshua B Tenenbaum. 'The Large-scale structure of semantic networks: Statistical analyses and a model of semantic growth'. In: Cogn. Sci. 29 (1 2005), pp. 41-78.

Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 'Yago: a core of semantic knowledge'. In: Proceedings of the 16th international conference on World Wide Web (Banff, Alberta, Canada). WWW '07. New York, NY, USA: Association for Computing Machinery, May 8, 2007, pp. 697-706. (Visited on 12/31/2023).

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 'Llama 2: Open Foundation and Fine-Tuned Chat Models'. In: arXiv [cs.CL] (July 18, 2023).

Ivan Voitalov, Pim van der Hoorn, Remco van der Hofstad, and Dmitri Krioukov. 'Scale-free networks well done'. In: Phys. Rev. Res. 1 (Nov. 5, 2018), p. 033034.

Denny Vrandeˇ ci´ c and Markus Krötzsch. 'Wikidata: a free collaborative knowledgebase'. In: Commun. ACM 57 (10 Sept. 23, 2014), pp. 78-85.

Jamelle Watson-Daniels, David C Parkes, and Berk Ustun. 'Predictive multiplicity in probabilistic classification'. en. In: Proc. Conf. AAAI Artif. Intell. 37 (9 June 26, 2023), pp. 10306-10314. (Visited on 04/28/2024).

Wikipedia contributors. Wikipedia:Who writes Wikipedia . (Visited on 01/24/2024).

David H Wolpert. 'The lack of a priori distinctions between learning algorithms'. In: Neural Comput. 8 (7 1996), pp. 1341-1390.

Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 'Improving recommendation lists through topic diversification'. en. In: Proceedings of the 14th international conference on World Wide Web - WWW '05 . the 14th international conference (Chiba, Japan). New York, New York, USA: ACM Press, 2005. (Visited on 01/04/2024).

Supplementary Information

A Notation

Random variables are denoted by italic uppercase letters, e.g., 𝐿, 𝑆, 𝑋 . Sets are denoted by calligraphic uppercase letters, e.g., X , S . Constants are indicated with lowercase greek letters, e.g., 𝜖, 𝜌 . Functions and scalar are denoted by lowercase letters, e.g., 𝑓 , 𝑔, ℎ and 𝑥, 𝑦 . Matrices and higher-order tensors are indicated with bold uppercase letters, e.g., F , U .

B Validity framework

In this work, I am interested in the validity of inference settings , i.e., whether assumptions and observations allow for any valid inferences at all. To formalize this, I will take the following high-level approach:

Inference setting An inference setting consists of a set of assumptions A , a fixed dataset D which is collected from a sampling distribution S , and a target distribution T for which we want to make inferences. Note that S is not guaranteed to be identical to T . Hence, we're concerned with out-of-distribution generalization settings.

Expected risk over possible worlds Assumptions A and observed data D define a set of possible worlds F that is consistent with A and D . Given a probability distribution F over F , I am then interested in the expected risk over all possible worlds that are consistent with A and D .

Validity An inference setting is valid, if the expected risk over possible worlds can be bounded meaningfully at all, i.e., if there exists at least one hypothesis class for which the generalization error of at least a single hypothesis can be bounded sufficiently.

To approach the question of validity, learning theory has traditionally focused nearly exclusively on sufficient conditions for valid inferences. Under active data collection, i.e., in scenarios where one can control exactly how data is collected, sufficient conditions are highly attractive since they provide exact specifications for inferences to be valid with high probability. However, under passive data collection, the situation is reversed. Sufficient conditions for the validity of inferences usually place highly restrictive demands on the data collection process (e.g., i.i.d. samples or simple random sampling) which are challenging to satisfy even when data is collected carefully in an active way. Since passive data collection, by definition, exerts no control over the sample generating process, these sufficient conditions are not met with near certainty. For this reason, I am focusing here on necessary conditions for validity, i.e., conditions that must always be satisfied for inferences to be valid. Under passive data collection, necessary conditions can provide important insights since they need to hold for any data collection process or, conversely, can be used to identify scenarios where inferences are not valid with high probability.

B.1 Connection to No-Free-Lunch theorems

The validity framework of section 3 and the No-Free-Lunch theorems are closely connected. First, consider the expected risk over all possible worlds relative to F , i.e.,

Equation (1) is then akin to the objectives considered in the seminal No Free Lunch (NFL) theorems [68, 60]. For instance, the NFL theorem for supervised learning can be written as ∀ Λ : E 𝑓 ∼ U E 𝑋 ∼ T [ ℓ ( ℎ Λ ( D ) ( 𝑋 ) , 𝑓 ( 𝑋 ))] = 1 / 2 , where U is the uniform distribution over all possible worlds in an assumption-free setting (i.e., A = ∅ ), ℓ is the 0 / 1 -loss, and ℎ Λ ( D ) is the hypothesis derived from a finite sample D with algorithm Λ . In contrast to the NFL theorems - where A = ∅ implies an induction-hostile universe - my focus is on induction-friendly settings ( A ≠ ∅ ) but where D is sampled from a complex social system. Since 𝐿 T 𝑓 ℎ is a non-negative random variable, we can then connect definition 4 and eq. (1) via upper and lower bounds based on Markov's inequality.

Definition 7 (Markov's inequality) . Let 𝑋 be a non-negative random variable and 𝑎 > 0 . Then

Hence, it follows that the expected risk over all possible worlds is large for invalid settings since it holds that

B.2 Importance of ontological parsimony and test validity in the i.i.d. setting

The strong appeal of the train-test paradigm is that, with careful data collection, we require no further ontological assumptions to ensure the validity of the model validation procedure. In particular, if we have a test set that is sampled independently from T , it follows straightforwardly from Hoeffding's inequality that we can meaningfully bound the approximation error over this test set [55, Theorem 11.1]. Let T ∼ T 𝑚 be a test set of size 𝑚 , sampled i.i.d. from the target distribution T . Then, it holds that

Importantly, this holds for any hypothesis ℎ , any algorithm Δ , and any training set D . Hence, under careful data collection where we know that if the test set is sampled i.i.d. from T , any hypothesis can be validated based on the observed data only .

This property, i.e., that we can evaluate the performance of a model without further assumptions on the model itself, is crucial to compare the performance of different methods since different architecture, inference, and hyperparameter choices correspond to different assumptions. Maybe more importantly, this property is also crucial to validate our model assumptions on observed data (given that the sampling assumption holds), since otherwise we could only make statements relative to that our model assumptions hold, which is, of course, much weaker and not informative. Hence, if we need to make model specific assumptions for the validation error to be informative for the generalization error, the train-test paradigm would be relatively meaningless. Of course, the (considerable) challenge is to collect 𝑚 i.i.d. samples from the true target distribution T which can not be guaranteed and is an important assumption on the data collection process.

𝑓

𝑖 𝑗

𝑆

𝑡

𝑖 𝑗

𝑡

=

1

, . . . , 𝑇

𝑖,

𝑗

=

1

, . . . ,𝑀

𝑖 𝑗

≠

𝑘ℓ

𝑡

′ →

𝑡

(a) Sampling bias

1

2

3

4

5

0

1

2

3

𝛼

1

1.5

2

2.5

3

(b) Pareto distribution

(c) Degree distribution of MOVIELENS 100k

C Complex social systems

In the following, I will discuss how sampling bias and heavy-tailed distributions can occur, and can be connected, in complex social systems. First, sampling bias is concerned with how S is collected. Most standard inference methods assume i.i.d. samples from T , but it is well know that this assumption can be easily violated when sampling in complex systems.

Definition 8 (Sampling bias) . Let S 𝑡 𝑖 𝑗 denote the random variable corresponding to entities ( 𝑖, 𝑗 ) ∈ X 1 × X 2 𝑗 being samples at time 𝑡 . Samples in complex social systems can then neither be assumed to be independent across time nor independent with regard to the target value 𝑓 𝑖 𝑗 , i.e.,

Higher arity relations are defined analogously. See also fig. 4a for the assumed sample dependencies.

Sample biases as in definition 8 can be causes by aspects such popularity bias, i.e., if popular items are more likely to be sampled, and quality biases, i.e., if items with higher values for 𝑓 𝑖 𝑗 are more likely to be sampled.

A prime example of how sampling bias in the form of popularity bias can lead to power-law distributions, is the influential Barabasi-Albert model [5]. In this model of complex networks, nodes are added to a network one by one and are connected to existing nodes with a probability proportional to their degree, i.e., popularity. Formally, this model is defined as follows:

Definition 9 (Barabasi-Albert model) . Let 𝐺 = ( X , E ) be a graph. Furthermore, let P ( 𝑖 ∼ 𝑡 𝑗 ) denote the probability that the edge 𝑖 ∼ 𝑗 is added at time 𝑡 to E and let 𝜅 𝑖 denote the degree of node 𝑥 𝑖 . The Barabasi-Albert model generates then a graph 𝐺 as follows:

Start with a small connected graph 𝐺 0 with 𝑚 nodes.

At each time step 𝑡 > 0 , add a new node 𝑥 to 𝐺 and connect it to 𝑚 existing nodes in 𝐺 with a probability proportional to their degree, i.e.,

It is then well known that definition 9 leads to a power-law degree distribution in 𝐺 , i.e., a distribution where the probability of a node having 𝑘 connections is proportional to 𝑘 -𝛼 for some 𝛼 > 0 .

D Proof corollary 1 (Necessary condition for test validity)

Corollary 1 (Necessary condition for test validity) . Let ( A , D , T , F ) be an inference setting, let ℓ : Y × Y → R + be a positive loss function, and let H be a hypothesis class. Furthermore, let 𝜃 ∈ R + be any risk estimate for ℎ . Then, if ( A , D , T , F ) is ( 𝜖, 𝛿 ) -test-valid, it must hold that

(a)

(b)

Linkprediction-Cora

ExtremeClassification-Wiki10-31K

Source

100

Bibtexentry

Target

e

scal

10-1

10-1

S

601)

g

10-2

Proportion

10-2

Proportion

10-3

10-3

P

10-4

0

25

50

75

100

125

150

0

500

1000

1500

2000

Observations

Observations

Assumptions

A

Possible worlds

F

Hypotheses

H

ℎ

𝑓

★

𝑓

(a) Validity relationships

𝑌

3

11

𝑌

3

12

𝑌

3

13

· · ·

𝑌

3

1

𝑛

𝑌

3

21

𝑌

3

22

𝑌

3

23

𝑌

3

2

𝑛

𝑌

3

31

𝑌

3

32

𝑌

3

33

𝑌

3

3

𝑛

.

.

.

.

.

.

𝑌

3

𝑚

1

𝑌

3

𝑚

2

𝑌

3

𝑚

3

𝑌

3

𝑚𝑛

𝑌

2

11

𝑌

2

12

𝑌

2

13

· · ·

𝑌

2

1

𝑛

𝑌

2

21

𝑌

2

22

𝑌

2

23

𝑌

2

2

𝑛

𝑌

2

31

𝑌

2

32

𝑌

2

33

𝑌

2

3

𝑛

.

.

.

.

.

.

𝑌

2

𝑚

1

𝑌

2

𝑚

2

𝑌

2

𝑚

3

𝑌

2

𝑚𝑛

𝑌

1

11

𝑌

1

12

𝑌

1

13

· · ·

𝑌

1

1

𝑛

𝑌

1

21

𝑌

1

22

𝑌

1

23

𝑌

1

2

𝑛

𝑌

1

31

𝑌

1

32

𝑌

1

33

𝑌

1

3

𝑛

.

.

.

.

.

.

𝑌

1

𝑚

1

𝑌

1

𝑚

2

𝑌

1

𝑚

3

𝑌

1

𝑚𝑛

X

3

X

2

X

1

(b) Tensor representation

(c) 2-core of a graph

Proof. First, note that | 𝜃 -𝐿 T 𝑓 ℎ | ≤ 𝜖 is equivalent to 𝜃 -𝐿 T 𝑓 ℎ ≤ 𝜖 ∧ 𝐿 T 𝑓 ℎ -𝜃 ≤ 𝜖 . Furthermore, we have

It follows then simply from the monotonicity of probability that

E Proof lemma 1 (Rank𝑘 underdetermination)

I will first introduce the concept of S -isomerism and connect it to 𝑘 -connectivity. I will then use these results to proof lemma 1.

First, let S 𝑖, · = { 𝑗 : ( 𝑖, 𝑗 ) ∈ S } denote the set of observed columns for row 𝑖 and S · , 𝑗 = { 𝑖 : ( 𝑖, 𝑗 ) ∈ S } denote the observed rows for column 𝑗 . Let S 𝑖, · [ F ] ∈ R 𝑚 ×| S 𝑖, · | be the sub-matrix of F ∈ R 𝑚 × 𝑛 which is obtained by restricting the columns of F to the indices in S 𝑖, · . Similarly, let S · , 𝑗 [ F ] ∈ R | S · , 𝑗 | × 𝑛 be the sub-matrix of F ∈ R 𝑚 × 𝑛 which is obtained by restricting the rows of F to the indices in S · , 𝑗 . Then, S -isomerism is defined as follows:

Definition 10 ( S -Isomeric) . Let F ∈ R 𝑚 × 𝑛 and let S ⊆ { 1 , . . . , 𝑚 } × { 1 , . . . , 𝑛 } with S · , 𝑗 ≠ ∅ . Then, F is called S -isomeric iff

Corollary 3 (Necessary condition for S -isomerism) . Let S be a sample graph and let rank ( F ) = 𝑘 . If F is S -isomeric, then it must hold that S is 𝑘 -connected.

Proof. First, note that rank ( X ) ≤ min ( 𝑚, 𝑛 ) for any X ∈ R 𝑚 × 𝑛 . Hence, it follows from definition 10 that for a rank𝑘 matrix to be S -isomeric, each column and row needs to have at least 𝑘 observed entries. Since this is equivalent to 𝑘 -connectivity, the result follows. □

Lemma 1 (Rank𝑘 underdetermination) . Let A = { 𝑓 | rank ( 𝑓 ) ≤ 𝑘 } . Then, if S is not 𝑘 -connected, the set of possible worlds F forms a non-empty vector space.

Proof. Since rank ( 𝑓 ) = 𝑘 and S is not 𝑘 -connected, it follows from corollary 3 that 𝑓 is not S -isomeric. Hence, it holds via [38, Theorem 3.2] that there exist infinitely many matrices 𝑓 ′ that all explain the observed data S perfectly, i.e.,

Moreover, it follows from [38, Lemma 5.1] that this set of possible worlds F forms a non-empty vector space V . □

F Proof lemma 2 (Rank𝑘 test invalidity)

To prove lemma 2, I will first show the following auxiliary proposition:

Proposition 1 (Risk inequality for Bregman projection) . Let ℓ : Y × Y → R + be a scalar Bregman divergence and let 𝑓 ★ = arg inf 𝑓 𝐿 T 𝑓 ℎ be the Bregman projection of ℎ onto a vector space F . Then, it holds that

Proof. First, note that 𝐿 T 𝑓 ℎ is simply a convex combination of scalar Bregman divergences, i.e.,

Hence, 𝐿 T 𝑓 ℎ itself is a (separable) Bregman divergence. Proposition 1 follows than from the generalized Pythagorean theorem for Bregman divergences [20, Eq. 2.3] since every vector space is a convex set and 𝑓 ★ is the projection of ℎ onto F , i.e., it holds that

Lemma 2 (Rank𝑘 test-invalidity) . Let A be identical to lemma 1, let ℓ be a scalar Bregman divergence, let F be the uniform distribution over F , and let T be the uniform distribution over X . Furthermore, let 𝜃 ∈ R + be any risk estimator on a test set. Then, if S is not 𝑘 -connected, ( A , D , T , F ) is test-invalid, i.e., it holds for any 𝜖 > 0 that

Proof. Since the standard uniform distribution is not defined on an entire vector space, I will instead consider the limit of the class of uniform distributions of balls of radius 𝑟 . Next, since ℓ is Borel measurable, B ( 𝑓 , 𝜖 ) = { 𝑓 ′ | 𝐿 T 𝑓 𝑓 ′ < 𝜖 } defines a measurable set around each 𝑓 ∈ F . Furthermore, let Vol B ( 𝑓 , 𝜖 ) denote the volume of such an 𝜖 -'ball'. Via the change of variables formula, we know

then that the volume of B ( 𝑓 , 𝑟 · 𝜖 ) , i.e., the volume of the original ball stretched in all directions by 𝑟 ≥ 1 is given by

Next, let U 𝑟 denote the uniform distribution over B ( 𝑓 , 𝑟 · 𝜖 ) . Then, the probability of sampling a point inside B ( 𝑓 , 𝜖 ) when drawing points uniformly from B ( 𝑓 , 𝑟 · 𝜖 ) with 𝑟 ≥ 1 is given by

Moreover, if V is non-empty it follows that dim V ≥ 1 . Hence, as we increase 𝑟 to span large parts of V , it holds that

Using eq. (2) I will then show lemma 2 by considering the two cases ℎ ∈ F and ℎ ∉ F .

If we assume ℎ ∈ F , lemma 2 follows directly from corollary 1 and F being a non-empty vector space according to lemma 1 (since S is not 𝑘 -connected).

Onthe other hand, if ℎ ∉ F , consider the projection of ℎ onto F according to ℓ , i.e., 𝑓 ★ = arg min 𝑓 𝐿 T 𝑓 ℎ . Since 𝐿 T 𝑓 ℎ is a Bregman divergence and F is a vector space, lemma 2 follows then from corollary 1, proposition 1 and the monotonicity of probability since 𝐿 T 𝑓 𝑓 ★ ≤ 𝐿 T 𝑓 ℎ . □

G Extensions to Higher-arity relations and bounded domains

G.1 Ternary and higher arity relations

First, note that higher arity functions can be represented as tensors of the same order as follows (see also fig. 6b for a visualization):

Definition 11 (Tensor representation) . For a function 𝑓 : X 1 × X 2 × · · · × X 𝑘 → R over finite sets of size | X 𝑖 | = 𝑚 𝑖 , we can construct its tensor representation F ∈ R 𝑚 1 × 𝑚 2 ×···× 𝑚 𝑘 via F 𝑖 𝑗...,𝑘 = 𝑓 ( 𝑥 𝑖 , 𝑥 𝑗 , . . . , 𝑥 𝑘 ) for all 𝑥 𝑖 ∈ X 1 , 𝑥 𝑗 ∈ X 2 , . . . , 𝑥 𝑘 ∈ X 𝑘 .

A trivial extension of theorem 1 and its related results can then be obtained by considering the rank of the projection of the tensor representation of 𝑓 onto its matrix presentation such that F 𝑖 𝑗 = ˝ 𝑘 𝑓 ( 𝑖, 𝑗 , 𝑘 ) (what equals the assumption that the rank of the predicate mode in 𝑓 is one). Since predicates in many knowledge graphs are very sparse, the sum over 𝑘 will often preserve this sparsity and the associated heavy-tailed distributions. In this case, the results of the matrix case extend directly to the tensor case. If this is not the case, it is necessary to extend the matrix analysis of lemma 1 to the tensor case and consider cases where the predicate mode can have rank larger than one. However, this is beyond the scope of this paper and reserved for future work.

G.2 Bounded domains

In many practical applications, the output domain of the function 𝑓 is bounded, i.e., Y ∈ [ 0 , 1 ] . In this case, the main results of this paper can possibly be extended if 𝑓 can be expressed in terms of a bijective link function 𝑔 , i.e., 𝑓 ( 𝑥 ) = 𝑔 ( 𝜙 ( 𝑥 )) where 𝜙 : X → R . Common link functions that are bijective include the logit and probit functions (binary variable) and the log function (Poisson variable). In the following, I provide a brief outline of the argument: Since 𝑔 is bijective, its inverse exists and we can work on Φ by applying 𝑔 -1 to 𝑓 . Next, for each column (or row) of Φ , consider the regression problem Φ 𝑖 = 𝑈𝑣 𝑖 with the goal of modeling the observed entries of Φ . If the dimensionality (rank) of 𝑈 and 𝑣 𝑖 is larger than the number of observed entries in Φ 𝑖 , this regression problem becomes an underdetermined system. For areas of the sample graph that are too sparse, we can then find again infinitely many matrices that match all the observed entries of Φ but are different on unobserved entries. The main results of this paper would then extend directly to Φ and via the link function 𝑔 to the bounded domain. One difference to the results in the unbounded case is that the above argument holds for the degrees in the sample graph, while the unbounded case holds for the stronger 𝑘 -core condition.

H Inefficiency of scaling and benchmarks

Lemma 1 allows to answer the scaling question by asking how many draws from S would be necessary such that all nodes are within the 𝑘 -core of S with high probability, i.e., how many samples are needed until arriving at a valid test setting. In the following, I will discuss different ways to approximate this question.

H.1 Scaling and the coupon collector problem

One way to lower bound the number of samples needed to arrive at a valid test setting would be to calculate the number of samples needed to sample each node outside the required k-core at least once. This is an instance of the coupon collector problem with unequal probabilities . In particular, let 𝑇 𝑘 be the number of draws from S until we have collected 𝑘 distinct nodes from X 2 for an arbitrary node in X 1 . Then, it follows from [26, Corollary 4.2] that

where 𝑃 𝐽 = ˝ 𝑗 ∈ 𝐽 𝑝 𝑗 and where ˝ | 𝐽 | = 𝑞 denotes the sum over all subsets 𝐽 of size 𝑞 . However, eq. (3) is hard to interpret and for that reason not very useful for our purposes. Moreover, eq. (3) is not even tractable to compute at the scale that we would require for the settings considered in this paper. For instance, assume that we are dealing with a relatively small-scale domain of | X | = 10 7 entities. Since eq. (3) requires to over all subsets of size 𝑘 -1 , for a model of rank 𝑘 = 10 this operation alone would require more than

H.2 Proof for scaling bound in corollary 2

Since the coupon collector problem is not computable, corollary 2 considers an even weaker lower bound and asks how many samples are needed to sample an average node at least once. For a fixed node 𝑥 𝑖 , this is an instance of number of trials until first success and follows a geometric distribution with expected value 𝑇 𝑖 = 1 / 𝑝 𝑖 . Next, for a power-law distribution with P ( 𝑋 > 𝑥 ) = 𝑢 ( 𝑥 ) 𝑥 -𝛼 it holds that 𝑃 ( 𝑋 = 𝑥 ) = 𝑢 ′ ( 𝑥 ) 𝑥 -( 𝛼 + 1 ) where 𝑢 ′ is also a slowly varying function. Hence, we have

To illustrate eq. (4), consider the following example using the Pareto distribution to instantiate 𝑈 ′ . In this case, we have 𝑇 𝑖 = 𝑥 𝛼 + 1 𝑖 / 𝛼𝑥 𝛼 min . For a random node in X os size 𝑛 , it holds then that

where the inequality follows from Jensen's inequality for 𝛼 > 0 . This concludes the proof.

I Experiments

All experiments were computed on a single NVIDIA Volta V100 GPU and implemented using Jax [12], Jaxopt [8], Numpy, and Scipy. All experiments were computed on the MOVIELENS 100k benchmark [27] which is available at ❤/a116/a116♣/a115✿✴✴❣/a114♦✉♣❧❡♥/a115✳♦/a114❣✴❞❛/a116❛/a115❡/a116/a115✴♠♦✈✐❡❧❡♥/a115✴✶✵✵❦✴ and released under a custom license ❤/a116/a116♣/a115✿✴✴❢✐❧❡/a115✳❣/a114♦✉♣❧❡♥/a115✳♦/a114❣✴❞❛/a116❛/a115❡/a116/a115✴♠♦✈✐❡❧❡♥/a115✴ ♠❧✲✶✵✵❦✲❘❊❆❉▼❊✳/a116①/a116 .

I.1 Computing possible worlds under rank constraints

To find possible worlds that fit the observed data under a rank-constraint, I will first compute a single subspace for which we can model all observed data with highest accuracy. For this purpose, I am first fitting a matrix F = UV ⊤ to the observed entries under a rank constraint, i.e., via min ∥ F S -Y S ∥ 2 𝐹 where the constraint rank ( F ≤ 𝑘 ) is enforced simply via U ∈ R 𝑚 × 𝑘 , V ∈ R 𝑛 × 𝑘 . Next, let U = QR

be the QR decomposition of U . Then, we know that the set of possible worlds within the subspace spanned by Q ∈ R 𝑛 × 𝑘 must be of the form P Q ∩ P S where

are the orthogonal projections onto the column space of Q and the observed entries, respectively. Furthermore, assume that we have already found 𝑝 matrices that are of rank 𝑘 and which fit the observed entries Y S with high accuracy. We can then find the 𝑝 + 1 -th matrix by minimizing the following objective:

Importantly, the experimental results in section 3 and fig. 3 hold already for only a single subspace U and considering further subspaces that also explain the observed data can only increase the differences between possible worlds reported in these experimental results.

I.2 Area over the eCDF as expected risk

In the following, I will discuss how the area over the eCDF curves in fig. 3b correspond to the risk 𝐿 U 𝑓 𝑓 ′ between these pairs of possible worlds. In particular, let 𝐸 be the random variable corresponding to the normalized absolute error of entries in possible worlds 𝑓 and 𝑓 ′ . Furthermore, let 𝐹 𝐸 ( 𝑥 ) = P ( 𝐸 ≤ 𝑥 ) be the CDF of 𝐸 . The expected error between both possible worlds (in terms of NAE) is then equivalent to the area over the curve of eCDF, i.e.,

(a) eCDF of Maximum NAE

(b) eCDF of Pairwise NAE

1.0

1.0

0.8

80thPercentile.

0.8

80thPercentile.

Proportion

Proportion

0.5

50thPercentile.

0.5

50thPercentile

0.2

20thPercentile

0.2

20thPercentile

0.0

0.0

0

0.7

0.00

0.230.31

1.00

(uwx- xewx) / l/x - !!xl xu

[Xi-X|l / (Xmax-Xmin)

X,X'

J Related work

In statistics, Meng [43] analyzed a scaling-related question similar to this paper: Given a carefully collected survey with low response rate (small data) or a large, self-reported dataset without data curation (big data), which dataset should you trust more to estimate population averages? For this purpose, Meng introduces an Euler-formula-like identity which connects estimation quality to data quality , data quantity , and problem difficulty . Similar to the results in this paper, Meng shows that data quantity is highly inefficient to overcome issues in data quality, especially sampling related issues. While related in spirit, the results in this paper go beyond the question of surveying and population averages and establish related results in the more general context of inductive inference via formalizing properties of complex social systems and their impact on validity of inferences.

In motivation, this paper is also related to the work of D'Amour et al. [19] who study underspecification of machine learning pipelines as a cause for inference failures. In this context, a machine learning pipeline is 'the full procedure followed to train and validate a predictor'. A machine learning pipeline is then considered underspecified when it can return many distinct predictors with equivalently strong test performance. This notion of underspecification is closed related to the concepts of possible worlds and validity in this paper.

Srebro et al. [57] studied the problem of matrix completion based on non-uniform observations such as power-laws. However, in contrast to this work, Srebro et al. [57] assume that S = T . The advantage of this assumption is that it leads to a much simplified learning setting in which valid inferences are indeed possible. However, as I discuss in section 2, I would argue that this is not the problem that many inference settings are concerned with (and that it is questionable in a matrix completion setting as well). Further important results in this context include [49] on low-rank matrix completion from deterministic samples as well as the work of Schnabel et al. [53] and Marlin et al. [40] on learning from biased samples. In contrast to these prior works, I am expanding the setting to the validity of inferences and validation, provide necessary conditions, and situate them explicitly in the context of complex social systems.

K Limitations

As most theoretical work, this work needs to make certain assumptions to make the phenomena of interest amenable to analysis. In this work, the core assumption is that samples in complex social systems follow a heavy-tailed distribution. While this is a very robust finding in social science and widely supported, as discussed in section 2, it limits the results of this paper to this specific setting. For further analysis, this paper further assumes that this heavy-tailed distribution follows a regularlyvarying power-law. This is again a supported assumption [64] and allows for a clean theoretical analysis. However, as discussed in section 2, it is still disputed whether samples in complex social systems actually follow this particular form. However, it is undisputed that they follow a heavy-tailed distribution, and as such, while the power-law based results might not apply exactly, their general implications are still supported.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Proofs for impossibility results as well as experimental evidence are provided in section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Limitations are discussed throughout the paper and further summarized in supp. K.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proof sketches are provided in the main paper, full proofs are included in the supplementary material. All results include a clear statement of assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The main paper includes an overview of the experimental results. All experimental detail can be found in the supplementary material supp. I. Datasets are publicly available (MovieLens).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We intend to release the code for experiments and proofs for the camera ready version. However, at current time, we are not able to provide access to the code. Yet, all experiments should be easily reproducible from the information in the main text and supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See supp. I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not applicable to the experimental analysis of this paper. However, fig. 3b shows the variability over possible worlds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See supp. I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: The research conducted in this paper conforms with the NeurIPS Code of Ethics, i.e., none of the potential harms apply to this work, societal impact of this work is discussed, and impact mitigations are considered as far as they apply.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper is largely theoretical and focused on the validity of the train-test paradigm in complex social systems. As such it has direct connections to understanding and improving the social impact of deployed system. While the evaluation of this aspect is beyond the scope of this aspect, I provide discussions of connections to societal aspects such as fairness and participatory data collection. I also discuss fairness aspects in the MovieLens experiments in section 4.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is purely theoretical and does not release models or data that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The only existing assest used in the paper is the MovieLens 100k dataset for which citations are provided and the license is linked in supp. I.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does currently not release new assets. In the future, code will be released under a CC-BY-NC license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work shows that the classic paradigm of training and testing in ML has validity flaws that makes it unreasonable to generalize about performance from test-set performance. They use the no free lunch theorems to theoretically demonstrate this, and then show empirical lens on the MovieLens benchmark dataset for recommender systems. They show that multiple models are able to fit the observed data but still perform arbitrarily well on unobserved data.\n\nSTRENGTHS:\nThe paper is very well-written overall. It tackles a very important question that has many implications given how popular the train-test paradigm is.\n\nWEAKNESSES:\nI’m ultimately not very sure what the prescriptive argument of this work is. I understand that there is value in work for pointing out a problem, but the issues with generalizing performance of ML systems into the real world is widely known and empirically shown (as demonstrated by the presence of the WILDS dataset [https://wilds.stanford.edu/datasets/]. This work provides a theoretical grounding for this, but from the abstract’s final sentence, I expected more implications rather than just gestures to participation and open science. I would have liked to see implications which demonstrate the strengths of approaching this problem from a theoretical perspective, as opposed to just the empirical evidence we have of it.\n\nThere is also increasingly a line of work on the limitations of using observational data (as opposed to data from, e.g., randomized controlled trials, as well as performative prediction, which I felt this work could connect to more, since that seems to be part of the premise of the issue with passive data collection.\n\nA smaller point of confusion, I was unclear sometimes whether you were referring to “benchmark” only as the evaluation dataset or also the training dataset.\n\nQUESTIONS:\nAbove, in weaknesses.\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding advanced and very large general-purpose ML models using extremely large datasets from sources such as the Internet has gained a lot of recent attention.\nParticularly, when the training set is sampled from a distribution S while the target distribution is T, the paper introduces the notions of (\\epsilon,\\alpha)-validity and (\\epsilon,\\alpha)-invalidity for inference validity and defines test validity accordingly. \nThe paper shows that there is \"no free delivery service\" of data that allows inference/test validity on a global scale for complex social systems.\n\nMore importantly, the paper provides the metrics and necessary conditions that limit the scope of AI in complex social systems.\n\nSTRENGTHS:\nS1. The paper studies a problem of immense importance: with the rapid growth of advanced ML models such as LLMs and their versatile usage across a wide range of tasks that impact societies and human lives, establishing the necessary conditions and checks and balances to limit their scope is vital. Formally introducing such metrics and conditions, this paper is an interesting step in that direction. \n\nS2. The paper follows a formal and careful writing scheme, which makes it easy to follow and fun to read.\n\nS3. The proposed validity framework and the inference and test validity concepts provide metrics for evaluating the validity extension of a model trained on a distribution S to T.\n\nWEAKNESSES:\nW1. While I enjoyed reading the formality and the definitions in section 2, I did not find the so-called no delivery service (NDS) of data surprising.\nAs the author also stated, based on the theory of ML and the NFL theorem, when there is a sampling bias, and the source and target distributions are different, the expected performance guarantee of the model does not carry over. This has also been extensively discussed under \"lack of generalizability\" concept.\n\nW2. It is not particularly a weakness, but if I am not mistaken, the NDS observation for complex social systems naturally follows the previously known heavy-tailed distribution property of these systems.\n\nQUESTIONS:\nNo specific questions\n\nLIMITATIONS:\nPlease see the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the critical issue of model validation in AI systems, especially those deployed in complex social environments. It argues that the prevalent train-test paradigm, commonly used for model validation, is often invalid in these settings due to the inherent assumptions it violates. The paper presents formal impossibility results, demonstrating that for many AI tasks involving complex social systems, the train-test paradigm cannot ensure valid model performance assessments. The study uses the MOVIELENS benchmark to illustrate these issues and suggests remedies like participatory data curation and open science to address the epistemic limitations identified.\n\nSTRENGTHS:\n1. The paper provides novel insights into the limitations of the train-test paradigm, especially in the context of AI systems interacting with complex social systems.\n2. It offers formal proofs of the epistemic limitations, providing a robust theoretical foundation for its claims.\nRelevance: The study addresses a highly relevant issue in modern AI, given the increasing deployment of AI systems in socially impactful contexts.\n3. Using the MOVIELENS benchmark to illustrate the theoretical points adds practical relevance and makes the arguments more tangible.\n\nWEAKNESSES:\n1. The formal proofs and theoretical discussions might be too complex for practitioners without a strong background in the relevant mathematical and statistical concepts.\n2. While the paper suggests remedies, it does not delve deeply into how these can be practically implemented on a large scale, which could limit their immediate applicability.\n3. The results are heavily dependent on the specific characteristics of the social systems and data collection methods considered, which might limit the generalizability of the findings to other contexts or systems.\n\nQUESTIONS:\n1. Could you elaborate on alternative model validation paradigms that could be more suitable for complex social systems?\n2. Could you provide more examples or case studies beyond the MOVIELENS benchmark to illustrate the validity issues in different types of AI systems?\n3. Could you discuss any potential limitations of your approach and how they might be addressed in future research?\n\nLIMITATIONS:\n1. To make the paper more accessible, consider simplifying some of the theoretical discussions or providing more intuitive explanations alongside the formal proofs.\n2. Include more detailed discussions on how the suggested remedies, such as participatory data curation, can be practically implemented in real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}