# All data is no data: a conjecture on learned representations with infinite information

Carmine-Emanuele Cella

<figure><img src="image_1.png" /></figure>

## To cite this version:

Carmine-Emanuele Cella. All data is no data: a conjecture on learned representations with infinite information. 2025. hal-05250956

HAL Id: hal-05250956

https://hal.science/hal-05250956v1

Preprint submitted on 12 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
All data is no data:

a conjecture on learned representations with infinite information

Carmine-Emanuele Cella

University of California, Berkeley

Abstract

In this paper, we present a theoretical paradox in machine learning that arises from the relationship between data representations, generality, and expressivity in the limit case of infinite information. We argue that as machine learning systems attempt to create increasingly expressive representations by learning from larger datasets, they asymptotically converge to maximally general and universal representations akin to the Fourier transform. We formalize this paradox using notions of generality and expressivity and formulate a conjecture that we name: all data is no data.

Our formulation connects with classical concepts from information theory (Shannon entropy), Bayesian inference (posterior collapse), and generalization theory (No Free Lunch theorems). We also propose an empirical framework for testing the paradox: training neural autoencoders on datasets of increasing entropy and measuring convergence to analytic bases. These findings have direct implications for large-scale pretraining, self-supervised learning, and model compression.

# 1 Introduction

Machine learning aims to discover useful representations of data. These representations, or “projectors,” reduce high-dimensional data to more manageable forms while preserving relevant information [1]. A classical example of a representation is the Fourier transform, which projects data onto a basis of complex sinusoids. While the Fourier basis is general, it may not be sufficiently expressive for certain datasets. Modern machine learning systems, in contrast, learn new projectors from data, achieving higher expressivity.

This paper investigates a paradox that emerges when considering the limit case of a dataset containing all possible data (infinite information). We argue that in this limit, the learned projectors become indistinguishable from universal representations like the Fourier transform, thereby questioning the very motivation for data-driven learning.

# 2 Data, projectors, and representations

Let $\mathcal{X} = L^2(\mathbb{R}^n)$ denote the Hilbert space of square-integrable functions over $\mathbb{R}^n$, with inner product

$$ \langle f, g \rangle = \int_{\mathbb{R}^n} f(x) \overline{g(x)} dx. $$

This space represents all possible data signals with finite energy.

Let $D = \{x_1, x_2, \dots, x_m\} \subset \mathcal{X}$ be a finite dataset. A representation is a mapping $R: \mathcal{X} \to \mathbb{R}^k$ or, more generally, an operator $P: \mathcal{X} \to \mathcal{X}$ that approximates the identity map while reducing dimensionality.
A projector is a linear operator $P$ defined as:

$$P(x) = \sum_{i=1}^{k} \langle x, \phi_i \rangle \phi_i,$$

where $\{\phi_i\}_{i=1}^k \subset \mathcal{X}$ is a family of (possibly orthonormal) functions. If the $\phi_i$ are learned from data, we denote the resulting operator as $P_D$; if they are fixed (e.g., Fourier basis), we denote it $P_F$.

## 2.1 Generality and expressivity

In this section we define two notions to characterize projectors, generality and expressivity, and we discuss the trade-off between them.

**Definition 2.1 (Generality).** A projector $P$ is general if it applies across many possible datasets. We can define generality as:

$$\text{Generality}(P) = \mathbb{E}_{D \sim \mathcal{P}(\mathcal{X})} [\mathbb{I}(P \text{ performs adequately on } D)],$$

where $\mathcal{P}(\mathcal{X})$ is a distribution over datasets, and performance is measured by a suitable loss function.

**Definition 2.2 (Expressivity).** A projector $P$ is expressive with respect to a dataset $D$ if it minimizes reconstruction loss:

$$\text{Expressivity}(P; D) = - \frac{1}{|D|} \sum_{x \in D} \|x - P(x)\|^2.$$

## 2.2 Trade-off

Generality and expressivity are often in tension. A highly expressive projector fits the specific dataset well, but may generalize poorly. Conversely, a highly general projector may perform adequately across many datasets, but poorly on a specific one. We model this trade-off abstractly as:

$$\text{Generality}(P)^{\alpha} \cdot \text{Expressivity}(P; D)^{1-\alpha} \leq C, \qquad (1)$$

for some constant $C > 0$ and $\alpha \in (0, 1)$. This expresses the idea that increasing generality typically requires sacrificing expressivity, and vice versa.

Non-trivial fixed projectors $P_F$ use basis funcions which are independent of $D$ and thus highly general. Since $P_F$ does not adapt to the dataset, its generality is maximal:

$$\text{Generality}(P_F) = \sup_P \text{Generality}(P),$$

but its expressivity is constrained:

$$\text{Expressivity}(P_F; D) \leq \sup_P \text{Expressivity}(P; D).$$

On the other hand, learned projectors $P_D$ use functions adapted to the dataset $D$, meant to optimize reconstruction loss or task-specific objectives. As a result,

$$\text{Expressivity}(P_D; D) = \sup_P \text{Expressivity}(P; D),$$

but generality decreases:

$$\text{Generality}(P_D) \leq \text{Generality}(P_F). \qquad (2)$$

Hence, fixed projectors such as $P_F$ maximize generality at the expense of expressivity, while learned projectors $P_D$ do the opposite. This encapsulates the generality-expressivity trade-off that lies at the heart of the conjecture presented below.
## 3 The limit of all data

Suppose we construct a dataset $D^* = \mathcal{X}$, the set of all possible data. Let $P^*$ be the learned projector from $D^*$.

**Conjecture 1 (All data is no data).**

$$P^* = \lim_{D \to \mathcal{X}} P_D = P_F, \quad \text{where } P_F \text{ is a fixed projector.}$$

In other words, as the dataset $D$ grows to encompass the full space $\mathcal{X}$, the learned projector $P^*$ converges to a universal representation $P_F$ (e.g., a Fourier basis), indistinguishable from pre-defined analytic bases.

### 3.1 Rationale

Learning projectors from the entire space of data forces representations to accommodate every possible structure. This leads to highly smooth, universal representations that must average out data-specific peculiarities. In the limit, only analytic, stationary, and orthogonal bases remain— properties exemplified by the Fourier basis.

The conjecture creates a paradox which suggests that extreme expressivity is equivalent to maximal generality and nullifies the value of data. When learning from everything, there is nothing to specialize to. Thus, infinite data erases specificity, just as no data.

### 3.2 Mathematical argument

As $D \to \mathcal{X}$, the learned projector must accommodate all possible structures within $L^2(\mathbb{R}^n)$. This necessitates extremely high generality:

$$\text{Generality}(P^*) \to \max_P \text{Generality}(P).$$

Using the trade-off relation provided in equation 1, we conclude that as generality approaches its maximum, expressivity must decrease:

$$\text{Expressivity}(P^*; D^*) \to \min_P \text{Expressivity}(P; D^*),$$

which contradicts equation 2, since $P^* \in \{P_D\}$.

Since the Fourier basis achieves maximal generality among orthogonal bases but is not tailored to any specific dataset, it represents the limiting case. Thus, learning from the complete data space $\mathcal{X}$ leads to rediscovering Fourier-like projectors, validating the paradox.

We note, however, that a full formal proof of the conjecture would require a more rigorous definition and assessment of the generality-expressivity trade-off inequalities introduced above. In particular, inequality 1—while conceptually motivated—is not uniquely determined and may depend on assumptions about the learning model class, the topology of the data space, and the metric used to assess representation fidelity. A complete treatment would also require characterizing the class of admissible projectors (e.g., via smoothness, spectral decay, or inductive bias constraints) and the geometry of data distributions. These questions are left open and provide a foundation for future theoretical refinement of the paradox.
# 4 Observations

## 4.1 The bias-variance trade-off

The paradox can be framed in terms of the bias-variance trade-off [5]. As data increases, variance in the learned projector decreases, leading to more stable representations. However, this stability arises at the cost of specificity—the learned projector begins to reflect the inductive bias of the learning algorithm.

In the limit $D \to \mathcal{X}$, the projector minimizes variance entirely and maximizes bias toward universal properties such as stationarity and translation invariance. The inductive bias becomes explicit, and the learning algorithm “defaults” to analytic projectors such as Fourier.

## 4.2 Information-theoretic perspective

From the perspective of information theory, Shannon entropy quantifies the uncertainty or information content in a source [12]. If $X$ is a random variable with distribution $p(x)$, its entropy is defined as:

$$H(X) = - \int p(x) \log p(x) dx.$$

Entropy measures the average information required to encode outcomes of $X$. In the context of learning representations, high entropy implies high variability or diversity in data.

If a dataset $D$ is a faithful sample of $\mathcal{X}$, then as $D \to \mathcal{X}$, the entropy $H(D) \to H(\mathcal{X})$. However, when the data becomes fully universal (i.e., infinite entropy), any representation must be invariant to all possible variations.

In this setting, a learned projector $P^*$ must reduce information while preserving invariance. By Shannon’s source coding theorem, any lossy encoding must trade precision for compression, and meaningful compression is only possible when redundancy exists. In the limit $H(\mathcal{X}) \to \infty$, the redundancy vanishes, and hence no specific compression—or specialized representation—is meaningful. Thus, only maximally smooth, invariant representations (like the Fourier basis) remain.

This information-theoretic viewpoint reinforces the conjecture: when all information is present, no compression—and therefore no specificity—is possible. All data becomes indistinguishable from noise, or equivalently, from pure symmetry.

## 4.3 Bayesian learning

The conjecture also resonates with a Bayesian understanding of learning [8]. In Bayesian inference, learning consists in updating a prior distribution over hypotheses $\theta \in \Theta$ in light of data $D$. The posterior is given by:

$$P(\theta | D) = \frac{P(D | \theta)P(\theta)}{P(D)},$$

where $P(\theta)$ is the prior, $P(D | \theta)$ is the likelihood, and $P(D)$ is the marginal evidence.

As the dataset $D \to \mathcal{X}$, i.e., as it becomes dense in the space of all possible observations, the likelihood term $P(D | \theta)$ tends to 1 for all models $\theta$ that are not pathological. In this regime, the posterior becomes increasingly dominated by the prior. If the prior is uniform (i.e., maximally ignorant), then:

$$P(\theta | D) \approx P(\theta),$$

and no significant update occurs. In effect, the inference process becomes vacuous: the model has seen everything, and so cannot prefer any hypothesis over another.
This mirrors our paradox. If expressivity corresponds to adapting to specific data and generality corresponds to insensitivity to particular distributions, then maximal generality entails minimal posterior refinement. In Bayesian terms, learning becomes inert in the absence of exclusion.

## 4.4 No Free Lunch theorem

Another theoretical foundation that supports our conjecture is the *No Free Lunch (NFL) theorem* in supervised learning [13]. The NFL theorem asserts that no learning algorithm can outperform all others when performance is averaged over all possible functions mapping inputs to outputs. Formally, for any two learning algorithms $A$ and $B$, and for any uniform distribution over functions $f: \mathcal{X} \to \mathcal{Y}$, the expected performance of $A$ and $B$ is the same.

This implies that, in the absence of prior knowledge or inductive bias, no learning algorithm has an advantage. In our context, as the dataset $D$ expands to include all possible functions or signals (i.e., $D \to \mathcal{X}$), the advantage of data-driven learning vanishes. Any learned projector must perform equally across all data and thus reverts to the universal analytic projector.

Our conjecture aligns with the NFL theorem by suggesting that maximal expressivity eliminates the gains from learning. Just as the NFL theorem warns against the expectation of universal performance, our conjecture warns that universal data collapses specificity and forces convergence to generic representations.

## 4.5 Mallat-Zeitouni conjecture

Finally, the conjecture we propose resonates with another open problem in representation theory: the *Mallat-Zeitouni conjecture* [10]. This conjecture addresses the optimality of the Karhunen-Loève (KL) basis for nonlinear reconstruction tasks. While the KL basis (equivalent to PCA in finite dimensions) is optimal for linear reconstructions under Gaussian assumptions, its status in nonlinear or general statistical regimes remains an open question.

Both conjectures suggest, however, a similar idea: when seeking general representations for arbitrarily complex data, learned systems tend to converge toward analytic, structure-preserving bases. In our case, this convergence arises due to the loss of specificity from excessive generality; in the Mallat-Zeitouni conjecture, it arises from the potential irrelevance of data-adapted nonlinear encodings when the KL basis already captures the statistical essence.

Together, these perspectives point to an intriguing principle: that universality and optimality often meet in analytic representations when data variation becomes unbounded.

# 5 Practical implications

Although the conjecture is rooted in abstract functional analysis, it has implications for practical machine learning systems. Several real-world behaviors can be seen as manifestations of our paradox.

## 5.1 Convolutional neural networks at scale

Convolutional neural networks (CNNs) build invariance to translation through weight sharing and convolutional layers. Initially designed to process structured data such as images, their learned filters resemble Gabor wavelets or localized Fourier atoms. As data increases, learned filters in CNNs often converge to increasingly smooth and symmetric forms [11].
This observation aligns with our conjecture: when CNNs are trained on massive datasets, their learned representations approximate handcrafted analytic functions. In extreme cases, such as self-supervised pretraining on web-scale data, CNNs may rediscover spectral features inherent in natural data statistics.

More formally, let $P_{\theta}$ be a neural network with parameters $\theta$, trained to approximate the optimal projector:

$$P_{\theta}(x) = \sum_{i=1}^{k} \langle x, \phi_{i}^{\theta} \rangle \phi_{i}^{\theta},$$

where $\{\phi_i^\theta\} \subset \mathcal{X}$ are learned functions implemented implicitly by the network (e.g., filters in a CNN).

In the overparameterized regime (i.e., when model capacity is very high), $P_{\theta}$ can approximate any projector arbitrarily well. As the dataset grows, the optimization objective implicitly penalizes solutions that overfit rare patterns and favors stable, smooth, and data-agnostic representations.

This leads to the same conclusion:

$$\lim_{|D| \to \infty} P_{\theta} \approx P_{F},$$

up to a choice of invariant, low-frequency, analytic bases. Hence, deep networks trained on sufficiently large datasets asymptotically should rediscover analytic representations, just as the conjecture predicts.

## 5.2 Universal transformers

Transformers, with self-attention mechanisms, aim to learn context-dependent representations. The attention mechanism allows the model to dynamically weigh the relevance of each input token to every other token, enabling a form of adaptable, global context modeling [2].

As the training corpus grows in size and diversity, the attention maps become increasingly structured and converge toward statistical regularities of the data, such as syntactic dependencies and semantic groupings. These structures resemble classical rule-based systems and exhibit behavior similar to analytic decompositions [9].

This convergence is a symptom of the same trade-off discussed throughout this paper: as the model becomes more general and attempts to encode a larger and more diverse dataset, its capacity to specialize diminishes. The model, in effect, aligns itself with the most invariant and general statistical structures available in the data—those which persist across domains and tasks.

In the extreme, with infinite data, the attention patterns of a transformer may approach universal patterns that encode distributional or grammatical properties common to all languages. These universal forms echo the role of Fourier bases in signal analysis: generic, general-purpose decompositions that are not tied to specific data instances. Thus, we believe that the behavior of transformers under data scaling further supports the introduced conjecture.

## 5.3 Compression and the recovery of specificity

Model compression, pruning, and distillation strategies often recover improved performance from large models by reintroducing sparsity, hierarchy, or task relevance. Pruning techniques eliminate redundant parameters after training [6], while knowledge distillation transfers useful behaviors from a large, general model to a smaller, more specialized one [7]. The Lottery Ticket Hypothesis suggests that within every large model lies a sparse subnetwork capable of training to high performance [3].
These methods reflect an empirical pattern: large models first absorb everything, and only later is specificity restored.

This practical workflow—first learn from massive, general data, then compress toward meaningful structure—suggests that unconstrained generality must be corrected after the fact. In this light, compression is not merely resource optimization, but a partial reversal of the generality-expressivity collapse predicted by our conjecture.

Meaningful learning requires constraints: models trained on everything risk learning nothing specific. The challenge is not just to process data, but to structure it—and to decide what to leave out.

# 6 Toward an empirical test

While the introduced conjecture is grounded in theoretical analysis, it also invites empirical investigation. A possible direction for testing the conjecture is to examine whether learned representations converge to universal analytic bases—such as the Fourier basis—when trained on increasingly large and structureless datasets.

## 6.1 Experimental outline

Let us consider a parameterized representation learner, such as a shallow autoencoder or convolutional neural network, trained to reconstruct inputs from a dataset $D \subseteq \mathcal{X}$. The goal is to monitor how the learned filters (e.g., the encoder weights) evolve as the size and diversity of $D$ increase.

A practical simulation could proceed as follows:

1. Define a space $\mathcal{X} = L^2([0, 1])$ and sample data from increasingly broad distributions over $\mathcal{X}$, ranging from simple structured signals (e.g., sparse sinusoids) to fully random noise.

2. Train identical architectures on each dataset $D_n$ of increasing size and entropy.

3. Extract the learned projectors (e.g., the encoder weights or first-layer filters).

4. Compare these projectors to analytic bases such as Fourier, wavelet, or Legendre polynomials using a suitable distance metric (e.g., alignment of Gram matrices or cosine similarity, or Procrustes distance).

## 6.2 Expected result and interpretation

If the conjecture holds, we expect that as $|D_n| \to \infty$, and as the distribution becomes increasingly uniform or structureless, the learned projectors will converge (in distribution or geometry) to the Fourier basis or similarly smooth, stationary functions. That is:

$$ \lim_{n \to \infty} P_{\theta_n} \approx P_F, $$

where $P_{\theta_n}$ is the learned projector and $P_F$ is a fixed projector. This convergence would serve as empirical evidence of the conjecture's core claim: that data abundance induces a collapse toward universal, analytic representations.

Such simulations would not only validate the paradox, but also inform the limits of representation learning in practice—highlighting when and why data diversity alone is insufficient for meaningful specialization.
# 7 Conclusion

In this short document we presented a conjecture that illustrates a fundamental tension in machine learning between generality and expressivity. In the pursuit of perfect representations via massive datasets, machine learning may unknowingly circle back to classical analytic tools like the Fourier transform. Recognizing this limit is crucial for understanding the true power—and limitations—of data-driven learning.

This fundamental tension is not confined to machine learning; it also reflects an epistemological dilemma between inclusion and exclusion as competing foundations of knowledge. In Bayesian terms, learning involves updating beliefs based on selective evidence. If evidence includes all possibilities, then no informative update can occur—the posterior is identical to the prior. In this view, selection is not a limitation of knowledge acquisition, but a precondition for it.

Recent work in cognitive neuroscience supports this view: the brain does not passively receive data but actively suppresses expected input while amplifying prediction error [4]. Learning thus depends on constraints, priors, and mismatches. In this light, the conjecture aligns with a philosophical warning: when we design learning systems to see everything, we may strip them of the very inductive constraints that make learning meaningful. Learning requires not just data, but *constraints on data*. Knowledge arises not from maximal exposure, but from selective ignorance.

# References

[1] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 35(8):1798-1828, 2013.

[2] Alexey Dosovitskiy, Lucas Beyer, et al. An image is worth 16x16 words: Transformers for image recognition at scale. *International Conference on Learning Representations (ICLR)*, 2021.

[3] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In *International Conference on Learning Representations (ICLR)*, 2019.

[4] Karl Friston. The free-energy principle: a unified brain theory? *Nature Reviews Neuroscience*, 11(2):127-138, 2010.

[5] Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. *Neural Computation*, 4(1):1-58, 1992.

[6] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In *International Conference on Learning Representations (ICLR)*, 2016.

[7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In *NIPS Deep Learning and Representation Learning Workshop*, 2015.

[8] E.T. Jaynes. *Probability Theory: The Logic of Science*. Cambridge University Press, 2003.

[9] Jared Kaplan, Sam McCandlish, Tom Henighan, et al. Scaling laws for neural language models. *arXiv preprint arXiv:2001.08361*, 2020.
[10] Stephane Mallat and Ofer Zeitouni. A conjecture concerning optimality of the Karhunen-Loeve basis in nonlinear reconstruction. *arXiv preprint arXiv:1109.0489*, 2011.

[11] Preetum Nakkiran, Gal Kaplun, et al. Deep double descent: Where bigger models and more data hurt. *International Conference on Learning Representations (ICLR)*, 2021.

[12] Claude E. Shannon. A mathematical theory of communication. *Bell System Technical Journal*, 27(3):379–423, 1948.

[13] David H. Wolpert and William G. Macready. No free lunch theorems for optimization. *IEEE Transactions on Evolutionary Computation*, 1(1):67–82, 1997.