Transcendence: Generative Models Can Outperform The Experts That Train Them
Authors: Edwin Zhang
Vincent Zhu
Naomi Saphra
Anat Kleiman
Benjamin L. Edelman
Milind Tambe
Sham M. Kakade
eran malach
Venue: NeurIPS 2024 poster (2024)
Keywords: theory
foundations
generative modelling
sequence modelling
Primary area: generative_models
TL;DR: We theoretically and empirically demonstrate that generative models can outperform the experts that train them by low-temperature sampling

Abstract: Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of *transcendence*: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence is enabled by low-temperature sampling, and rigorously assess this experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.

Transcendence: Generative Models Can Outperform The Experts That Train Them
Edwin Zhang OpenAl Harvard University
Vincent Zhu UC Santa Barbara Humanity Unleashed
Naomi Saphra Harvard University Kempner Institute msaphra@g. harvard. edu
Humanity Unleashed edwin@openai. com
vincentzhuGucab.cm edu
Anat Kleiman
Benjamin L. Edelman
Milind Tambe
Harvard University
Princeton University
Harvard University
Apple anatkleiman0g www.aarvaa....... edu
Harvard University
tambeCg. harvard. edu
bedelman0g. harvard. edu
Sham Kakade
Eran Malach
Harvard University
Harvard University
Kempner Institute
Kempner Institute
sham8g. harvard edu
emalachog harvard. edu
Abstract
Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model outperform the humans their original objectives. In this work, we study the phenomenon transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset] We theoretically prove that transcendence can be enabled by now-temperature sampling, and rigorously assess this claim experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in broader setting.
Introduction
Generative models (GMs) are typically trained to mimic human behavior. These humans may be skilled their various human objectives: answering a question, creating art, singing a song. The model has only objective: minimizing the cross coos-ey loss with respect the output distribution, thereby adjusting it to match the distribution human labely Therefore, one might assume the model can, at best, match the performance of expert their human objectives. Is it possible for these models to surpass- to franscend their expert sources in some domains?
'To play with our models, code, and data, please see our website at wttttps//taassceeeecc..deee win Athouug chatbots are subject to variety of post-training tuning methods, e.g., RLHF, restrict scope by assuming that the specialized knowledge and capacities already provided by cross-entropy loss.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

1800
1020
Max Rating, During, Training 1500
1300
Max Rating During Training: 1300
1400
Batng
Roeng
1330
1200
Max Rating Seen During, Training: 1000
100
1050
1000
800
50
500
6 001
0.01
0.1
0.3
05
0.75
15
b.00L 0.01
0.1
0.3
0.5
0.75
1.5
00000 001
0.01
0.1
0.3
0.5
0.75
3 1.5
Tempersture (+)
Tempersture [r)
Temperature (r)
Figure 1: Ratings of our autoreggressivee decoodeeooy transformer, ChessFormer, over several different temperatures. We refer to our models as "ChessFormer <.Maximum Glicko-2 rating seen during training>" to easily distinguish between different models in subsequent sections. Each model is trained only games with players up certain rating (1000, 1300, 1500, respectively). We report 95% confidence intervals calculated through taking 1.960.
We illustrate an example of such transcendence in Figure 1] which measures the chess ratings (Glicko-2 (7] of several transformer [35] models. Our experimental testbed is generative modeling chess, which we choose as a domain for its well-understood, constrained nature. The transformer models are trained on public datasets of human chess transcripts, autoregressively predicting the next move in the game. To test for transcendence, we limit the maximal rating of the human players in the dataset below specified score. We find that ChessFormer 1000 and ChessFormer 1300 (the latter number being the maximum rating seen during training) achieve significant levels of transcendence, surpassing the maximal rating seen in the dataset. Our focus is this capacity of a GM to transcend its expert sources by broadly outperforming any one expert. The key to our findings is the observation that GMs implicitly perform majority voting over the human experts. As these models experts with diverse capacities, predilections, and biases, this majority vote oftentimes outperforms any individual expert, phenomena that is known "wisdom of the crowd".
trained collection of many
Our objective is to formalize the notion of transcendence and focus narrowly on this source of improvement over the experts: the removal of diverse human biases and errors. We prove that this form of denoising is enabled by low-temperature sampling, which implicitly induces a majority vote. Our result draws subtle but deep connection from ensembling [ 19), enabling several key results. We precisely characterize the conditions under which transcendence is possible, and give rigorous theoretical framework for enabling future study into the phenomenon. To test the predictive power of our theory, we then empirically demonstrate these effects. Digging deeper into the effects of majority voting, show that its advantage is primarily due to performing much better on a small subset of states-that is, under conditions that are likely key determining the outcome of the game. We also find that diversity in the data is necessary condition for practically effective majority voting, confirming our theoretical findings. In short:
setting rich prior literature on model
We formalize the notion of transcendence in generative models (Section
We find key insight explaining one cause of transcendence by connecting the case of denoising experts to model ensembling. In low temperature sampling settings, we prove that generative model can transcend if trained
single expert that makes mistakes uniformly at random. We then extend this result to transcending a collection of experts that are each skilled in different domains (Section 3)
We train a chess transformer on game transcripts that only include players up to a particular skill level. We confirm our theoretical prediction that this model only surpasses the maximum rating of its expert data generators at low temperature settings (Section[4)
We visualize the distribution of changes in reward by setting lower sampling temperature, attributing the increased performance to large improvements on relatively small portion of states (Section
We explore the necessity of dataset diversity, and the inability of ChessFormer to transcend when trained on less diverse datasets (Section 4.2

---

2 Definition of Transcendence
Denote by X' the (variable-length) input space and by y the (finite) output space. Let F be the class of all functions mapping A'~> P(Y)) (where we use the notation P(Y) to denote probability distributions over 3/). That is, the functions in FF map inputs in X' to probability distributions over y, so each function f E F defines conditional probability distribution of y y given x E X. We denote this distribution by f(]zx)
Fix some imput distribution over X' such that p has full support (namely, for every x X' we have p(z)>> 00). Throughout the paper, we assume that our data is labeled by à experts, denoted fi....,f fi E F. Namely, we assume that the inputs are sampled from the imput distribution p and then each input E is labeled by some expert chosen uniformly at randon This process induces a joint probability distribution over X' x xV, which we denote by D. Specifically, D(z.y) p(x) (2)(2) where f is the mixture the expert distributions, namely
k F(yl=) 1>A(vlee 1=1
(I)
We measure the quality of some prediction function Fuing reward assigned to each input-output pair. Namely, we define a reward function r X' x R. s.t. for all æ, the function r(z,-) is not constant (i.e., for every input x not all outputs have the same reward), We choose some test distribution Peest over X, and for some f E F define the average reward of f over Prest by:
Re (f) ) Exreoss [f)],, where Tz (x( = (r(2,))
(2)
A learner has access to the distribution D. and needs to find function that minimizes the cross-entropy loss over D. Namely, the learner chooses some function f E F s.t. f argmin Exep (f.f)) where H the cross-entrops function.
Definition 1. We define "transcendence" to be setting of fi.... EF and p P(X) where:
Rp (f) > max
Peace (fr)
SE[K]
In other words, transcendence describes cases where the learned predictor performs better (achieves better reward) than the best expert generating the data. Note that we are focusing on an idealized setting, where the learner has access infinite amount of data from the distribution D, and can arbitrarily choose any function to fit the distribution (not limited to particular choice of architecture or optimization constraints). As we will show, even in this idealized setting, transcendence can be impossible to achieve without further modifying the distribution.
We have made various simplifying assumptions when introducing our setting. For example, we assume that all experts share the same input distribution, we assume that all inputs have non-zero probability under the training distribution p. and we assume the experts are sampled uniformly at random. We leave a complete analysis of more general setting to future work, and discuss this point further in nisection 6
Remark 1.
3 Conditions for Transcendence
In this section we analyze the necessary and sufficient conditions for transcendence in our setting. We begin by showing that low-temperature sampling is necessary for transcendence in our specific setting. Then, we analyze specific sufficient conditions for transcendence, both in the case where the data is generated by single expert and when the data is generated by multiple experts. We defer all proofs to Appendix A
3.1 Low-Temperature Sampling is Necessary for Transcendence
Observe that by definition f. and using standard properties of the cross-entropy loss, we get that = as defined in Eq. (1). Therefore, the conditional probability distribution generated by is simply average of the distributions generated by the expert. Since the reward is linear function of these distributions, get that / never achieves transcendence:
Equivalently, assume that each example is labeled by all experts.

---

Reward RS (3)
MZX
WI
A
A
A
A
T = 1.0
T = 0.75
7 = 0.001
0.5
33
05
Figure 2: Visualizing the denoising effects of low temperature on the action distribution: an example of Chess- Former shifting probability mass towards the high reward move of trapping the queen with the rook as the temperature decreases. Opacity of the red arrows represent the probability mass given to different moves. The color of the square represent the reward that would be given for taking the action that moves the given piece that state. Purple here is high reward, while blue is low. For more visualizations, Appendix
Proposition 1. For all choice of f1.--.kk and Prest, there exists some fa Rpis (fi) Repeaa (f)-
Note that in our setting, we assume that experts are sampled uniformly for given input æ. instead this assumption is removed, then it may be possible to achieve transcendence with bayesian weighting. We leave this analysis for future work.
3.2 Transcendence with Temmeemeerr Sampling
Now, we consider temperature sampling scheme over the learned function f. Namely, for some temperature r >> 0, and some probability distribution E P()), denote the softmax operator with exp(g,/T)
temperature t by softmax(g;T) E P(Y) s.t. softmax(4:TT, = Σ explyy
Additionally,
we define argmax E P()) to be the uniform distribution over the maximal values of g, namely argmax(4) = 1/|XX, if y Y, and 0 if ÿ Yg. where (,= e V/: max(())) Now, define fr to
be the temperature sampling of i.e. f+(-(=)= softmax)] (-(())): and fmax the arg-mas "sampling" of f. i.c =aaggaa argmax(f((()) f-(==)). We now show that if the arg wrx predictor fmax is better than the best expert, then transcendence possible with low- te-perateee sampling.
Proposition 2. Rppeaa (faxx) > max Reroas (fi) if and only if there exists some temperature T (0,1) s.t. for all T,, it holds that Ryron (fr) maX Rprnes (fi).
The above shows that, distribution, it can be achieved by temperature sampling, assuming that the arg max predictor achieves higher reward compared to all experts. In other words, make the subtle connection here that low- temperature sampling can be thought of as performning majority vote [ between the experts. Please ApenAppe for formal proof of this connection. When the experts put non-negligible mass the best actions, the resulting majority vote may find the best action (9] which improves performance compared to individual experts (i.e., "wisdom of the crowd") and thus achieve transcendence.
though transcendence cannot be achieved when directly modeling the
3.3 Denoising a Single Expert
now turn to study particular cases where low-temperature sampling can lead transcendence. The most simple case is of a single expert that outputs correct but noisy prediction. Denote by f the
5(y V*)
optimal expert, s.t. for all x we have (y(2z)
where Y = { Vy max, max, (xx,)))
Y

---

and )(condition) is 1 if the condition is true and 0 otherwise. Now, for some p E (0,1), let fo be "noisy" expert, s.t., for all z, with probability p chooses a random output, and with probability 1 1-p chooses an output according to the optimal expert f (-Ix), namely [0(u|æ)) p/ (1 p) (vlz). We show that transcendence is achieved with low -eeteeperr sampling for data generated by Sp:
Proposition 3. Assume the data is generated by single expert fe- Then, there exists some lemperature TE (0,1) s.t. for all ST, the predictor f achieves "transcendence".
3.4 Transcendence from Multiple Experts
Next, we consider the case where the dataset is generated by multiple experts that complement each other in terms of their ability to correctly predict the best output, For example, consider the case where the input space is partitioned into k disjoint subsets, X' = &'.... s.t. the i-th expert performs well on the subset X; but behaves randomly on other subsets. Namely, assume the expert fi behaves as
5(xgx,) where Y is as previously defined and 8( (condition) follows: f((l=))= )((EE)))(2EX.) + 1
1Y:1
is 1 if the condition is true and 0 otherwise. We show that, assuming that the test distribution Peest is not concentrated on a single subset X,, we achieve transcendence with lowttempeatture sampling:
Proposition 4. Let Peest be some distribution s.t. there are at least two subsets X, + X2 s.t. Pueat (Xt) (2x) > 0. Then, if the data is generated by fi....,k. there exists some temperature TE (0,1) s.t. for all T' S T, the predictor for achieves "transcendence".
In order to build intuition for Propoosisn 4 see Appendix Cfor an intuitive diagram.
4 Experiments
To evaluate the predictive power of our impossibility result of transcendence with no temperature sampling (Proposition 1) as well as our result of transcendence from multiple experts with low temperature sampling (Proposition 2), we turn to modeling and training chess players. Chess stands out as an attractive option for several reasons. Chess is well-understood domain and more constrained than other settings such as natural language generation, lending to easier and stronger analysis. Evaluation of skill in chess is also natural and well-studied, with several rigorous statistical rating systems available. org
In this paper, we use the Glicko-2 rating system [7]. which is also adopted by the free and open-source online chess server from which we source our dataset.
4.1 Experimental Setup
Training Details. We trained several 50M parameter autoregressive transformer decoders following best practices from modern large model training. including a cosine learning rate schedule and similar batch size-lcarning rate ratios as prescribed by the OPT-175B team (37]. Our dataset consists of human chess games from the 1i chess. org openn source database from January 2023 to October 2023. total, this dataset contains approximately one billion games. In this setting, an expert is a specific individual player. To test for transcendence, we truncate this dataset by a maximum rating, so that during training model only sees data up to a given rating. We train our model on the next-token prediction objective, and represent our chess games as Portable Game Notation (PGN) strings, such as e4 e5 2. Nf3 Nc6 3. .b5...... 1/2-1/2. Note that we do not give any rating or reward information during training- only input the model sees are the moves and the outcome of the game. We tokenize our dataset the 32-symbol character level. (For further details, see Appendix EI) Our model plays chess "blind"- without direct access to the board state- saate-nd,, furthermore, is never explicitly given the rules of the game: at point is play constrained to valid outputs for given piece board state. Nontrivial chess skill is therefore not straightforward to acquire, and if not for the surprising capabilities of modern large transformers, one might imagine such model would fail to learn even the basic rules of playing chess. This blindfolded setting has also been studied by prior work 23 30), as discussed further in ssscctioon 5)
One gap between our theory and practice is that in our theory, we assume that each expert is defined over the entire input space X. However, in the chess setting such full coverage extremely unlikely to be the case after around move 15, as there are more unique chess games than atoms in the universe due to the high branching factor of the game tree. To address this gap, we visualize the latent representation of model in Figure 3 where we find the model is able to capture meaningful semantics regarding both
5

---

1.0
Reward
0.5
0.25
0.0
f. Ir.. 1.0
B B [7
B Bg- [ra] 0.0
Eg Pa 0.54
ChessFormer's 's last hidden layer latent representations of game transcripts during training time. The colors represent the prohability of winning, with +1 corresponding to state where White has and O to Black. Probabiliy of winning is computed through the Stockfish analysis engine. We also visualize several board states associated with different clusters the t-SNE embedding, and their associated expected reward when following model distinguishes between states where the outcome has already been determined (the two left boards), versus opening states that are extremely similar (the two right boards). See the full t-SNE in Appendixx G
Figure 3: Inspired by Mnih et 20. we generate a t-SNE embedding
expert Stockfish distribution, Note that the
the relative advantage of a state, as well as the identity of the black and white player. This visualization illustrates the ability of our model to generalize by compressing games into some shared latent represen- tation, enabling experts to generalize to unseen states, bridging this gap between theory and practice.
Evaluation. We evaluate each model by its Glicko-2 ratings against Stockfish 16.1 [29]. a popular
open-source chess engine. Stockfish uses traditional minimax search equipped with bespoke CPU- efficient meural network for evaluation 22 and pruning for further efficiency. We evaluate Stockfish levels 3, and with 100ms timeout directly Lichess' platform against the Maia (18 1,5, and bots (human behavior cloned convolutional networks trained at rating bins 1100 1200, 1500-1600, and 1900-2000, respectively) for several hundred games, obtaining calibrated Glicko-2 ratings for Stockfish specifically on Lichess' platform (1552-14.22 1842 44522 45.2, 21 2142 + 59 for Stockfish Levels 1, and 5, respectively). Next, for evaluating our models, then play against Stockfish levels of 1, and for 100 games each, reaching final rating calculation with 300 games. We then report both the Glicko-2 rating R as well rating deviation R.D of our models, where 2 R.D provides 95% confidence interval. To play against Stockfish, we successively prompt our model with the current game PGN string. Note that our output is entirely unconstrained, and may either illegal in the current board state or altogether unparsable. If our model fails to generate valid legal move after 5 samples, we consider it to have lost. After generation, we give the updated board state to Stockfish and pass new PGN string appended with the prior move of Stockfish back to our model. We repeat this process until the game ends.
4.2 Experimental Results
Low-temperature sampling enables transcendence. In this section we attempt to answer primary research question, can low-temperature sampling actually induce transcendence in practice? We test PProposition by evaluating several ChessFormers across different temperature values, from 0.001 (nearly deterministic), to 1.0 (original distribution), to 1.5 (high entropy). In Figure Te definitively confirm the existence transcendence. Our ChessFormer 1000 (where the latter number refers to the maximum rating seen during training) and ChessFormmr 1300 models are able to transcend to around 1500 rating at temperature 7 equal to 0.001. Interestingly, ChessFormer 1500 is unable transcend at test time, result further analyze in Dataset Diversity
Main Result:
To more deeply understand when and why transcendence occurs, we investigate two questions. (1) How does the reward function defined in Equation 2/sh with respect to owwteemeeaature sampling? (2) Does transcendence rely on dataset diversity, as introduced theoretically in infsubsection 3.4
Lowering temperature increases rewards in expectation on specific states, leading to transcen- dence over the full game. When playing chess, a low-skilled player may play reasonably well

---

until they make significant blunder at a key point in play. If these errors are idiosyncratic, averaging across many experts would have a denoising effect, leaving the best moves with higher probability. Therefore, low-temperature sampling would move probability mass towards better moves in specific play contexts. Without oo-tempeeatuur sampling, the model would still put probability mass onto blunders. To gain intuition for this idea, we visualize it theoretically in [ppppddix Cland empirically in Figure 2aadd Appendix B) This hypothesis motivates our first research question in this section: Does low-temperature sampling improve the expected reward very much for just some specific key game states, or a little for many game states?
To formalize this notion, we first define a "favor" function, which captures the improvement in reward by following some new probability distribution over some baseline probability distribution. Our definition is inspired by the Performance Difference Lemma (PDL) [TO] from Reinforcement Learning (RL), which establishes an equivalence between the change in performance from following some new policy (a probability distribution of actions given state) over some old policy, and the expected value of the advantage function of the old policy sampled with respect to the new policy. RI., the advantage function is defined as the difference between the value of taking single action in given state versus the expected value of following some policy distribution of actions in that state.
Here, we define the "favor" of f' over / in a as the change in the reward function by comparing what would have done when following f' for given input
F( ; Badd w (-lw) (1))))(((r)))))))))))))
Byzal' w (
(r(r.y))
Where d' refers to the state visitation distribution 31 when following f in sequential setting- informally, this variable can be thought of the distribution of states seen when sampling from f with a fixed transition function that takes in an input x, a output y, and outputs a next input x. Here, that transition function is given by the rules of chess and the opponent player. Given this favor function, we can now quantitatively explore the effects that lead to transcendence by setting the baseline / to be the original imitation- learned probability distribution (temperature 7= 1), and f' intervention on ff (c.g. temperature T = O). We can empirically calculate the reward by using the evaluation function 22 of Stockfish, expert neural reward function that Stockfish uses to calculate its next move. This reward function is neural network trained to predict the probability of winning through sigmoid linear combination of handcrafted expert heuristics, such amount of material versus opponent material, and number of moves to a potential checkmate.
0.6
f
f'
f
0.4
Baseline B[F) 0.0
0.2
B|F] 2.15
[3[4] 0.99
0.0
10.0
7.5
5.0
2.5
0.0
2.5
5.0
7.5
10.0
Change in expected reward over base fr-1 probability distribution over individual states: F(
Figure 4: The favor probability distribution, or change in expected reward by setting temperature lower than T T= 1.0. We plot the favor distribution across two different lemperatures: setting = and r 0.001 by running the Stockfish analysis engine across 100 total Chessformer 1000 games played at 0.001 temperature against Stockfish level (as theoretically justified by PDL (0)). We calculate favor sampling 100 counterfactual potential moves at T 1.0 per actual move made at 0.001 to compute baseline expected reward. In total, we gather an empirical probability distribution with n = 382,000 2000 total samples per (38.2 2. moves on average per game). Note that we plot the distributions with transparency, so the brownish area is where the two overlap. We visualize several long-tail examples in Appendix B
In Figure 4
we find that lowering the temperature has the effect of skewing the expected reward distribution to the right, especially for the green 7 = 0.001 distribution. This result implies that the model does not improve the expected reward by small amount for many game states, but rather improves the expected reward by relatively large amount for a few game states. Thus, 0.001 improves the expected reward (probability of winning) by an average 2. 2.15 0.11%, but for some states, this expected improvement is over 5%. Note that the original temperature expected reward can
7

---

be thought of as Dirac distribution centered at O. The above finding answers our research question in this section: Low-temperature sampling able improves the expected reward by relatively large amounts for some specific game states, which is likely why the ChessFormer 1000 and 1300 model
was able to achieve transcendence.
Temperature
E[PP, (%))
E[P,- P1.o]
Top 1 Acc (%)
Top 3 Acc A%)(%)
Top 5 Acc (%)
7 7=0.00 0.000
39.95=0.92
29.61 2.15
54.26=1.57
16.86$1.44
T --0.5
38.79=0.90
0.99
25.08+0.995
47.84 1.09
60.37 = 1.04 56.27
7 1.0
07.8
0+:0
22.61 0.88
44.00=9.96
Table I: Table of several statistics describing the relationship between reward at 117-0vs 00.. In the column, we display the expected reward across our dataset, which is P of winning calculated by Stockfish 16.1). In the second column, we display F, the change reward for the given temperature 7 versus the haseline. In the last three columns we display the accuracy for the best moves ranked by Stockfish analysis run at time cutoff 1 second. Here, the top-k accuracy is the percentage of games where the actual move sampled by the model was in the top-k moves as ranked by Stockfish. We report 95% bootstrapped confidence intervals with 10K resamples.
In Table IJ we present the statistics of the favor function for different temperature values. From this table, we observe that the temperature decreases, the top-k accuracies monotonically increase, suggesting that the model becomes more consistent in selecting good moves. We also observe that although the model improves as temperature decreases, the probability of winning is still below 50%, meaning our model should tend to lose more games than it wins against Stockfish 1. This result matches with our results in Figure I the rating of Stockfish 1 is also higher than the reported rating for = (1550 for Stockfish vs ~ 1450 for Chessformer 1000). Overall, the analysis of the advantage statisties provides further evidence for the effectiveness of low-temperature sampling in inducing transcendence in chess models.
Dataset diversity is essential for transcendence. As we note in suusectioonnn 3.4, our theory requires dataset diversity as a necessary condition for enabling transcendence. Importantly, we find in Figure I that not all models are able to transcend. Unlike ChessFormer 1000 or 1300, the Chessformer 1500 fails to transcend. We hypothesize that this results is due to the fact that in the band of ratings from 1000 to 1500, diversity does not significantly increase. If so, a 1000 rated player can be thought of as noisy 1500 rated player, but a 1500 rated player cannot be thought of as a noisy 2000 rated player. In this section we ask the following research question: Is diversity in data required for enabling transcendence?
we explore this research question by quantifying dataset diversity through the normalized entropy on the action distribution HY IX) = (fl=://t/=xx(loo/// /1)lg//(///////// X)] /log D'I. To gain intuition for this metric, imagine the action distribution of moves taken for any given state. Entropy will be higher for more uniform action distributions, and lower for more deterministic, peaked action distributions. The average entropy of these action distributions can therefore serve as a measurement of the diversity of the dataset. We normalize this entropy to the range [0.1] by dividing by the binary log of the number of legal moves: log, D*/.
In Figure 5]
Importantly, we cannot calculate this normalized entropy for every state, as most states after move 16 the midgame and before the engame are unique within the dataset and we therefore observe just single action for thus states. Therefore our metric is limited in that it only considers opening moves, the beginning of the midgame, and the endgame. We consider only common states with greater than 100 actions by sampling 1, .000,,0000 000 games from each dataset. The average entropy confirm our hypothesis: The < 1500 cut off dataset has average less diversity than the < 1300 dataset, which has is again less than the < 1000 dataset. This result suggests that measured by the average normalized entropy Chessformer 1500 likely is not transcendent due to a lack of over different chess rating dataset cutoffs diversity in its dataset. If the entropy instead stayed constant with = 3681,3037,50 2169 common states for each dataset, it would imply that each had a similar level of for ratings 1000, 1300, 1500, respectively. diversity. In such a case, we would expect that ChessFormer 1500 likely would also transcend. Instead, as predicted, it is the empiricial frequencies of our datasct, and likely not transcendent due lack of diversity.
N action distribution over common states
0.15
Max Maring
Ratieg
1900 (X))
Max Roting 2500
EIM 1020. 6.70
0.10
si 1300 0.05
0.06
1500 D:4
0.00
0.0
0.2
0.4 01.5
0.5
3.0
Figure 5: Action distribution diversity, as
These entropies are calculated directly from
are model-agnostic
8

---

4.3 Additional Settings
SQuADv2 Natural Language Temperature Denoising Experiment. We extend our analysis to the Natural Language Processing domain by running experiments on the Stanford Question Answering Dataset (SQuAD 2.0). We tested the effects of temperature denoising on the performance of several large language models (LLMs) of varying sizes. The SQuAD task involves reading comprehension and question-an -nwwerrnn based on Wikipedia articles, making it an ideal setting to evaluate the impact of denoising on language models. We measured the exact-match, semantic-match, and F1 scores of the model outputs at different temperatures. The results show that temperature denoising leads to improved performance, corroborating the findings of our chess experiments and providing broader validation of the underlying mechanism of temperature denoising in diverse domains.
Score vs Temperature
Exact Match (%) vs Temperature Semantic Match (%) Temperature
70
60
70
Seanarcie Matah (%)
60
Exaet Match (%6)
50
60
P1 SSSme
50
50
40
30
10
2ered (25)
Goaaaa 21133
30
Grress
Queid (20)
Varra CTC
30
40
Qhraad (TB)
(795
(S90))
CIT (1034)
CPE52 (1536)
0.001
0.25
0.75 Temperstare (F)
0.5
1.5
0.001 0.25 0.5
0.75 Temperature (T)
1.5
0.001
0:25
0.75 Temperature (r)
0.5
1.5
Figure 6: We evaluate several pretrained language models on the SQuADv2 Question-Answering reading comprehesion dataset, a task consisting of answering question given some snippet from Wikipedia article. We report FI, 'Exact Match', and 'Semantic Match' scores of several different language models of varying size from 163M parameters to 7B parameters, over several different temperatures. Semantic Match is calculated by using another LLM (Ilama3. 1) to judge if two responses are equivalent, even if the exact strings slightly differ between the model output and the correct response. Wc also report 95% confidence intervals calculated through taking +1.66.
Toy Model Setting and Results. In addition, we develop toy theoretical model to further study when transcendence is possible. This model involves classificatin task with Gaussian input data and linearly separable classes Experts label the data with noisy versions of the ground truth separator. We trained linear model on a dataset labeled by random experts and observed the test accuracy for different temper- ature settings. The synthetic experiments demonstrated that transcendence occurs when expert diversity is high and temperature is low, aligning with our theoretical and empirical analysis in the chess domain.
std: 0.1
std: 0.2
std: 0.4
std: 0.6
1.0
1.0
1.0
Best
0.8
0.8
Best Expert
0.8
Deet Pepect
0.6
0.6
0.6
Best Expert
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.5
0.22
0.4
0.5
3.0
0.2
0.3
0.A
0.5
0.0
0.2
0.5
0.4
0.5
0.2
0.5
Jemperatur
'Amppruiuuee
Demperatture
Temparaturr
Figure 7: Toy model for demonstrating transcendence. Input data d-dimensional Gaussian, with d= 100. Output is classification with 10 classes. Ground-truth is generated by a linear function, i.e. y argmax;; W x for some W" C 10xd We sample l: experts, with k= 5, to label the data, where the labels of each expert are generated by some W I ioxd s.t. W = W' + where E., -~(,0a"")) ), for some standard deviation Namely, each expert labels the data with a noisy version of the ground truth separator, with noise std a. We then train linear model on a dataset with 10K examples, where each example is labeled by a random expert. We plot the test accuracy, measured by the probability assigned to the correct class, for different choices of temperature, and compare to the best expert.

---

5
Related Work
Chess and AL. Chess has been motivating Al research since the field began. In 1950, before anyone had used the term "artificial intelligence", automated chess were explored by both Claude Shannon [26] and Alan Turing [32]. Arguably, this history goes back even further: the famed "mechanical turk" of the 18th century was fraudulently automated chess player. These centuries of mechanical ambitions were finally realized in 1997, when world champion Garry Kasparov was defeated by IBM's Deep Blue [3]. Since then, chess program developers have drawn on neural approaches, with the RL-based convolutional network AlphaZero 2 far surpassing prior world champion engines such as Stockfish 25 Our chess model testbed is inspired by number of existing approaches, including other models trained on lichess data [8]. and other transformer- based sequential chess agents [2 5
Diversity beats Strength. Another historical thread in Al research is the strength of diverse learners. Long since the development of ensemble methods that exploit learner
bagging [.] boosting [6], and model averaging [19] researchers have continued to articulate this insight across set- tings. Similar to our chess setting, a diverse team of go playing agents have been proven and empirically shown to outperform solitary agents [9] and homogeneous teams 28] even when the alternative models individually outperform the diverse team members 17.. We draw a connection to this deep literature through our theory, which shows that imitation learning objective and then performing low- lowteemperature sampling subtly implies the same principle of majority voting. Teacher diversity has also been explored in the machine learning literature. One related method is ensemble distillation [16], in which model is trained with an additional objective to match a variety of weaker teacher models. Closer to our setting, ensemble self-training approaches [4] train learner directly on the labels produced by varied teachers. Large language models supervised by smaller or less trained models are said to exhibit "weak to strong generalization" (2). Overall, evidence continues to accrue that the general phenomenon we address is pervasive: that is, models can substantially improve over the experts that generate their training data.
Offline Reinforeement Learning. Our work also draws connections to the Offline Reinforcement Learning [4] setting, where one attempts to learn a new policy x that improves upon a fixed dataset generated by some behavior policy TTs. However, our setting of imitation learning differs substantially from this literature, as we do not explicitly train our model on RL objective that attempts to improve upon the dataset. Importantly, such an objective oftentimes introduces training instabilities [15) and also assumes reward labels. We defer a more extended discussion of related work to lo/ppendixx D
6 Discussion and Future Work
This paper introduces the concept of transcendence. Our theoretical analysis shows that low- temperature sampling is key to achieving transcendence by denoising expert biases and consolidating diverse knowledge. We validate our findings empirically by training several chess models which, under low-temperature sampling, surpass the performance of the players who produced their training data, as well as further experiments in natural language question-answeree and toy Gaussian models. We additionally highlight the necessity of dataset diversity for transcendence, emphasizing the role of varied expert perspectives.
Limitations. While our work provides strong foundation for understanding and achieving transcen- dence in generative models, several avenues for future research remain. Future work may investigate transcendence and its causes in domains and contexts beyond chess, such natural language process- ing, computer vision, and text-to-video, to understand the generalizability of our findings. Additionally, our theoretical framework assumes that game conditions at test time match those seen during training; in order to extend our findings to cases of composition or reasoning, we must forego this assumption.
Future Work. Future work could also explore the practical implementations of transcendence, and ethical considerations in the broader context of deployed generative models. Ultimately, our findings lay the groundwork for leveraging generative models to not only match but exceed human expertise across diverse applications. pushing the theoretical boundaries of what generative models can achieve.
Broader Impact. The possibility of "superintelligent" AGI has recently fueled many speculative hopes and fears. It is therefore possible that our work will be cited by concerned communities as evidence of a threat, but we would highlight that the denoising effect addressed in this paper does not offer any evidence for a model being able to produce novel solutions that human expert would be incapable of devising. In particular, we do not present evidence that low temperature sampling leads to novel abstract reasoning, but just denoising of errors.
10

---

Acknowledgements
Sham Kakade acknowledges this work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence; support from the Office of Naval Research under award N00014-22-1-2577, and the National Science Foundation Grant under award #IIS 2229881.
References
[1] L. Breiman. Bagging predictors. Machine Learning,
24:123-140, 1996.
URL
htttp:://api.sseantti www.seeeaaccooioioeeee..
arg/CorpusID:477288..
[2] C. Burns, P. Izmailov, H. Kirchner, B. Baker, L. Gao. L. Aschenbrenner, Y. Chen, A. Ecoffet,
M. Joglckar, Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with
weak supervision. arXiv preprint arXiv:23 2023.
[3] M. Camphell, J. Hoane, and F.-h. Hsu. Deep Blue. Artificial Intelligence, 134(1):57-83,, Jan.
2002. ISSN 0004-3702. doi: 10. 0011016/00004330020000 10011091))
[4] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
[5] X. Feng. Y. Luo, Z. Wang, H. Tang, M. Yang, K. Shao, D. Mguni, Du, and J. Wang. ChessGPT: Bridging Policy Learning and Language Modeling. Advances in Neural Information Processing Systems, (0667216-72666 Dec. 2023.
[6] Y. Freund and R. E. Schapire. A short introduction to boosting, 1999. (BorpusID:9621000
URL
M. E. Glickman. Example of the glicko-2 system. Boston University, 28, 2012.
[8] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as big sequence modeling problem, 2021.
[9] A. Jiang, L. Soriano Marcolino, A. Procaccia, T. Sandholm, N. Shah, and M. Tambe. Diverse randomized agents vote win. Advances Neural Information Processing Systems, 27, 2014.
[10] S. M. Kakade and J. Langford.
Approximately optimal approximate reinforce- ment learning. In International Conference Machine Learning, 2002. URL www.seeeeaaentiiieeeeeeeeiiiiete httpp://ap...mmmtticcchh.l httpeccooaar.orgrooopussD::::::. orgg/CorppssD::1142999
[11] A. Karpathy. NanoGPT, httpp: hotpaa/cccccaaaaataaaaaaooooo 2022.
[12] A. Karvonen. Emergent world models and latent variable estimation chess- paasss-playing language
models. arXiv preprini arXiv:2403. 15498, 2024.
[13] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[14] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
[15] L., Zhang, M. Yin, Q. Bai, Y.-X. Wang, and Y. Wang. Offline reinforcement learning with closed-form policy improvement operators. In International Conference on Machine Learning, pages 20485-20528. PMLR, 2023.
[16] T. Lin, L. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-225 2263, 2020.
[17] L. S. Marcolino, A. X. Jiang, and M. Tambe. Multi-agent team formation: Diversity beats strength? In IJCAI, volume 13, 2013.
[18] R. Yellrooy-Yuung S. Sen, J. Kleinberg, and A. Anderson. Aligning Superhuman Al with Human Behavior: Chess as a Model System. Proceedings of 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1677-1687, Aug. 2020. doi:
10.1145/394488634033199
[19] H. McMahan, F. Moore, D. Ramage, S. Hampson, and B. A.y Arcas. Communication-efficeenn learning of deep networks from decentralized data, 2023.

---

[20] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland. G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, nalure, 5187(540))5777 23333333399-333,,,0 2015.
[21] R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
[22] Y. Nasu. Efficiently Updatable 2018.
Evaluation Functions for Computer Shogi,
[23] D. Noever, M. Ciolino, and Kalin, The Chess Transformer: Mastering Play using Generative Language Models, Sept. 2020.
[24] A. Odonnat, V. Feofanov, and I. Redko. Leveraging ensemble diversity for robust self-training in the presence of sample selection bias, 2024.
[25] Pete.
New 1,000-Gamm Match. wttpssswwwm/ooowwww/i/ttttttddld crapaazzcro--uusssssssockikiiie -iniiih-----ee--000
AlphaZero Crushes Stockfish In
game-match, Dec. 2018.
[26] C. E. Shannon. XXII. Programming computer for playing chess. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 41(314):256-275 Mar. 1950. ISSN 1941-5982, 1941-5990, doi: 10.1080/147888 4786445057774
[27] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, Graepel. T. Lillicrap. K. Simonyan, and D. Hassabis. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Dec. 2017.
[28] L.. Soriano Marcolino, H. Xu, A. Xin Jiang, M. Tambe, and F. Bowring. Give a Hard Problem to a Diverse Team: Exploring Large Action Spaces. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), June 2014. ISSN 2374-3468, 21 2159-5399. doi: 10. 1..16669aai..25555 wtt.ai..22818888
[29] The Stockfish developers (see AUTHORS file). Stockfish, 2024.
URL
https
//stockfishchess org/
[30] S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel. Chess as a Testbed for Language Model State Tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11385-11393, June 2022. doi: 10. 16009//ai..33611....
[31] A. Touati, A. Zhang. J. Pineau, and P. Vincent. Stable policy optimization via off-policy divergence regularization. In Conference on Uncertainty in Artificial Intelligence, pages 1328-1337. PMLR, 2020.
[32] A. Turing. Chess (1953), B. Copeland, editor, The Essential Turing, page 0. Oxford University Press, Sept. 2004. ISBN 978-0-19 1--119979--1.1 doi: 10. /000///////7/019825001000001 100009900.000000.....
[33] A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio. Learning from disagreement: survey. Journal of Artificial Intelligence Research, 72: 14700 2021.
[34] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[37] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arX5v: 01066 2022.
12

---

A Proofs
Here we prove Proposition I
where rranscendenceeannot oceur by purely using imitation learning in our setting where all experts are sampled uniformly across the input distribution.
Proof. From linearity of the expectation
Rpccn (f)
Exceprcn
[Tx (F)
k
Σ
(fi)
Σ k
1
Peeet (fi) maxx maxRpposs
Press (ft)
We now give the proof of Proposition 2 that if the arg-ma prediction is better than the best expert, then transcendence possible with low-temperature sampling.
Proof. Observe that for all g, it holds that lim, lim--ussfttmax((77 = =argmax(g)) Therefore, for all x
limr=(f+) lim 7-0
(r(a))) (m)l -r (fmax)
and so,
lim Ryron (fr)= lim E T-0
Preat ro (fr)
7-
lim limra (f+))
-Exrproer Pro (f((fax)== Rerec (max)
Therefore, the required immediately follows.
To prove Proposition 3.3 we directly use the result Proposition 2
Proof. Notice that this expert, (rgmax(f((1x))) (ylax, which achieves higher reward compared to f. Therefore, Theorem iimplies that we achieve transcendence in the setting where all the data is generated by single expert
Next, we give the proof for that now-temperature sampling can thought of as performing majority vote 6 between the experts:
Proposition 5. Let z [nn be vector and > 0 be temperature parameter. Define the
softmax function as
e/s/T
(+(2))
Then, as ++ the limit of the softmax function is given by
=f 2max:
lim o- (z())==
k
7-++
0,
otherwise,
where Zmax maax-mmaxjjjsa and l is the number of indices such that z; Zmax-
Proof. Let 2max maxigj and define the set
S [ = Zmax}.
with cardinality A: [S] each i, let
A Zmax 50
13

---

Then the softmax function becomes
@(FmaxtA.)/5
0A1/T
or (2i)
Σn E
since e=max// cancels out in the numerator and denominator.
We analyze the behavior of the terms as 7 -+
For E have A, 0 and so:
ei/7 =e=11.
For S we have A <0 so
lime A./r =0 T-30
Therefore, the denominator simplifies to
lim
lim e
lim e
- jdS
Do=k.
+-++
jes -0-
jES
jES
Similarly, the numerator becomes
lim e A./r +
1, ifieS,
0, if
Thus, for each i,
lim, -o+ AA/T eA/T
1
K ifieS,
lim (2)) +
lim, E/ Cj=1
0,
if
This concludes the proof.
Finally, we give the proof Proposition 4) or the statement that transcendence can occur from multiple experts if the test distribution Ptest is spread across multiple disjoing subsets of X;
Proof. In this case, observe that for all
Repeat (fi) = Piest -.))-Eyrmeeelele lx; +=(ff)++oon (P
< RRpeee (f*)
Therefore, we get that for all x
f(yl2)) = En h
k
yyeY:
k 1=1
D
k- yl
k|Y
Thus, we get Fmax = f", and the required follows from Proposition 2
14

---

B
Additional Denoising Visualizations
Reward R.
1
7 7= 1.0
T 0.75
T 0.001
Figure An example of where denoising helps black find the only correct move. White has pinned the black rook to the Queen: any move where the rook does not move to o4 results in heavy loss of material. As decreasses, the expected reward increases substantially and converges onto the correct move.
Reward R2 Ra(
1.0
0.75
0.001
Figure 9: Another example where denoising helps avoid errors. Moving the queen to either d1 h1 takes bishop rook, respectively, but loses the queen in the following turn. While queen does not put the queen in immediate danger, allows white to push the pawn bishop on cl. The queen then must move out of danger, losing its opportunity to take the free pawn on h4 and giving white valuable space towards the center of the board. As decreases. the expected reward converges to the move queen to d4, taking the pawn and checking the black king.
d3, where threatens queen and is protected by
Reward R. (9)
1.0
0.75
7 0.001
Figure 10: this setup, a higher temperature shows two plausible moves for the black rook: gl or fl. temperature decreases, the expected reward converges to gl. If the black rook were to move to fl, the white rook would take the black rook. blocking the black pawn on f2 from promoting and protecting the promotion square from the h2 pawn. If the rook h2 pawn without being at any immediate risk. If white responded by moving bishop to g2. protecting promotion squares from both of the advanced black pawns, black could respond by taking the rook on al, gaining significant material.
to to gl, the other hand, would open the promotion square from
15

---

C Intuition of low temperature sampling inducing transcendence
To build intuition for the primary mechanism of transcendence that explore in this paper, give the following toy progression of distributions in order to clearly illustrate how low-temperature sampling can induce transcendence through majority voting. Here, the middle purple action represent the correct, high-reward output, whilst the left and right actions are low-reward bad outputs. We plot probability of each output as label on the x axis.
100
75
50
25
60
40
Figure The first expert output distribution. Although puts non-neglie mass on the purple, high-reward action, still samples low-reward action the majority of time.
100
75
50
25
40
60
Figure 12: The second expert output distribution. Symmetric to to the first expert, also puts non-ne non-negligible mass on the purple, high-reward action. However, it samples a low-reward action the majority of the time on the right.
100
25
30
Figure 13: By taking the average of the first and second expert, we observe that this distribution now the majority of mass onto the correct action.
100
75
P(Y X)
50
90
Figure 14: Finally, by setting temperature be 1, more weight shifted towards the high probability action, leading to a gain in the expected reward.
16

---

D Further Related Work
D.1 Label Disagreement
Label disagreement in training data, in particular, can improve models in practice. Xie et al. [36 empirically show that adding random noise pacher-generated labels can improve student model. Uma et al. [33] even survey the literature on human interannotator disagreement and find a trend of improvements when models are trained on the full of disagreeing labels rather than on majority vote labels or only on data where labelers agree. Our theoretical claims build on these findings by
making the point that the learner can even improve on these original diverse labelers.
D.2 Offline Reinforcement Learning
Although most Offline Reinforcement Learning algorithms train on an RL objective, perhaps most similar our work Decision Transformer [4] and Trajectory Transformer {8 prior models trained on just the sequence prediction of trajectories. Most notably, Decision Transformer also finds an alternative form of transcendencethan the one explored in this paper: by conditioning the trained transformer by the performance of the trajectory, at inference time they can then prompt the model to perform better than the best trajectory seen during training. This remains another promising direction to explore transcendence under.
Interestingly, analogue low-temperature sampling also has been noticed and exploited by Reinforcement Learning practitioners in the context off-policy learning, where a different exploration policy TE used than the final learned target policy иT. Oftentimes тт will just be to greedy version of g [21]. such as choosing to take the argmax action of TE, which we note directly equivalent to setting temperature 0.
17

---

E
Training Details
We give full list of the hyperparameters we used for training here. Note that we largely follow the same hyperparameter set as [37. but lower the batch size to 125K as we found training to still be stable ta this level. We also release our code openly support further research into transcendence, which was built off the wonderful work done by Karvonen [2] and Karpathy [1]]
Hyperparameter
Value
ChessFormer
Optimizer
AdamW
13
Activation Function
ReLU
Mini-batch size
125K tokens
Gradient Accumulation Steps Transformer num. layers
Transformer num. heads
Transformer embedding dim.
512
Dropout
0.0
Learning Rate
3e-4
Number of gradient steps
100K
Weight Decay
0.1
Critic hidden layers
Adam 5
0.90
Adam B2
0.95
Gradient Clip
1.0
Cosine Learning Rate
True
Warmup Iterations
2000
Minimum Learning Rate
3e-5
Learning Rate Deacy Iterations
400K
Tensor datatype
bfloat16
Table 2: Hyperparameters for our ChessFormer model.
18

---

Compute Resources
We train all of our models on the Nvidia H100 80GB GPU, To train one of our models takes around 6 to 12 hours.
19

---

G
Full t-SNE
We visualize the full I-SNE here, coloring by the reward of the game. We see that the model has learned some representation of the reward, with high absolute reward states being more likely to be near each other the latent space. This also points towards evidence that the model has learned some sort equivariant representation of the player identity, as the region of symmetric high reward states indicate. Note that reward is not directly given to the model during training.
t-SNE of Latent Board States Colored by Reward
1.00
60
0.875
40
00.55
20
00.25
Reward
0-
0.50
-20
0..755
-0.25
-40
0..25
-60
0.00
-60
40
20
0
20
40
60
20

---

We visualize the same t-SNE, but this time coloring by game length rather than reward. We see that games with high reward tend to be longer, which makes logical sense the result of the game will tend to be clearer as the game proogresses.
t-SNE of Latent Board States Colored by Game Length
180
60
160
40
140
-20
20
Game Length
100
80
-20
600
440
40
20
60
-60
40
-0
20
40
60
21

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided a proper justificionn is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims in the introduction are cross referenced with the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations provided in discussion section
22

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should diseuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: proof provided in the appendix, assumptions provided in preambles
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims conclusoon the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: data sources are given, and open source
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We link to our website on the first page, which provides open access data, code, and extensive documentation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https
//nips.cc/public//eeeccc
for more details.
While we encourage the release of code and data, we understand that this might not pos- sible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https://nips.. http/iiisscccpppi//pe httpccc/ppbllccc httpp//ss...pp.iicciiiieee/////s CodeeSuum iemiisssioni wwwiisssiiooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
24

---

6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide all information for reproducibility in our code, well as Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: plots that demonstrate transcendence include standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eигоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: provide this information in Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary failed experiments that didn't make it into the paper).
25

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethoss/tttps::/nuuippp... http://pw.p.p...ooooooccccaaaaaa
Answer: [Yes]
Justification: the authors have read the code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Provided in discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), priups) considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the models in question domain specific chess agents
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all original sources of data (lichess) and write all of training code from scratch.
Guidelines
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppperswii thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide code and release the dataset that we train our models on, well the figure generation code. We give extensive documentation on both, and take pride in the reproducibility of our work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects, data used from public sources without individual identities.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "En2sj2mbN5", "reviewer": "Reviewer_mPZQ", "summary": "The paper explores the phenomenon where generative models, trained to mimic human behavior, can surpass the performance of the experts generating the training data. The study focuses on autoregressive transformer models trained on chess game transcripts, demonstrating that these models can outperform the best human players in the dataset. The key to this transcendence is identified as low-temperature sampling, which effectively denoises human errors and biases through a majority-voting mechanism. The paper provides theoretical proof and empirical evidence supporting this phenomenon and discusses the necessity of dataset diversity for achieving transcendence.", "questions": "1. How do the authors plan to test the concept of transcendence in domains beyond chess, particularly in more complex and less structured tasks?\n2. What are the potential impacts of relaxing some of the theoretical assumptions made in this study? For example, how would a non-uniform sampling of experts affect the results?\n3. What ethical implications might arise from models that can transcend their training data, particularly in sensitive domains like healthcare or finance?\n4. How can dataset diversity be ensured or measured effectively in other domains to facilitate transcendence?\n5. Can the authors provide specific examples or case studies where transcendence could lead to significant real-world benefits?", "limitations": "1. Domain Specificity: The study is currently limited to chess, and its applicability to other domains is speculative at this point. Further research is needed to confirm the generalizability of the findings.\n2. Simplifying Assumptions: The theoretical results are based on several simplifying assumptions that may not hold in practice. Future work should aim to relax these assumptions and test the robustness of the findings.\n3. Resource Intensity: The models and experiments require substantial computational resources, which may not be accessible to all researchers, potentially hindering reproducibility and further investigation.\n4. Ethical and Practical Implications: The paper briefly touches on the broader impact but does not delve deeply into the ethical and practical implications of deploying transcendent models in real-world applications.\n5. Future Work: While the paper lays a strong theoretical foundation, it leaves many avenues for future research, including exploring other domains, addressing ethical concerns, and developing practical applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Some Strengths:\n1. Novel Concept: The paper introduces and formalizes the concept of transcendence in generative models, providing a new perspective on model performance.\n2. Theoretical Foundation: The authors offer rigorous theoretical proofs that support their claims, grounding the concept of transcendence in solid mathematical principles.\n3. Empirical Validation: The study includes extensive experiments with chess models, demonstrating the practical applicability of their theoretical findings.\n4. Comprehensive Analysis: The paper not only shows that transcendence is possible but also delves into the mechanisms (low-temperature sampling) and conditions (dataset diversity) required to achieve it.", "weaknesses": "Here are some weaknesses:\n1. Limited Scope: The empirical validation is confined to chess, a well-defined and constrained domain. It remains to be seen if transcendence can be generalized to other, more complex tasks.\n2. Assumptions: The theoretical framework relies on several simplifying assumptions, such as uniform sampling of experts and the nature of reward functions, which may not hold in real-world scenarios.\n3. Practical Implications: While the concept of transcendence is theoretically and experimentally validated, the practical implications and real-world applications of this phenomenon are not thoroughly explored.\n4. Computational Resources: The training of large transformer models, particularly with extensive datasets and multiple temperature settings, requires significant computational resources, potentially limiting reproducibility for researchers with fewer resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> We thank the reviewer for praising the novelty of the concept of transcendence and the rigor of our theoretical proofs while noting concerns about the limited scope of our experiments and the lack of exploration of the practical implications of transcendence.\n---\nLimited..\n\n> To help address this concern, we have run a new preliminary experiment on the Stanford Question Answering Dataset ([SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/)), testing the effects of temperature denoising performance on several LLMs of various sizes. For more details, please see the main rebuttal PDF and the first rebuttal to Reviewer Se7B.\n\nAssumptions..\n\n>  We acknowledge this both in the theory section itself and in the limitations section at the end. As with many theoretical works in the field, simplifying assumptions is necessary for proving strong results, and these may not always capture the complex nature of real-world settings. To alleviate this problem, we have performed thorough empirical experiments to show that our theoretical findings indeed apply to real-world settings.\n\nPractical..\n\n> While we agree that practical implications and real-world applications are not thoroughly explored, doing so would exceed the scope of our work and lead to a less focused paper. The 9-page limit constrains us from thoroughly exploring theoretical validation, empirical analysis, and practical applications of Transcendence.\n\n> For preliminary discussion, we can identify tasks that benefit most from AI, like copyediting, by considering those with errors aligning with Transcendence's theoretical conditions. Generally, AI excels in situations where the \"wisdom of the crowds\" surpasses individual judgment, such as question-answer platforms like StackOverflow or detecting spam in Gmail. These domains already leverage AI effectively (e.g., LLMs for copyediting and QA, Naive Bayes for spam).\n\n> [1] \"Studying the 'Wisdom of Crowds' at Scale.\" Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 7, no. 1, 2019, pp. 171-179. \n\nComputational..\n\n> We would like to kindly clarify a misunderstanding here. We do not use significant computational resources, and training our model requires only a single consumer-level GPU. As detailed in section 4.1, our transformer model is only 50M parameters, which is very small compared to large transformers such as GPT-3 (175B).\n\nHow do..\n\n> As noted above, we additionally have new experiments validating that denoising improves performance in Natural Language Question-Answering and Reading Comprehension. Here, it is clear how evaluation is done as human labels are provided. A baseline can easily be measured by having humans perform the task and using their performance as the threshold needed for transcendence. In tasks where human labels cannot be assumed, Elo or Glicko-2 Ratings is a powerful metric for any task where two models can be compared against, which is a much weaker assumption than assuming gold labels or the ground truth is known. In fact, our experiments are run with Glicko-2 Ratings, demonstrating a clear path for measuring transcendence in future more complex tasks.\n\nWhat are..\n\n> Relaxing some of the theoretical assumptions such as uniform sampling of experts, would in fact potentially enable other forms of transcendence. To give one example, imagine that each expert has a domain of expertise where they are sampled more often and are almost always correct, and outside that domain, they are almost always wrong. Thus, whilst we had $\\overline f(y|x) = 1/k \\sum_i^k f_i(y|x)$ before, we would now have $\\overline f(y|x) = \\sum_i^k f_i(y|x) p(i|x)$, where $p(i|x)$ is the prior likelihood. Intuitively, this would enable a new form of transcendence, as the mixture distribution would \"choose\" the correct expert given different domains, improving performance over any one expert. We leave a more formal exploration of this form of transcendence for future work.\n\nWhat ethical..\n\n> It is hard to address ethical implications without speculating well beyond the empirical and theoretical results currently available within our paper and related work, and we would not feel comfortable making claims that cannot be backed with rigor or supporting evidence.\n\nHow can dataset..\n\n> There have been several works published on measuring and ensuring dataset diversity, using information-theoretic [1] and geometric approaches [2,3]. \n\n> [1] Dieng, Adji Bousso, et al. \"The Vendi Score: A Diversity Evaluation Metric for Machine Learning.\" 2 Jul. 2023.\n\n> [2] Han, Jiyeon, et al. \"Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images.\" 26 June 2022.\n\n> [3] Sajjadi, Mehdi S. M. et al. “Assessing Generative Models via Precision and Recall.” 2018.\n\nCan the authors..\n\n> Pre-trained language models have already demonstrated the potential of transcendence in providing significant real-world benefits, including already cited use cases such as copyediting, question-answering, and spam filtration. In addition, Khan Academy has launched Khanmigo [1], an AI-powered assistant that uses advanced language models to offer personalized education, enabling tailored learning experiences for students at scale. \n\n> [1] https://www.khanmigo.ai/\n\nLimitations Section:\n\n> We have addressed all these limitations in the above responses to the weaknesses and questions you have raised.\n\n---\n> We are grateful to you for your insightful queries and comments. We hope our clarification and comprehensive responses adequately address the concerns raised and encourage a review of the score assigned. We are confident that the transcendence phenomena and subsequent research agenda could have a significant impact on understanding the capabilities and limitations of future generative models and enable future research on this concept. We kindly request a reevaluation of our work, considering its potential contribution to this area, and thank you for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpsNL8KfRn", "reviewer": "Reviewer_Se7B", "summary": "In this work the authors focus on formalizing and investigating the concept of *trascendence*, i.e. the capacity of a model to outperform the experts it was trained on. To do so, they train transformer-based models for chess limiting experts used for training to a certain maximal ranking. In this setup they postulate and prove a connection of trascendence to low temperature scaling and validate it empirically. Then they continue their analysis by showing that the gain in performance due to low temperature scaling is to be attributed to large improvements over a relatively small subset of states. Finally, a link between trascendence and dataset diversity is drawn.", "questions": "- In Eq.4 there appears to be a typo, since the two terms subtracted on the right-hand-side are exactly the same. In my understanding it should be $y \\sim f(.|x)$ in the second expectation.\n- In line 237 it seems that \"left\" should be replaced with \"right\".", "limitations": "The authors are throughout the paper clear in the assumptions that define the scope of their claims, hence implicitly outlining some limitations of the study. Limitations are also explicitly addressed in the last section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The topic of the work is relevant, and the paper provides interesting insights with solid empirical validation.\n- The choice of chess as an experimental setting is very smart for this study. This setup also allows the authors to perform targeted ablations and validate specific claims.\n- The experiments section is clear, with convincing results, and it is easy for the reader to find and understand the evidence that backs up specific claims in the paper. \n- The paper is well-structured and well-written.", "weaknesses": "- Chess is the only experimental setting considered. While the experiments are thorough in this setting, validating the claims in the paper on multiple and diverse experimental settings would be stronger evidence. \n- Theorem 2, basically easily follows from the assumption that $\\hat{f}_{max}$  is better than the best expert. Since it is so central, can the authors elaborate more on the assumption that $\\hat{f} _{max}$  is better than the best expert? Is it always realistic to assume? \n- The connection between majority voting and low temperature sampling (line 116) should be more formally proved, rather than left to intuition. One could substitute the expression for $\\hat{f} =\\frac{1}{k} \\sum_{ i=1}^k f_i$ in $\\text{softmax}(\\hat{f}(\\cdot|x); \\tau)$ and show that for low $\\tau$ (e.g. taking the $\\lim _{\\tau \\rightarrow 0}$) it results in majority voting. In addition, I am not sure how much this connection is precise. In fact, looking at the Appendix C, doing majority voting of the single experts would not lead to the same outcome as low temperature sampling. It seems more precise to say that low temperature sampling results in a sharp (i.e. low-entropy) distribution with its peak corresponding to the action that is put more mass on by the consensus of experts.\n- To my understanding, in Section 3.4 the mathematical definition of the i-th expert, that performs well on the subset $\\mathcal{X}_i$, should rather be $f_i(y|x) = \\biggl( \\frac{\\delta(y \\in Y^\\star_x) \\delta(x \\in \\mathcal{X}_i)} {|Y^\\star_x|} + \\frac{\\delta(x \\notin \\mathcal{X}_i)} {|\\mathcal{Y}|} \\biggr)$.  Am I missing something?\n- In key parts of the work there are typos/inaccuracies (see above and Questions section), which hamper the clarity of the work and confuse the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> We thank the reviewer for their perceptive questions and feedback on the choice of chess as a setting and clarifications about the theory. In the responses following, we have sought to address each point in detail. We hope our clarifications will elucidate a better understanding of our approach and its potential to influence this research field.\n---\n- Chess is the only experimental setting..\n\n> While chess provides a well-understood and controlled environment for initial validation, we agree that demonstrating our findings across diverse domains would strengthen our claims. Thus, we have run a new preliminary experiment on the Stanford Question Answering Dataset ([SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/)), testing the effects of temperature denoising performance on several LLMs of various sizes. The SQuAD task is a reading-comprehension question-answering task on Wikipedia articles. This setting highlights how denoising improves performance in language models as these transformers begin to make fewer errors in language generation. Empirically, we measure the exact-match and F1 score on the output generations of the LLM at different temperatures. Other prior work has also found ensemble models outperform humans on this task [1,2,3]. As we explain within our paper, ensembling and majority-voting can be thought of as another form of temperature denoising, as these methods all exploit the same underlying property of learner diversity. This expansion provides a broader validation of better performance through low-temperature denoising. We include these results in the rebuttal response.\n\n> In addition, we cite in Appendix D (\"Further Related Work\") that several prior works have also found that pure imitation learning can lead to generative models that outperform the dataset in both the domains of Reinforcement Learning [4] (Atari games) and Natural Language Processing [5] (next-token prediction task), although they do not elucidate the mechanisms under why such phenomenon may occur. In our work, we give one such mechanism of transcendence through temperature denoising and support it with rigorous theoretical and empirical evidence.\n\n> [1] Li, Shilun, et al. \"Ensemble ALBERT on SQuAD 2.0.\" 19 Oct. 2021, arxiv.org/abs/2110.09665\n\n> [2] Zhang, Zhuosheng, et al. \"Retrospective Reader for Machine Reading Comprehension.\" 11 Dec. 2020, arxiv.org/abs/2001.09694v4.\n\n> [3] Rajpurkar, Pranav, et al. \"SQuAD Explorer.\" Stanford Question Answering Dataset.\n\n> [4] Figure 4, Section 5.2. Chen, Lili, et al. \"Decision Transformer: Reinforcement Learning via Sequence Modeling.\" 24 June 2021, arxiv.org/abs/2106.01345.\n\n> [5] Shlegeris, B., Roger, F., and Chan, L. Language models seem to be much better than humans at next-token prediction. Alignment Forum, 2022.\n\n- Theorem 2, basically easily follows..\n\n> Thank you for asking for further clarification. We kindly refer you to Section 3.2, where we elaborated further on the assumption that ${f_{max}}$ is better than the best expert. We discuss scenarios where this assumption holds true, particularly in contexts where the model benefits from aggregating diverse expert knowledge, leading to performance that exceeds individual experts. To be clear, this is not always realistic to assume. For instance, in the paper, we demonstrate that one cannot assume that $\\hat{f}_{max}$ is always better than the best expert in the 1500-rating chess example. Due to the lower data diversity, aggregating the different experts does not result in better performance than the best expert. Another example is given in the new toy model that we propose (see the Rebuttal Summary), where training on data generated by non-diverse experts (low std) does not result in transcendence.\n\n- The connection between majority voting and low temperature sampling..\n\n> We will give here a short proof for this result, and add a more detailed proof to the final revision of the paper. Let $z$ be the vector of outputs, i.e. $z = \\hat{f}(\\cdot|x)$, and denote $s = \\text{softmax}(z|\\tau)$, where $\\tau$ is the temperature. Let $a = \\max z$ be the maximal value in $z$, or the value of the logit(s) for the majority vote. Let $i$ be some coordinate of $z$ that is strictly smaller than $a$, namely $z_i < a$. Then, observe that $s_i = \\frac{\\exp(z_i/\\tau)}{\\sum_{j}\\exp(z_j/\\tau)} \\le \\frac{\\exp(z_i/\\tau)}{\\exp(a/\\tau)} = \\exp((z_i-a)/\\tau) \\to_{\\tau \\to 0} 0$.\n> Therefore, all the coordinates that are smaller than the maximal value will converge to probability $0$ as $\\tau$ converges to $0$. It is therefore easy to show when $\\tau \\to 0$ the vector $s$ converges to a vector that is the uniform distribution over the maximal values of $z$ (which we denote by $\\hat{f}_{\\text{max}}$).\n\n- To my understanding, in Section 3.4..\n\n> It is possible that our original notation was not clear enough. We denoted by $\\delta(\\text{condition}) = 1$ the function that outputs $1$ if the condition holds, and wrote $\\delta(y \\in Y^*_x, x \\in X_i)$ to indicate that the function is $1$ if both $y \\in Y^*_x$ and $x \\in X_i$. Your suggested notation is equivalent but easier to understand, so we will update our paper to use this notation instead.\n\n- In Eq.4..\n\n> Thank you for pointing this out. After correcting, the second expectation in Eq. 4 now correctly reads $y∼f(.∣x)$, which clarifies the calculation of the reward difference.\n\n- In line 237..\n\n> We have made the suggested correction, replacing 'left' with 'right' in line 237 to accurately describe the direction in the context.\n---\n> We appreciate the feedback and suggestions, which have significantly improved the clarity and rigor of our paper. We hope that these revisions address your concerns and help our paper meet the standards for publication. We believe that our work on transcendence in generative models has the potential to influence future research in this area, and we are grateful for your review and insights. We look forward to further feedback you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LrqwupNcFi", "reviewer": "Reviewer_np7r", "summary": "1. Definition of Transcendence: The paper defines transcendence as a phenomenon where a generative model, trained on data from human experts, achieves capabilities that surpass the abilities of those experts.\n\n2. Theoretical Framework: The authors provide a theoretical analysis of the conditions under which transcendence can occur. They prove that low-temperature sampling is necessary for transcendence in their setting.\n\n3. Chess as a Testbed: The researchers use chess as a domain to empirically test their theories, training transformer models (called \"ChessFormers\") on datasets of human chess games with various skill levels.\n\n4. Main Findings:\n   - Low-temperature sampling enables transcendence: Models trained on games from players rated up to 1000 and 1300 were able to achieve ratings above 1500 when using low-temperature sampling.\n   - Dataset diversity is crucial: The model trained on games up to 1500 rating did not achieve transcendence, likely due to less diversity in the dataset.\n   - Denoising effect: Low-temperature sampling helps by shifting probability mass towards better moves in key situations.\n\n5. Theoretical Insights: The paper draws connections between their findings and existing literature on model ensembling and the \"wisdom of the crowd\" effect.", "questions": "See weaknesses\n\nTypo in line 100 we make assume \nTypo in line 123 outputs a correct but noisy prediction(s).\nReference in line 201 - reference to the current section \nTypo in line 237: to the right?\nFigure 4, improve legend and generally readability/visualizations of the Figure, E[F] two times, not clear what the difference is \nTypo in line 249 This is matches \nTypo in line 265 We, we\nReference Theorem line 142", "limitations": "Looks fine, would be great to  discuss future research directions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies an interesting questions theoretically and shows the insights relate on a nicely accessible empirical problem namely chess. One strength of the paper are also the simple theoretical arguments presented, which seem relevant in non-trivial settings. Paper is quite well written (one could improve here quite a bit) and results nicely presented, although the paper seems quite rushed.", "weaknesses": "I think the theoretical section could be improved by integrating the proofs, when given the extra page if accepted. Also I wouldn’t call the theoretical insights „theorems“ but rather propositions. \nI think the paper would greatly benefit from a toy model which allows to study when transcendence is possible, theoretically and practically. I think the major weakness of the paper is the missing understanding when transcendence can occur and when it can’t.\n\nDidn’t have much time to review this paper unfortunately, will study it in more detail in the next weeks and consider raising my score, after the authors feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> We appreciate the reviewer's insightful questions and valuable critiques. To summarize, we have addressed concerns about a toy theoretical model, and typographical errors. We trust that our responses below will provide a more comprehensive understanding of our research and its implications.\n\n---\nI think the theoretical section could be improved by integrating the proofs, when given the extra page if accepted. Also I wouldn’t call the theoretical insights „theorems“ but rather propositions. \n\n> We agree with the reviewer's sentiment, and will rename the theorems to propositions instead, and will integrate the proofs given the extra page if accepted.\n\nI think the paper would greatly benefit from a toy model which allows to study when transcendence is possible, theoretically and practically. \n\n> We agree that the paper can benefit from a toy model for studying transcendence. Following your suggestion, we introduce the following toy model, demonstrating transcendence in the case of Gaussian data and linearly separable classes. Specifically, our input data is $d$-dimensional Gaussian, and the target is classification with $n$ classes. The ground-truth is generated by a linear function, i.e. $y = \\arg \\max_i W_i^\\star x$ for some $W^* \\in \\mathbb{R}^{n \\times d}$. We sample $k$ experts to label the data, where the labels of each expert are generated by some $W \\in \\mathbb{R}^{n \\times d}$ s.t. $W = W^* + \\xi$, where $\\xi_{i,j} \\sim \\mathcal{N}(0, \\sigma^2)$, for some standard deviation $\\sigma$. Namely, each expert labels the data with a noisy version of the ground truth separator, with noise std $\\sigma$. We then train a linear model on a dataset with $m$ examples, where each example is labeled by a random expert. We observe the test accuracy, measured by the probability assigned to the correct class, for different choices of temperature, and compare to the best expert. As can be seen in our synthetic experiments (**in the attached rebuttal PDF**), we are able to observe transcendence in exactly the same situation as in our chess setting, i.e. - when the diversity of the experts is high (high std) and the temperature is low. This also aligns with our theoretical analysis.\n\nI think the major weakness of the paper is the missing understanding when transcendence can occur and when it can’t.\n\n> We would like to clarify that our paper provides a thorough analysis of the conditions necessary for transcendence both theoretically and empirically. Specifically, we detail in Section 3 the theoretical conditions for transcendence, emphasizing the necessity of low-temperature sampling (Section 3.1) and dataset diversity (Section 3.4). Our empirical results in Section 4 further validate these theoretical insights.\n> * Theoretical Analysis (Section 3): We analyze the necessity and sufficiency of low-temperature sampling and demonstrate that without it, transcendence cannot be achieved (Theorem 1). We also explore conditions involving multiple experts and the role of data diversity.\n> * Empirical Validation (Section 4.2): Our experiments confirm that low-temperature sampling enables transcendence to occur on high-diversity datasets by denoising errors (Figure 2 and Figure 4). We show that models trained on less diverse datasets, such as ChessFormer 1500, fail to transcend, highlighting the importance of dataset diversity (Figure 5).\n> * Toy Model: We note that our new toy model (see Rebuttal Summary) also cleanly demonstrates that transcendence is possible only in cases where the expert diversity is high, and the sampling temperature is low.\n\n\nSee weaknesses\n\n* Typo in line 100 we make assume \n* Typo in line 123 outputs a correct but noisy prediction(s).\n* Reference in line 201 - reference to the current section \n* Typo in line 237: to the right?\n* Figure 4, improve legend and generally readability/visualizations of the Figure, E[F] two times, not clear what the difference is \n* Typo in line 249 This is matches \n* Typo in line 265 We, we\n* Reference Theorem line 142\n\n> We apologize for the typographical errors and unclear references you noted. We have corrected the following issues in our local version and will upload the revisions in the final version of our paper if it is accepted.\n\n> In response to the Figure 4 concerns, we would like to clarify that it presents the expected reward distributions at two different temperatures, specifically $\\tau=0.75$ and $\\tau=0.001$. The two instances of E[F] represent the expected reward under these two temperature settings. We will revise the figure legend and accompanying text to make this distinction clearer.\"\n\nLooks fine, would be great to discuss future research directions.\n\n> We would like to kindly clarify that we do lay out future research directions in Section 6, Discussion and Future Work. Some possible future work may include seeking other causes of Transcendence, expanding our analysis to more domains such as NLP, CV, and RL, and finally potentially using Transcendence in an iterative fashion to achieve a stable RL-like algorithm without learning a critic function but purely through Imitation Learning. All of these directions would be quite interesting to pursue.\n\n---\n\n> Given these clarifications and the thorough addressing of your concerns, we hope the reviewer may consider raising their score. We are confident that our detailed response and amendments underscore the rigor and scalability of our work, potentially driving future insights into generative models. We trust that these responses adequately address your concerns. Your insight has been highly constructive and we appreciate your time in this review process. We hope to hear from you soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Note 

**Thank You.** We thank the reviewers for their insightful feedback and comments. We are encouraged to find that the reviewers recognized the paper as novel, introducing "a new perspective on model performance" (mPZQ). We are also grateful that the reviewers appreciated the evaluation of our method in chess, "an experimental setting very smart for this study" (Se7b) and "nicely accessible" (np7r), adding credibility to our findings with "clear, convincing results" and "solid empirical validation" (Se7b). We are pleased to hear that the reviewers found the paper "well-structured and well-written" (Se7B,np7r). We also appreciate that the theoretical section was noted as insightful (Se7b) and "relevant in non-trivial settings" (np7r), and offers "rigorous theoretical proofs that support their claims, grounding the concept of transcendence in solid mathematical principles" (mPZQ). 


**New Experiments.**  To address the feedback regarding the generalization of our findings beyond chess, we have conducted two new experiments:

1. **NLP Experiment:**
   We extended our analysis to the Natural Language Processing domain by running experiments on the Stanford Question Answering Dataset (SQuAD 2.0). We tested the effects of temperature denoising on the performance of several large language models (LLMs) of varying sizes. The SQuAD task involves reading comprehension and question-answering based on Wikipedia articles, making it an ideal setting to evaluate the impact of denoising on language models. We measured the exact-match, semantic-match, and F1 scores of the model outputs at different temperatures. The results show that temperature denoising leads to improved performance, corroborating the findings of our chess experiments and providing broader validation of the underlying mechanism of temperature denoising in diverse domains.

2. **Toy Theoretical Model:**
   We developed a toy theoretical model to further study when transcendence is possible. This model involves a classification task with Gaussian input data and linearly separable classes. Experts label the data with noisy versions of the ground truth separator. We trained a linear model on a dataset labeled by random experts and observed the test accuracy for different temperature settings. The synthetic experiments demonstrated that transcendence occurs when expert diversity is high and temperature is low, aligning with our theoretical and empirical analysis in the chess domain.

These additional experiments support the generalizability of our findings and provide a more comprehensive understanding of the conditions under which transcendence can occur in different settings. Please find the new results and detailed analysis attached to the 1-page rebuttal PDF.

Meta-review: The paper studies the phenomenon where a generative model, trained on data from human experts, achieves capabilities that surpass the abilities of those experts. The authors provide a theoretical analysis of the conditions under which transcendence can occur. A theoretical framework is provided that allows to analyse the conditions under which transcendence can occur and it is shown that low-temperature sampling is necessary to observe transcendence. The relevance of the findings is demonstrated on chess. In the rebuttal further experimental results on a question-answering set as well as a theoretical toy-model were added, strengthening the paper. 

I agree with the two more elaborated reviews that voted for clear acceptance.
Decision: Accept (poster)