On Masked Pre-training and the Marginal Likelihood
Authors: Pablo Moreno-Muñoz
Pol G. Recasens
Søren Hauberg
Venue: NeurIPS 2023 poster (2023)
Keywords: Marginal likelihood
masked pre-training
Bayesian inference
TL;DR: Masked pre-training maximizes the model's marginal likelihood.

Abstract: Masked pre-training removes random input dimensions and learns a model that can predict the missing values. Empirical results indicate that this intuitive form of self-supervised learning yields models that generalize very well to new domains. A theoretical understanding is, however, lacking. This paper shows that masked pre-training with a suitable cumulative scoring function corresponds to maximizing the model's marginal likelihood, which is de facto the Bayesian model selection measure of generalization. Beyond shedding light on the success of masked pre-training, this insight also suggests that Bayesian models can be trained with appropriately designed self-supervision. Empirically, we confirm the developed theory and explore the main learning principles of masked pre-training in large language models.

On Masked Pre-training and the Marginal Likelihood
Pablo Moreno-Muñoz Section for Cognitive Systems Technical University of Denmark (DTU) pabmo@dtu.dk
Pol G. Recasens"
CROMAL, Barcelona Supercomputing Center
Universitat Politécnica de Catalunya (UPC) pol. gawcciabbbbc.es
Soren Hauberg
Section for Cognitive Systems Technical University of Denmark (DTU) sohauddtu.dk
Abstract
Masked pre-training removes random input dimensions and learns a model that can predict the missing values. Empirical results indicate that this intuitive form of self-supervised learning yields models that generalize very well new domains. A theoretical understanding is, however, lacking. This paper shows that masked pre- training with a suitable cumulative scoring function corresponds to maximizing the model's marginal likelihood, which is de facto the Bayesian model selection mea- sure of generalization. Beyond shedding light on the success of masked pre-training, this insight also suggests that Bayesian models can be trained with appropriately de- signed self-supervision. Empirically, we confirm the developed theory and explore the main learning principles of masked pre-training in large language models.
1 Introduction
Masked pre-training (MPT) is a family of self-supervised learning methods (Dosovitskiy et al., 2020; Devlin al., 2018; Caron al., 2021), that empirically has been demonstrated result in models that generalize very well to new settings. In essence, masked pre-training removes random features of the data and learns model to recover these from the remaining input. While empirical results are impressive, deeper understanding of why pre-trained models generalize so well is lacking, Is it due to the use of transformer architectures (Vaswani et al., 2017), the vast over- per-parametrization (Neyshabur et al., 2019), something entirely different?
The marginal likelihood evidence is commonly used the measure of generalization ability in Bayesian models (Tenenbaum and Griffiths, 2001; MacKay, 2003). While computationally expensive, the blessing of the marginal likelihood comes from the probabilistic integration of hypotheses. Whenever we are considering a latent variable model in the Bayesian framework, such integration can be thought of the average over all the possible latent variable mappings, weighted by our prior beliefs. Since masked pre-tras pre-training drives generalization so well, the lingering question in the Bayesian modeling community is then: masked pre-traiin pre-training somehow related the maximization of the marginal likelihood?
In this paper, provide positive to a stochastic gradient of the log- marginall likelihood (LML). Importantly, the log-marginal likelihood equivalent the cumulative sum of masked pre-trate pre-training losses shaped with different sizes for the random mask. Even if its practical use avoids this cumulative sum, we demonstrate that choosing a
We show that masked pre-training optimizes according
Work done during an Erasmus exchange in DTU (Denmark).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

fixed masking rate, e.g. 15% as in BERT (Devlin et al., 2018), leads to a stochastic biased estimation which still maximizes the log-marginal likelihood.
Our proof relies on a previous observation from Fong and Holmes (2020), who show that the log-marginal likelihood equals the average of exhaustive leave-M-ow cross-validation (CV) given posterior predictive scores. Intuitively, our formal results can be seen as the transposed version of Fong and Holmes's results: where CV removes random observations measure generalization. masked pre-training removes random random.eatuuess. While the seminal link between CV and the marginal likelihood was purely formal result that pointed out the underlying presence of Bayesian principles in well-known class of learning, our work extends the theory behind the marginal likelihood to comprehend the impressive behavior of the latest generative models.
Masked pre-training
Masked pre-training (MPT) is variant of self-supervised learning (Dosovitskiy et al., 2020; Devlin et al., 2018) that removes random input dimensions (also known as masking) in the observed data and learns model that accurately predicts the missing values. This family of methods, well-known due to their success in natural language understanding, typically adopts a transformer architecture (Vaswani al., 2017) as the feature extractor, that together with positional encodings and random masked dimensions allows capturing the bidirectional context in the data.
In BERT (Devlin et al., 2018), each sentence is usually considered vector, x = (I1, Z2 ...,................. x.)) where dimensions F2 are named tokens. Given a random mask M of size MrD, as set of indices drawn uniformly from {l...., D), D), each token whose index belongs to M is considered to be in the subset I. = (I M(2). "M(M)).......,M)) M- We refer to these as the masked tokens. The rest of indices R {1, 21,2,..., D) \ M induce the complementary subset RR. such that M U ER. Under this notation, MPT learns the parameters 0 of model pa(() by maximising an average of the following objective
D dimensional observation
M log po (1 (ZRR)
log log (20
for every observation in the dataset DD The stochastic choice of a, makes predictive conditionals po(x (œR) to be different for every observation and training step. Once the pre-training of A has converged, this naturally allows the model to capture the underlying structure between dimensions of the data. One additional remark is the number of random masks needed cover all combinations
between masked and observed tokens, which can be obtained as Ca (D)) In the particular
example of BERT, where the masking rate is 15% with D = 512 and M = 76, the total number of random masks needed cover all combinations of tokens is CM R 1.21 x 10 This shows the inner combinatorial problem behind MPT. We provide empirical results on why this is not a limitation for learning with MPT in Sec. 3.1.
3 A probabilistic perspective, theory, and analysis
Our key objective is to demonstrate that the good generalization of MPT can be explained as an equivalence with the model's high marginal likelihood. Indeed, we will prove that MPT implicitly maximizes marginal likelihood according to some latent variable model of the form po pz]
Marginal likelihood. For our theory, we consider some dataset consisting of n i.idd observations Vime where each sample x; could be either continuous also assume that there exists a latent space Z E IR K where we can find unobserved variables 2im which are part of the generative process of the data. This assumption is inspired in the common use of latent encodings in recent models fitted with MPT. In this direction, we also consider the observations to be samples of a likelihood function po ax z2), where the mapping between the latent and observed variable is controlled by some parameters 6, which might also include likelihood or prior hyperparameters.
discrete and is of dimensionality D. We
Importantly, we consider the parameters be deterministic, while we are interested integrating out the latent variables that cannot observe. Automatically, this leads to the -ogamnginaa

---

likelihood (LML) of the model, which may factorize as sum of marginals and can be also written as log Po (Zi:.) - 1 Po (o,),, where the 7th probability density comes from the integral pe( = pe (=: This definition coincides with the target LML used in the lower bound of variational autoencoders (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) and it is widely used in probabilistic generative models.
Masking and conditional probabilities. From the properties of probability distributions, we can decompose the individual L.ML. functions log po (ar;) as a sum of log-conditionals between tokens. Omitting the zh observation subscript in x: to keep the notation uncluttered, the sum takes the form
(a(
Σ po (Te (LL+1:D)
However, the previous sum imposes a particular order on the selection of variables for conditioning,
e.g. (21 (7,,, -....-- ), {ae #4, ..4,,...,,, ), etc. Moreover, the order of tokens in the observation vector remains predetermined, as dimensions are not exchangeable. Thus, we can consider a different com- bination of conditional probabilities in the sum for instance, {2 ", .2,... }.
{2a[21 1 ....... etc. Here, the key insight is that the rules of probability applied to the log-marginal likelihood make it invariant the combination of different conditional factors, as we are observing different views of the same graphical model.
This combinatorial process between tokens in be understood as the selection problem of indices. For that reason, we can assume mask M of the targest |M = D, such that /hat = {1, 21,..... ............ D). Using similar properties of combinatories, we can also obtain D! different choices for M. While all the indices are always in the set, the order of indices differs between combinations. This principled order in M indicates how sum the conditional probabilities in Eq. 2.
Since the LML is invariant to random choices of M, we can re-write the sum in Eq. 2 as an expectation with a countable set of possible outcomes. Each outcome corresponds to one of the D! choices for M, such that
1 DI Σ
log pa (x:)
(x) log pa (aM))) Mm M.HLD:) Σ Ex [oggott (n) (x) M()) Mctt)))
(3)
D!
where the superscript (x) denotes which mask M are we using for indexing the tokens. We also swapped the order of the sums to obtain the desired expectation in the r.h.s. of the formula.
The role of random masking. we now take a particular index (f) and we look at the Tth summand in the previous expression, can see that the LML is still invariant to the order of the conditioning tokens a 3 in the log-probabilities: log (I a (M)++:)))) in the sum. Intuitively, we can use both {.1 [x2), {22} or {22 [r1) {01) independently of the conditional factors previously considered. In practice, this indicates that we can insert a second set of indices to the r.h.s. variables, which is the key point to link negative MPT loss and L.I...
Now, assume that M indexes less than 100% of tokens, while the rest is indexed by R as defined in Sec. 2. If we match both complementary masks to be aligned with the conditional and conditioning variables in the log-probabilities this allows us to rewrite the 1th summands in Eq. 2 as
D! log pe (zM (o(())(()))))
1 Σ
log po
(T)
1
(T)
Here, we can easily see that there are (P))) choices for the unmasked tokens in the r.h.s. of the conditional distribution, where we have previously fixed the index t. If we set the binomial coefficient C, ((2)) as the maximum number of choices, we can obtain the following equality
C. D-++1
(5) Σ log pa (IM la (I(.D-) = Σ Σ
logps (2) (TMS) x (5)
since D! C. VL E (1,2...., D}. Notice that once we have chosen a specific order (T) in the masking pattern of M and R in Eq. there are still (D t 1) choices for the masked tokens
3

---

under evaluation in the probability distribution. Alternatiely, we can think of this method as taking advantage the properties of probability to split the D! choices in the order of log-conditional into the two sums in Eq. 4. The driving idea is then that the two sums in the previous expression still remain invariant given any E (1,2....,4 D).
Using the previous notion in Eq. 2, we obtained our main result, which holds under the assumption of i..d. observations with correlated tokens and the previous definition of the LML as the integral over the stochastic latent variables in the model.
Proposition 1 - The cumulative expected loss of masked pre-training along the sizes of the mask of tokens M E f1,2...., D} is equivalent to the log-marginal likelihood of the model when using self-predictive conditionals probabilities. such that
log (o
Σ M=1
Sa M),
(5)
where the score function So(:; M) corresponds to
CM
M
M
5o(x;M) = Cm M
1
Σ
1
Σ
log po (o MM) Mb) -)()
(T)
EM
Σ
log po (o(
M
j=1
Proof: In the supplementary material.
is remarkably important link the sum of log-conditionals log (1)) (R) in our proposition with the main objective used in MPT in Eq. 1. The main message of our result is that the score function ()(=;t) acts as an average over the different random masks. These shape the structure of conditioning in probabilities. The cumulative sum of the score function Sa t) over different sizes of the MPT's mask formally leads to the true value of the model's LML. This result exact whenever we consider the closed-form self-predictive probabilities of the model and all the possible choices for the masking pattern M. Since this usually not affordable, due to the combinatorial cost and the lack of tractability, we usually have biased estimator. However, it is still sufficient to prove that MPT maximizes LML during training as we will show later. This point will be discussed in the following empirical studies. Further details on the derivations are included in the supplementary material.
3.1 Formal results in tractable models
To verify that masked pre-training effectively maximizes LMI., we need a tractable probabilistic model based on latent variables the proof-of-conceptt Probabilistic PCA (PPCA) (Tipping and Bishop, 1999) is perhaps the best option here, as it has been previously used to understand other empirical observations in generative methods, e.g. posterior collapse in VAES (Lucas et al., 2019), even considered as the starting point of GPLVMS (Lawrence, 2005). In particular, the PPCA model assumes that Gaussian observations map linearly to sets of real-valued latent variables Zi:n. such that æ Wz z + f + where e ~ (00 I1). Importantly, the prior is conventionally defined as isotropic. where p(z) (0, 1). We are therefore interested in the closed form expression of the PPCA's 's LML, which also factorizes across samples as follows
log pa (x =
Σ log pe (x:), where ==(x:)==N( = ((t::)=((x((u.S)) S),
(6)
and obtain the covariance matrix using S = WWT +of.. For our analysis, the Gaussian nature of po(x;,) of fundamental importance. Given the random mask M, the self-predictive conditionals used in MPT naturally emerge from the formulation using properties of Gaussian marginals, such that po pa(x ØR))= = (mm (M(R) is parameterized according to
mRRR SJeR R UMRR SMm + SRt
(7)
where we split the LMI. covariance matrix S into the blocks corresponding to the indices included in M and R. We use these mean and variance parameters of the self- pelfitredeetive density to recursively

---

evaluate the log prg-bbbbbbilitis in Prop. 1. In practice, two elements become critical for the computation. one is the size M of the mask and another the number random masks P < Ca considered. These induce trade-off between accuracy and computational cost. Moreover, their role in approximating L.I. using a biased estimate carefully analyzed the following empirical studies. We also included additional details on the previous derivations in the supplementary material.
Fast asymptotic convergence. Our theory indicates that we should evaluate all Cas random masks of the tokens to achieve the exact value of the LML. However, even if the combinatorial nature of the sum in the r.h.s. of the last equation in Prop. 1 becomes very large when the dimensionality of data augments, we suspect that it might converge relatively fast to the true value of the LMI.. This hypothesis would explain why large models that are fitted with standard MPT generalize well using just one random mask per training epoch.
Here, we empirically study if the cumulative MPT loss converges to the true value of the LML under the definition of the PPCA model. In particular, to the LML obtained with the original parameters that generated the data. The results in Fig. 1 and Tab. indicate that as long as we average over more random masking patterns, the cumulative MPT loss approximates the LML of the model very well. Thus, having defined a PPCA model with a latent space of K 2 dimensions, we observe in the left and middle plots that the asymptotic convergence happens for both small (D = 5) and large (D 50) number of tokens per observation. Additionally, we observe that the estimation of LMT. is clearly unbiased if we use the cumulative MPT loss according to Eq. 1, which is an important insight. Notice that P is usually set up MPT in practice. Additionally, we tested the tractable model
Table 1:
Evolution of negative MPT loss w.r.t. max. number of random masks P.
TRUE L.M. (t)
P=1
p= 10
p= 100
(- 60.34)
-60.44 = 0.47
-60.22 + 0.12
60.34 0.03
Converganre of MPT vs. LML
TOKENS
# TOKENS 50
TOXENS
-IX MASK
22
0.10
-3.230
14
0.50
Diferroc
32
0:25
-10225
-1030
-15
-0.00
-34
-0.00
6.70
-0.000
100
10C
10c
-1300 10
150
Num. of random masks
Num of rendom masks
Num. random masks
Figure 1: Asymptotic convergence of the cumulative MPT loss to LML as the number of random masks P augments. Curves indicate the relative difference, where 0.0 means that MPT equals LML. (Left). Each observation consists of 5 tokens. (Center) Each observation consists of 50 tokens. (Right). Observations have 512 tokens and the rate of masking is fixed to 15% (76 tokens). As the theory indicates, when the size of M is fixed, the cumulative MPT loss becomes a biased estimator of the L.M... The curves converge asymptotically to the bias.
using a dimensionality similar to the input data used in BERT (Devlin et al., 2018), where the number of tokens is typically D = 512 per observation and the mask rate is fixed to 15%. The fact of fixing the rate of masking in MPT produces that the sum in Eq. 5 is incomplete. Thus, we have biased estimation of the L.MI.. However, this bias is known and constant during the training of parameters 0, which does not prevent the general maximization of LML. This point is carefully analyzed in the next empirical study with learning curves. One additional finding here is that as P + Car, the cumulative MPT loss also converges asymptotically to the biased estimator of the LML shown in the right plot in Fig. I.
LML maximization and biased estimation. We next seek to extend the previous study to under- stand the behavior of the cumulative MPT loss in training curves. So far, we have observed how the number of random mask patterns affects the precision around the unbiased estimation of the
5

---

LML. Theory and previous empirical results indicate that we are targeting LML or at least decent biased estimate of LML when averaging self preddccieitive conditionals as MPT. However, we still want to examine if this maximizes L.I. in all cases and under stochastic gradient optimization. This principal hypothesis is confirmed in Fig. 2. where different training curves are shown for different initializations and setups of the same PPCA model. The key insight showed by this experiment is that the exact LML is iteratively maximized at each epoch, in parallel with the maximization of the negative MPT loss. On the other side, we also have that MPT is an unbiased stochastic approximation of LML, as in Fig. 1, whenever we consider different rates of random masking M. We can also observe that as soon as we fix the size of the mask to index the 20% of tokens, the MPT loss becomes a biased estimate. Intuitively, this is equivalent to fixing M in the sum in Eq. 5. Again, it converges to the same value from different initializations of parameters 0. Additionally, we highlight that the LMI. is still maximized in this case, which is of high similarity to practical uses in larger models. Overall, this result first confirms the main insight of the work the link between generalization when using MPT and the maximization of the model's LMI..
Training curves/ # TOWENSS 16 MASIS
Training curses & TOCKERS 10 MASSCS
FIXED MASE
EMT bissed estimnle
Noxeetvv Ner 1/155
Tres model 1t0 Commirive SPP
True model 1itk Chumiaaivve SIP
Epocks
Epocki
Spooks
Figure 2: Training curves of the negative cumulative MPT loss in PPCA vs. the ground truth (GT)
LML. The number of samples is N 2000 and the number of tokens is D = 10. All plots used P= 1 random masks per epoch and five different initializations. (Left). The rate of masking is unfixed and it varies from 1% until 100%. The negative MPT loss converges to the GT-LML (dashed line). Darker curves are the exact LML per epoch. (Center). Convergence with fixed mask to 20% of tokens. The negative MPT loss is no longer centered around the LML and it converges to biased estimate. (Right). Zoomed curves of convergence. The bias is constant and all MPT losses converge to the same point. The LML per epoch is also maximized and converges to GT-LML.
Beyond tractable models and implicit integration. One remaining question in our analysis is how the probabilistic theory around MPT adapts to intractable or non-linear models. In practice, self-predictive probabilities imply integrating out the latent variables, often given the posterior distribution. In most cases, performing this integration is extremely difficult or not possible in training time. Therefore, we are interested in finding if alternative approximations qe to the true self-conditional probabilities still produce accurate estimation and maximization of the LML. This point is confirmed in Fig. 3. Inspired by the experiments of Lucas et al. (2019) with linear VAES, we set up Bernoulli likelihood top of the latent variable model. The tractable formulation in the Gaussian example coincides with PPCA. Since predictive conditionals are no longer tractable for us, we use numerical integration to obtain the probabilities of masked tokens. In Fig. 3, test the training with the cumulative MPT loss as well as compare with standard variational inference using the model's 's evidence lower bound (ELBO). For the mini-dataset with MNIST samples, we observe that both models converge to similar value of the LMI.. Thus, the fundamental insight here is that MPT maximizes LML even under training with approximate self-predictive conditional probabilities. For the LML curves, we also used numerical integration.
Beyond linear models, our theory is useful when applied to non-linear models. Moreover, in Fig. 3 we also include the results for deep VAES based on NNs. While the estimation of LML was obtained via Monte Carlo (MC) samples, we used iterative encoding-decoding decoding produce the self-conditional probabilities for masked tokens- - see Sec. F in Rezende et al. (2014). In this scenario, we also observe the maximization of the LML according to the evolution of the MPT loss.
Another key insight showed by this study is the ability of MPT to perform implicit integration. The cumulative sum over the different rates of random masking is another way to discrete integral

---

BERNOULLI LINEAR VAE MNIST
BERNOULLI LINEAR VAE FMNIST
200
NEGATIVE LOss
-200
-00
-120
XL.50
KL3O
1312 XLB0
LM E2.10
-1000
XPPT
-1260
100
200
330
10C
200
690
750
3000
610
Epochs
Epockss
VANILLA VAE / MNIST / (1ML))
VANILLA VAE MNIST (MPT)
-30
-1802
NEGATIVE LOSS
-390
-430
-2303
-49-
-7623
XLBO
-590
1302 ELBO
-2802
25 75 166
135
17o
175
-3002
80
100
Epochs
Epockks
Figure Training curves for linear VAE and deep VAF models with variational inference (V1) and MPT. Data consist of subsets of MNIST and FMNIST. (Upper Row). A lincar VAE model with Bernoulli likelihood function in N 2000 samples of MNIST and FMNIST. Shaded curves correspond to the target losses used in the optimizer (ELBO and MPT). Darker lines indicate the evolution of the LML, which are approximated via numerical integration in a latent space Z of dimensionality K = 2. (Lower Row). Vanilla VAE with Gaussian likelihood for MNIST. The LML curves are approximated via Monte Carlo (MC) samples. Self- celf-predietive conditional probabilities are obtained via recursive encoding-decoding. The size of the random masking is fixed and 33%.
/ Mac permutaaionss
# TOXENS 200 Mix. pertuntations
K prem
3/10
-33
MPT hoss
-100
-1.5
-155
-12
15
175
276
1
32%
10.05
1.5
10% 28
176 405
1000
125 30.
195
06 of masked tolems
56 masked tolocts
of maslon tolens
Figure 4: Area under the curve described by So(:; M). The area is approximately equal to the model's L.I. according to the theory. Larger probability values are obtained with smaller rates of masking. (Left). Area described with P = 1 random masking per epoch. The curve is more noisy and the area slightly loses precision w.r.t. LMI... (Center). Area under the MPT curve for 100. (Right). Latent space is augmented to be of K 50. Decay of predictive probabilities begins at around 50% masking rate.
under the curve described by the score function (a M) ) in Eq. 5. In Fig. 4, we show the areas under the curve and the effect of reducing the number random masks P. The blue plots correspond to trained PPCA model and the area corresponds to the LML estimate, The long tail in the right part of the curves, when the rate of masking is larger than 90%, indicates that the model is longer able to produce good estimates of the tokens with only 10% of the input dimensions observed. This explains, why the probabilities have an approximately exponential decay. However, this effect not constant, and it might depend on the latent structure of the model. In the r.h.s. plot we observe that the decay of conditional probabilities happens earlier at approximate 50% random masking or larger. The role of the masking rate is perbaps the missing part in the picture (Wettig et al., 2022), as it is the one that
7

---

determines the approximation to the LML. With the purpose of providing an intuition on how rates of 15% or 85% affect to the area under the curve, we indicate with two black lines the approximate area that approximates the I.M... A longer discussion is provided in the supplementary material.
Table Area under the MPT curve for BERT model and four GLUE datasets.
GLUE DATASETS -
AX
COLA
QNLI
MRPC
AREA/ RANDOM INIT. (f)
5245.31 -1715.75
5283.52 1657.68
5343.98 1770.28
5362.21
AREA / PRE-TRAINED (1)
1773.45
SAILT AX
52172 COLA
TRACTABLE MODEL (PPCA)
(Tam EAIL 17.13)
From trdde to IMI TOKENS 50
SEPT
AREA 17 .3
16.88
Altis
B2RT MIPC
DERT QNIL
ARBA 196.05
NPE boes
ARBI -25 86
12%
3014
55%
13%
85%
10791
% maskeed tokens
toloms
masland toloma
Figure 5: Evolution of the area under the MPT curve. Comparison between one tractable model (PPCA) and BERT. The area under the curves is approximately the LML. Random initialization of the parameters produces MPT curves with similar low probabilities for all % of masking. As the number of epochs increases, the curve brings higher values of pog-prbabbility for lower ratios of masking. The area also converges to the true value of LMI.. (Left). PPCA model trained for 600 epochs. Each curve represents {0, 100, 200, 300, 400, 500, 600) epochs of training with MPT. (Right). Random initialization and end-of-pretraining curves for the MPT loss w.r.t. the % of masked tokens. Curves are similar but not identical for the different datasets given the pre-trained BERT model.
3.2 Applied theory on large language models
In this section, we aim to understand how the area under the MPT curve evolves and behaves for large language models (LLMs). While the direct computation of the LML is not feasible for non-lincer transformer models, are interested in checking how the rate of masking affects the curve compared with the tractable PPCA model. The results provided in Fig. 5 and Tab. 2 give us insights into this behavior. First, observe that the MPT curve is approximately flat for every rate of masking in the PPCA when parameters are randomly initialized. Intuitively, this indicates that the model is not able to correctly predict any token given some context. In some way, produces noise independently of the number of conditional tokens, which explains the low log-probabilities. Second, we can also notice that the curve changes its shape as more training epochs epochs produces high probability values for different rates of masking, while the long tail of low probabilities appears when masking more than 85% of tokens. Moreover, the area under these curves is the estimation of the LML, which accurately converges to the ground truth value of the LML with the original generative parameters.
considered. The curve after 600
For the study of the curves in LLMS, we used four datasets from the General Language Understanding Evaluation (GLUE) (Wang et al., 2019). Additionally, we consider 110M parameters BERT model using pre-trained c and random initializations. To draw the MPT curves, we computed the mean cross-entropy per each rate of masking between 1% and 99%. In Fig. 5, we observe that random initializations of BERT parameters lead to flat curves of low self-predictive probabilities. On the other hand, the pre- pre-trained curves show similar behavior as in the tractable model, where the area
Pree-traine parameters for the BERT model are available in the library httph:ttp/// httpp::/hugginggface.om
8

---

is reduced and a long tail of low probabilities happens when the rate of masking becomes larger. This result supports our hypothesis that MPT in LLMS might be performing implicit integration of the latent space and maximizing the marginal likelihood of the model.
Results on vision models. In addition to the results shown in Fig. 5 with the BERT model, we also interested in the behavior of the theory on large models oriented to vision. For this study we provide the curves of the area under the MPT loss for VIT-MAE (Dosovitskiy et al., 2020; He et al., 2022) with different masking rates. In a similar way as in Sec. 3.2, we use an (already) pre-trained VIT-MAF model loaded from a public repository. To draw the curves shown in Fig. 6, computed the losses per each rate of masking between 5% and 95% for samples from three different test image datasets (FASHION-MNIST, CIFAR- (FFAR-100 and TINY-IMAGENET). We can observe that the curves described by the pre-trained VIT-MAE model show similar behaviour the one we obtained with BERT and they are also aligned with the analysis done in this work with tractable models. We highlight that one must be aware that masking on vision models is often performed via patches. In practice, this could affect the expectation in Proposition 1, so this point should be taken into consideration if theory is applied to this case.
VIT-XAE
TINY DNN-DDECEEET
VIT-MAAE PA SASIIN--NNISTT
V:T-MAK CIFAR100
lose
NIPT l008
MPT
To of masked tokms
%. of mashed tokens
9% of waelerd tolents
(a) TINY-IMAGENET
(b) FASHION MNIST
(c) CIFAR
Figure 6: Evolution of the area under the MPT curve. Comparison between three different datasets with VIT-MAE. The area under the curves is approximately the LML. Random initialization of the parameters produces MPT curves with similar low probabilities for all % of masking. As the number of epochs increases, the curve brings higher values of pe-probaailltyy for lower ratios of masking. The area also converges to the true value of LMI..
Reproducibility. All the empirical studies and results are reproducible. We provide the code and details for every figure in the public repository at tttts://gittube cotp///prrrreeo////PMMM/L/
4 Related work
Masked pre-training and large scalability are key elements of the current success of transformer models (Vaswani et al., 2017) on natural language processing (NI.P) tasks. Vision transformers (ViT) (Dosovitskiy et al., 2020) bridged the architectural gap between NLP and computer vision, making masked language modeling (MLM) suitable for images. In this regard, BeiT (Bao et al., 2022) adopted ViT and proposed to mask and predict discrete visual tokens. Most recently, masked autoencoders (He et al., 2022) also adopted masked pre- me--raining by predicting pixel values for each masked patch, and BEİT3 (Wang et al., 2022) performs MLM on texts, images, and image-text pairs, obtaining ofte-off-hhearr performance all-vision and vision- vision-language language tasks. Additionally, masked pre-training has been successfully adapted to video (Tong et al., 2022), where random temporal cubes iteratively masked and reconstructed.
The surprising ability of recent generative models to generalize and do impressive in-context learning has inspired earlier works to study this phenomenon from the Bayesian lens. The notion that LLMS might be performing implicit Bayesian inference was first described in Xie et al. (2021) where in-context learning is described as mixture of IIMSs. However, the equivalence between the -og-magiiaa likelihood and exhaustive cross-validatio (2020). Earlier works (Vehtari and Lampinen, 2002; Gelman et al., 2014) also provided a Bayesian perspective of Cv. Additionally, Moreno-Muñoz et al. (2022) leveraged this link for training Gaussian process models according to a stochastic approximation to the marginal likelihood. Similarly to current masked pre-training, the size of the conditioning variable (masking rate) was held constant. This was reported improve notably upon traditional variational lower bounds.
was first provided in Fong and Holmes

---

5 Discussion and outlook
In this paper, we have shown that masked pre- pre-training implicitly performs stochastic maximization of the model's marginal likelihood. The latter is generally acknowledged as being an excellent measure of model's ability to generalize (Fong and Holmes, 2020), and our results help to explain the strong empirical performance associated with masked pre-training. We have further seen that the developed theory matches the empirical training behavior well. Moreover, we illustrated the role that the rates and the number random samples of masking play in the estimation of the LML. We have also provided insights and new perspective to study masked pre-training in tractable models while also finding strong similarities with LLMS.
Limitations. We have developed a formal probabilistic theory that links masked pre-training with the Bayesian principles. While we provide evidence that the impressive performance in recent large models is related to the maximization of the marginal likelihood, these methods usually introduce new elements of improvement but may not entirely fit the propositions provided in this work. In practice, this is not a limitation but remark that there is still room for understanding the abilities of recent generative modeling. In this regard, one example might be autoregressive modeling between the masked tokens. While these are not currently analyzed in our work, we hypothesize that they could also be linked in further development to our formal propositions.
Relevance for large models using masked pre-training. We have shown empirical results of the connection between MPT and LMI.. This link sheds light the understanding of generalization, particularly in recent pre-trained models. One positive outcome of our studies is the notion of having biased Bayesian estimators whenever a practitioner fixes the masking rate, e.g. to 15%. Currently, there is a significant interest in the role of masking rates in LLMS (Wettig et al., 2022). These studies could benefit from the insights provided in this paper. We also argue that the theory offers hints that may be beneficial, for instance, for uniformly sampling the mask size, instead of the current fixed-rate practice. This practice is empirically shown in the supplementary material, and it leads unbiased estimation which may result in better performance for certain scenarios (Tay et al., 2023).
Relevance for Bayesian models. Current Bayesian modeling is dominated by approximate methods. Variational inference foregoes the ambition of training according to the marginal likelihood and instead resorts bounds thereof. This inherently yields suboptimal models. Our theory suggests that if we can design Bayesian models in which conditioning is cheap, then we can stochastically optimize w.r.t. the true marginal likelihood easily. Beyond shedding light the success of masked pre-training, the theory also suggests that large-scale Bayesian models could be successfully trained in the future with appropriately designed self-supervision.
Acknowledgements
The authors want to thank Yingzhen Li for her inspiring talk MPT during the GENU'22 meeting in Copenhagen. For us, this was the starting point of fruitful discussions that led to this final work. This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 757360). This work was in part funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF200C0066666), SH was also supported in part by a research grant (42062) from VILLUM FONDEN.
References
H. Bao, L. Dong, Piao, and F. Wei. BEIT: BERT pre-training image transformers. International Conference on Learning Representations (ICLR), 2022.
M. Caron, H. Touvron, Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEFE/CVE international conference on computer vision, pages (650-9660, 2021.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint 1r00v:1810004005, 2018.
10

---

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer. G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2rXiv:2010.111 1999, 2020.
E. Fong and C. C. Holmes. On the marginal likelihood and cross-validation. Biometrika, 107(2):489 496, 2020.
A. Gelman, J. Hwang, and A. Vehtari. Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):97--0016, 2014.
K. He, X. Chen, S. Xie, Li, P. Dollár, and R. Girshick. Masked autoencoders scalahle vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv: 2013.
N. Lawrenee. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Leurning Rexearch (JMLR), 6(11), 2005.
J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi. Don't blame the ELBO! A linear VAE perspective on posterior collapse. Advances in Neurol Information Processing Systems (NeurIPS), 32, 2019.
D. J. MacKay, Information theory, inference and learning algorithmss Cambridge University Press, 2003.
P. Moreno-Muñoz, C. W. Feldager, and S. Hauberg. Revisiting active sets for gaussian process decoders. In Advonces in Neural Information Processing Systems (NeurIPS), 2022.
B. Neyshabur, Z. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro. The role of over parametrization in generalization of neural networks. In International Conference Learning Representations (ICLR), 2019.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), pages 1278-1286, 2014.
Y. Tay. M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang. H. W. Chung. D. Bahri, T. Schuster. S. Zheng, D. Zhou, N. Houlsby, and D. Metzler. UL2: Unifying language learning paradigms. In International Conference on Learning Representations (ICLR), 2023.
J. B. Tenenbaum and T. L. Griffiths. Generalization, similarity, and Bayesian inference. Behavioral and Brain sciences, 24(4):629 640, 2001.
M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Sociery: Series (Statistical Methodology), 61(3):611-62) 1999.
Z. Tong, Y. Song. J. Wang, and L. Wang. Videomae: Masked autoencoders data-efficient learners for self supervised video pre-training. arXiv preprint arXiv:2203. 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention all you need. Advances Neural Information Processing Systems (NIPS), 30, 2017.
A. Vehtari and J. Lampinen. Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation, 14(10):2439-25 -2468 2002.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. International Conference on Learning Representations (ICLR), 2019.
W. Wang, H. Bao, L. Dong. J. Bjorek, Peng. Q. Liu, Aggarwal, O. K. Mohammed, S. Singhal, Som. ct al. Image as a forcign language: BEIT pretraining for vision and vision Mainuala tasks. arXiv preprint arXiv:2208, 10442, 2022.
A. Wettig, T. Gao, Z. Zhong, and D. Chen. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.0R8. 008005 2022.
M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian 2021.
inference, arXiv preprint


Reviews:
[{"review_id": "P79tqy3Ky3", "reviewer": "Reviewer_EYbT", "summary": "This paper proposes viewing masked pretraining methods (like BERT or MAE) as optimizing a biased form of the log marginal likelihood $\\log p_\\theta(x)$. Using a parallel to exhaustive leave-M-out cross validation, the authors derive the exact relationship between the LML and a weighted sum of masked pretraining losses across mask sizes. The authors show that maximizing the masked pretraining loss improves the LML, across a wide variety of settings, from probabilistic PCA to VAEs to BERT. ", "questions": "**Major questions**: \n- If maximizing LML is the ultimate reason why MPT is good, then why are other methods of directly maximizing LML (e.g., autoregressive models) worse?\n- Can you run the same BERT experiments with pretrained MAE models as well, to show applicability to vision models?\n\n**Minor questions and suggestions**:\n- Where did this \"log-marginal likelihood\" name come from? It looks just like the $\\log p_\\theta(x)$ that all generative models maximize, so I'm confused about the naming. \n- L19-26: This paragraph doesn't really convince me that the marginal likelihood is that important. \n- L27-28: \"masked pre-training optimizes according to a stochastic gradient of the log-marginal likelihood\" -- I think this sentence should be removed, since the following sentence states the contribution more clearly. \n- L33: \"who shows\" -> \"who show\"\n- L74: \"which might also include likelihood\" -- what is likelihood?\n- L86: \"previous sum\" -- is this referring to Eq. 2?\n- L87-92: I understand what it's trying to say after reading the entire paper, but this paragraph is very confusing.  What do exchangeable and invariant mean?\n- The block of text between L108-109: \"to be aligned\" -- this is also confusing. It's explained much better in the appendix!\n  - Generally, Appendix A explains this proof much better than the main text. \n- L128: \"we usually have a biased estimator\" -- wouldn't a Monte Carlo estimator, with masks drawn from the right distribution, be unbiased?\n- L157: \"just one random mask per training epoch\" -- my understanding is that masks are different for each sample in each epoch. I'm reading this as using the same random mask for all samples in a given epoch. \n  - L165: \"notice that P=1 is usually set up in MPT in practice\" -- same issue as above. What exactly is being claimed about the number of masks?\n- L170: \"this bias is known\" -- known in what sense? The numerical value of the bias, the symbolic form of the bias, or just the fact that the MPT objective is not equal to the LML objective?\n- L183: \"the exact LML is iteratively maximized at each epoch\" -- I wouldn't use the word \"maximize\" here, as it seems like it increases but is not guaranteed to even reach a local minimum unless there are further assumptions on the structure of the model. \n  - Same issue with L204-205\n- Fig. 2 caption: \"The rate of masking is unfixed and it varies from 1% to 100%\" -- I may be misunderstanding here, but with 10 tokens, shouldn't the masking rate start at 10%?\n- L194-195: probabilities imply integrating out the latent variables, often given the posterior distribution. In most cases, performing this integration is extremely difficult or not possible in training time.\" -- why do there need to be latent variables at all? Latent variables are one way to induce a structured joint distribution over all dimensions of x, but this doesn't have to be used. In fact, latent variables aren't used in the proof of Prop 1. \n- L202: \"as well as we compare\" -> \"as well as compare\"\n- L209-210: \"we used iterative encoding-decoding to produce the self-conditional probabilities for masked tokens — see Sec. F in Rezende et al. (2014)\" -- it would be good to explain this procedure here. I glanced at Sec F and it seems like it only gives a way to sample from the marginal over masked tokens, not a way to compute the probability? I may be misunderstanding, which is why it would be nice to have a clear explanation in this paragraph.\n- L212-214 on implicit integration: these lines are confusing and would benefit from being rewritten.\n- Fig. 3 caption: \"Darker lines indicate the evolution of the LML\" -- the figure legend does not match this. Also, i don't understand why the lower right plot is separate from the lower left plot, or why the value is so different. \n- Fig. 4: \"looses\" -> \"loses.\" Also, I don't understand what the black lines in Fig. 4 (right) is supposed to denote. \n- L218-219: \"This explains why the probabilities have an approximately exponential decay\" -- I don't see how anything explains this.\n- L224: \"effect to\" -> \"affect\"\n- L225: \"a longer discussion is provided in the supplementary material\" -- Appendix C.2 doesn't seem very relevant to this section. \n- Table 2 caption: \"area under the MPT curve\" -- is this weighted by the number of masks or not? Wouldn't the weighted sum based on Prop. 1 directly measure LML here?\n- Fig. 5: y axis label is cut off on the far left plot. \n\n\n\n", "limitations": "A major limitation that isn't addressed by the authors is this: why is MPT the right way to improve LML, instead of another method that directly maximizes LML? Empirically, masked pretraining methods like BERT/MAE are much better representation learners than alternate approaches that optimize the LML. The authors mention \"autoregressive modeling\" in their limitations section, but I don't think enough weight is put on it as a large question that this paper on MPT<-> LML introduces yet doesn't answer. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- I found Prop. 1, the connection between LML and the weighted sum of masked pretraining losses, to be original and quite enlightening. \n- Paper is thorough and provides extensive empirical experiments to confirm the theory. \n- Finding seems like a good step towards understanding masked pretraining objectives, which are extremely widespread in both NLP (BERT) and vision (MAE), and potentially understanding how to improve them. \n- Paper is well written and clear, especially if the appendix is used to supplement the derivations in Section 3. ", "weaknesses": "- The biggest issue is that the paper shows that masked pretraining performance improves LML, but does not address why it does better than other training objectives that directly maximize LML (such as an autoregressive model).\n- No experiments on the vision side, even though MAE is an extremely popular method that falls neatly into the framework. \n- Some parts of the paper are a tiny bit confusing. Surprisingly, I felt that Appendix A and B could be swapped into the main paper and it would significantly improve the clarity of Proposition 1's derivation or the PPCA setting. \n  - I felt that the second paragraph of the intro (L19-26) didn't sufficiently motivate why the LML is important. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive consideration and the useful feedback on our work. We are particularly glad for the scores marked as *excellent* and for hearing that the connection between LML and the weighted sum of masked pretraining losses is considered *original* and *enlightening*. Overall, we appreciate the time you clearly took to review our paper, including the key questions and the rest of minor details attached. We've done our best to fit in this rebuttal clear answers to the principal points of your review. Additionally, we have performed extra experiments on the vision side with MAE to address one of the reviewer's main concerns.\n\n**Point 1A** & **1B**\n> *If maximizing LML is the ultimate reason why MPT is good, then why are other methods of directly maximizing LML (e.g., autoregressive models) worse?*\n\n> *why is MPT the right way to improve LML, instead of another method that directly maximizes LML? (..)*\n\nOur ambition is not to say that maximising the LML is the **ultimate** reason why MPT works so well. Perhaps we need to revise our communication if a final update of the paper is allowed.\n\nInstead, we are strictly providing a positive answer to the question: *Is MPT related to the maximisation of LML?* The complete answer included in the paper is that MPT optimises according to a stochastic gradient of the LML. As the latter is a well-known measure of generalization ability in Bayesian models, we believe that the connection provides a new tool for analysing MPT. However, we do not claim it to be the unique reason, as MPT is a learning algorithm that might be combined with different models. \n\nIn the particular case of LLMs, we should not ignore the role that the chosen architecture for the model plays. One additional example that supports this point is the work of (Neyshabur, ICLR 2019), which says the following: *Our capacity bound correlates with the behavior of test error with increasing network sizes (...), and could partly explain the improvement in generalization with over-parametrization*. Even in this example, the explanation is assumed to be *partial*, and our perspective is similar in this way.\n\nWith respect to the *direct maximisation of LML*, we cannot claim that it is generally *worse* than MPT. However, we have observed that MPT maximises according to a stochastic approximation to the training objective. From a practical perspective, this gives advantages with respect to the direct maximisation of the objective, where the lower computational cost excels and the stochastic optimization might be beneficial to avoid local minima.\n\n**Point 2**\n> *Can you run the same BERT experiments with pretrained MAE models as well, (..)*\n\nYes. In particular, we reproduced the same empirical results obtained in **Section 3.2** with BERT, but for the MAE approach instead. We considered the exact model included in He et al., CVPR 2022. The results are shown in the additional PDF attached to the global response.\n\n```\nK. He, X. Chen, S. Xie, Y. Li, P. Dollar and R. Girshick, Masked Autoencoders Are Scalable Vision Learners, CVPR 2022\n```\n\n**Point 3**\n> *The authors mention \"autoregressive modeling\" (..) but I don't think enough weight is put on it (..).*\n\nFair point. We understand the interest around *autoregressive modelling* and its connection with MPT. However, we do think that this topic slightly falls out of the scope of our work, as our main goal was to build the link between LML and MPT in a thorough manner. \n\nImportantly, autoregressive modelling is mentioned in the discussion section to point out that there is still room for understanding the link with LML beyond the simplest version of MPT. This is not really a limitation, but rather potential future work. Our early hypothesis is that the theory proposed could accept conditioning between the tokens to-be-predicted. This could make the pre-training algorithm less of an approximation to the true LML. However, further analysis and theoretical development should be needed to prove this crucial point.\n\n**Point 4.1**\n> *Where did this \"log-marginal likelihood\" name come from? (...).*\n\nThe *marginal likelihood* name has been longly used in the Bayesian ML for more than 25 years, at least. In his seminal works around 1998 with Bayesian NNs and Laplace approximations, David JC MacKay already mentions marginal likelihood as a method for model comparison also providing even earlier references. A common synonym for marginal likelihood is *evidence*, which also gives name to the acronym ELBO (Evidence Lower BOund) widely used in variational inference and VAEs, for instance. This probability is indeed the one being bounded by the ELBO.\n\n```\nD. JC MacKay. Choice of Basis for Laplace Approximations. Machine Learning, volume 33, pp. 77–86, Springer, 1998\n```\n\n**Additional comment:** We would've liked to include the rest of answers to the questions in the minor comments and suggestions (those marked as point 4.1, 4.2, etc..). Unfortunately, we did not have enough space in this rebuttal box. As soon as the discussion phase will begin, we will include the mentioned answers in an additional comment for the reviewer. "}, {"review_id": "VKCPdaaK2P", "reviewer": "Reviewer_mVT4", "summary": "They show empirically that randomly masking a fixed number of M tokens and predicting with the remaining tokens produces a biased estimate of the log marginal likelihood $\\log p(x)$ (LML).\nFurthermore, they prove that repeating the masking for M from 1 to D (the total number of tokens), and summing up the estimates, leads to an unbiased estimate of the LML.", "questions": "- Could you comment on my main concern above?\n- Why does the LML change with the number of epochs?\n- What is GT-LML?", "limitations": "see main concern.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and mostly easy to follow.\n- Intuitive proofs and arguments are well supported by empirical results for tractable and intractable models.\n- The work might give insights/ideas for setting masking rates in pre-training.\n", "weaknesses": "Main Concern:\nThe main result Proposition 1, requires to condition on different subsets of the remaining the tokens $R$. But this is not what is done in practice for masked pre-training, where normally all tokens in $R$ are used. It would have been interesting to see some discussion/experiments about the effect.\n\nOthers:\n- It would have been interesting to see some experiments about the ideas stated in line 295.\n- Given the works like Fong and Holmes (2020), there is no so much novelty from a theoretical point of view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the acknowledgement of the main contributions of our work, the useful comments and the relevant feedback provided on the technical side. We have addressed _all_ your comments individually in the lines below.  If there is still something unclear, we would be also happy to clarify with extra responses.\n\n**Point 1**\n> *The main result Proposition 1, requires to condition on different subsets of the remaining the tokens $R$. But this is not what is done in practice for masked pre-training, where normally all tokens in $R$ are used. It would have been interesting to see some discussion/experiments about the effect.*\n\nWe understand that from a first sight, the main result provided in the l.h.s. of **Proposition 1** seems to indicate that we are conditioning on different subsets of remaining tokens $\\boldsymbol{x}_{\\mathcal{R}}$. As the reviewer indicates, this would not fit with MPT, that in practice uses all remaining tokens $\\mathcal{R}$. \n\nHowever, the last expression in **Proposition 1** indicates a slight different thing. Particularly, the variable $x_{\\mathcal{R}(1:D-j)}$ does not indicate a selection of subsets but the number of the remaining tokens instead. This is due to the sum in this proof considers different ratios between the masking size $M$ and the rest of tokens $R$. (See the sum which goes from $j=1$ until $M$). In the end, this corresponds to averaging over the number of the masked tokens $M$, where $R$ is always set up to the rest of items accordingly. \n\nPerhaps this point can be easily observed in the expression in the r.h.s. of the previous equation where we condition on all the remaining tokens in $\\boldsymbol{x}_{\\mathcal{R}}$. This one is also the one that matches Equation (1), building the final link with MPT. We apologise for this point that might lead to a little confusion. We would be happy to update this detail, also keeping the notation uncluttered, if the paper moves forward.\n\n**Point 2**\n> *It would have been interesting to see some experiments about the ideas stated in line 295.*\n\nThanks for pointing this out. We have now included new experiments that perform what is described in L295. In particular, we  uniformly sample the number of masked tokens to obtain an unbiased estimate of the LML with the MPT losses. This generates what the theory indicates, the MPT losses are now centered with respect to the LML and they also make it converge to the true LML. Additionally, we also provide two extra Figures where we sample uniformly between the 0--50% and 50--100%. Obviously, as it is not sampling the entire range of the mask size, provides again biased estimates as we could have guessed. \n\nAn important detail is that we initially discarded the uniform sampling due to it might become prohibitive on the computational cost. This could happen if the drawn masking rate is, for instance, around the 0%-5%, due to the model needs to recursively use all tokens at each iteration. Of course, this depends on the type of model chosen. But this is just for clarifying all details for the reviewer.\n\nThe figures can be accessed in the additional PDF page attached to the general rebuttal response (to all reviewers).\n\n**Point 3**\n> *Given the works like Fong and Holmes (2020), there is no so much novelty from a theoretical point of view.*\n\nPerhaps, we have not correctly highlighted some the main theoretical differences with respect to the work of Fong and Holmes (2020) in our paper. Thus, to address the concern of the reviewer, we remark some important details that might help to understand better why the theoretical ideas brought in Fong and Holmes (2020) are significantly different from the ones shown in our submission.\n\nThe work of *Fong and Holmes (2020)* introduced a first connection between leave-p-out *cross-validation* (CV) and the LML in such cases where CV uses posterior predictive probabilities as the scoring function. The equivalence is theoretically analysed from a Bayesian statistical perspective, where the main focus is placed on exhaustive leave-p-out CV and the prohibitive computational cost.\n\nHonestly, *Fong and Holmes (2020)* is a relevant reference because it inspired the probabilistic ML community to take a further look for understanding how LML is linked to many probabilistic settings. However, they successfully did it only for cross validation. In this way, we have found, proved and provided empirical results on the equivalence between MPT and LML for different models. This achievement was initially inspired on recent work like Chen et al. ICLR 2023, where Gaussian processes (GPs) are introduced in the architecture of transformers. \n\n```\nW. Chen and Y. Li, Calibrating Transformers via Sparse Gaussian Processes. ICLR 2023\n```\n\n**Point 4**\n> *Why does the LML change with the number of epochs?*\n\nIf we guess correctly, the reviewer makes reference to the maximisation of the LML in the experiments and particularly, on the curves drawn in Figure 2. This plot shows three curves: the LML of the true model which generated the data, the cumulative MPT loss under analysis and the LML. The latter corresponds to the log-probability in Equation (6) for the PPCA model. In this experiment, we also maximise this expression of the LML to understand how it behaves compared with the MPT loss. Thus, the LML changes with the number of epochs as we are optimising wrt. the parameters $\\boldsymbol{W}$ in L138 and L141. Interestingly, notice that both the LML and the MPT losses converge to the true value of the LML in the left hand side plot.\n\n**Point 5**\n> *What is GT-LML?*\n\nGood catch. GT-LML makes reference to *ground truth LML* in the caption of Figure 2, but we used the sentence \"true model LML\" in the label of the plots for the dashed line. It is a typo that will be corrected in the main paper.\n"}, {"review_id": "MfHeXYijfF", "reviewer": "Reviewer_Gnha", "summary": "This paper shows that MLM is effectively maximizing the model's marginal likelihood, perhaps explaining why MLM has been successful. Beyond providing a proof, they run several empirical experiments suggesting their results hold in practice.", "questions": "On line 207 (and beyond) you make some distinction between linear vs non-linear models. Does your proof make assumptions along these lines? If so, can you bring these assumptions further up into the abstract?", "limitations": "\"The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, authors are encouraged to provide a short proof sketch to provide intuition.\" More of a proof sketch might help the reader understand the footing of the paper.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Clear introduction; Clear highlighting of paper strengths.\n2. Clear background on MLM.\n3. They provide a proof in the appendix and a number of empirical tests that support their findings in practice.\n4. Their discussion showing connections both to model design in AI as well as relevance to improving Bayesian Models seems well put together.\n\nI don't deeply understand the proof and math, so am I unable to fully judge the strength of the contribution. For what I do appreciate, I would recommend the paper to be accepted. I found the paper to be interesting, overall well-presented, and noteworthy.", "weaknesses": "1. Providing more hand-holding between the various subsections of section 3 would help the reader understand your goals for each section. Right now I can see the specific finding, but: Stressing the greater point that you're trying to verify your proof in practice (?) in each of these subsections and where and how you find support would be great.\n2. Providing slightly more hand-holding on what marginal likelihood is and what it would be capturing in the context of, say, BERT, would make the paper stronger.\n\nNits\n1. Figure 1: The y-axis get smaller (key detail!) but I could not see this initially. The text was very light. Perhaps increasing the font size somehow or making more of a note of it in the caption.\n2. Figure 5: y-axis is both small and cut off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive consideration and the useful feedback on our work. We also appreciate the highlighted strengths and contributions. We addressed the main comments and concerns included in your review. If any other concern remains, we would be also happy to clarify during the discussion period.\n\n**Point 1 & 2**\n> *Providing more hand-holding between the various subsections of section 3 would help (...)*\n\n> *Providing slightly more hand-holding on what marginal likelihood is and what it would be capturing in the context of, say, BERT, would make the paper stronger.*\n\nWe appreciate the insightful feedback for making the paper *clearer* and *stronger*. At the beginning, our main concern was on the structure of the paper. Mainly, we had to choose between the two views described in the end of **Section 5**. That is, writing the paper to highlight the relevance for Bayesian ML practitioners (who might not be aware of the role that MPT plays), or for the audience who knows MPT very well and would like to use the proposed link for a better understanding. Perhaps, we implicitly chose the first view, whose main consequence are the two points mentioned by the reviewer. We will update it accordingly to your comments if the paper goes on for a final version.\n\n**Point 3**\n> *On line 207 (and beyond) you make some distinction between linear vs non-linear models. Does your proof make assumptions along these lines? (...)*\n\nFair point. The paper is organised in a way where we first focus on the connection between masked pre-training (MPT) and log-marginal likelihood (LML), independently of the type of probabilistic model chosen (e.g. linear as PPCA or non-linear as VAEs). Importantly, the proof we provide **does not make any assumption** along the lines on the linear and non-linear nature of the probabilistic model. This is an important point that we would like to emphasize.\n\nOnce we provide the main theoretical results for the connection between MPT and LML, we jumped into the second part, where our interest was providing new empirical results to the previous theory. Since for linear models the calculus of the LML is generally **exact**, we used PPCA as the first probabilistic model for verification. This is inspired by Lucas et al., NeurIPS 2019, who also used PPCA for analysing the behavior of VAEs and particularly, the *posterior collapse* effect. \n\nThe sentence on L207 makes reference to the applicability of the theory (i.e. connection between MPT and LML) beyond tractable linear models, that is, non-linear models. We address this point in the following sections with empirical results with VAEs and LLMs (i.e. BERT).\n\n```\nJ. Lucas, G. Tucker, R. Grosse and M. Norouzi. *Don’t Blame the ELBO! A Linear VAE Perspective on Posterior Collapse*. NeurIPS 2019\n```\n\n**Point 4**\n> *\"The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, authors are encouraged to provide a short proof sketch to provide intuition.\" More of a proof sketch might help the reader understand the footing of the paper.*\n\nWe appreciate your recommendation. Our first idea was to include the main proof in the paper, however, we chose an intermediate solution which can be checked between L103-L130, including the **Proposition 1**. We shaped this as a proof sketch, but considering your feedback, we would be happy to restructure this part and include extra details if the paper moves forward. "}, {"review_id": "7pyndB4Bus", "reviewer": "Reviewer_LKWQ", "summary": "This paper derives the equivalence between log marginal likelihood and negative masked pre-training loss which is often used for training text models. The paper argues that as marginal likelihood is the Bayesian way to do model selection, doing training using the masked pre-training loss inherits good generalization properties.\n\nExperiments are done on a probabilistic PCA model, VAE and BERT. In PPCA, where the ground truth is known and the equivalence is confirmed empirically. On this model, it is empirically confirmed that even when we optimize a biased estimation of the log marginal likelihood, different training runs still converge to the same log marginal likelihood, with a slight offset to the true value.\n\nIn VAE, we don't know the ground truth, but verify that optimizing using the masked pretraining loss is similar to optimizing with ELBO, in terms of the log marginal likelihood value that the model converges to.\n\nIn the BERT model, we simply observe that the shape of the S(x, M) term is similar for PPCA and BERT during different points of training.", "questions": "- In equation (1), do we not condition on $\\mathbf x_{\\mathcal M \\setminus \\{\\mathcal M(t)\\}}$ -- how is the marginalization over this variable done in language models?\n- Is there a reason why we condition on $\\mathbf x_{t + 1:D}$ instead of $\\mathbf x_{1:t}$?\n- In equation before equation 4, we assume $M < D$, so the value $\\mathbf x_D$ which is part of $\\mathbf x_{\\mathcal M(t+ 1:D)}$ in the left hand side doesn't exist\n- In line 109, shouldn't it be ${D \\choose t}$ instead of ${D \\choose t - 1}$ since there are $t$ tokens we condition on?\n", "limitations": "-", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is an important connection to make and the authors discuss relevance for LLMs and Bayesian learning in the discussion.\n- The experiments study the realistic case in which we can't enumerate over all masks and verify that masked pre-training is still similar to log marginal likelihood training", "weaknesses": "- The derivations are likely correct but I couldn't establish this 100%, see questions below.\n- Isn't the Transformer just trained with the autoregressive log likelihood (and not with masked pre-training)? And isn't ViT simply trained to predict object class -- where is the masking?\n- The jump to Proposition 1 was big for me and the preceding intuition building parts didn't help", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive consideration of our work, and the useful feedback including the technical questions on the derivations. We are interested in addressing all the questions brought in the review for a full understanding of the reviewer. If there is still something unclear, we would be happy to provide extra responses.\n\n**Point 1**\n> *Isn't the Transformer just trained with the autoregressive log likelihood (and not with masked pre-training)? And isn't ViT simply trained to predict object class -- where is the masking?* \n\nFair point. As the reviewer indicates, Transformers and ViT architectures themselves don't use masked pre-training (MPT). But we do not mention Transformer or ViT architectures as the models considered in our analysis of MPT. Neither on the related work section. However, BERT and MAE models do use Transformers and ViT as the backbone method respectively while, at the same time, they use MPT to generate the self-supervisory signals. Precisely, this pre-training methodology is the point were we focus our theoretical analysis and connection with LML.\n\nRegarding ViT and vision models, we have now included new empirical results with the ViT-MAE model. These are shown in the PDF attached in the global response to all reviewers.\n\n**Point 2**\n> *In equation (1), do we not condition on $x_{\\mathcal{M}/\\mathcal{M}(t)}$ -- how is the marginalisation over this variable done in language models?*\n\nAssuming you refer to the masked tokens $x_{\\mathcal{M}(t)}$, we do not condition on them as they are the ones we want to predict. These ones are indeed the self-supervisory signals that makes the model to learn a holistic structure of the data. Notice that we are recurrently conditioning on the rest of tokens $x_{\\mathcal{R}}$. Importantly, the observed data is never marginalised and the log-marginal likelihood (LML) builds a probability metric over all the tokens.\n\n**Point 3**\n> *Is there a reason why we condition on $x_{t+1:D}$ instead of $x_{t+1}$?*\n\nIf we understand correctly, the reviewer is referring to Equation (2), were we apply properties of conditional probabilities to factorise the $\\log p_{\\theta}(x)$ distribution or LML. We highlight  that in the object $x$ we have $D$ variables $x_1,x_2,\\dots, x_D$. Thus, the goal of the sum is to obtain a recursive summation of uni-variate conditional distributions, for instance, $\\log p_{\\theta}(x_1|x_2,x_3, \\dots ,x_D) + \\log p_{\\theta}(x_2|x_3, \\dots ,x_D) + \\dots$ That's why we use $x_{t+1:D}$ instead of $x_{t+1}$ in the conditioning variables. \n\n**Point 4**\n> *In equation before equation 4, we assume $M < D$, so the value $x_D$ which is part of $x_{\\mathcal{M}(t+1:D)}$ in the left hand side doesn't exist*\n\nRight, maybe this was not clear enough. The equation indicated by the reviewer says that we have multiple choices for the order of conditioning objects in the right hand side of log probabilities. Importantly, the letter $\\mathcal{M}$ indicates the indices, such that $\\mathcal{M} = \\{1,2,\\dots,D\\}$ as is described in L94. \n\nThe key point in this equation is that we are fixing the index $t$, where $1<t<D$ such that we can analyse with all the mentioned combinations for $x^{(\\pi)}_{\\mathcal{M}(t+1:D)}$  Thus, the tokens included in the list of indices $\\mathcal{M}(t+1:D)$ depend on the choice $\\pi$ and there is not really a problem with the variable $x_D$. \n\nWe highlight that the variable $M$ (size of mask) is different from the variable $\\mathcal{M}$ (indices of tokens), which could be the cause of a confusion here. We hope that this clarifies this point and we will do our best to update the section on this regard.\n\n**Point 5**\n> *In line 109, shouldn't it be $\\binom{D}{t}$ instead of $\\binom{D}{t-1}$ since there are tokens we condition on?*\n\nFair point. This critical point is perhaps a bit counterintuitive. We will update it in a way that is clearer. The key idea in L109 is that, fixing an index $t$, we have $D-t+1$ tokens to predict probabilities on (left hand side variables $x_{\\mathcal{M}(t)}$ in the conditional distribution), and D-choose-(t-1) distinct orders for the conditioning tokens $x^{(\\pi)}_{\\mathcal{M}(t+1:D)}$ (*unmasked*). That is, we represent the latter choices with the binomial coefficient $\\mathcal{C_t} = \\binom{C}{t-1}$. Extra details on these derivations are also included in the Appendix A of the paper, where all elements are placed together for a better understanding."}, {"review_id": "7MBXd9wicK", "reviewer": "Reviewer_vzMk", "summary": "The authors set out to show that the good generalization of Masked Pre-Training can be explained as the equivalence with model's high marginal likelihood. The exchangeability assumption in sequential inputs is handled via a combinatorial choice over the subset of masking features, such that the masked pre-training objectives is a marginalization over uniform distribution of different maskings. The rest of the paper is a discussion on the implications of such a formulation.", "questions": "1. Perhaps I am misunderstanding, but shouldn't Eq. (2) be conditioned on all feature variables $1:D$ except for the one being predicted? Eq. (3) seems to present is correctly, since it clearly distinguishes masking indices.", "limitations": "Yes. See my comments on weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is reasonably written, and the authors have made a decent effort to convey the significance of the work.", "weaknesses": "I am not very convinced by the arguments in the paper that claim that maximizing the marginal likelihood is the reason behind generalization of MPT.\n\nIn Line 182, the authors claim an \"insight\" that MPT targets a biased estimate of LML by showing that fixing the masking rate to 20% of  tokens, MPT becomes biased and converges to a different value. I feel like this argument could be made for any two objectives, and something remains missing in this justification. From the reverse-engineered theory, it is already obvious that MPT introduces bias and positioning this practical result as an insight seems to not add any new information. I'd appreciate a clarification here.\n\nIn Line 212, the authors claim that MPT performs implicit integration. However, this claim also seems a little unfounded. When averaging over the % of masked tokens in Figure 4, if bulk of the integrated mass is constant, then an alternate way to approximate the integral would be to just take the constant mass (since the integral contains averaging terms as well). Doesn't this result confound the conclusion and just a matter of coincidence?\n\nIn any case, marginal likelihood just answers a question very different from generalization - marginal likelihood tells us the likelihood of the data under the \"prior\" model, i.e. how well the priors explain the data. It does not have anything to do with what happens to the model after training, in principle. There could be confounders to the \"insights\" claimed by the authors here, especially with large models, since the choice of number of tokens, number of maskings, are far-off from what model large language models use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback and the time taken to review our paper. We have addressed all your comments individually below. If any concerns remain, we would be also happy to clarify.\n\n**Point 1** & **Point 2**\n> *I am not very convinced by the arguments in the paper that claim that maximizing the marginal likelihood is the reason behind generalization of MPT.*\n\n> *In any case, marginal likelihood just answers a question very different from generalization - (...)* \n\nWe would like to remark that this is not the exact message that we want to send in our paper. Precisely, we are interested in the link between MPT and the log-marginal likelihood. Thus, the main discovery that we show is that log-marginal likelihood is equivalent to the cumulative sum of MPT losses shaped with different sizes of the mask. The final conclusion of our work is that MPT implicitly performs stochastic maximisation of the model's marginal likelihood. That is what we prove in a rigorous manner. Additionally, these points are confirmed with the empirical results provided on different tractable and intractable models.\n\nThe fact that MPT implicitly maximises the model's marginal likelihood is important. Indeed, the marginal likelihood (or evidence) has been longly considered as the measure of generalization ability in Bayesian models. As we are sure the reviewer knows, LML is usually the desired loss that one would like to obtain in Bayesian modelling, with a large list of examples where Laplace approximations for Bayesian NNs recently excell. However, this one is usually difficult to compute due to the prohibitive computational cost or intractability in the integrals. \n\nHence, having this connection is perhaps not the final answer to explain why MPT works so well in the recent advances, but we believe that the empirical evidence is a step forward to explain at least one of the causes why pre-trained models generalize so well. \n\n**Point 3**\n> *In Line 182, the authors claim an \"insight\" that MPT targets a biased estimate of LML by showing that fixing the masking rate to 20% of tokens, (...).*\n\nFair point. So far, before the section that begins in L175, we have proven and provided empirical results which indicate that the cumulative MPT losses with a masking rate from 1% to 100% equals the LML of the model. However, we know that this is not what MPT does in practice, where the masking rate is usually fixed. \n\nWe were particularly interested in the understanding the effect of fixing the masking rate. Perhaps, the main surprise is not having a biased estimate of the LML probabilities, but that the biased is **fixed** with respect to the learnable parameters. This is not a common thing, and having biased estimation is a usually a huge problem. However, in the results that we provide this bias is not negatively affecting the maximisation of the LML as it is indicated in Figures 2,3 and 5.\n\nOn the other hand, we were interested in understanding the meaning of setting a fixed masking rate of  20, 30, 40 or even 50%. For that reason, we included additional empirical results in Figure 4, where the $x$-axis is the % of masked tokens. Notice that the area under the curve is the LML, which is maximised as the MPT losses decrease.\n\n**Point 4**\n> *In Line 212, the authors claim that MPT performs implicit integration. However, this claim also seems a little unfounded. (...)*\n\nWe understand the concern here, but we do not think that the claim on *implicit integration* is *unfounded*. We remark that the **expectation integral** plays a key role in the main equation in Proposition 1. Moreover, as the reviewer correctly points out, we focus on the area under the curve described by the MPT losses with respect to the % of masking rate. These curves can be seen in Figures 4 and 5. \n\nThe area under the curve is not necessarily constant, and fixing the masking rate basically sets an approximation based on the constant mass. Depending on which decisions are taken, the integration of the area is performed way with one approximation or another. But importantly, none of this is done on purpose, but *implicitly* when we average losses in MPT. We do not think that all of this is just a **coincidence**, as theory goes first and empirical results later. In that sense, we encourage the reviewer to check the left-hand plot in Figure 5, where the area under the curve converges to the true value of the model's LML.\n\n**Point 5**\n> *Perhaps I am misunderstanding, but shouldn't Eq. (2) be conditioned on all feature variables $1:D$ except for the one being predicted? (...)*\n\nIn particular, Equation (2) uses rules of probability (i.e. chain rule) to make the factorisation of $\\log p_{\\theta}(\\boldsymbol{x})$. Notice that the object $\\boldsymbol{x}$ included all the observed variables $x_1, x_2, x_3, \\dots, x_D$. In this way, the density $\\log p_{\\theta}(\\boldsymbol{x})$ works as a joint distribution $\\log p_{\\theta}(x_1, x_2, \\dots, x_D)$. If we apply one time the chain rule of conditional probability to this joint distribution, we easily get the following factorisation\n\n$$\\log p_{\\theta}(x_1, x_2, \\dots, x_D) = \\log p_{\\theta}(x_1| x_2, \\dots, x_D) +\\log p_{\\theta}(x_2, \\dots, x_D).$$\n\nDoing it again on the right hand side log-density $\\log p_{\\theta}(x_2, \\dots, x_D)$ gives us again an extra conditional term to the sum, such that\n\n$$\\log p_{\\theta}(x_1, x_2, \\dots, x_D) = \\log p_{\\theta}(x_1| x_2, \\dots, x_D) +\\log p_{\\theta}(x_2 | x_3, \\dots, x_D) + \\log p_{\\theta}(x_3, \\dots, x_D).$$\n\nIf we recursively apply this property, we obtain the sum expressed in Eq. (2), which does not requires to have distributions conditioned on all the feature variables $1:D$. We hope that this clarifies a bit this point, as it is a key property in the rest of derivations used for the proof."}]

Author rebuttal:
**General comment to all reviewers**

We thank all reviewers for their useful comments, positive consideration and relevant feedback on our paper. It seems that the reviews are positive in general and acknowledges our main contributions and soundness of our work. We have addressed each comment and question individually below and we would be glad to engage in discussion in case of more questions or concerns exist. We also provided an extra PDF with the new results from the experiments that some of the reviewers asked for. We will also update the main paper in the future with the main requested changes for improvement, if the submissions moves forward.

Moreover, as some reviewers already did in their comments, we use the acronyms MPT (masked pre-training) and LML (log-marginal likelihood) in our response.

**Note on the additional experiments**

Additionally to the main response in this platform, we provide the results for the suggested experiments in the attached PDF. In particular, the plots included in Fig. 1 and the response **E.1** addresses the **Point 2** of the reviewer **mVT4**. Additionally, the results shown in Fig. 2 and the section **E.2** correspond to the **Point 2** of the reviewer **EYbT**.

**E.1.  Experiments on uniform masking rate sampling**

These experiments answer the question around the effect of uniformly sampling the masking ratio with MPT losses. So far, we have observed that the cumulative MPT loss is equivalent to an \*unbiased* estimate of the log-marginal likelihood (LML) when we consider all possible numbers for the amount of masked tokens. To avoid having a *biased* estimate when fixing the masking rate (e.g. to 20\%), one option is to use an uniform distribution. In this way, we sampled the rate of masking at each epoch in the range ($0\%-100\%$). The results in the (A) plot of Fig. 1 indicate that we are able to obtain such unbiased target loss. Importantly, we should notice that the cumulative losses oscillate around the true value of the LML that is also maximised. For completeness of the experiments, we also included the empirical results when the masking rate is sampled in different ranges (i.e. $50\%-100\%$). In this case, we have different biases in the plots (B) and (C) in the Fig. 1. This biases are related to the area under the curves described in Fig. 4 of the main paper.

**E.2. Experiments on vision.**

Additionally to the results included in Fig. 5 of the main paper with the BERT model, we now provide the curves produced in our experiments with the ViT-MAE model. For this study with vision self-supervised learners, we have loaded a pre-trained model of ViT-MAE from a public repository. To draw the curves in Fig. 2, we computed the losses per each rate of masking between 5% and 95% for samples from three different test image datasets. We can observe that the curves described by the pre-trained ViT-MAE model shows a similar behaviour to the one we obtained with  BERT. These curves are also aligned with the *canonical* analysis performed with the tractable models. On the hand, the curves described by ViT-MAE with randomly initialised parameters are *flat* and of low self-predictive probability. 

Meta-review: The paper received mixed reviews leaning toward acceptance (1 accept, 2 weak accepts, and 2 borderline rejects) after the rebuttal period. The AC checked the reviews and discussions, and believes the reviews have covered most aspects of the paper -- some checked the proofs in detail, others relied more on intuitive understandings. Overall, while the paper has some claims and descriptions that are not worded ideally, it establishes an interesting connection between masked pre-training and the maximization of log-marginal likelihood. This is by itself a valuable theoretical contribution, and the authors also did additional experiments during the rebuttal to enhance the empirical justifications of the work. Therefore recommending acceptance -- the authors are expected to incorporate useful feedbacks from the reviews for the final version.
Decision: Accept (poster)