Big Learning: A Universal Machine Learning Paradigm?
Authors: Yulai Cong
Miaoyun Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Foundation models
big learning
incomplete data
GAN

Abstract: Recent breakthroughs based on big/foundation models reveal a vague avenue for AI, that is, \emph{big data, big/foundation models, big learning, $\cdots$}. Following that avenue, here we elaborate on our newly introduced big learning. Specifically, big learning exhaustively exploits the information/tasks inherent in its large-scale \emph{complete/incomplete} training data, by learning to simultaneously model many/all joint/conditional/marginal data distributions (thus named big learning) with one universal foundation model. We reveal that big learning is what existing foundation models are implicitly doing; accordingly, our big learning provides high-level guidance for flexible design and improvements of foundation models. Besides, big learning ($i$) is equipped with great flexibilities for complete/incomplete training data and for customizing trustworthy data tasks; ($ii$) potentially delivers all joint/conditional/marginal data capabilities after training; ($iii$) significantly reduces the training-test gap with improved model generalization; and ($iv$) potentially unifies conventional machine learning paradigms and enables their flexible cooperations, manifested as a universal learning paradigm. Preliminary experiments verified the effectiveness of the presented big learning. 


Under review as a conference paper at ICLR 2023
BIG LEARNING: A UNIVERSAL MACHINE LEARNING PARADIGM?
Anonymous authors Paper under double-blind review
ABSTRACT
Recent breakthroughs based on big/foundation models reveal vague avenue for AI, that is. big data, big/foundation models, big learning. ... Following that avenue, here we elaborate on our newly introduced big learning. Specifically, big learning exhaustively exploits the information/tasks inherent in its large-scale completefincomplete training data, by learning to simultaneously model many/all join//conditionalmarriiii data distributions (thus named big learning) with one universal foundation model. We reveal that big learning is what existing foundation models are implicitly doing; accordingly, our big learning provides high-level guidance for flexible design and improvements of foundation models. Besides, big learning (i) is equipped with great flexibilities for complete/incomplete train- ing data and for customizing trustworthy data tasks; (ii) potentially delivers all joint/conditiona//aarinaa data capabilities after training; (iıi) significantly reduces the training-test gap with improved model generalization; and (iv) potentially unifies conventional machine learning paradigms and enables their flexible coop- erations, manifested as a universal learning paradigm. Preliminary experiments verified the effectiveness of the presented big learning.
INTRODUCTION
AI is undergoing a paradigm shift with the rise of big/foundation models (Bommasani al. 2021 Yuan et al., 2022), ) e.g., BERT (Stickland & Murray 2019), GPT-3 (Brown et al.] (020). LALLLESS (Ramesh et al.) 2022 2022). MAE (He et al.] 2021). etc. Foundation models, often based on mask- and-predict pretraining and downstream finetuning, are capable of benefiting from pretraining on broad data scale and accordingly, demonstrate diverse downstream task capabilities with impressive robustness (Stickland & Murray| 2019), adaptability (He et al.) (021). and generalization (Ramesh) al. 2021). Therefore, they are rapidly being integrated into real-world AI systems, e.g. BERT into Google e search, Codex (Chen et al. 2021a) into GitHub's 's Copilot, elc.
Despite the impressive capabilities and characteristics of foundation models, a unified theoretical framework justifying their successes remains missing (Bommasani et al. [2021] Yuan et al. 2022). which is crucial for their further improvements and is likely a milestone for the foundation model community (Tamkin et al. 2021).
To address that challenge, we first notice that the successes of foundation models are mainly attributed to the following two properties, in addition to increasingly powerful parallel computing techniques.
Data comprehensiveness. Foundation models are not picky about their training data and therefore embrace flexible training on massive aasily-accessible data with great diversity (e.g., those crawled from the Internet). These training data, thanks to their massiveness, diversity, and minimal human interventions in the collection, are likely more consistent with the "true" data distribution that underlies both training and test phases, leading to narrowed training-test gap from the data perspective and serving as one reason for the improved generalization and robustness of foundation models.
Task comprehensiveness. A foundation model is often pretrained in massive-task manner on a wealth of data tasks (like mask-and and---nd-d--ee sodddect),),, which can be flexibly specified as modeling some conditional data distributions across potentially diverse domains (see Section for details). Such massive-task and potentially diverse pretraining of foundation models narrows the pretraining- finetuning/training-e gap from the learning perspective, i.e., it's likely the downstream task

---

Under review as a conference paper at ICLR 2023
resembles pretraining one, and therefore contributes to the successes of foundation models.
Moreover, the massive-task pretraining may encourage learning compositional intrinsic meta-
knowledge encoded in the model parameters (Lu et al. 2021] Aghajanyan et al. 202I), which
may hold the key for out-of-distribution generalization (Bommasani et al. 2021
Based on the above observations and by reviewing the development history of deep learning, we perceive a vague avenue for AI, that is big data, big/foundation models, big learning. .... Specifi- cally, one leverages big data to comprehensively represent the underlying data distribution, develops big/foundation models to serve sively and exhaustively convey data information into that container, and so on. Accordingly. different from existing machine learning paradigms that only exploit limited information contained in training data, we present big learning for exhaustive data information exploitation, following that Al avenue.
big information "container," relies big learning to comprehen-
The presented big learning further strengthens the above-mentioned data and task compre- hensiveness by leveraging a universal foundation model to simultancously model many/all voin//conditoonalmaainnal data distributions (across potentially diverse domains), manifested as "big" training task that exhaustively exploits the data information. Such big learning behavior closely resembles the fundamental unconscious mind and the vision system of human brains, which are excellent comprehensive information exploitation in multitasking manner (Bargh & Morsella 2008 Mesquita 2015; Ludwig et al. 2014 Saarela Landy 2015).
Our big learning comes with three main contributions.
It serves as a theoretical platform for analyzing, justifying, and improving big/foundation models, because most of them are implicitly doing (parts of) big learning, as revealed in Section 3
By modeling many/all join//ondiiionalmaarinaa data distributions, big learning (i) compre- hensively exploits the available data information and embraces statistical sharing power to encourage summarizing intrinsic compositional knotl-enoweedge within model parameters and (ii) potentially delivers all joint/conditional/aargnaall data capabilities after training, which are of great value egg., for arbitrary data completion, flexible counter-factual analysis, and reasoning.
It delivers extraordinary data and training-task flexibilities by enabling large-scale training with complete/incomplete data diverse learning tasks across different domains. leading to (i) minimal human interventions in data collection and learning-task specification, (iz) significantly reduced training-test (or pegrraining-inetunnnn gap, and (izi) potentially universal machine learning paradigm that unifies and enables cooperations among conventional ones.
2 RELATED WORK AND PRELIMINARY
Big/Foundation models. Taking shape in NLP, big/foundation models have drastically changed the research and practice of AI (Bommasani et al. 2021 Yuan et al. 2022). BERT (Stickland & Murray| 2019) and GPT series (Radford et al. 2019: Brown al. 2020) significantly accelerate the development of natural language processing, while models like DALL-Es (Ramesh et al.) [021 2022) effectively promote interdisciplinary research among different research fields. Most foundation models are pretrained in a mask-and- and-predict manner, i.e., holding out a portion of the input followed by training the model use the remaining parts to predict that held-out portion. We will reveal in Section 3 that such mask-and-predict pretraining is a special case of the proposed big learning, which accordingly reveals the underlying principle of foundation models and serves as a theoretical platform for their analysis, justification, and further improvements.
Transformers and Vision Transformers (ViTs). Based on the self-attention mechanism Vaswani et al., 2017), Transformers have been serving as the de facto model architecture for foundation models. Often Transformers (like BERT) take as input sequence of discrete indexes x E 7 L with lenghh I. and output the corresponding latent embedding h Lx D with embedding dimension D for downstream applications; attentions are implemented among the L locations layer-wisely. ViTs (Dosovitskiy et al. 2020) are Transformers modified for dealing with continuous images, which have been empirically proven to have better generalization and robustness than convolutional neural networks (Naseer et al.) 2021p. Different from Transformers embedding discrete indexes into high- dimensional continuous features, VITs directly employ flattened image patches as those features, as demonstrated in Fig. Tb It's well known that Transformers/ViTss are werapparaettrrized Lan et al.
2

---

Under review as a conference paper at ICLR 2023
(2019) and therefore data/information hungry; will reveal that this property of Transformers/ViTs. together with their great modeling flexibility, exactly matches our big learning.
Multi-mode learning objectives. Two well-known multi-mod learning objectives are (i) the cross- entropy loss, often used in maximum likelihood learning with discrete categorical observations, and (ti) the GAN loss (Goodfellow et al. 2014) for adversarial learning continuous observations.
Given feature-labelhhisttryccuum word pairs (æ: y) ~ q(z, v), {1, po (y (z) modeling the categorical distributed y given æ, the cross-entropy loss is identical to
; and model
KL[g(æ v) Ipe (y ( [))( xc ()(()) log (e(y)<))
where the optimal po- (ulæ²) ((yla:). Note the categorical distribution is capable of modeling multiple modes, e.g., consider the diverse generation from the GPT-3 (Brown et al. 2020).
Generative adversarial nets (GANs) are widely used for synthesizing highly realistic images Karras et al. 2019a bj (02I). A standard GAN (Goodfellow et al.) 2014) consists of a generator Go and a discriminator DΦ, both of which are trained in adversarial manner via
min max Em~a(z) log (x) + pae log(1 (x))))
0
where (xx) is the underlying data distribution and (x( is the generated distribution with the generative process x Gp(z), 2 p(z). p(z) is an easy-to-sample distribution, like normal distribution. With optimal D =.. Eq. minimizes the Jensen-Shanne divergence [S[g(z) Ipe (z) (Goodfellow et al., 2014). Recently, the community begins to exploit integrat- ing ViTs into GANs to benefit from their modeling flexibility (Jiang et al. 2021 Lee et al. 202T
Zhao et al. (00T Zhang et al. 2021). We also employ the ViT-based GAN generator and discrim- inator in the experiments, so as to leverage ViTs' modeling flexibility and over-paramettriizaion
property to meet the modeling requirement of our big learning.
BIG LEARNING: A UNIVERSAL. MACHINE LEARNING PARADIGM
For better introduction of our big learning, we first present its main idea in simplified unsupervised settings. where a data sample X = (x) contains only a feature x E R LLxD (with length L and dimension D. like flattened patches of an image or I. words with D = 1), followed by generalizing its scope to the general settings with a data sample X (y, æ)) containing an additional supervision E R 1. x D (e.g., when L D 1. y {1, ... ,C} may represent label). Note in both cases, our big learning can naturally handle "incomplete data," which are defined as either æ missing values along the L-dimension (like missing image patches) or y missing values along the L.M-dimension.
3.1 UNSUPERVISED BIG LEARNING
In unsupervised settings, we focus on generation tasks for the introduction. Given a collection of data samples {æ1, ....... from the underlying data distribution g(x), the mainstream machine learning paradigms concentrate solely on the joint modeling, i.e., to construct a joint model pa (x) to resemble
the joint data distribution q(x), informally po
q(z), using GANs (Brock et al. 2019
Karras et al.] (0119)), VAEs (Kingma & Welling 2013 Dai & Wipf 2019), Flows (Dinh et al. 2014
Kingma & Dhariwal 2018 diffusion models (Ho et al. 2020 Song al. 2020). etc.
Motivations. We highlight two practical situations where the joint modeling is restricted.
1.
For most practical applications like those in medical/biological scenarios, only a limited portion of the accessible data are complete and can be utilized by joint modeling. It's inexpedient to simply discard all incomplete ones (and the valuable information therein), especially where the data collection is expensive. Moreover, discarding incomplete data likely introduces unexpected interventions that violate the i.i.d. assumption, which lays the foundation of deep learning.
2. It's worth highlighting that, given dataset with complete data, already receives the data sam- ples from all joint/coniitional/maaggg distributions; therefore, ideally, one should comprehen- sively exploit that valuable information e.g., to form all the associated data capabilities (like vari-
3

---

Under review as a conference paper at ICLR 2023
target index T
HOG
MAE
MAE/HOG
Model
Po (xTT |xgs)
SUT=D
M
M
M
M
source index $
(a) Unsupervised Big Learning
(b) MAE
et al.
2021 and HOG
(Wei
2021
Figure 1: Unsupervised big learning (a) and its special cases (b). Often mask token [M] is inserted to the input locations outside for forward propagation, while no loss is back- propaaaepeg to the output locations outside T. Note inserting the |M] tokens later in a middle layer (but at same location) often lightens the computation and memory burdens but improves the performance (He et al. 2021
ous conditional sampling for data completion) or leverage different loin//ooditiiona//margaaalll perspectives (formed as different training tasks) to regularize each other"
The above analyses motivate us to model all join/condiitionalmarrignn distributions simultaneously (manifested as "big" learning with massive tasks), to enable flexible training with all available complete/incomplete date and, at the same time, comprehensively "collect" data capabilities via exhaustive data exploitation. Ideally, to collect all joint/condittonal//aargina data capabilities, one need construct Nall E Ci(E Ct models in total (See Appendix A for details), which is clearly prohibitive. Alternatively, we propose the following unsupervised big Tearning.
Unsupervised big learning. For the unsupervised settings with E L and the index set = (1,... L}. the proposed unsupervised big learning leverages universal foundation model Po po( (T a2)) to model many/all join/cconditional/maagnna data distributions simultaneously, i.e.
po xs) - q( q(<r/<3),
where S C L and T L, ( denote any non-overlapping source/target index sets, respectively. q( ax)) the underlying conditional data distribution, whose samples are readily selected from training data. Note $ T need not be II, indicating that unsupervised big learning can naturally handle incomplete data (with the model architecture and training objective detailed below). Fig. demonstrates unsupervised big learning and Table I compares it with the conventional joint modeling.
Table Comparing joint modeling with unsupervised big learning.
Compared Methods
Joint Modeling
Unsupervised Big Learning
Intuitively Training Data
trraight-forware Complete Data
Complicated/Intractbbl Complete/Incomplete Data
Data Information Exploitation
Single Joint Perspective
Exhaustive Many/All Perspectives
Capabilities After Training
Joint Limited
Maat/Cooditional/MMrraaallll
Potential Downstream Applications
Extremely Abundant
Model architecture and training objective. Since the lengths of source Dş and target zr are not fixed, it's not easy to model the universal (o (an)) based on convolutions. Motivated by the modeling flexibility of Transformers/Vit and the fact that most foundation models are built on top of them, we propose to model the universal (a wg) based on the transformer architecture.
If the joint modeling is learned perfectly, it's possible but often computationally expensive to recover all conditional//mariinal capabilities. However, that perfect modeling assumption is usually violated in practice. Incomplete data are readily exploited the corresponding conditional/marrinal tasks.
CC denotes the number of i-combinations from set with I. elements. Note in this paper, only consider the soint/ondiiiiona//magnnnn distributions the length I., e.g., consider the NL.P setup with D 1.
4

---

Under review as a conference paper at ICLR 2023
The training objective is data- data-specific and can be specified as commonly- used machine learning objectives, as exampled below, where we reveal that common foundation models are implicitly doing big learning.
1. Let x denotes a sequence of continuous features, such as flattened patches of an image (He et al., (22T)) as demonstrated in Fig. [b] Unsupervised big learning aims at acquiring the data capabilities of generating a subset of image patches ær given another subset ægg, manifested as versatile data completions (S A 0) or joint/marginal generations ($- = 0). Considering the varying lengths of rg/ we resort to the attention mechanism to handle that challenge and, accordingly, construct po (z)) as modified ViT, which models the generative process of or conditioned on Øg. similar to a (conditional-)GAA) generator. For practical applications, TT are usually multi-mode therefore, the training objective need have the multi-mode capability.
We propose to use the GAN loss as detailed in the following Eqs. (4) and (), where we again leverage a ViT-based GAN discriminator handle varying-length as wa/20. wr.
One can of course consider other multi-mode extensions (such maximum likelihood learning via e.g., VAEs, Flows, diffusion models, and EBMs (LeCun 2022) and even the simple unimodal Gaussian case as in the MAE (He al., ). Specifically, the MAE employs (or()( = ( (2z)), I), where Ho (Zs) is a modified ViT, S is 25% subset of L, and T L.S. It's thus clear that the MAE is a special case of our unsupervised big learning.
2. Let x denotes a sequence of discrete tokens, like text words or vector- cecttrrquantifieed image patches |Ramesh et al., 202I). Unsupervised big learning predicts the target/masked tokens æT given the source ones Øy for many/all (s, T) pairs. To model the correlations among the target tokens ær. we propose to construct the universal porrr/z x2)) as a conditional language model, which autoregressively generates DT (W.r.. the order specified by T) conditioned on the bidirectionally extracted aea-features Therefore, thanks to the random ordering in T, unsupervised big learning will deliver versatile generation/compleett
w.r.f. any predicting orde after training.
The unsupervised big learning is closely related to the permutation language modeling (PLM) proposed in the XLNet (Yang et al. 2019), except that PLM only considers the conditional modeling with SUT L and a fixed 85% 15% split for S/T; it's therefore clear that the XLNet implicitly implements (a special case of) unsupervised big learning. Similarly, the masked language modeling of the BERT can be recovered by further specifying conditional independent assumption among 2r tokens. With S = 0 and the only forward autoregressive order for T, unsupervised big learning readily reduces to the causal language modeling of GPTs.
Take the former continuous settings for an example. For simplicity, we illustrate with the standard GAN loss (Goodfellow et al.) 2014). Given universal model po (wT (azs) that models the generative processes of Øt given Dg for all (S, T) pairs, one can
1. match any model distribution po ( (r)((=))(())))) to the corresponding underlying (subset) data distribution ((FSU))) with
mjn max (yg(asur) log [f S, T)] + Epo (wr (s))) log af (x; T)).
(4)
where the optimal fa- (#; T) log pa (ary (an (m(s) log gl mm)
o(Fsur)
po (zo (a=| [ To handle the varying
lengths of S/T, fz; S, T) is also constructed as a modified ViT; see Appendix for details.
2. enable "communications" among any two model distributions with St UU = S T via pp lagi (((ma)) log (f fx; T
min max
(5)
+ Epo Epp (anga mga (mmga) log lf - - fa; (
where the "communication" discriminator can be implicitly constructed with the same neural network fz S, T) from Eq. (4). Proofs are given in Appendix B
Note (S, T) can be flexibly sampled from all possible pairs (or predefined subset) according to the actual situations of the problem of interest. How to "optimally" specify the pretraining settings for (S, T) is beyond the scope of this paper. Here, we focus on demonstrating the feasibility of (unsupervised) big learning, i.e., one can train a universal foundation model to yield many/all joint/conditional//aaginn data capabilities, informally pe- p0-(2olaa)) (as)) = g (#3) for all (S, T) pairs.
'The previous continuous example need not consider the predicting order, thanks to its (conditionally) joint modeling and generation of æy via GANs.
5

---

Under review as a conference paper at ICLR 2023
3.2 DISCUSSIONS ON UNSUPERVISED BIG LEARNING
The following discussions are readily extended to our big learning presented in Section 3.3
Can share universal foundation model po po(ar (zg) among all (S, T) pairs? Yes, and it's what we should do. It's clear that all conditional/marginaa data distributions g( (2s) can be derived from the joint one g(æ), meaning that their perfect modelings should share the same set of parameters. Being consistent with that fact, (unsupervised) big learning employs the universal po po (gg) Y(S T) with shared parameters data distributions. The great successes from existing foundation models (like BERT, XLNet, MAE, etc.) have extensively verified the effectiveness of that universal modeling of (unsupervised) big learning, under certain special settings with & U T = IL. fixed S/T ratio, but various (S, T) pairs.
simultaneously model all foin/conditional/maainnn
On the model capacity of po po (rs). To collect many/all data capabilities within one universal foundation model po(ar)(zs) brings tremendous challenges to its model capacity. Fortunately, Trans- formers/ViTs are well-known to be data/information hungry, along with their modeling flexibility and parallel-ommputng amenability, making them well suited to model pe (æT (zg). Moreover, huge Trans- formers are emerging, e.g., the BaGuaLu with 174 trillion parameters (Ma et al., 2022). Therefore, the model capacity is likely not an issue for (unsupervised) big learning (see the experiments).
Generalization to diverse domains. Considering practical situations, to directly model within the observed domain, i.e,, po po (arrlar)) may not be good choice (He et al. 2021 Wei et al. (021). We reveal that, with trustworthy domain knowledge, one may alternatively do (unsupervised) big learning in diverse transformed domains, e.g., via (i) pe (ØT (xg) with & g(xz) or (ii) pe (h( (xr) ) (k=g)) as exampled in Fig. lb where g()), fa-), and k--) are wemnannn-nowledge---e
functions.
On the generalization of model parameters and latent features. As aforementioned, exiting big/foundation models, showing extraordinary robustness, adaptability, and generalization capabilities, are implicitly doing (unsupervised) big learning. Accordingly, we try to explain from the big learning perspective why they have such amazing characteristics.
Firstly, by referring to Eq. 3 and Fig. 1 both the model parameters and latent features of po po(ar (as)) are shared among many/all data tasks 5 manifested massive multi-task learning that exhaustively exploits the data information with statistical sharing power. Because all data tasks share consistent goal model (from diverse perspectives) the one underlying data distribution (xx). it's expected that (unsupervised) big learning would encourage the parameters f (and also the latent features) to summarize the intrinsic data information or compositional data meta- knowleen (Wu et al. 2021 Lu et al. 2021). manifested as those amazing characteristics.
Secondly, thanks to its comprehensive training nature, (unsupervised) big learning comes with extraordinary data and training-task flexibilities, enabling training with massive com- plete/incomplete data on many/all data tasks across potentially diverse domains. That signif- icantly expanded training experiences (associated with both data and tasks) are expected to effectively reduce the training-test (or cetrraniing-inetuniin gap and therefore improves the robustness/geeeraiizatio of big-learned foundation models.
On the weighting of massive data tasks. We highlight that (unsupervised) big learning comes with flexible weighting of its massive training tasks, e.g., via predefined sampling strategy for (S, T). How to "optimally" weight those tasks is challenging and is likely downstream-task dependent; we will not cover it here expect presenting several thoughts. (i) Most real-world datasets naturally consist of both complete and incomplete samples, meaning the corresponding (S, T) pairs are already given; therefore, one may prefer to "let the data speak for themselves." (ii) One can of course employ a specific sampling strategy for (S, T), e-g., according to the available domain knowledge. It's worth emphasizing that, despite different weighting strategies, the optimum is the same, i.e., 0 informationally identical to the underlying data distribution q(x).
3.3
BIG LEARNING IN GENERAL SETTINGS
Based on unsupervised big learning with X (x) containing only feature æ R2 we next present its generalized version, i.e., big learning, where X = (y. x)) contains both feature æ and
'The diversity of the training objectives of such trustworthy data tasks is further enlarged in Section 3.3

---

Under review as a conference paper at ICLR 2023
Model Pa X
BERT
S'= (2/ 31
[MU
Food
[M]
Taki
[M]
Toe3
Tak?
Toko
(a) Big Learning
(b) BERT (ttcckland & Murray 2019)
Figure 2: Big learning (a) and its special case of BERT (b). Similar the mask token [M] for (see Fig. lb
we employ another mask token M, for y. which works identically to the classification token ICLS in BERT settings (Stickland & Murray 2019) and the start-of-sentence token in GPT settings (Brown et al. 2020). Often inserting [M]/[M, tokens later in middle layer improves performance (He ct al. 2021 Touvron et al. 2021
supervision y E RL*xD The non-overlapping source/target index subsets S/T are correspondingly expanded as S' = Sy and T = [TV, T), respectively, where L' [LV L. S' c C', T C L', and
T' * 0.
Thanks to the modeling flexibility of unsupervised big learning, to generalize it into big learning is trraght-forward, where the main idea is to model po (x) - q(X (xx) for many/all (S', T') pairs, demonstrated in Fig. 2a
(X) q(y, x)) is the underlying data distribution.
Model architecture and training objective. For situations where X = (y. x) has the same data type (e.g., both y and x denote a sequence of continuous features). big learning works basically the same as its unsupervised variant. We next elaborate on the situations with multimodality (Gupta al. 202 Li et 2021 Ramesh et al. 2021 2022 Baevski et al.) 2022), where e.g., y denotes discrete token sequence but x is a continuous feature sequence. We reveal two solutions.
1. To transform one data type into the other for alignment. For example, quantize the continuous x into a sequence discrete tokens, similar to the DALL-E (Ramesh) al. 2021), followed by employing similar techniques introduced in Section 3.1
vector-
2. To recursively reuse (e(X (x X to model the correlation between æ and The key idea is to further exploit the flexibility of big learning. Specifically, we can unfold the learning via
(6) where XT (yrv œT) and X TT= gx g has one unique data type after unfolding. Big learning first forward propaaapsopagates twice through the model, with the output æy of the first propagation inserted to the input of the second one; after calculating the objective and thanks to the continuity of PT, the gradients can be back coce-propggtted to parameter twice for model updating.
Po X X )= = Po (yT /TT, X (Pr Xx) = Po Tu X Tus )Po X
BERT pretraining as a special case of big learning, Section revealed that (unsupervised) big learning contains the masked language modeling of the BERT pretraining special case. We next reveal that big learning also covers the next sentence prediction (NSP) task (Stickland & Murray 2019), where x indicates two sentences and label y E {0. indicates whether they are next each other. It's readily verified that NSP is recovered with S' = [0. L] and T' = [1], 0] (refer to the 1st column of Table Z. summarize, big learning contains the BERT pretraining as a special case.
Big learning serves as a universal machine learning paradigm. Benefiting from its modeling flexibilityy, big learning has most machine learning paradigms as special cases, as illustrated in Table That universality of big learning. combined with its data/task flexibilities, enable flexible combinations and communications among different learning paradigms, e.g., via the shared parameters or training objectives like Eq. (5). Therefore, the proposed big learning might potentially facilitate semantically diverse multi-task self-learning on the Internet, producing brain-scale big/foundation models with reinforced performance, robustness, and generalization.
Big learning versus self-supervised contrastive learning. Contrastive learning focuses on exploiting domain prior knowledge to learn generally applicable data representations for downstream tasks (He
7

---

Under review as a conference paper at ICLR 2023
Table Example special cases of big learning po (r of generality, we assume y E {1.... CJ is the label paired with æz where L = {1}- We focus on the core
[X with = [3" and [TV, Tj. Without loss
idea for demonstration and highlight that the model/objective implementations can be task-specific
Supervised Learning
Self-supervised Learning
Unconditioned Generation
Conditioned Generation
po (y((zr) [0.L] L
po [t [ms)
pe
po ( (a1yy (y)
S' [0, 8]
[0. 8
[[1]. (1]..0 0]
(1).0)
T' [0, T
T [0, TL.]
T [8, T.]
ct al. 2020; Chen et al. 2020; Grill et al. 2020; Chen & He 2021). From the perspective of prior
exploitation, contrastive learning is orthogonal to big learning that is mostly data-driven. One can of course consider leveraging the flexibility of big learning to combine it with contrastive learning for incorporating trustworthy domain priors; please refer Appendix for further discussions.
On the i.i.d. assumption. Due to the space constraint, please refer to Appendix E for the details.
4 EXPERIMENTS
The data/task flexibilities of big learning significantly expand its scope of application, which, however, also bring tremendous challenges to the comprehensive evaluations of its robustness, adaptability, and generalization capabilities. We emphasize that the great successes of existing big/foundation models have provided concrete evidences that support the presented big learning.
In what follows, we concentrate on demonstrating the main idea that (unsupervised) big learning is indeed capable of delivering many/all joint/conditional/maargr extraordinary data/task flexibilities and exhaustive exploitation of data information. We conduct unsupervised big learning with all ciintcconditional/maargnn data tasks (via (S, TT)-smpliigg on image datasets of MNIST and CelebA, based on Eqs. 44) and (5) (details are given Appendix After training, we (i) diversely test the data completion capabilities of the big-learned model and (ir) tasks.
data capabilities, thanks to its
ferociously challenge its generalization capability with abused anomalous
4.1 VERSATILE DATA COMPLETION CAPABILITIES WITH ADAPTIVE GENERATION DIVERSITY
We first test the big-learned data generation/completion capabilities with different ratios rs of S in LL For specific rs. we either randomly sample r3 image patches or choose the first rs rs-portion to form the source æg, which is then input to the model p0 zg) for image completion. Fig. shows the cor- responding results. It's clear that the big-learned model masters many/all joint/condiiionalmmarinnaa data capabilities simultaneously. Besides, big learning also learns from the data an adaptive gen- cration diversity conditioned on Øg. Specifically, with increasing/decrrasing r3g (i.e., more/less source information), big learning delivers increasingly deterministic/ddierss generations controlled by rgg/random-noise following our intuition (see Appendix for more results).
We then test the big-learned capabilities with respect to various S and noise settings, with the results summarized in Fig. 4 On the one hand, given an image æ and a random noise z, big learning clearly delivers for various 35 diverse realistic generations on both MNIST (see the variations in class/stroke-
1777
7
7
7
7 7 2 0 4 5 9
/
2
2
2
2
2
2
2
2
2
3
6
0
0
0
0
0
0
0
0 4 5
3
4
4
4
4
4
4
4
4 5 9
3
5
5
5
5 5
3
2
7
8
9 9
9
7
9
(x) MNIST with random
(b) CelchA with initial portion 3
Figure 3: Versatile data generation/completion capabilities from big learning. The first row with light-blue boxes shows different Ss, with increasing S-ratio from left to right. The rightmost column gives the real image.
8

---

Under review as a conference paper at ICLR 2023
4
4
O
4
L
9
0 0 9
0 8
9
4
4
8
Ze
27
8
5
(4) MNIST
ColebA
Figure 4: Versatile data completion capabilities from big learning w.t.. various 3 (left) and noise = (right). Es are shown in upper-right light-bluc boxes, while the red boxes show (left) and (right), respectively.
thickness/shape/angll) and CelebA (see the varying dennity/aiisssllemkkke/////sssse other hand, given a specific æe with limited information, the big-learned model, when input different noises 24, also generates realistic images with diversity.
On the
The experimental results in Figs. and 4 demonstrate that, by comprehensively exploiting the available information inherent in large- scale complete/incomplete data, big learning is capable of delivering versatile data generation/completiin capabilities with learned adaptive generation diversity.
4.2 GENERALIZATION ON ABUSED ANOMALOUS OUT-OF-DOMAIN COMPLETION
We design abused completion tasks to ferociously challenge the generalization of our big learning. Specifically, we intentionally design æts with (i) abused interventions to source patches (e.g,, random relocation and duplication, as shown in Fig. 5 aa)); (ii) mixed-up patches from different data samples (see Fig. ())):: and (iti) unseen out-of-domain image patches, shown in Figs. (d))
3
3
3
2
2233
32
3
D
8
(a)
(b)
(c)
(d)
Figure 5: Abused anomalous completion for demonstrating the generalization of big learning. (a) iss constructed with random center patches replaced in the upper-left corner (top) and duplicated and replaced in the center (bottom). A model big-learned on CelehA is used in (b)-(d). (b) combining patches from di different images.
Out-of-domain ag from Flowers (Nilsback & Zisserman, 2008 and MctFaces (Karras et al. 2020 (d).
It's clear that big learning manages to handle these abused æs with reasonable image completion; egg. see the realistic characters with overall consistent style and smooth strokes in Fig. [ a), the harmoniously completed faces with mismatched face frame and hair color in Fig. 5 b), and
the fluent oft-o--domain completion with smooth junctions in Figs. [(c)-(d). These surprising
results from abused anomalous out-of-domain completions (as well as the great successes of existing foundation models) justify the remarkable generalization capability of the presented big learning.
5 CONCLUSIONS
We propose the big learning that exhaustively exploits the available data information during training and potentially delivers all oinn/conditionna/maaginall data capabilities after training. We reveal that big learning (i) comes with marvelous training flexibilities for complete/incompllte data and for customizing training tasks, (ii) is what existing foundation models are implicitly doing, and (Hii) unifies conventional machine learning paradigms and enables their flexible cooperations. Though inspiring, big learning also shares the constrains of foundation models that are discussed in detail in Bommasani et al. (2021); Yuan al. (2022); e.g., to comprchensively verify its effectiveness (for general downstream tasks) is extremely challenging and time-consuming. Therefore, we believe that big learning needs our community, and vice versa.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zeulemoyer. and Sonal Gupta. Muppet: Massive multi-task representations with pre-finctuning.. arXiv preprint arXiv:2101.J 1038, 2021.
Alexci Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.08555, 2022.
John A Bargh and Ezequiel Morsella. The unconscious mind. Perspectives on psychological science, 3(1): 73-79, 2008.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bemsstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07256 2021.
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019. URL
net htte://ooommddddl1.. 009m
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subhiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Mark Chen, Jenry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on codc. arXiv preprint ar arXiv:2107.0333 2021a.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geotfrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMIR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021b.
Dai and D. Wipf. Diagnosing and enhancing VAE models. In ICLR, 2019. URL httpt:t/oppeeeeveaaeee net/ Forum? 1d=B1c0X3C99t
Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for
language understanding. arXiv preprint arXiv: 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-lincar independent components estimation. arXiv preprint 41Xiv:1410.8516, 2014.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11990 2020.
I. Goodfellow, Pouget-Abadie M. Mirza, Xu, Warde- Warde-Farley, Ozair, A. Courville, and Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Jean-Baster Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised lcarning. Advances in neural information processing systems, 33: 21271-21284, 2020.
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. arXiv preprint arXiv:2104.00743, 2021.
Raia Hadsell. Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) volume 2, pp. 1735-12 1742. IEEE, 2006.
Kaiming He. Haoqi Fan. Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF conference confereceo computer vision and pattern recognition, pp. 9729-9738, 2020.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, arXiv preprint arX6:2211..0637, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:684006851,, 2020.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In CVPR, June 2019a.
Karras, Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN, arXiv preprint arXiv; 19XX::://22.49555 2019h.
Tero Karras. Miik Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33: 33:12104-11 12114, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. arXiv preprint arXiv: /9999999999999,90.1 2019.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.
Kwonjoon Lee, Huiwen Chang, Lu Jiang. Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint 2rXiv:2 2021.
Qing Li. Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du, Ming- Hiung Yang, and Matthew Brown. Towards unified foundation model: Jointly pre-training transformers on unpaired images and lext. arXiv preprint arXiv:2112.07074, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1711001111.. 2017.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. arXiv preprint arXiv:2103.05227, 2021.
Casimir JH Ludwig, Rhys Davies, and Miguel P Eckstein. Foveal analysis and peripheral selection during active visual sampling. Proceedings of the National Academy of Sciences, 111(2):2211-E29) 2014.
Zixuan Ma. Jiaao He, Jiezhong Qiu, Huanqi Cao. Yuanwei Wang. Zhenbo Sun. Liyan Zheng, Haojie Wang. Shizhi Tang, Tianyu Zheng, al. BaGual w: targeting brain scale pretrained models with over 37 million cores. In Proceedings of the 27th ACM SIGPIAN Symposium on Principles and Practice Parallel Programming. pp. 192-204, 2022.
L. Mescheder, A. Geiger, and Nowozin. Which training methods for GANs do actually converge? In ICML, pp. 3478-3487, 2018
Anabela Mesquita. Human behavior, psychology, and social interaction in the digital era. IGI Global, 2015.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat. Fahad Shahbaz Khan, and Ming- Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-233000 2021.
M. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICCV, Graphics & Image Processing, pp. 722-729. IEEE, 2008.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18))9, 2019. 2019.
11

---

Under review as a conference paper at ICLR 2023
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh. Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, arXiv preprint arXiv:2204.06/ 06120000.615.5. 2022.
Toni P Saarela and Michael Landy. Integration trumps selection object recognition. Current Biology, 25(7): 920 220-927, 2015.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic diffcrential equations. arXiv preprint 1xXi::2011.13455 2020.
A. Stickland and L. Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. arXiv preprint 1roXiv:902.006777 2019.
Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Goodman. DABS: A domain- agnostic benchmark for self- supervised learning. arXiv preprint 1:Xi:2211112062. 2021.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 32 42, 2021.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurlPS, pp. 5998 000866008-- 2017.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2018.
Chen Wei, Haoqi Fan, Saining Xie. Chao- Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for supr-super visual pre-training, arXiv preprint arXiv:2112.09133 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen. Clara Ma. Yacine Jernite, Julien Plu. Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. Procedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. httpshttss://// wwwp:://wwww........o wcw.:wa.wwwl.. org/ 2020. emn pwmpo.dee.....
antho ogwp//20..co..
Yuhuai Wa, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pp. 11251-1 1- 1262. PMLR, 2021.
Z. Yang. Z. Dai, Y. Yang, Carbonell, R. Salakhutdinov, and Q. Le. XLNet: Generalized autoregressive pretraining for language understanding. NeurlPS, pp. 5753-5763, 2019.
Sha Yuan, Hanyu Zhao, Shuai Zhao, Jiahong Leng, Yangxiao Liang, Xiaozi Wang, Jifan Yu, Xin L.,, Zhou Shao, Jiaao He, et al. A roadmap for big model. arXiv preprint arXiv:2203.1 /4/01, 2022.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. arXiv preprint arXiv:2 2021.
Long Zhao, Zizhao Zhang, Ting Chen, Dimitris Metaxas, and Han Zhang, Improved transformer for high- resolution gans. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Under review as a conference paper at ICLR 2023
Appendix of Big Learning: A Universal Machine Learning Paradigm?
Anonymous Authors
A ON NAIVE MODELING OF ALL JOINT/CONDIIIONALMAAGIINN DATA DISTRIBUTIONS
We present with the unsupervised settings, where a RLXD with length L and dimension (like flattened patches of an image or L words with D 1). It's straightforward to generalize the following analyses to the general settings with a data sample X = (y. x) contains an additional supervision R2 D Considering > 1 and D = for image patches and text words, respectively, we concentrate on analyzing the modeling of all voin//oondiiiona//aagina data distributions w.E.f. the length below.
As mentioned in the main manuscript, one need construct Na = EI Ci models to naively model all biin//coddiiionaa/maninnal data distributions, collect all joont/condiiooaa/mmainnn data capabilities. C₁ denotes the number of i-combinations from set with L elements
To elaborate on that, consider a simple 3-length 1-dimensional problem with x: = [2], 22, 03, where T. 3, D 1, 3y E IR, and the length index set IL (1,
The goal of the joint modeling is to deliver (ax) - ((x) with one model (ar)..
By contrast, naively model all join/conditionaalmariinn data distributions, one need construct 19 models for such a simple 3-length problem, i.e.,
Po: (21), Poe (12), (xa), Po (211,2),, Pos (x2, Po Poe (I1, Por (41, 22, #2)
Pø (222 [x)) Po (xaa Pa (x2, (222 x1),
Por (zi (22), Po (23/22), (11,
Para (2n (ra); Po (22) Po (21, x2
12 (3);
Pair (2n (222, .3), Pais (xe [2), 23), Pa Paw (22)=1,,02).
Based on the above 3-length problem, one can readily summarize the following two steps in calculat- ing the number of models in naively modeling all joont/conditionalmmarggn data distributions, i.e., q(< æs), VS C L, T L.
1. Sample S. The source index set S may contain fo. 1} indexes/locations, where S containing 0 index corresponds to joint/marginal generations and S containing 2 1 indexes corresponds to conditional generaaions/ccompltiions.. For special case with indexes in S with i E [0, J. 1), one has Cj ways to specify that source index set S.
2. Sample T conditioned on S. Given S consisting of indexes, the target index set could contain [1.... L, i} indexes/locations outside S. For a special case of T containing k indexes where i E [1, T. - 1), one has Ch ways to specify the target T.
Therefore, to naively model all joint/conditiona//aargna data distributions, one need construct Na Ei=1 Ci.EEIi Ct.,,) models, which, however, is prohibitive in practice.
Note with ideal modeling of ((=m(ar)) the orders in S/T should not matter. However, that may not hold true considering practical constraints, e.g., where existing joint modeling techniques fail to model the multi-mode characteristics of TT. Besides, in the NLP application of language modeling, one may be interested versatile (conditional) generation ordering (as defined in T), mimicking the permutation language modeling (Yang et al. 2019). In that case, to naively modeling all pointcooddiionaa/maainnn data distributions, one need construct Ni ΣL= C C A models to take into consideration the order Tf where the order is ignored and At denotes the number of the ordered arrangements of k elements from set with L - elements. Similarly, one need construct NIi Et- A (E A/_.)) models model the orders in both S and T.
13

---

Under review as a conference paper at ICLR 2023
po (a: S1)
po Po(2;5S S1)
((x))
q x
pe (x; S2)
pe (x; S3)
Pe (æ: S2)
po ax; 8z)
(a) Case
(b) Case 2
Figure 6: Demonstration of unsupervised big learning based on GANs.
B DERIVATIONS OF THE GAN EXAMPLE ASSOCIATED WITH EQS. (4) AND (5)
Here we present the detailed derivations/proofs for the GAN example associated with Eqs. of the main manuscript. For better understanding, we begin with simplified case where followed by generalizing the results to the general situations with T C L.S.
4 and
B.1 T
To leverage the GAN training framework (Goodfellow al., 2014), needs the sampling capabili- ties from the distributions interest. With T LAS, here we are interested in the joint distributions with accessible sampling capabilities, including
((x)
(8)
pa (w; S) = po (aLLy (
VS.
Note one can of course exploit the flexibility of big learning to define other joint distributions with sampling capabilities, such as an recursively defined distribution
pe(x;; 82) = pe (#L))3) Øgaa (pe
where pa (xga)) f pe (LLA [g (q( 1,82. For simplicity, focus the simplified settings in Eq. (8) and leave the interesting but complicated recursive case for future research.
Given the underlying data distribution ((x) and "model" distributions po (z:; in Eq.
1. one can match any pa [e; S) to q(æ)) adversarially with GAN. Take the standard GAN (Goodfellow et al. 2014) for an example, the objective
min max Eytan) log (fo((x; ()) + (eo log(1 - ((f((x;)))
(10)
0
where the optimal fa- (ar; S) = log (
log [ Ideally, opti-
(
mizing the above objective is identical minimizing the Jensen-Shannon divergence JS[g(æ)]|Ppg (p) S)], as illustrated with the blue solid arrows in Fig.
2. one can also conduct matching among any two model distributions (e.g.. Pe( = pe ( [z (g(xz)) ) and po (x S?) pe (ZLAsz (Zga (p(=))) )) enable communica- tions/cooperations among them, via optimizing
Epols leg (((1))) log o (w; (8)))
min max &
(11)
Ep (mznea ( log (x,
where the optimal K/ (a:; S1, 82) no ( ma ]
po ( |a )g(agL)) log
The orange dotted arrows in
Fig- demonsttate such idea.
14

---

Under review as a conference paper at ICLR 2023
At first sight of Eqs. (0) and [1,, it seems should at least construct two discriminators, with Sx; and ls (x; S S²)
respectively. However, we notice that
(x))
((x)
Sar (2; 8²)
log
log po (LL)
po (1 agga (g(()))))
agg ) (2))))
far (x: S²) for (x: St).
Accordingly, we propose to employ further simplification that builds (a; , top of (o; i.e, we reformulate Eq.
as
p a Jo(a,)) log [f (ax; S1)] fo (x;
min max
(12)
p laga (e( log o [So( l
Till now, we present the derivations associated with T L\S, i.e, matching the joint space. In what follows, we generalize to the settings with T C L,S, to deliver (unsupervised) big learning in all join//conditiona//maagnaa spaces.
B.2 LAS
Similar to the previous section, also consider simplified situations with recursiveness, that is, we do not consider a model distribution pa(T (ts po (xxz), even though such recursive flexibility of big learning is quite interesting. We leave that future research.
Accordingly, the considered joint/conditiona//aargnaall distributions with sampling capabilities are (
(13)
(D(UT) = Po x (<)))
vS,
where $ U T need not be L. Note S TT L means the corresponding q( ((Esur) is marginal data distribution, whose data samples are readily accessible from those of ((x).
Similar to the previous section,
one can match any model distribution pa (EBUT) to the underlying joint/marginal data distribution q via the standard GAN objective
min max Ba((aar)) log (f T)) + E pe (ar((a)))(())) log(1 - ((; (14)
0
where for S, T) log 7(msur)) po(ay) (12)) log
a(aa [s))
pa (xy ing)
one can also conduct matching among any two model distributions, e-g., Po (2TT agg )q( and po( ag (ggg))) as long as S' UT! 52 U T2, with the corresponding objective
E (a ja (a(a) log a (x; T (T)))
min max 0
(15)
+ E Epo ( log(1 (fs ,,,,,,,,,,,,)),
where f/- (z; St. T 82 T log (m (m )=((m)))
no (my aa )=(g)))
For further simplifications, we again resort to
1/ (x: 88 .. T..Ts T2) = log (
Po ( ag )4 (gg2) log a( (p
Po (a l ) (Fg))
for (x; S2 T2)
før (x; T
and build (x; T T. T2) on top of fo (x: S, T). Accordingly, Eq. 115) is reformulated
E (mp ( (m log o Ifs 5² T f (
min max
(16)
0
Epp l )m(aa))) log logo [f T (2;
Accordingly, conclude the proofs for the GAN example of the main manuscript.
15

---

Under review as a conference paper at ICLR 2023
C ON MODEL ARCHITECTURES OF THE GAN EXAMPLE IN EQS. AND 5
We next focus on discussing the model architectures of the GAN generator and discriminator employed in Eqs. 14 and (T6) (i.e., Eqs. 44) and (5) of the main manuseript).
Recently, y, the community begins to exploit integrating ViTs into GANs (Jiang et al. 2021 Lee al. 2021 Zhao et al. 2021 Zhang et al. 2021). For example, the ViTGAN (Lee al. delivering SOTA generative performance, employs simple modifications to the VIT architecture to construct the generator and the discriminator, but adopts many techniques regularize the ViT-based discriminator for stable training. Motivated by the modeling flexibility of ViTs, we also employ ViT-based GAN generator and discriminator in the experiments, but similarly, find it challenging to stabilize GAN training with ViT-based discriminator. It's worth highlighting that it's possible to design other alternative model architectures for our big learning; we employ what's presented below for demonstration.
source index $
G-Encoder
[M]
G G-Decoder
[M]
target index T
Generator
[M]
Po XT
X
[M]
Embedding
Patch
Ep:
Ega
Esa
Maak
E,
Eu
Ex
[M]
[M]
[M]
M
Enbedsing
Nake
[M.
M..]
Embeddinng
E
Embadding ng?
Postion
E
(a) GAN Generator
[CL)]
real/fake
P2
Discriminator
P5
trasdding
Eu
[M
Eeseadiigg
(b) GAN Discriminator
Figure 7: Example implementations of GAN generator discriminator employed Eqs. [4 and (6
(i.e.. Eqs. 4 and (5) the main manuscript).
Fig. demonstrates the employed GAN generator and discriminator, both of which are constructed with Transformers/Vitss exploit their modeling capabilities and flexibilities.
16

---

Under review as a conference paper at ICLR 2023
GAN Generator. Following the MAE (He et al. 2021). design the GAN generator Pe (a (e)) with an autoencoder-like architecture, which employs an encoding G-Encoder and a decoding G-Decoder, shown in Fig. The G- G-Encoder encodes the source patches zg (if any) to their latent codes; then, these codes are combined with the mask tokens [M], patch-wise noise embeddings, and new positional encodings serve the input of the G-Decoder; finally, the G-Decoder transforms its input to generate the target patches 2T.
[M] tokens are inserted later in middle layer, because doing this often improves per- formance and lowers the computational burden (Touvron et al. [02T] He et al.) (02T). A noise is mapped with an 8-layer MLP to produce the patch-wise noise embeddings {nl..... , Note we also introduce another toke [M,)] to indicate no noise embeddings are necessary the corresponding source locations in S.
GAN Discriminator. As shown in Fig. 7b we also modify the Transformer/Vii architecture to construct the universal GAN discriminator o (f (x; S, T)) that applies to all (s, T) cases. We employ additional CLS token mimicking the BERT, whose output indicates whether the input patches are realistic or not (more specifically, whether they form "real" data from ((FSUT) or a fake one from po ( ( (x)), by referring Eq. (14). The input of the discriminator consists of patch embeddings, positional embeddings, and two new special tokens [M.] and [M,)) that indicate source or target patches mimicking the sentence tokens in the BERT.
D BIG LEARNING VERSUS CONTRASTIVE LEARNING
Contrastive learning (Hadsell et al. 2006) aims at learning a latent representation space, where the representations of different views of the same image ("positive pairs") near each other but those from different images ("negative pairs") " are far away from each other.
As discussed in the main manuscript, the self sule-super contrastive learning focuses on exploiting domain prior knowledge to learn generally applicable data representations, while the presented big learning mostly data-driven. From that perspective, they are orthogonal to each other. However, we reveal below that big learning and contrastive learning have lot in common.
Both of them are based on massive multi-task training, associated with source/target indexes (S, T) and online/target augmentation pairs (A, B) (see Fig. Bap, respectively.
Both of them share universal model among massive training tasks.
From the information perspective, both of them predict (or retrieve) the T/B-associated information conditioned on the information related to S/A, as detailed below.
Existing contrastive learning methods can be roughly grouped into two groups, based on whether the method uses negative pairs (like SimCLR (Chen et al. 2020) and MoCo (Chen et al. 2021b)) or not (like BYOL Grill al. 2020) and SimSiam (Chen He 2021)
Group 1. Contrastive learning methods using negative pairs, like SimCLR and MoCo, can be interpreted as retrieving (in the latent representation space) the target positively paired sample B; from the negative samples within the mini-batch, conditioned on the source augmented sample A, as illustrated in Fig. 8a
Group 2. Contrastive learning methods not using negative pairs, like BYOL and SimSiam, directly predict/gennerate (in the latent representation space) the target/teacher projection as- sociated with B,, conditioned on the student projection associated with A,, as demonstrated in Fig. 8b
Either group of contrastive learning methods retrieves predicts the B-associated information conditioned on the information related to A, which is quite similar to the proposed big learning that prediets/generates ær conditioned æs. Therefore, from the information perspective, both of them predict (or retrieve) a piece of data/prior information conditioned on another piece of data/prior information.
It's interesting to consider combining big learning with contrastive learning to exhaustively exploit the available information from both data and domain-prior perspectives.
17

---

Under review as a conference paper at ICLR 2023
4
B
A
B
X
1
0
O
O
X
2
X/3
1
X
O
O
O
O
1
O
4
B
4
B
A
B
F
0
1
X
O
0
O
X
0
0
0
1
x
(a) Group predicts/retrieves the positively paired sample the negative samples within a mini-batch. A and B denote the online and target augmentation, respectively.
student
teacher
student
teacher
grad
similarity
grad
similarity
predictor
predictor
moving averuge
encoder
momentum encoder
encoder
encoder
image
image
BYOL
SimSiam
(b) Group 2 directly generates/predicts the target/teacher projection conditioned the student projection.
Figure 8: Demonstrations of contrastive learning methods. (b) is adapted from Fig. 3 of Chen & He 2021
E ON THE i.i.d. ASSUMPTION
The i.i.d. assumption is one of the key foundations of deep learning. But it's also well-known that the training data collected for practical applications rarely i.i.d., leading training-test gap (i.e,, novel/unseen data information emerges during testing) that significantly hinders the practical reliability of existing deep-learnin models.
Recently, foundation models begin to demonstrate increasing robustness and generalization towards that gap. likely due to their large-scale pretraining (special cases of big learning) effectively reduces the probability of observing novel information during testing. We next elaborate on that and present 3 reasons on why big learning expected significantly reduce the werereeereeeeraiinnniieeeeeeeeee gap for improved robustness.
1. Thanks to its remarkable flexibilities on complete/incomplete training data and massive diverse learning tasks, big learning delivers abundant "exercise experiences" that significantly enlarges the training scope of the model, making it less likely to be "surprised" by novel test data/tasks.
2. Manually collecting or filtering data samples will likely introduce unintentional interventions that violate the i.idd assumption. The data flexibility of big learning makes possible conduct training with minimal human interventions in data collection, and accordingly, "let the data speak for themselves."
3. Even with the same dataset with all complete samples, big learning is expected to behave more robustly to the i.id assumption, because (t) to collect perfectly i.i.d. complete samples is often intractable for practical applications; (18) the conditional tasks of big learning always implemented with perfect and trustworthy i.idd. samples; and (fit) big learning enables (implicit) communications among tasks, which is expected transfer advantages that benefit each other.
18

---

Under review as a conference paper at ICLR 2023
F EXPERIMENTAL SETTINGS
We employ the same model architectures in the previous Section C for the experiments on the MNIST and CelebA datasets, with the detailed hyperparameters summarized in Table 3 Despite the relatively small models used, we find that big learning is capable delivering potentially all toin//condiiiona/mmargnnnn data capabilities simultaneousll. We adopt the AdamW optimizer (oshchilov Hutter 2017) with 8 = (0.1,0) 0.999) and constant learning rates for both the generator and the discriminator. Code will be released upon publication.
Table 3: Hyperparameters used in experiments.
Dataset
MNIST
CelebA
Image size
64
120
Patch size
8
10
G-Encoder depth
6
G-Encoder #heads
8
8
G-Encoder dim
256
256
G-Decoder depth
G-Decoder #heads
8
8
G GeDeoeer dim
512
512
D depth
6
D #heads
8
D dim
256
256
GP (Mescheder et al. 2018)
real
real
AGP
10
10
Learning rate
10-4
10-4
Batch size
256
128
Source ratio 1 1/111
Beta(0.5,3)
Beta(0.5,3)
Target ratio IT 1/
Beta(3,0.5)
Beta(3,0.5)
Communication source ratio |s9////s'um
Beta(0.5,3)
Beta(0.5,3)
Overall, we find it's quite straightforward to implement the MNIST experiments with the standard implementations discussed in Sections B and C without resorting to any "tricks" like warm-up or gradient clipping. However, the more complicated CelebA experiments, we find it's necessary to employ some, as detailed below.
We employ warm-up in the first 10 epochs for both the GAN generator and discriminator; after that, we use the constant learning rate given in Table 3
We apply gradient clipping, with the max norm of 5, to both the generator and discriminator optimizers.
Similar to Lee et al. (222 we also find it challenging to stabilize GAN training with a ViT-based discriminator. To deal with that, we additionally (i) overlap image patches (Lee et al.] (02T) with e.g., 2 pixels at the input of the discriminator (different from the non-overlapping image patches used in the vanilla ViT); and (ii) use a larger hyperparameter = 10-5 in the AdamW optimizer.
Other empirical experiences listed below.
We empirically find that the last normalization layers of both the GAN generator and discrim- inator have a significant influence on the learning stability and final performance. Specifi- cally, replacing the last LayerNorm of the G-Decoder the generator with LeakyRe leads to improved generative performance, whereas replacing the last LayerNorm of the discriminator with other normalization/actiatton layers results in training collapse.
Employing additional convolutional head (like 3-layer CNN) to the output of the generator often leads to improved performance and training stability.
Instead of only introducing noise embeddings the first layer of the G-Decoder of the generator, as shown in Fig. 7a| we find it's beneficial to concatenate the same set of noise embeddings layer-wisely into the G-Decoder layers.
19

---

Under review as a conference paper at ICLR 2023
G EMPIRICAL EVALUATIONS ON THE GLUE BENCHMARK
Concerning the empirical comparisons between existing methods for foundation models and the presented big learning, intuitively, one would consider first using our big learning as the pretraining strategy in place of existing ones, followed by applying the same naive fine-tuning on downstream tasks, to evaluate the effectiveness our big learning. Unfortunately, we cannot afford the pretraining cost; for example, to pretrain XLNet-Large takes about 5.5 days on 512 TPUs according to Yang iet al. (2019). We leave that to the community, as mentioned in the Conclusion.
To demonstrate the advantages of our big learning over existing methods for foundation models, we alternatively consider leveraging it to serve as the less expensive fine-tuning strategy. It's worth highlighting that, from another perspective, such experiments also verify the advantages of the big learning in the fields of supervised learning, when compared to existing supervised learning methods.
Specifically, design experiments based the Hugging Face transformers library Wolf al. (2020), the GLUE benchmark Wang et al. (0188) and the XLNET Yang et al. (2019) that outperforms the BERT many NI.P tasks. We employ the same pretrained xlnet-ba bast- casedd model and continually train it on the downstream KTE/MPP//SST- classification tasks via (1) the naive fine- tuning (i.e., identical to the original XLNET, termed FT) and (ii) our big learning (termed big-Icarn), respectively. In other words, the pretraining phase (i.e., the permutation language modeling Yang al. 2019). special case of our big learning) is the same and we compare our big-learn with the naive FT during the finetuning phase.
Because the data of the downstream classification tasks contain both feature æ and label y. we resort to the big learning settings of Section 3.3 of the main manuscript. Specifically, X = (y, ax) and the universal foundation model po X Tr
Sr has network architecture similar to the one shown in | consists of the pretrained XLNET backbone and a tas sppeciicc head that is attached to the output of the <CLS> token; for simplicity, we abuse to represent all the parameters. For specific (S', T)) pair, pe [X T Xx)) recovers pe po(y)az), ) i.e., conventional classifier.
Fig. 2 of the main manuscript. Note Pe X
With the above notations, we next formalize the objective for both FT and our big-learn.
FT. Often cross-entropy is employed, which is identical to
(r) E R (2. [- log (o(((
(17)
where Idownstream (ar, y) represents the training data of the downstream classification task. Big-learn. For direct comparisons, we formalize the big-learn objective
Lbig learn (0) LFr( BBigt C(O),
(18)
where SBigteam is hyperparameter and
(()) F. T E gamammm (x) log Po X (X X3
(19)
with ((S, T') denoting the sampling process of TY). We simply reuse the same sampling process in Table
Note Eq. 19 is equivalent to minimizing E (T) KT. (downstreeam
X
Po
X
X
by
referring to Eq. of the main manuscript.
Table 4: Tested hyperparameters when comparing FT with big-lcarn on the GLUE benchmark.
Task's Hyperparameter
Learning Rate [2e-5, 4c-5, 6e-5] [2e-5, 4c-5, 6c-5] 12e-5, 4c-5, 6e-5]
#Epochs
WarmUp Steps
BrigLeam
RTF
[3, 4, 7, 10, 15]
[0, 120]
[0., 0.2, 0.4, 0.6, 0.8]
MRPC
[3, 4, 7, 10, 15]
[0, 120]
[o., 0.2, 0.4, 0.6, 0.8]
SST-2
[2,
[0. 1200]
[0., 0.2, 0.4]
We extensively compare FT with big-learn on the downstream RTE/MRPC/SST-2 classification tasks, by evaluating the accuracy and/or F1 score the Dev set across the combinations of the tested
20

---

Under review as a conference paper at ICLR 2023
hyperparameters shown in Table 4 The hyperparameters chosen following Devlin et al. (2018); Yang et al. (2019).
Table 5: Empirical evaluations showing superiority of big-learn to FT. The best/median metrics are calculated among the combinations of the tested hyperparameters in Table
Metric
Best Accuracy FFI
Median Accuracy /IQR
Task
FT
big-learn
FT
big-learn
RTE
71.84
75.09
66.06/2.34
70.75/1.44
MRPC
88. 88.97/92.09
90.20/93.03
87.00/2.45
87.74/1.10
SST-2
94.15
95.18
93.75/0.45
94.66/0.28
The best/median metrics are summarized in T and Fig. shows the corresponding boxplots; it's clear that our big-learn consistently outperforms FT. Accordingly, our big learning can serve as a superior fine-tuning strategy. It's worth highlighting we did not carefully tune our big-learn; therefore, it's likely that its performance could be further improved by e.g, tuning the sampling process (S', )
0.90
0.700
0.675
0.87
0.050
0.86
0.625
0.84
0.83
Asption
(a) RTE
MRPC
0.000
0.915
0.910
0,35
SST-2
Figure 9: Boxplots of the Dev-set accuracies from FT and our big-learn. Note big-learn with SoigLeam 0 is identical to FT (see Eq. (8)). It's clear that hig-learn consistently outperforms FT on all three tasks.
We'd like to emphasize that our big learning can reduce the
gap because
it can act as the pretraining and finetuning objectives, simultaneously; rely
big learning to completely merge the pretraining and finetuning phases, leading to zero gap.
Motivated by the performance boost from the BERT the XLNET and discussions "on the generalization of model parameters and latent features" of Section 3.2 of the main manuscript, we posit that our big learning serve better pretraining and finetuning strategies than existing
21

---

Under review as a conference paper at ICLR 2023
methods, leading to universal machine learning paradigm. We leave the corresponding verification as future research.
H ADDITIONAL EXPERIMENTAL RESULTS
More experimental results, complementing the limited demonstrations of the main manuscript, are given below. Please refer to the captions for details.
17
7
7
7
7
7
7
7 2
7
7
1
2
2
2
2
2
2
2
2
2
3
6
0
0
0
0
0
6
6
0
0 4
3
4
4
4
4
4
4
4
4
4 6
3
5
5
6
-
5
6
3
2
7
9
9
9
9
7
9
9
9
(a) MNIST with random S
(b) CelebA with random
1
2
7
7
7
7
7
7 2
7
7 7 2
3
2
2
2
2
2
8
8
6
6
0
6
0
0
0
0
6 4 5 9
7
7
8
4
4
4
4
4
4
4 5 9
3
8
0
8
5
5
5
5
7
7
9
9
9 9
(c) MNIST with initial-portion S
(d) CelebA with initial- portionn S
Figure 10: Demonstrating the gencration/complltion capabilities of big learning when gradually increasing the ratio from (joint generation) to 0.9, from left to right. Shown in the light-blue boxes of the first row are the masks of zs applied in cach column; white/black indicates S/T. The right-most most column shows ground-truth æ shared in each row. Note each row also employs the same noise. It's clear that the generations become increasingly similar/dissimilar the ground-truth the ratio of S increases/decreasss, as expected. See the category, style, and thickness
MNIST generations as the ratio decreases, as well as identity, expression, hairstyle, and gender of the CelebA generations. Big learning produces realistic and diverse generations/coomlletinss in all situations.
22

---

Under review as a conference paper at ICLR 2023
0,0
1
3
5
5
4
9
8
2
4
7
2
9
5
3
3
5
5
3
9
8
0.1
7
2
/
0
4
/
4
9
5
7
0
0
7
0
1
6
7
5
4
0.2
7
2
/
0
4
4
9
5
0
6
9
0
1
1597
3
4
0.3
7
/
0
4
9
9
0
6
9
55977
3
4
0.4
7
/
0
4
4
9
5
9
0
6
9
0
159759
3
4
0.5
2
6
4
1
4
9
5
9
0
6
9
0
9
4
0.6
7
7
2
/
6
4
1
4
9
5
9
0
6
9
0
]
5 9
8
4
0.7
7
2
/
6
4
4
9
9
0
6
9
0
]
5
7
8
4
0.8
7
2
/
0
4
1
4
9
9
0
6
9
0
]
5
7
8
4
0.9
7
2
/
6
4
4
9
9
0
6
9
0
1
5
9
7
8
4
GT
7
2
/
6
4
4
6
9
0
6
9
0
]
5
9
7
8
4
Figure 11: More MNIST generations/coommlliion from big learning when gradually increasing the ratio of from 0.0 0.9.
23

---

Under review as a conference paper at ICLR 2023
0.0
113
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
FHATETAL
GT
Figure 12: More CelebA generationsscoomlltiioss from big learning when gradually increasing the ratio of from 0.0 0.9.
24

---

Under review as a conference paper at ICLR 2023
6
6
22
0
8
117700170
4
6
077700510
22 | 656
3
6
0
66
3
3
3
6
4
2
27
2
(
8
6
6
L
3
Q
4
D 6 9 O O 9 S 9 O
2
6
6
3
3
4
5
4
/
8
9
/
/ /
4 4 9
/
/
O
8
9 9 9
/
/
9
O
O 7 OOO
9
O
9
O
O
4
O O O O O
8
0
O
O
O I 7
O
9
9
4
4 4 6
4 9 9
/
6
4
7
4
4
6
666
5
6
//
3
/
/
5
5
S
/
8
H
/ 5
5
3
/
4 9
9
7
4
4
8
7
1 9 9 7
4
4
4
9
9
9 4
9 7
4
4
4
4
4
8
9 9
7
4
44
8
8
5
9 8
7
9
9
7
O
7 8
7 7 7 L 1 2
7 7
7
9 9
7 >
7
7
9
9
1999
7 7
P
1
8
5 5993
3 O
9
8 8
4 4
2 L 3
3 2 R
C. 3
3
8
8
3
3
7 2
4
S
4
7
7 7 9
4
7
/
9
2
7 9
4 4
9
9
9
9
9
S
9
7
7
9
7
H
S
(a)
(
(c)
(d)
Figure 13: The diverse generations/completions big learning with (a)(c) various settings and (b)(d) different noises. Shown in red boxes are either the ground-truth images æ or the source ass. Big learning delivers diverse realistic generations w.l. different S/noise settings.
25

---

Under review as a conference paper at ICLR 2023
52
-
48
3 333
-22
3 3
_
3
23
22
A
22
2
222
222322
6
7
4
A
7 7
T
2
2
2
232
222
7
9
7
4
2
9
9
2
2
2
222
223
97
9
9
0
2 2
2
2
6
6
6
3
132
9
9
0
34 9 9 9 4
4 4
3 34433
235
1
999
0
Q
0
44
/
9
3 0
2223
223
1
9
999
9
0
O
O
4
/
4
333
222
999
9
0
4
22 125
0
e
44
2222222
O
G
222 222
A
0
G
222 2/2
e
G
22
U
2 222
4 444
U
E
75 232
444
0
0
5 232
444
O
O
0
33 22
6
3
0
0
2
2
333 222
699
6
3
3
0
0
0
2
2
3 2 222
222
197
3
3
0
0
0
5
3
3 333 22
999
3
0
O
O
/ 777
B
32 22
3
L L
7
2449
9
333 222
333
Q. L
9 7 7
4 4/4
9
9 9
233 222
333
L 2 0
4 1/44 4
9
9 9
223
333
2
4 4 4
9 9
22 22
33
733
8
R 5 3
7
77
222331
333 333
8
8
B
6 $ 6
7
223
33 3
8
8
H
5
6
5 4
7 3
223
333333
8
8
B
S
5
2.2 22
66 3
J
222
6663
V
222321
6663
33
232
366337
4
2.2 22
33
9
9
4
4
222
224
333
464
9
9
9
9 9
9
22
2
33
3
609
9
9
9
9
9
9
4
4
4
2 2
2
3 2
3 3 3 (b)
6
4 6
9 3
7
4
4
4
9 9
4
4
4
(a)
(C)
(d)
Figure 14: The strong generalization capability of big learning w.r.f. anomalous testing cases of the training domain. Big learning generalizes well on ægs that are constructed with (a) random center patches replaced in the upper-left corner, (b) random center patches replaced in the upper part, (c) random center patches duplicated and replaced in the center, and (d) random patches and more complicated manipulations (including duplication, relocation, and mix-up).
26

---

Under review as a conference paper at ICLR 2023
(b)
Figure 15: The strong generalization capability of big learning w.r.f amomalous/unseen lesting cases out of the training domain, on (a) CelebA, (b) Flowers, and (c) MetFaces. Big learning generalizes well on æs constructed by (a) mixing-up up patches from different CelebA images, (b) sampling oftmain image patches from the Flowers dataset, and (c) sampling out-of-domain image patches from the MetFaces dataset.
27

---

Under review as a conference paper at ICLR 2023
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
GT
Figure 16: Out-of-domain generations/completion from big learning the Flowers, when gradually increasing the ratio of from to 0.9. The tested model is big-learned on the CelebA.
28

---

Under review as a conference paper at ICLR 2023
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
F 8
GT
Figure 17: Out-of-domain generations/commlltionss from big learning the MetFaces, when gradually increasing the ratio of from 0.0 to 0.9. The tested model is learned on the CelebA.
29


Reviews:
[{"review_id": "nX-mZieNEg", "reviewer": "Reviewer_ZM7W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper attempts to unify many different machine learning models by re-stating them as ‘big learning’. The authors claim that this paradigm is better than existing methods since it can handle incomplete data. They conduct experiments on conditional image generation.\n\n", "summary_of_the_review": "The contribution of this paper is unclear - while they attempt to unify existing methods under their big learning paradigm, they do not demonstrate concrete theoretical or practical benefits of doing so. The paper also contains many unsubstantiated claims.", "strength_and_weaknesses": "Strengths:   \n1. The paper reviews various training objectives in literature and shows how they may be restated under their ‘big learning paradigm’. \n2.  They conduct experiments on image completion that produce good images completions (though they do not provide any metrics to compare with existing results). \n\nWeakness:\n1. The main goal of the paper is unclear. Are the authors proposing a new theoretical paradigm to analyze foundation models? If yes, there are no stated proposals for how their phrasing gives us more insight into existing methods than the existing phrasing. Are they proposing a new experimental paradigm? Is yes, the authors need to clearly state what the downstream goal and use of their experimental paradigm is.  \n2. Unsubstantiated claims:   \n- The authors claim that their big learning setup gives you more downstream applications than joint modeling (table 1), but they do not state any particular way to test this claim. For example, in their conditional joint modeling experiments, they show that you can generate images from many different ratios of missing pixels, but they don’t show what the downstream use of this capability is? For instance, they don’t compare to joint modeling methods to show that this method learns better representations for classification.  \n- “potentially delivers all joint/conditional/marginal data capabilities after training” it is not at all obvious that a model trained on many diverse datasets will be able to model all distributions and this has not been shown via experiments in their paper.  \n- The paper makes some vague claims like “big learning behavior closely resembles the fundamental unconscious mind and the vision system of human brains”. \n- They claim that “big learning is what foundation models are implicitly doing” but again they don’t lay out any clear testable hypothesis for this claim. Which behavior exactly are they referring to, and why is big learning any more explanatory than the loss function of existing foundation models like autoregressive modeling.   \n\n\nMinor comments. \n1. Table 1 is unclear. For eg: what do authors  ean that ‘capabilities after training’ are ‘joint’? What is the basis to claim that downstream applications are limited for joint modeling vs. abundant for unsupervised big learning?   \n2. The paper does not provide any metrics for measuring the quality of results in their paper.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lUG1RYQ-73B", "reviewer": "Reviewer_qcrZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed \"big learning\", a framework for understanding and modeling complex probability distributions. Big learning is defined to be a modeling of all possible factorizations of a given probability distribution, with all conditionals, marginals, and joints fully learned. The paper shows that existing modeling frameworks fit under the umbrella of big learning as special cases. Experiments with masking show compelling conditional generation results.", "summary_of_the_review": "The paper presents a new framework, but fails to demonstrate its value, both theoretically and experimentally. Many arguments are described with text, but ultimately are not proven or substantiated.", "strength_and_weaknesses": "Strengths\n1) The paper approaches an interesting idea. Understanding learning problems as specific instantiations of modeling the complete joint distribution over data is a compelling goal, and the basic formulation of learning all P(S|T) for all subsets S,T makes intuitive sense.\n\nWeaknesses\n1) The paper ultimately fails to prove the value of its proposition. The proposed framework is never explicitly defined, and seems to have a varying definition to fit the argument being made at any point in the text. It appears to encompass \"all possible probability distrubtion modelling\", but its not clear how this generalization is valuable.\n\na) 3.2 Discussions has a lot of strong claims that are not well substantiated. \nhow exactly can you share one model? this needs to be explicit: how does p_\\theta(x_T|x_S) learned for all S,T yield all joint/conditional/marginals that are close to q(x_T|x_S) for a specific S,T? The empirical argument is fine but does not justify this new “big learning” paradigm unless it provides some new theoretical advantage or insight.\n\nb) Why or how does the framework generalize past the transformer/ViT models? Because they are “data/information hungry” is odd, it's not clear what the authors mean by this. Not enough evidence is provided that “big learning” generalizes past the transformer/ViT/BERT models presented. \n\nc) “model capacity likely not an issue” the discussion here seems unnecessary if the assumption is that the model is \"big enough\".\n\nd) How is this different from simple masking or conditional generation? what new insight should a reader be getting?\n\ne) weighting massive data tasks:\nthe optimum is the same regardless of different weighting strategies? how? The problem is still highly nonconvex, how is it guaranteed that the optimum will be the same if the training procedure is different via different sampling?\n\nf) Eq.6 looks like it is exactly making some assumption about the factorization of the distribution. How does this fit into the model of learning all joint/conditional/marginals? If I wanted to sample from p_\\theta(X_T’|y_S’), how would I do that?\n\n\ng) How is this different from masking/conditional generation that already exists with different mask levels? All experiments seem to be showing various masking results with no comparison to other frameworks, and do not seem to provide much new observations that are only possible because of “big learning”.\n\n\nNotes/Other\n\n1) The structure of the arguments are difficult to follow. It's hard to see what is a technical result and what is just discussion. It would benefit significantly from definition/theorem/proposition blocks.\n\n2) Figures 1 and 2 do not add much over the text, and table 1 is very vague and not well substantiated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7XdKRa533A", "reviewer": "Reviewer_hoLo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work provides a new method named “Big learning” that provides a theoretical platform for analysing, justifying, and improving foundation models such as BERT. ", "summary_of_the_review": "This paper presents a new method named Big Learning that attempts to improve upon popular NLP models such as BERT by exploiting the information inherent in large scale data through modelling marginal data distributions with a single universal model. \n\nOverall, although the proposed method is novel,  the paper is poorly written ambiguous as the problem is not simply and clearly defined. Several different words are used to describe the same thing which might lead to confusion and misinterpretation by the reader. Although the mathematical expressions are clearly defined, this gets lost in the unclear text. ", "strength_and_weaknesses": "Strength: The mathematical descriptions are well defined. Also, the proposed Big learning method has an advantage of handling incomplete data and missing values. \n\nWeakness : The abstract is poorly written, ambiguous,  and does not describe the problem. For example, the statement “ Recent breakthroughs based on big/foundation models reveal a vague avenue for AI, that is, big data, big/foundation models, big learning, · · · .”. Exactly what avenue is being explored here? Also, decide on a single term to describe the problem and not include multiple terms such as “big data, big/foundation models, big learning”.  This leaves a lot of room for confusion and misinterpretation.  This happens throughout the rest of the paper as well. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject