Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper attempts to unify many different machine learning models by re-stating them as ‘big learning’. The authors claim that this paradigm is better than existing methods since it can handle incomplete data. They conduct experiments on conditional image generation.


strength_and_weaknesses: Strengths:   
1. The paper reviews various training objectives in literature and shows how they may be restated under their ‘big learning paradigm’. 
2.  They conduct experiments on image completion that produce good images completions (though they do not provide any metrics to compare with existing results). 

Weakness:
1. The main goal of the paper is unclear. Are the authors proposing a new theoretical paradigm to analyze foundation models? If yes, there are no stated proposals for how their phrasing gives us more insight into existing methods than the existing phrasing. Are they proposing a new experimental paradigm? Is yes, the authors need to clearly state what the downstream goal and use of their experimental paradigm is.  
2. Unsubstantiated claims:   
- The authors claim that their big learning setup gives you more downstream applications than joint modeling (table 1), but they do not state any particular way to test this claim. For example, in their conditional joint modeling experiments, they show that you can generate images from many different ratios of missing pixels, but they don’t show what the downstream use of this capability is? For instance, they don’t compare to joint modeling methods to show that this method learns better representations for classification.  
- “potentially delivers all joint/conditional/marginal data capabilities after training” it is not at all obvious that a model trained on many diverse datasets will be able to model all distributions and this has not been shown via experiments in their paper.  
- The paper makes some vague claims like “big learning behavior closely resembles the fundamental unconscious mind and the vision system of human brains”. 
- They claim that “big learning is what foundation models are implicitly doing” but again they don’t lay out any clear testable hypothesis for this claim. Which behavior exactly are they referring to, and why is big learning any more explanatory than the loss function of existing foundation models like autoregressive modeling.   


Minor comments. 
1. Table 1 is unclear. For eg: what do authors  ean that ‘capabilities after training’ are ‘joint’? What is the basis to claim that downstream applications are limited for joint modeling vs. abundant for unsupervised big learning?   
2. The paper does not provide any metrics for measuring the quality of results in their paper.   

clarity,_quality,_novelty_and_reproducibility: Clarity: The papers goals could be stated more clearly
Quality: The paper does not benchmark any results or provide clear downstream applications
Novelty: The experiments in the paper employ existing methods with some modifications 
summary_of_the_review: The contribution of this paper is unclear - while they attempt to unify existing methods under their big learning paradigm, they do not demonstrate concrete theoretical or practical benefits of doing so. The paper also contains many unsubstantiated claims.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposed "big learning", a framework for understanding and modeling complex probability distributions. Big learning is defined to be a modeling of all possible factorizations of a given probability distribution, with all conditionals, marginals, and joints fully learned. The paper shows that existing modeling frameworks fit under the umbrella of big learning as special cases. Experiments with masking show compelling conditional generation results.
strength_and_weaknesses: Strengths
1) The paper approaches an interesting idea. Understanding learning problems as specific instantiations of modeling the complete joint distribution over data is a compelling goal, and the basic formulation of learning all P(S|T) for all subsets S,T makes intuitive sense.

Weaknesses
1) The paper ultimately fails to prove the value of its proposition. The proposed framework is never explicitly defined, and seems to have a varying definition to fit the argument being made at any point in the text. It appears to encompass "all possible probability distrubtion modelling", but its not clear how this generalization is valuable.

a) 3.2 Discussions has a lot of strong claims that are not well substantiated. 
how exactly can you share one model? this needs to be explicit: how does p_\theta(x_T|x_S) learned for all S,T yield all joint/conditional/marginals that are close to q(x_T|x_S) for a specific S,T? The empirical argument is fine but does not justify this new “big learning” paradigm unless it provides some new theoretical advantage or insight.

b) Why or how does the framework generalize past the transformer/ViT models? Because they are “data/information hungry” is odd, it's not clear what the authors mean by this. Not enough evidence is provided that “big learning” generalizes past the transformer/ViT/BERT models presented. 

c) “model capacity likely not an issue” the discussion here seems unnecessary if the assumption is that the model is "big enough".

d) How is this different from simple masking or conditional generation? what new insight should a reader be getting?

e) weighting massive data tasks:
the optimum is the same regardless of different weighting strategies? how? The problem is still highly nonconvex, how is it guaranteed that the optimum will be the same if the training procedure is different via different sampling?

f) Eq.6 looks like it is exactly making some assumption about the factorization of the distribution. How does this fit into the model of learning all joint/conditional/marginals? If I wanted to sample from p_\theta(X_T’|y_S’), how would I do that?


g) How is this different from masking/conditional generation that already exists with different mask levels? All experiments seem to be showing various masking results with no comparison to other frameworks, and do not seem to provide much new observations that are only possible because of “big learning”.


Notes/Other

1) The structure of the arguments are difficult to follow. It's hard to see what is a technical result and what is just discussion. It would benefit significantly from definition/theorem/proposition blocks.

2) Figures 1 and 2 do not add much over the text, and table 1 is very vague and not well substantiated.
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is quite readable, but the arguments presented are not clear. Many points are made without sufficient explanation or proof. Many comparative sentences just say it is “more” or “better”. Better than what? Concretely, theoretically, empirically, how is it better and what is it better than? There is a lot of language but not sufficient concrete evidence to support the claims made.

Quality
The paper has typos, and because of the poor evidence for the claims fails to meet a high quality standard.

Novelty
The paper claims extreme novelty, but does not justify it. The current presentation does not seem novel.

Reproducibility
I have not gone through the appendices in full detail, but experimental details are provided along with more discussion of some points made in the main paper.
No code is provided to replicate the results presented in the experiments section.
summary_of_the_review: The paper presents a new framework, but fails to demonstrate its value, both theoretically and experimentally. Many arguments are described with text, but ultimately are not proven or substantiated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work provides a new method named “Big learning” that provides a theoretical platform for analysing, justifying, and improving foundation models such as BERT. 
strength_and_weaknesses: Strength: The mathematical descriptions are well defined. Also, the proposed Big learning method has an advantage of handling incomplete data and missing values. 

Weakness : The abstract is poorly written, ambiguous,  and does not describe the problem. For example, the statement “ Recent breakthroughs based on big/foundation models reveal a vague avenue for AI, that is, big data, big/foundation models, big learning, · · · .”. Exactly what avenue is being explored here? Also, decide on a single term to describe the problem and not include multiple terms such as “big data, big/foundation models, big learning”.  This leaves a lot of room for confusion and misinterpretation.  This happens throughout the rest of the paper as well. 
clarity,_quality,_novelty_and_reproducibility: The paper presents a novel method as it aims to improve foundation models such as BERT by providing a theoretical framework that explains the success of the foundation models. The authors do not share the code and data used in the experiments. As a result, this may be difficult to reproduce. 
summary_of_the_review: This paper presents a new method named Big Learning that attempts to improve upon popular NLP models such as BERT by exploiting the information inherent in large scale data through modelling marginal data distributions with a single universal model. 

Overall, although the proposed method is novel,  the paper is poorly written ambiguous as the problem is not simply and clearly defined. Several different words are used to describe the same thing which might lead to confusion and misinterpretation by the reader. Although the mathematical expressions are clearly defined, this gets lost in the unclear text.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new framework called "big learning" which unifies the training of large-scale transformer models. The authors show that most of the commonly used objectives are special cases of big learning and that big learning can model the joint distribution better.

However, all three reviewers think that the claims in the paper are overstated and are not clear. This paper would benefit from a complete revision based on the reviewers' feedback. While the authors made a revision, the revision is not highlighted by a colour change and hence it is not easy for the reviewers to read the whole paper once again.

I recommend the authors work on the claims and writing and resubmit to a future venue. Also, I find the tone of the authors to be a bit rude towards the reviewers. While I understand that it is frustrating that the reviewers did not get what you intend to convey, it also shows that more work needs to be done from the authors' side in terms of explaining your ideas clearly.
justification_for_why_not_higher_score: The paper definitely needs to be rewritten with toned-down claims and needs to go through a detailed review process once again.
Decision: Reject