Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The Author's investigation reveals critical insights into the gap of uniform convergence for analyzing pre-trained representations, their stochastic nature under gradient descent optimization, what model convergence means to them, and how they might interact with downstream tasks. Authors propose a simple approach which contributes to both applied and theoretical research of representation learning. 
strength_and_weaknesses: Strengths:
1) Authors explain concerns with existing methods clearly.
2) Authors divide each of their theoretical results and give a takeaway summary for better understanding for readers 
3) Authors back their theoretical claims with empirical results 

Weaknesses:
1) Setting considered in the paper is too simplistic to be broadly used in the industry. 
2) More connection between theoretical results and empirical results is not discussed well enough. 

clarity,_quality,_novelty_and_reproducibility: Paper is clearly written and the contribution is novel. 
summary_of_the_review: 
1) In section 3, why is \Theta d x k when y is just a scalar? How to make sense of loss in this case?

2) Section 4.2 "instability of ˆh(x): the exact position of ˆh(x) in Rd is stochastic, depending on the initialization and the order of the pretraining data that is fed to SGD": What if the initializations are same every time? 

3) "In addition to improved stability and performance": How is stability measured in Table 1?

4) How was hyperparameter tuning done for the main results?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper observed an instability issue of downstream prediction based on features from pretrained models (i.e., the accuracy of a downstream task is varing, which depends on the pretrained representation trained on the same data, model configuration, training setup, and stopping criteria). This empirical observation is theoretically analyzed via a novel uniform convergence bound for pretrained representations, and eventually proposes a solution, called featurizing pretrained representation, which is empirically justified to outperform a baseline (i.e., a downstream task with usual pretrained features) using the MovieLens1m dataset and an in-house real-world production dataset. 

strength_and_weaknesses: **Strengths**:
* Provide theoretical interpretations on empirical observations. 
* A solution to address the claimed instability issue is simple.

**Weaknesses**:
* The main empirical observations (Figure 1) are made in weak setups
* Proposition 1 is not necessarily true. 
* The experiment result is weak

(Weakness 1)

The paper first made empirical observations on MovieLens-1m along with the traditional Doc2vec model as a pretrained representation, along with logistic regression for a downstream task. Considering a huge advance on a large language model, this experiment setup does not represent the current state of representation learning. If this observation is made with IMDB (or any larger dataset) along with BERT (or RoBERTa), the story would be much stronger. 

Moreover, the following statement needs to be more carefully explained: 
“Since we use logistic regression as the downstream model, the fluctuation can only be caused by the instability of pretrained representations.”
Why logistic regression cannot be the source of the instability? In particular, do you mean the optimization for the logistic regression is done until it converges to the global optimum?


(Weakness 2)

Proposition 1 implies that the uniform convergence bound for pretrained representation has an irreducible term, contributing to the instability of the downstream task performance. However, I think Proposition 1 is not necessarily true. In particular, (A.1) decomposes the generalization bound, and the proof claims some terms are non-negative, which is not always true. 
Consider the second term in (A.1), where the proof said “by definition, the second term is non-positive “. First, I think "non-positive" is a typo as if it is non-positive, it can cancel out the irreducible term, thus the claim does not make sense; so I assume that the term is "non-negative". Here, h* is the optimal representation w.r.t. the expected loss R(h, f) instead of the empirical loss, thus the second term can be negative. In other words, if $h^* = argmin_h \sum_i \ell(f \circ h(X_i), Y_i)$, the second term cannot be negative; but the if statement does not hold. Additionally, the non-negative argument on the last term is the same issue. 

This proof and the affected arguments in the following sections need to be fixed and adjusted. 


(Weakness 3)

I liked the experiments on real-world production experiments, which demonstrates how knowledge in academia is actually used in practice. But, to justify the efficacy of the proposed approach, it is always good to evaluate multiple public datasets, but currently only one is used. 

For the ML-1m, I’m not sure if this is a widely acceptable dataset in representation learning; the main reason is that its input dimension looks not large enough (e.g., based on the paper, saying “each movie is provided with its title and genre, in the form of English words or sentences. There are 18 genres in total.”). Based on this limited information, classifying five ratings sounds very challenging (contrast to this, a human rates each movie based on richer information). I personally think this unusual property of the dataset may lead to the unstable results in Figure 1. Appendix F1 includes some results on IMDB, but I cannot see the downstream accuracy results as in the figure in page 2. 

In short, evaluating the proposed approach at least 2 widely acceptable datasets would make the experiment results stronger. I guess IMDB's downstream accuracy results need to be presented to strongly show the observed instability issue. Moreover, I’d recommend using features from BERT or RoBERTa for pretrained model features. Based on Table 1, the featurting pretrained representation looks promising, and hoping that the same trend holds for other datasets and representations. 

clarity,_quality,_novelty_and_reproducibility: (clarity) I think the paper is well written. It would be better if details on the data is provided (e.g., the dataset size), as the stability issues may be related to the small number of samples. 

(quality) If my concerns are addressed, the paper quality is good as the main claim is supported by both theories and empirical results.

(novelty) I think the claim could be novel if my concerns on the proof are addressed. 

summary_of_the_review: I like the way of this paper to attack the observed representation instability; but the paper does not follow the actively changing trends — large language models are widely used, but didn’t account for this line of work (including in the related work section). Moreover, as mentioned in the weaknesses, I have concerns on the proof that supports the paper’s main contribution. Thus, I vote for rejection for now, but I’m willing to adjust my understanding. 


=== After the rebuttal

Thanks for providing details.

For the Proposition 1 proof, it needs to provide that the slack term does not disappear for some f and h. In particular, the generalization error in Proposition 1 is decomposed in three terms in the proof. And each term is *independently* bounded by a uniform convergence bound. However, this does not mean that there exist f and h such that the slack term is not canceled by other terms. Moreover, the argument on the existence of an irreducible slack term may not be achievable via upper-bounding the generalization error since I can take any upper bound and then artificially add some slack term to claim that there is an irreducible slack term. 

I think having a correct proof Proposition 1 is necessary to generalize the paper’s intriguing observation — “In real-world production, however, we have encountered key problems that cannot be justified by existing knowledge. They raise concerns that the naive use of pretrained representation as feature vector could lead to unwarranted and suboptimal solution.” I agree that Proposition 1 is an intermediate step, but this is the basis of the paper’s argument. Otherwise, authors’ need to empirically justify their observations with a large number of experiments with various models and datasets.

For these reasons, I still lean toward rejection, but I would see the value of the proposed algorithms (as heuristic) and empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to solve the suboptimal problems caused by the pretraining models in industrial systems. And the authors analyze the problem with four aspects( the gap of uniform convergence for analyzing pretrained representations, their stochastic nature under gradient descent optimization, what model convergence means to them, and how they might interact with downstream tasks). Then they propose a method to refine the features of pretraining models in various ways to improve the stability of the downstream task model. Experiments on six tasks prove the effectiveness of the proposed method.


strength_and_weaknesses: Pros:

-The problem addressed has high practical value: it tries to make pre-trained model more accessible to a range of industrial systems. The "Featurizing Pretrained Representations" idea will promote the performance of downstream tasks compared to other approaches.

-This paper verifies its thought with a detailed formula. By analyzing the features of the pre-training models, the author proves that the features from the pre-training models are related to the stability of the performance of the downstream tasks. In detail, the variance of the pre-training features h (x) plays an essential role in the learning process of downstream tasks, and the variance will increase with the increase of random variables in the pre-training model. 

-The proposed method (Featurizing Pretrained Representations) is intuitive and effective, which uses a more stable h (x), e.g., the representation of weights in the NW estimator in Section 4.3, in downstream tasks. Moreover, experiments on six downstream tasks indicate that the method has superior performance and stability.


Cons:

-It may have a more suitable h(x). The authors think the optimal h(x) can be obtained through Fourier Transform. But it is not easy to get the representation of the sample by the inverse Fourier transform. So they replaced this method by increasing the number of samples and the dimension of features. Nevertheless, this method may not be the most suitable one in industrial systems due to the limited sample and computing resources. Maybe the authors can seek a better method from the perspective of math or Fourier Transform.
-The analysis is not comprehensive enough. Although the paper proves that the features of pre-training model do affect the stability of downstream tasks, it ignores the features of downstream tasks, which are more essential for downstream tasks. Have the authors tried to analyze the downstream tasks, and what does the result look like?

-The essence of h(x) is still not clear. The paper discussed the impact of the variance of h(x) on downstream tasks, and then the authors improved the performance with the optimization of h(x). However, they did not give any description of the reason why the pretraining models generate such features h(x). Maybe it is more important.

clarity,_quality,_novelty_and_reproducibility: Have met the standards.
summary_of_the_review: Overall, the paper proposed an interesting idea and showed strong empirical results, but it should have a more comprehensive analysis of both pretraining and finetuning. Hence, I would suggest the author complete more related works to give more convincing conclusions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work investigates the challenges that face using pretrained representations in industrial systems. This is an important topic given the shift in deep learning toward leveraging large pretrained models. The authors provide theoretical analysis on the gap of uniform convergence for analyzing pretrained representations, as well as other analysis such as how this interacts with downstream tasks.

After discussion, reviewers recommended reject with the exception of 1 reviewer who did not participate in the discussions. The major concern is around the proof behind Proposition 1, which provides a uniform convergence bound arguing effectively for a "no free lunch” result for using pretrained representations. Reviewers agreed that Proposition 1 is crucial to the work as the rest of the analysis is based on it, and the additional assumptions the authors added after rebuttal still don't resolve the issue. The manuscript is also not updated to reflect any such changes. On the empirical side, one also needs to have variety of datasets and models to back their claims, but their empirical analysis is restricted to one pretrained model's embeddings and one dataset.
justification_for_why_not_higher_score: The paper's writing could be improved. There's also not as strong a connection between empirical practice and theory due to a few simplified settings that they conducted for validation (also raised by multiple reviewers, e.g., the focus on Doc2Vec).
Decision: Reject