Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework for training Vision Transformers at any given budget by reducing the inherent redundancies of the model at the early training stages.

This paper proposes a training strategy to dynamically adjust the activation rate of the model along the training process.

strength_and_weaknesses: This paper investigates three redundancy factors in Vision Transformers, including attention heads, hidden dimensions in MLP, and visual tokens.

Extensive experiments show the effusiveness of the proposed framework with competitive performances on a wide range of training budgets.
clarity,_quality,_novelty_and_reproducibility: I have no other concers.
summary_of_the_review: This paper take a step forward and focus on the problem of budgeted training and aims to achieve the highest model performance under any given training budget.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents training strategy that allows to training optimal transformers under user-defined budgets.  The budget training strategy is based on investigating the redundancy in attention heads, hidden dimensions in MLP, and
visual tokens. The training strategy could adjust the activation rate of the model along the training process to make use of the redudancies. Extensive experiments have demonstrated effectiveness. 
strength_and_weaknesses: Strength:
1. the analysis of the redundancy in modules of  VT will be very important not only restricted to this work, but potentially many research on designing efficient VT models.
2. The paper is quite easy to follow and all the contents are easy to follow (despite a few typos: Sec 6, effusiveness -> effectiveness?)

Weakness:
1. the motivation could be improved. There are multiple solutions for reducing training cost, they could be but not limited to data set pruning, using smaller models, or more effective training schemes. Using training cost as motivation seems not enough to justify this paper. And this will make readers wonder how this proposed method compared to other treatments. Perhaps one scenario unique is continuous/online training?  
2. more discussions might be needed to convince the effectiveness of the proposed training strategy: 1. how it compares to different original training schedules, e.g., what if we reduce the original training epochs from 300 to 150, using some quick schedule (cosine learning rate)?
clarity,_quality,_novelty_and_reproducibility: This paper is clear written, All the experiments are well described and should be straightforward to reproduce. 
summary_of_the_review: See strength/weakness
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method to train a Vision Transformer given a budget defined by a total training time or computation cost. Training a Vision Transformer from scratch is usually expensive, so designing a budgeted training is a way to make the training a Vision Transformer more accessible. The paper analyzes and identifies some redundancies in the attention heads, MLPs and visual tokens of Vision Transformer architectures. The redundancies are usually high at the beginning of the training and then gradually decrease. To make training faster, the paper introduces a 3 phase training strategy, where the training starts with a small architecture and the architecture grows after each phase. The proposed method is evaluated on ImageNet with multiple Vision Transformer architectures.
strength_and_weaknesses: **Strengths**
- To reduce the training cost of Vision Transformer, the paper identifies some redundancies in the architectures. Section 3.2, 3.3 and 3.4 analyzes the redundancies in the attention heads, MLPs and visual tokens of Vision Transformer architectures. For each part, an empirical analysis performed on a subset of ImageNet is shown. The analyses show that the redundancies are usually high at the beginning of the training and then gradually decrease.
- After identifying some redundancies, the paper introduces a 3 phase training strategy that takes advantage of the redundancies. The training starts with a small architecture and the architecture grows after each phase because small architectures can be trained faster than large architectures and large architectures have more redundancies.
- I like the research area on budgeted training. I think that making from scratch training more accessible/affordable is positive for the community because more research labs will be able to develop new model architectures. I think it can also help to reduce the environmental impact of ML model training.
- The proposed method is evaluated on ImageNet with multiple Vision Transformer architectures (DeiT, Pyramid Vision Transformer v2 (PVTv2), and Swin Transformer). The proposed training strategy improves the performances for several training budgets (25%, 50%, 75%, and 100%). The paper also contains ablation studies of the important hyper-parameters of the proposed method. 


**Weaknesses**
- The performances are evaluated only on ImageNet. It could be interesting to evaluate the pre-trained models on downstream tasks to analyze the transfer learning performances because it will increase the quality of the paper.
- The exponential function used to define the training budget for each phase does not seem well justified.  
- The proposed training strategy has 3 phases but there is no justification about this design choice. It could be interesting to show results with more phases.


clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and understand. Making training more efficient is an interesting research area and can have a lot of impact. The proposed method seems novel and shows good results for several architectures and training budgets. There are some implementation details in the appendix so it should not be too difficult to reproduce some experiments. 
summary_of_the_review: Overall, I think the paper is well motivated and is about an interesting research area. The proposed method seems novel and improves the performances in multiple scenarios but the experiment section could be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at the interesting budgeted training problem: How to train models under a given budget (training time/compute) to achieve a good accuracy under such constraints. The paper takes the perspective of identifying redundancies in the attention heads, MLPs, and visual tokens of Vision Transformer architectures and uses this insight to suggest a progressive growing mechanism used across a three-phase training regime. 

The reviewers all appreciated the writing clarity, problem setting (relevant given the current growth of compute in training these models), analysis demonstrating redundancy in different components, and evaluation across multiple architectures/ablations showing benefits across different budgets. Some suggestions were made to improve various aspects of the paper, including improving the narrative of the motivation to include discussion of other potential approaches and the angle this paper takes on the problem, comparison to some of the other approaches (e.g. pruning), comparison of different learning schedules, and demonstration of generalization across additional downstream tasks. The authors provided an extensive rebuttal, including significant new experiments addressing these, showing that the method is indeed general to other vision tasks. 

  Overall, the problem of budgeted training is extremely important from a variety of practical, research, and societal aspects and this paper takes an interesting perspective that is well-motivated through analysis to yield an effective method. Therefore I recommend acceptance. I recommend that the authors update the manuscript to include all of the discussion items, especially the new experiments, in the camera-ready version, as they significantly add to the demonstrated effectiveness and generality of the method.
justification_for_why_not_higher_score: This paper provides a nice analysis and method for budgeted training, which would be of interest to the community. The paper could be accepted as spotlight although I don't believe that the scores and strength of the paper necessarily justifies that. 
justification_for_why_not_lower_score: Overall, all of the reviewers appreciated this paper's contributions, especially after the rebuttals which included significant additional experiments showing the generality of the method. The two positive reviewers confirmed that they vote for acceptance.
Decision: Accept: poster