Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propose a new LM architecture that trains branches of LMs on different dataset domains and ensemble the results by taking a weighted average of the output logits or the model parameters. Adding a new LM branch can be done by first taking a weighted average over the existing weights, and then training on the new domain.
strength_and_weaknesses: The paper proposes a promising modification to the current LLM training scheme. Specifically, I found the way the authors combine the outputs with domain posterior interesting. However, I have the following concerns regarding the proposed method and its evaluation:

- The technique of ensemble of many deep learning models is a well-known idea in deep learning. The paper uses a standard ensemble method with limited algorithmic innovations. Please highlight the difference between your work and previous works on
- The results in Table 1 are not fair because different models have different numbers of parameters. A more meaningful experiment is to compare the accuracy of the ELM model with a Transformer-LM with the same number of parameters and check whether the ELM model can match the performance with less training cost, or can outperform Transformer-LM under the same training budget.
- In addition, one can also ensemble the baseline Transformer-LM across different checkpoints and this can also boost the final inference performance. How does this baseline compare with the proposed method?
- The study on the speed/efficiency of the proposed method is not complete. Table 2 only provides the per-step speed during training. Please provide a more detailed study on the convergence/inference speed of the expert-based LM.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. The result should be able to be reproduced with the code provided. However, I do have concerns on the novelty of the paper as indicated above.
summary_of_the_review: I have concerns about the novelty of the work and the experiments to evaluate the proposed idea.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces BTM training, a new algorithm to train an ELMForest, which contains many expert LMs that can be ensembled or parameter averaged at any time for efficient training and rapid customization. 

In the training process, it first trains a seed model and then make k copies of the model, and train the k models independently, which makes the training process highly parallelizable. 
strength_and_weaknesses: Strength
1. The experiments are very extensive, the ELMforest ensembles outperform baselines at no additional training cost.

Weekness
1. The evaluation for the parameter averaged ELMforests seems not fair, given that during evaluation the parameter averaged model can activate domain related experts on the test set. The hidden assumption is that the consecutive sentences in the test set are from the same domain. 
2. If the parameter average approach doesn't give reasonable performance, the inference cost can be very high if we want to gain good performance. 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and understand. The idea of training multiple LMs and then assemble them as a forest seems not new. The code will be released, so the reproducibility is good.
summary_of_the_review: The paper proposes a new way to train ensemble of neural language models, in order to reduce the model inference cost, the paper proposed using domain posterior to activate only related trained experts. The assumption is not used in other models, so it seems unfair.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces BTM training to train a collection of Expert LMs that built an ELMFOREST. Each component in the forest can be dynamically added/removed/ensembled (merged) at any time. The idea of the method is to train scalable and parallel individual models that each specialize in a particular domain. The paper presents comprehensive experiments and analyses. The study improves in- and out-of-domain perplexities as compared to GPT-style transformer LMs.

Main contributions:
- Propose a new framework to efficiently train language models
- Evaluate the proposed method on various datasets (in- and out-domains)
strength_and_weaknesses: **Strengths:**
-  Interesting approach of learning models in parallel and running on multi-node. The paper introduces a new learning framework to train multiple models on different domains and merge them.

**Weaknesses:**
- The evaluation is limited to language model perplexity. The work would be more complete to have the results on other downstream tasks. Did you have any chance to evaluate the model on downstream NLP tasks? And does the model have the same capability as standard GPT models (i.e., in-context learning or generation tasks)?
- The motivation is unclear on why the authors proposed to branch-train-and merge. Why do you choose to branch the LMs? How can they be effective?
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
- The paper is well-written and the method is clear. 
- Missing details: On Table 1, the authors only show the standard deviation on the smallest model. Can you also put the number on larger models? and did you run on 8 different seeds as well?

**Quality:**
- The writing is good. No critical typographical issues.

**Novelty:**
- The proposed method has novel ideas to branch and grow a forest by training models. 

**Reproducibility:**
- The authors plan to release the code.
summary_of_the_review: The paper has new ideas for learning efficient language models in a parallel and non-synchronized manner. And the proposed method can be effectively used to build language models via ensemble.

I would gladly see the paper at the conference and give a "marginally above the acceptance threshold" score. However, there are some questions and concerns, and I will adjust the score if the authors address my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is on build language models via embarrassingly parallel training mechanism. More specifically, the author propose a branch train and merge method, which could build the large-scale language models by independently training subparts of a new class of LLMs on different subsets of the data. Such a way is able to eliminate the massive multi-node synchronization currently required to train LLMs. The results show that this training paradigm could be used to efficiently scale LLMs. 

strength_and_weaknesses: Strength:
The studied topic is at the core of community's interest. How to scale the large scale language models  with reduced costs is important for many tasks. The branch, train and merge framework has a clear design and the empirical studies supported the design idea and indicate the problems that the authors has faced. The proposed method is validated on the large scale datasets and suggest the effectiveness of the paradigm.

Weakness:
1. Some of issues still remain unknown and insights behind this design is also lacking. For example, why the model need to branch from some LMs on the similar domains? Why does the merging of models trained on different domains lead to better performance rather than the models trained on random splits?
2. The trained LLMs are not evaluated on the downstream tasks. The evaluation of downstream tasks are useful to understand the effectiveness of this training paradigm. GPT style models usually have strong few-shot abilities and strong generation abilities. Look forward to seeing the corresponding results to support the effectiveness claims. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow.
The paper has its merits and shows important findings. However, some findings will be more valuable if insights and hypothesis behind the findings could be provided.
The novelty is clearly shown. Even thought merging the weights of pre-trained model is explored, the pre-training design based on weights merging is still under explored. 
The reproducibility could be validated based on the released checkpoints. 
summary_of_the_review: The authors design an elegant and impactful pre-training design, which could potential change the current pre-training mechanism and reduce the training costs. The findings are valuable and will be more impactful if more insights and hypothesis could be provided. The performance on the downstream tasks are missing. The paper quality will be further improved if the downstream task experiments could be added.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to train an ensemble of LMs based on some labeled division of the dataset. 

The paper proposes to train different LMs on different domains and merge them as a model ensemble. 

Reviewers recognized that the paper contains extensive experimental efforts. However, concerns are raised about the fairness of the experiments (which was not addressed in the author response). Moreover, reviewers find that model ensemble does not seem to be novel.

justification_for_why_not_higher_score: Lack of novelty and unfair comparison.
Decision: Reject