Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This blog post provides a summary of the ALBERT --- a lightweight and more efficient variant of the BERT architecture that is more amenable to scaling to larger model sizes. In particular, this blog post summarises the three ways in which ALBERT differs from a standard BERT model: (i) factorised embedding, (ii) cross-layer parameter sharing, and (iii) a sentence-order prediction loss. The blog post then proceeds to summarise the key empirical findings that compare the performance of BERT and ALBERT on various dimensions (performance, training time, etc.).

Although this blog post provides a more accessible overview of ALBERT (which is an influential model in representation learning within NLP), it does not offer more additional points and added value that go beyond a more accessible summary of the ALBERT paper. A more detailed assessment of the blog post is provided below:

1. The blog post misses an opportunity to provide a critique---above and beyond a more accessible summary---and relate the ALBERT paper with more recent developments in the field. For instance, the findings of the ALBERT paper (e.g. that scaling BERT to more and more parameters does not always yield better performance) seem to contradict more recent findings on the scaling laws for language models (Kaplan et al., 2020; Brown et al., 2020). In particular, this line of prior work argues that---for left-to-right/generative language modelling---increasing model size almost always leads to better performance at both generative language modelling and downstream tasks, in addition to improved few-shot learning ability. This would be an interesting angle to explore in this blog post: What are the possible reasons for these seemingly contradicting trends? Are masked language models like BERT inherently less able to take advantage of larger model sizes, whereas generative/left-to-right language models like GPT-3 are inherently better suited to do so? Or can BERT instead benefit from larger model sizes as well if they have access to much larger training sets? Acknowledging this point and encouraging more work in this area would have made the blog post stronger.

2. The presentational aspect of the blog post can also be improved. For instance, there are some dubious assertions that "this state-of-the-art architecture [BERT] deals with millions if not billions of parameters ...". To my knowledge, BERT Large only has ~300M parameters (i.e. less than billions). In the conclusion section, it is stated that "ALBERT is computationally more expensive than BERT". This statement warrants more explanation, and even seems to contradict the previous section on "Comparison with training time", where ALBERT can train much faster than a comparable BERT model. Another such statement is on the Section "Comparison with number of parameters", where it is stated that "From the comparison above, it is obvious that ALBERT performs better than BERT!". This statement is a bit strange, because at that point, the blog post had only compared ALBERT's number of parameters vs BERT's number of parameters, whereas the statement "ALBERT performs better than BERT" is only established on the next section "Comparison with Benchmarks". I also found the statement "... is built upon the concept of transformers where every output element is connected to every input element and their weights are dynamically calculated." to be a bit imprecise.

3. Another presentational aspect to improve is that the Tables seem to be taken directly from the ALBERT paper. It would have been better to recreate the tables and highlight only the *relevant* information, because the Latex-style Tables can be harder to read, and may contain extraneous information that is not directly discussed in the blog post text.

For these reasons, I am recommending an overall score of "5", and hope that my feedback can be useful for improving the blog post going forward.
review_title: Review

Decision: Reject