Systematic Review of Large Language Models: Applications, Limitations, Practical Usages and Future Directions

Authors: Enoch Solomon, Abraham Woubie Zewoudie
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Systematic Review

Abstract:
Large Language Models have revolutionized natural language processing with their remarkable ability to understand and generate human-like text. This review explores the various applications of large language models, highlighting their versatility across different domains. The paper begins with an introduction to LLMs, followed by an overview of their types and a detailed literature review. We then examine their limitations before delving into specific applications such as text generation, translation, summarization, and more. Finally, we discuss future directions for research and development, concluding with a summary of key findings and the potential impact of large language models on various industries.

Decision: reject
Reviewer scores: 1, 1, 1, 1

Reviews and discussion:
<thread>
creation date:
1730579271760

thread_type:
official_review

<thread_object>
creator: reviewer_4db8

<content>
summary:
This paper provides a brief summary and review of LLM architecture, and its limitations, as well as various application areas (e.g., text generation, translation, summarization, etc), and existing LLM benchmarks. They also discuss future directions for LLM research.

strengths:
1. Provides a summary of LLM architecture, application areas, limitations, practical usage, and future directions. Which could be useful to get a quick idea for new LLM researchers to get the basic idea. However, it is more effective as a blog post and not at all a research survey paper.

weaknesses:
i. This survey does not provide any new novel insights in comparison to what is known already about LLMs (e.g., [1], [2] etc.). It is more of a straightforward summary of the architectures, applications, and limitations of LLMs, lacking in-depth critical review.

ii. Moreover, although the paper title claims the paper as a systematic survey, the discussion on different topics in this review is also superficial. 

iii. Even though the paper mentions LLMs, the discussion is more around typical transformer-based language models like BERT, GPT, and T5 without offering new insights. 

iv. The paper relies heavily on vague citations. Moreover, some of the citations have "?" marks. This demonstrates that the paper lacks attention to detail. Potentially, this paper was written without any comprehensive research.

1. Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X. and Gao, J., 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196.

2. Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z. and Du, Y., 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

v. The discussed limitations and proposed future directions also do not offer anything new. 

vi. Figure 2 also looks pretty bad. The text size in the caption is also very large in comparison to the paper text.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649650944

thread_type:
official_review

<thread_object>
creator: reviewer_kj2v

<content>
summary:
This is a review paper on a broad topic of LLMs regarding the types, applications, and limitations, etc., of LLMs.

strengths:
It is important to provide a review on LLMs in the era of the vast development of LLMs.

weaknesses:
I believe this paper is not appropriate for ICLR. This is not a "systematic" review. The content is superficial and outdated. The insights are not valid.

questions:
Dear author, I think it is more realistic to focus on a certain smaller aspect and conduct a really "systematic" review on that topic. The current paper is not a good review, as the topic is very big and you didn't properly address this field in such a small paper. I would recommend to rethink the scope.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733373817

thread_type:
official_review

<thread_object>
creator: reviewer_1rve

<content>
summary:
This paper presents a literature review on the topic of Large Language Models. The paper characterizes different types of LLMs: Generative Models, Masked Language Models, Sequence-to-Sequence Models, and, Hybrid Models. The survey discusses a range of topics, from "Deep-learning methods and techniques used to develop LLMs" to "Recent developments and Benchmarks".

strengths:
None, as can be seen in the manuscript, Section 3.11 (Comparison with Recent Reviews):
>To provide a more comprehensive overview, we compare our findings with recent reviews on LLMs. Notably, Bommasani et al. (2021) and Zhao et al. (2023) offer extensive analyses of the latest advancements and applications of LLMs, including ethical considerations and deployment challenges. These reviews highlight the importance of continuous benchmarking and evaluation to ensure that
LLMs are developed and used responsibly. **By integrating insights from recent benchmarks and reviews, this section provides a broader perspective on the current state of LLM research, highlighting both the progress made and the challenges that remain.**

That is, this paper does not offer any contribution other than those mentioned in Bommasani et al. (2021) and Zhao et al. (2023).

weaknesses:
* The paper does not present any novel or meaningful contribution.
* The content is outdated.
* The organization, particularly in Section 3 is very poor.
* There are missing or wrongly cited references.
* **The paper feels empty:** Most of the subsections in Section 3 contain a single paragraph with (in the best case) a single reference.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730862298629

thread_type:
official_review

<thread_object>
creator: reviewer_zppz

<content>
summary:
This paper provides an overview of the development, applications, and comparative analysis of Language Models (LMs). It begins by detailing the methodologies used in the construction of LMs. Following this, the paper explores various applications of LMs, and finally, the study concludes with a side-by-side comparison of four LMs, evaluating their strengths and weaknesses.

strengths:
The paper provides an overview of the technologies employed in the development of Language Models (LMs).

weaknesses:
The paper’s objectives are not clearly defined. While it purports to review Large Language Models (LLMs), the models it examines are not currently regarded as large by contemporary standards (such as open models like LLaMa-2, OLMo, etc.) [1]. Furthermore, despite claiming to explore future directions for LLMs, the paper fails to address this topic adequately. For instance, specific future applications of LLMs in new fields or emerging challenges associated with the expansion of LLM could have been explored [2].

[1] Bommasani, Rishi, et al. "On the opportunities and risks of foundation models." arXiv preprint arXiv:2108.07258 (2021).

[2] Li, Sha, et al. "Defining a new NLP playground." arXiv preprint arXiv:2310.20633 (2023).

questions:
I have no questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper attempts to provide an overview of large language modes including architecture, data, training and algorithms. No strengths were highlighted by the reviewers. All reviewers agree that this paper is poorly written, does not offer new insights not previously known, contains outdated information, and contains many typos, missing citations, figures. Reviewers have also raised concerns that it potentially might be LM-generated, and I share that concern.