LLM-based Affective Text Generation Quality Based on Different Quantization Values
Authors: Yarik Menchaca Resendiz, Roman Klinger


Reviewer paper summaries: The paper "LLM-based Affective Text Generation Quality Based on Different Quantization Values" explores the impact of quantization on large language models (LLMs) in the context of affective text generation. The study investigates the trade-offs between precision bits (8, 16, and 32), GPU RAM usage, inference time, and the quality of generated affective text. The research uses two models, LLAMA 2 (three sizes) and Mistral (two sizes), and evaluates their performance using an emotion classifier and text quality metrics through GPT-3.5.

The paper titled LLM-based Affective Text Generation Quality Based on Different Quantization Values mainly discusses the impact of reducing precision bits on large language models (LLMs) for affective text generation. It aims to provide a comprehensive understanding of the trade-offs involved in using quantization with LLMs for affective text generation. The study investigates the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation.

The authors show that reducing bits could lead to a 76% memory saving. However, this optimization has trade-offs: larger models experience up to a 10 percentage point drop in F1 score, while smaller models see a 10 percentage point increase, and inference time roughly doubles. Regarding text quality, larger models at lower quantization levels generally perform better than smaller, higher-precision models, despite needing similar memory.
Summary of strengths: -The study addresses a relevant problem in NLP, particularly the computational resource demands of LLMs and how quantization can mitigate these.
-The paper is well written and easy to follow.
-The paper evaluates the trade-offs using multiple configurations and metrics, including F1 scores, GPU RAM usage, inference time, and text quality, showing that under certain conditions, an increasing quantization is not only beneficial in terms of computational resources but also does not affect latency and text quality.
-The methodology is well-structured, detailing seed prompts, model configurations, and evaluation techniques.

1. Although quantization has been a popular topic in the field of NLP, previous works have not focused on the trade-offs between affective text generation and quantization. This research addresses this gap by investigating the trade-offs between GPU RAM, inference time, and text quality when using different precision bits in affective text generation. This motivation is very reasonable.

2. The paper also employs various prompt templates, models, and model sizes to conduct experiments, exploring not only classification accuracy but also text quality.
Summary of weaknesses: -The effect of explicitly including the category to be predicted in the generated text over the F1 scores is not addressed. e.g. If for the category “Anger” the generated text includes adjectives such as “angry”, it might be that the classifier would have an easy task compared to a situation where the emotion is not stated explicitly in the generated text. Probably a differentiation between implicit and explicit mention of the underlying emotion might benefit the study by providing more clarity. I would hypothesize that a simple look up function could also achieve high scores for the explicit scenarios.
-The study focuses on a specific set of models (LLAMA 2 and Mistral) and may not generalize to other architectures.
-The evaluation is based on a single emotion dataset
-Very often the results don’t give a clear direction in terms of the benefits or disadvantages of quantization in terms of text quality, F1 score or GPU requirements. Some additional analysis to investigate possible causes of this mismatch could benefit the quality of the study (e.g. potentially the varying proportion of explicit mentions of the underlying emotion in the generated text).
-While the paper is well documented, it is difficult to reproduce the experiments if the code base, version of the models and version of the dataset are not provided/made publicly available.

1. The authors use GPT-3.5 for text quality evaluation, but this should be further verified by comparing the evaluation results of GPT-3.5 with those from a small-scale human evaluation. Additionally, why not use GPT-4, which is more reliable?

2. The experimental results and figures are not convincing. Conclusions such as "Llama-2-7b’s F1-score drops at the highest quantization level (32), while Llama-2-13b shows variability but generally decreases at mid-level quantization (16)" are unreliable without providing reasons and justifications.

3. The final conclusion, "The analysis suggests that the impact of quantization on LLM performance varies, with some models benefiting from higher quantization levels and others experiencing a decline, guiding the choice of quantization based on desired performance and resource efficiency," is too vague and offers no meaningful insights.

4. The experimental datasets are surprisingly small. 3 sentences are generated for each template, resulting in a total of 30 data points per configuration per emotion. This limited number of data points cannot support strong conclusions.
Comments, suggestions and typos: coherence instead of coherency?
line 172: involved instead of involve?

Use more data.

Explore reasons behind the observations. Do not just describe the results aimlessly.
Overall assessment scores: 3

1