Kati8/ICLR_2023_2025 huggingface dataset:
authors: Haoran Qi
authors: Zehua Wang
reviews: 
Comments: This paper studies the use of text based data augmentations techniques for learning models which can role play as specific characters in diverse scenarios. The paper compares 6 data augmentations strategies on two characters whose content they sample from games as the basis of training Llama 3 using LoRA.
Strengths: - At present, the paper does not offer any clear value to the community. The problem of data gathering and evaluating LLMs ability to roleplay is an important challenge, but the present work fails to engage with the broader literature both on these particular models and more specifically in this research space nor to offer carefully controlled experiments which answer questions from the wider field.
Weaknesses: - The methods studied don't address the core challenges the authors themselves present. While the authors highlight issues that most training data used for this task occurs in a particular thematic framework as a limitation, the data augmentations they use, which are all methods which have been carefully studied in existing work but are not attributed, only perform local word and structure level permutations on data.
- The paper, as written now, spends much more time explaining (somewhat tangential) background than seems necessary. The concrete experimental setup doesn't begin until almost halfway through the work.
\
\
For example, the section on Unsloth in section 2.4 and the inclusion of external model evaluations on benchmarks that are unrelated to personalization in *both* Figure and Table 1 have little to do with the core research questions. These sections distract from the narrative of the paper and should be moved to the appendix to make room for more substantive experiments on the primary topic.
- The work primarily presents a study on just two characters from role-playing that the paper both introduces itself and compares minimally to any external baselines. It is poorly contextualized in the field without evaluations on datasets from the broader community or comparisons to even simple baselines like prompting the model to roleplay.
- There are many signs that the work was minimally refined before submission. Despite having only 11 citations , in these minimal citations there are obvious errors such as listing "Journal Name" as the publication venue and "Unknown Author et al" as the authors (the work in reference has both publicly known authors and a venue of publication: https://openreview.net/forum?id=rc2h1h89aDi). The figures have mislabeled legends (in figure 3 `BT Loss` is repeated twice) and there are typos where content is duplicated at the start of paragraphs (Line 79: "Wei and Zou (2019) Wei & Zou (2019) introduced")
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: The paper explores different data augmentation strategies (i.e., random deletion, synonym replacement, swapping, random insertion, back translation, and paraphrasing) to train personalized AI models. The model is evaluate by using LoRA and two dataset proposed by the authors.
Strengths: None
Weaknesses: - Overall lack of clarity. It is unclear to me what is the major claim of the paper. The paper mention several techniques on data augmentation methods, but none of the it's explained in details nor formally. The results are also mysterious, leaving out most of the details. 
- The results on the paper are non-sensical, the plots are meaningless, showing no particular trends. 
- The paper contains screenshots from other papers (llama3), without a clear citations.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: The paper investigates the relevance of data augmentation techniques for the training of personalized dialogue agents. It gives an overview of existing text augmentation methods, of existing open-weights LLMs and of low-rank adaptation techniques. It introduces a custom dataset based on a TV series and a video game, and a task consisting of enacting a specific character within these environments.
Strengths: # Originality

The proposed approach consists of low-rank fine-tuning applied on a training dialogue dataset with data augmentation. The approach in itself is not particularly novel, and its presentation would benefit from a more thorough literature review.

The introduced dataset is novel, and its elaboration would benefit from a more detailed presentation.

# Quality

Only one set of experiments has been run, using one available open-weight model and the dataset that the paper introduces. This is insufficient to get to sound conclusions as to which data augmentation techniques are better in general for the task of dialogue modeling.

# Clarity

The paper is easy to follow, but misses key details in how the dataset has been constructed: the paper should mention where the data comes from, how it was processed, what are the key characteristics (# of episodes, of tokens, of characters...), how the training/validation split was done...

# Significance

The understanding of which data augmentation techniques are most relevant in order to adapt LLMs to small datasets is of prime importance for the field. However, it seems that the proposed method and the results do not give very strong indications of winning strategies.
Weaknesses: Regarding the proposed dataset:
* It is not clear where the training data comes from and how it was curated
* It is not clear how the proposed dataset relates to existing work, and which specific problems it aims at solving
* There is no measurement of training data leakage. Given their popularity, it seems likely that these TV shows are quite prevalent in LLM training data

Regarding the fine-tuning results:
* There is no comparison with a baseline that does not use data augmentation
* There is no comparison to any few-shot prompting scenario
* It is not clear what the fine-tuning for a specific character consists of: how is the autoregressive loss computed when multiple characters are talking?

Regarding the paper presentation:
* The paper spends several pages presenting topics that are not directly relevant for the subject at hand. While understanding low-rank fine-tuning is a topic of importance, it is not particularly relevant to spend 1 page on it, at the expense of experimental details and results.
* The abstract mentions 3 different datasets, but the results cover only two of them.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 3
reviews: 
Comments: This work investigates various data augmentation techniques (including random removal, lexical substitution, token reordering, random addition, machine translation-based augmentation, and text generation) for developing customized machine learning models. The evaluation employs LoRA fine-tuning methodology and datasets created by the research team.
Weaknesses: - Significant issues with presentation and organization. The central contribution remains ambiguous throughout the manuscript. While multiple data augmentation methodologies are discussed, they lack comprehensive explanation and formal definition. The experimental findings are presented vaguely with insufficient methodological details provided. 
- The empirical results appear incoherent, with visualizations that fail to convey meaningful information or demonstrate discernible patterns. 
- The manuscript incorporates figures sourced from external works (e.g., llama3) without appropriate attribution or referencing.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1