Venue: NAACL 2025

# Navigating the Path of Writing:

Outline-guided Text Generation with Large Language Models

 Yukyung Lee1,2Soonwon Ka2Bokyung Son2Pilsung Kang3Jaewook Kang2

1Boston University \({}^{2}\)NAVER AI Platform \({}^{3}\)Seoul National University

ylee5@bu.edu pilsung_kang@snu.ac.kr {soonon.ka, bo.son, jaewook.kang}@navercorp.com

###### Abstract

Large Language Models (LLMs) have impacted the writing process, enhancing productivity by collaborating with humans in content creation platforms. However, generating high-quality, user-aligned text to satisfy real-world content creation needs remains challenging. We propose WritingPath, a framework that uses explicit outlines to guide LLMs in generating goal-oriented, high-quality text. Our approach draws inspiration from structured writing planning and reasoning paths, focusing on reflecting user intentions throughout the writing process. To validate our approach in real-world scenarios, we construct a diverse dataset from unstructured blog posts to benchmark writing performance and introduce a comprehensive evaluation framework assessing the quality of outlines and generated texts. Our evaluations with various LLMs demonstrate that the WritingPath approach significantly enhances text quality according to evaluations by both LLMs and professional writers.

+
Footnote †: Work done as a research intern at NAVER

## 1 Introduction

Writing is a fundamental means of structuring thoughts and conveying knowledge and personal opinions Collins and Gentner (1980). This process requires systematic planning and detailed review. Hayes (1980) describes writing as a complex problem-solving process and explores how planning and execution interact in writing. That is, writing involves more than merely generating text; it encompasses developing a proper understanding of the topic, gathering relevant subject matter, and implementing thorough structuring.

Recent advancements in Large Language Models (LLMs) have advanced the writing workflow, enhancing both its efficiency and productivity. One significant area of exploration is the collaborative use of LLMs in writing processes Lee et al. (2022); Mysore et al. (2023), as demonstrated by tools like Notion AI, Jasper, and Cohesive. The typical approach to incorporating LLMs involves the establishment of a writing plan and iterative improvement of interim outputs through revision Schick et al. (2023); Yang et al. (2022), as illustrated in Figure 1 (b) with a focus on utilizing the generative capabilities of LLMs to improve fluency, consistency, and grammatical accuracy. While these tools support users in creating content more efficiently, there remains room for improvement in maintaining consistent quality that accurately aligns with specific user intentions in production environments Wang et al. (2024).

To address this, we propose **WritingPath**, a methodology designed to incorporate user intentions such as desired topic, textual flow, keyword inclusion, and search result integration into the writing process. WritingPath emphasizes the importance of systematic planning and a clear outline from the early stages of writing. Inspired by the structured writing plan of Hayes (1980) and the reasoning path of Wei et al. (2022), the WritingPath collects ideas and creates outlines that encapsus

Figure 1: Comparative overview of writing approaches: (A) direct generation, (B) iterative writing involving planning, editing, and explaining, and (C) WritingPath method, which starts with a consistency-focused plan, incorporates information-rich browsing, and results in an augmented, consistent, and rich outline.



late the user's intentions before generating the final text. Furthermore, the initial outlines are further augmented with additional information through information browsing. Such a structured approach offers enhanced control over the text generation process and improves the quality of the content produced by LLMs.

We also utilize a multi-aspect writing evaluation framework to assess the intermediate and final productions from the WritingPath, offering a way to evaluate the quality of free-form text1 without relying on reference texts. Taking into account that conventional Likert scales (1-5 ratings) Clark and Hinkin (1998) make it challenging to systematically compare and evaluate diverse writing outputs, particularly in creative tasks Chakrabarty et al. (2023), our evaluation framework aims to provide more precise and reliable assessments for the outlines and final texts. For evaluation purposes, we construct a free-form blog text dataset incorporating a wide range of writing styles and topics from real users, including Beauty, Travel, Gardening, Cooking, and IT. Using this dataset, we evaluate how well the LLM outputs reflect the user's intentions. Applying the WritingPath to various LLMs shows significant performance gains across all evaluated models. These results validate that our approach enables the models to maintain a stronger focus on the given topic and purpose, ultimately generating higher-quality text that more accurately reflects user intentions. Furthermore, to validate real-world applicability, we applied WritingPath to a commercial writing platform for beta testing from October 2023 to March 2024. The deployment demonstrated its effectiveness in supporting real users with structured content creation across diverse writing needs.

Footnote 1: Free-form text generation focuses on creating diverse texts tailored to specific information and user intentions, unlike story generation, which develops narratives with plots and characters

The main contributions of this study can be summarized as follows:

* We propose WritingPath, a novel framework that enhances the ability of LLMs to generate high-quality and goal-oriented pieces of writing by using explicit outlines.
* We customize a comprehensive evaluation framework that measures the quality of both the intermediate outlines and the final texts.
* We construct a diverse writing dataset from unstructured blog posts across multiple domains, providing useful information such as aligned human evaluation scores, such as metadata that can be used as input for LLM-based writing tasks, and aligned human evaluation scores for the generated texts.
* Our evaluation results indicate that the WritingPath markedly improves the quality of LLM-generated texts compared to methods that do not use intermediate outlines.

## 2 Design of WritingPath

We propose WritingPath, a systematic writing process to produce consistent, rich, and well-organized text with LLMs. Inspired by human writing processes, it consists of five key steps: metadata preparation, initial outline generation, information browsing, augmented outline creation, and final text writing (Figure 2). Each step is guided by a specific prompt configuration that aligns LLM output with specific step requirements.

The core components of WritingPath are those that generate outlines as they establish a structured writing plan. Research suggests that a well-structured outline significantly impacts the quality of the written text Sun et al. (2022); Yang et al. (2022); Yang et al. (2022). The initial sketch is transformed into a detailed outline, including the flow, style, keywords, and relevant information from search results. This outline provides a clearer view of the final text to the LLMs. The specific steps are described as follows:

**Step 1: Prepare Meta Data** The first step establishes the writing direction and target reader using metadata \(m\), which includes i) purpose, ii) type, iii) style, and iv) keywords. To simulate this process, we converted human-written texts into metadata (see Section 4.1 for details of the dataset).

**Step 2: Generate Title and Initial Outline** The second step generates the title \(t\) and initial outline \(O_{\text{init}}\) based on the metadata \(m\) from step 1, using the LLM function \(f_{\text{lim}}\) with a prompt configuration function \(\phi_{s}\). Here, \(s\) indicates the step index, and for step 2, the prompt configuration is \(\phi_{2}\):

\[t,O_{\text{init}}=f_{\text{llm}}(\phi_{2}(m)), \tag{1}\]

The initial outline \(O_{\text{init}}\) consists of main headers \(h_{i,0}\), where \(i\) denotes the header sequence. This outline serves as the scaffolding of the text, organizing the main ideas and laying out the key points 



**Step 3: Browse for Information** The third step enriches the text by collecting additional information and keywords to reinforce the initial outline. We use the search function \(f_{\text{search}}\) with the generated title \(t\) as the query to retrieve the top-1 blog document, \(D_{\text{sim}}\):

\[D_{\text{sim}}=f_{\text{search}}(t) \tag{2}\]

In our implementation, we employ the NAVER search API2 to retrieve the top-1 document among similar blog posts. From the blog document, we extract keywords \(K\) using the \(f_{\text{llm}}\) with a prompt configuration \(\phi_{3}\):

Footnote 2: [https://developers.naver.com/docs/serviceapi](https://developers.naver.com/docs/serviceapi)

\[K=f_{\text{llm}}(\phi_{3}(D_{\text{sim}})), \tag{3}\]

The extracted keywords constitute the additional information from the search results, leading to a more specific writing plan, improving the quality of the generated text.

**Step 4: Generate Augmented Outline** The fourth step refines the initial outline by adding subheadings and specific details to each section based on incorporating the keywords collected from the previous step. The augmented outline \(O_{\text{aug}}\) is generated using the LLM function \(f_{\text{llm}}\) with a prompt configuration \(\phi_{4}\) that takes the title \(t\), keywords \(K\), and initial outline \(O_{\text{init}}\) as inputs:

\[O_{\text{aug}} =f_{\text{llm}}(\phi_{4}(t,k,O_{\text{init}}))\] \[=\{(h_{1,0},\{h_{1,1},h_{1,2},\ldots\}),\] \[(h_{2,0},\{h_{2,1},h_{2,2},\ldots\}),\] \[\ldots\} \tag{4}\]

The resulting augmented outline, \(O_{\text{aug}}\), comprises headers (\(h_{i,0}\)) and their corresponding subheaders (\(h_{i,j}\)), where \(i\) denotes the header index, and \(j\) indexes the subheaders. This detailed structure serves as a comprehensive writing plan, breaking down the text into manageable parts and providing clear direction for the content.

**Step 5: Write the Text** Finally, the text for each section \(d^{i}\) is generated using the LLM function with a prompt configuration \(\phi_{5}\) that takes the title \(t\) and the corresponding section of the augmented outline \(O_{\text{aug}}^{i}\) as inputs:

\[d^{i}=f_{\text{llm}}(\phi_{5}(t,O_{\text{aug}}^{i})) \tag{5}\]

The final blog document \(D\) is then compiled by concatenating all sections:

\[D=\{d^{1},d^{2},\dots,d^{n}\} \tag{6}\]

WritingPath organically connects all steps of the writing process, employing an outline to aggregate and manage diverse information, and assists users in producing high-quality writing. The prompts utilized for the WritingPath are detailed in Figure 12 and 13.

## 3 Evaluation of WritingPath

Evaluating the effectiveness of WritingPath compared to existing writing support systems is challenging. Most previous studies do not directly utilize outlines in the writing process, resulting in a lack of systematic methods to assess outline quality. Even when outlines are used, evaluation relies only on human evaluation Yang et al. (2023); Zhou et al. (2023). Moreover, current approaches heavily rely on human evaluation, which poses challenges for assessing full texts Schick et al. (2023); Yang et al. (2022); Lee et al. (2023), as it requires evaluating multiple aspects of the written work. This

Figure 2: Main architecture of WritingPath, our proposed framework for guiding LLMs to generate high-quality text following a structured writing process. The WritingPath condenses text generation into five key steps. Inspired by human writing planning, it ensures alignment with specified writing goals.



challenge can arise in content creation workflows where scalable and consistent quality assessment helps maintain content standards.

To address these limitations, we propose an evaluation framework that combines human and automatic evaluation to assess the quality of generated outlines and final texts from multiple perspectives. This hybrid approach is designed to support real-world content creation workflows by combining systematic automated metrics with human assessment of nuanced writing aspects that require subjective judgment. The proposed method establishes clear evaluation criteria, enabling objective and reproducible validation of WritingPath's effectiveness as a writing support system.

### Outline Evaluation

#### 3.1.1 Automatic Evaluation

We adapt various metrics to evaluate the logical alignment, coherence, diversity, and repetition in outlines, following criteria established in linguistic literature Van Dijk (2017); Pitler and Nenkova (2008); Tang et al. (2019); Elazar et al. (2021). Logical alignment, assessed through NLI-based methods, ensures that headers and subheaders are logically connected. Coherence evaluates thematic uniformity across sections, while diversity measures the breadth of topics covered. Repetition is analyzed to minimize redundancy and improve information efficiency. Note that coherence and diversity exhibit a trade-off relationship; maintaining coherence while covering a wide range of topics is essential to ensure the effectiveness of the outline in guiding the writing process. Detailed evaluation definitions are available in Appendix C.2.

#### 3.1.2 Human Evaluation

In addition to automatic evaluation metrics, we conduct a human evaluation to assess aspects of the generated outlines that are difficult to capture solely with automatic measures. These aspects include cohesion, natural flow, and redundancy. For augmented outlines, we also evaluate the usefulness of added information and overall improvement compared to the initial outline. Detailed evaluation definitions are available in Appendix C.3.

### Writing Evaluation

Traditional evaluation metrics such as Likert scales are not well-suited for assessing creative tasks like long story generation Chakrabarty et al. (2023). Acknowledging the need for more specific writing evaluation methods, we employ CheckEval Lee et al. (2024) to assess writing quality3. CheckEval decomposes the evaluation aspects into more granular sub-questions, forming a detailed checklist. These aspect-based checklists can make performance evaluations by either humans or LLMs more fine-grained. Moreover, by explicitly capturing the evaluator's reasoning behind each rating, this approach enhances the explainability of the evaluation process. To adapt CheckEval, we identified 7 aspects and selected relevant sub-aspects for each. We formulated them as binary (Yes/No) questions. This resulted in a checklist-style evaluation sheet for each sub-aspect, enabling an intuitive and structured assessment of the generated texts. The prompts utilized for the writing evaluation are detailed in Figure 11. The evaluation criteria were selected based on prior linguistics research Wolfe (1997); Knoch (2011); van der Lee et al. (2019); Celikyilmaz et al. (2020); Chhun et al. (2022); Sai et al. (2022); van der Lee et al. (2021) and finalized through a review and refinement process involving 6 writing experts. Details of the evalua

Figure 3: Breakdown of the seven key aspects used in writing evaluation, each with corresponding sub-aspects, employed in a Boolean QA-style checklist for human and LLM evaluation. This comprehensive framework ensures a multi-dimensional analysis of text quality.

 

ation criteria are in Figure 3, and the instructions and checklist used during the evaluation process are presented in Table 7.

## 4 Experimental Setting

### Dataset

In this study, we constructed a Korean dataset based on real user-written blog posts to assess the effects of the WritingPath in real content creation scenarios. The dataset covers five domains frequently handled in content creation:: travel, beauty, gardening, IT, and cooking. We created a total of 1,500 posts for each model, resulting in 4,500 instances in total. For human evaluation, we randomly sampled 10% of the outlines and texts and assessed their scores. Final texts were evaluated by human experts, aligning model outputs with professional quailty standards. Details of the dataset are in Appendix B.

### Model

We conducted experiments using three models: GPT-3.5-turbo Brown et al. (2020), GPT-4 Achiam et al. (2023), and HyperCLOVA X Yoo et al. (2024)4. For evaluation, we used GPT-4-turbo5. Additionally, we attempted to adapt the WritingPath approach to open-source models, including Llama2, Orion, and KoAlpaca. However, their outputs did not meet the quality standards necessary for fair comparison, and they were excluded from our analysis.

Footnote 4: gpt-3.5-turbo, gpt-4-0125, HCX-003

Footnote 5: gpt-4-turbo; we chose GPT-4-turbo as the evaluation model because of its best performance at the time of this study.

### Human Evaluation

We conducted two separate human evaluation processes, involving a total of 12 carefully selected evaluators. For outline evaluations, which are relatively simple and short, we employed 6 native Korean speakers with experience in LLM. For the more detailed and rigorous writing evaluation, we recruited 6 professional writers and teachers as writing experts, each with over 10 years of expertise in Korean writing.

## 5 Experimental Results

### Effectiveness of WritingPath

To verify that going through the WritingPath improves the final writing quality, we designed an analysis incorporating three cases (Figure 4):

writing from metadata,

writing from the initial outline,

writing from the augmented outline,

where this final case corresponds to the complete WritingPath pipeline.

Figure 5 shows results from both (a) LLM and (b) human evaluation using CheckEval. Both consistently show progressive improvement as more components of the WritingPath are incorporated, while the model rankings are in different order between the two evaluation methods6. Specifically, The results show that t

\begin{table}
\begin{tabular}{l|c c c c|c c c c} \hline \hline
**Model** & \multicolumn{4}{c|}{**Automatic Evaluation**} & \multicolumn{4}{c}{**Human Evaluation**} \\ \hline
* & 0.60/0.31 & 0.60 & 32.03 & **3.38** & 2.70 & 2.77 & 2.73 \\ _augmented_ & 0.61 & **1.33**/**0.51** & **0.61** & **17.33** & 3.15 & **2.78** & **3.54** & **3.13** \\ \hline GPT-4 & _initial_ &
- & 0.80/0.49 & 0.67 & 24.81 & **3.40** & 2.86 & 3.06 & 2.86 \\ _augmented_ & 0.66 & **1.61**/**0.52** & **0.68** & **13.12** & **3.40** & **2.98** & **3.74** & **3.43** \\ \hline HyperCLOVA X & _initial_ &
- & 0.75/0.41 & 0.74 & 18.04 & **3.47** & 2.96 & 2.82 & 3.22 \\ _augmented_ & 0.67 & **1.82**/**0.54** & **0.75** & **11.50** & 3.41 & **3.48** & **3.93** & **3.79** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Automatic and human evaluations on the quality of initial and augmented outlines from GPT-3.5, GPT-4, and HyperCLOVA X. Bold indicates the best result within a model.

Figure 4: Overview of the main analysis steps in the WritingPath framework, covering meta-data only, initial outline, and augmented outline scenarios, respectively.

 

(aug) leads to better writing quality compared to using only metadata (meta), indicating that the quality of writing improves significantly when the full WritingPath pipeline is employed. Furthermore, the augmented outline (aug) outperforms the initial outline (init), indicating that the content enrichment process further enhances writing quality. For a comprehensive analysis of writing quality, including human evaluation results for the final text across models, detailed improvement of text quality through the WritingPath, and Kendall tau correlations between various writing aspects and overall text quality, see Appendix D.

### Outline Evaluation

Section 5.1 showed that using the augmented outline in the WritingPath pipeline led to better performance compared to using only the initial outline or metadata. To assess not only the impact of initial and augmented outlines on the quality of the final writing but also any differences in quality at the outline stage itself, we evaluated the initial and augmented outlines independently.

**Automatic Evaluation** To see the effects of the outline augmentation module, we conducted automatic evaluations on the initial and augmented outlines using criteria described in Section 3.1.1. The results in Table 1 show significant improvements in Coherence and Repetition aspects for the augmented outlines compared to the initial ones, indicating that the outline augmentation process enhances content consistency and reduces unnecessary repetition. Notably, although Diversity and Coherence are often considered trade-offs, the augmented outlines in our study maintained Diversity while improving Coherence. This suggests that the outline expansion module can increase consistency without compromising content diversity. Detailed performance across various domains is in Table 3.

**Human Evaluation** As described in Section 3.1.2, we conducted human evaluations to assess the cohesion, natural flow, diversity, and redundancy of initial and augmented outlines. The augmented outlines demonstrated significant improvements in all aspects except cohesion, which slightly declined or remained stable. Nevertheless, the overall performance of the augmented outlines surpassed that of the initial outlines. Further evaluations of the augmented outlines were conducted on usefulness and improvement, which indicated the extent of useful information added and overall quality enhancement compared to the initial outlines. As shown in Figure 6, all models demonstrated improvements in both metrics, validating the power of the browsing step. Detailed performance across various text domains is in Table 4.

## 6 Real-World Deployment

WritingPath was integrated into a commercial blogging platform as a writing assistance feature and tested for six months. In the service environment, additional considerations such as safety filtering and content quality control measures were necessary for reliable content generation. The system architecture of CLOVA for Writing by NAVER is depicted in Figure 7.

The serving pipeline integrates multiple components for reliable service operation. It integrates user request handling, content filtering, Kafka pipeline, and retrieval. Requests pass through a Gateway with rate limiting and are filtered for harmlessness. Specifically, the system includes emergency filtering and safety classification before pass

Figure 5: Main analysis steps on writing evaluation results by (a) LLM and (b) Human Evaluation.

Figure 6: Evaluation of augmented outlines showing all models surpass the effectiveness threshold with scores in Usefulness above 2 and Improvement over 0.5, indicating universal enhancements from the initial outlines.

 

ing requests to WritingPath. Additionally, a token event monitoring system tracks model usage, followed by token event filtering over output anomalies.

## 7 Conclusion

We introduced WritingPath, a framework that enhances the ability of LLMs to generate high-quality and goal-oriented writing by employing explicit outlines. Designed for real-world content creation, our approach uses structured guidelines from the early stages to ensure consistent quality control.

We verified the impact of WritingPath by conducting a comprehensive evaluation that incorporates automatic and human evaluations covering a wide range of aspects. Our experimental results demonstrate that texts generated following the full WritingPath approach, which includes the use of augmented outlines, exhibit superior performance compared to texts produced using only initial outlines or without any intermediate outlines. We also proposed a framework for assessing the WritingPath's intermediate outlines, which found that augmented outlines have better inherent quality than initial outlines, demonstrating the importance of outline augmentation steps. We hope that this work will contribute to the research and development of more reliable AI-assisted writing solutions.

## 8 Acknowledgments

This research project was conducted as part of the NAVER HyperCLOVA-X and CLOVA for Writing projects. We express our gratitude to Nako Sung for his thoughtful advice on writer LLMs and to the NAVER AX-SmartEditor team. Additionally, we would like to thank the NAVER Cloud Conversational Experience team for their practical assistance and valuable advice in creating the blog dataset. We appreciate Jaehee Kim, Hyowon Cho, Keonwoo Kim, Joonwon Jang, Hyejin Lee, Joonghoon Kim, Sangmin Lee, and Jaewon Cheon for their invaluable feedback and evaluation. We also thank DSBA NLP Group members for their comments on the paper.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint:2303.08774. Cited by: SS1.
* A. Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi (2024)Self-RAG: learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: SS5.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. Cited by: SS6.
* A. Celikyilmaz, E. Clark, and J. Gao (2020)Evaluation of text generation: a survey. ArXivabs/2006.14799. Cited by: SS1.
* T. Chakrabarty, P. Laban, D. Agarwal, S. Muresan, and C. Wu (2023)Art or artifice? large language models and the false promise of creativity. arXiv preprint arXiv:2309.14556.

Figure 7: Real-world deployment pipeline of WritingPath.

 

* Chen and Eger (2023) Yanran Chen and Steffen Eger. 2023. MENLI: Robust evaluation metrics from natural language inference. _Transactions of the Association for Computational Linguistics_, 11:804-825.
* Chhun et al. (2022) Cyril Chhun, Pierre Colombo, Chloe Clavel, and Fabian M. Suchanek. 2022. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In _International Conference on Computational Linguistics_.
* Clark & watson (1995) LA Clark. 8cwatson, d.(1995). constructing validity: Basic issues in objective scale development. _Psychological Assessment_, 7(3):309-319.
* Collins and Gentner (1980) Allan Collins and Dedre Gentner. 1980. A framework for a cognitive theory of writing. In _Cognitive Processes in Writing_, pages 51-72. Erlbaum.
* Dieng et al. (2020) Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. _Transactions of the Association for Computational Linguistics_, 8:439-453.
* Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilsha Ravichander, Eduard Hovy, Hinrich Schutze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. _Transactions of the Association for Computational Linguistics_, 9:1012-1031.
* Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. _arXiv preprint arXiv:2302.04166_.
* Galinski et al. (2019) Filip Garlinski, Anna Wroblewska, Tomasz Stanislawek, Kamil Grabowski, and Tomasz Gorecki. 2019. GEval: Tool for debugging NLP datasets and models. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 254-262, Florence, Italy. Association for Computational Linguistics.
* Hayes (1980)JR Hayes. 1980. identifying the organization of wi iining processes. _Cognitive processes in writing: An interdisciplinary approach_, pages 3-10.
* Hinkin (1998) Timothy R Hinkin. 1998. A brief tutorial on the development of measures for use in survey questionnaires. _Organizational research methods_, 1(1):104-121.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* Ke et al. (2022) Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. CTRLEval: An unsupervised reference-free metric for evaluating controlled text generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2306-2319, Dublin, Ireland. Association for Computational Linguistics.
* Knoch (2011) Ute Knoch. 2011. Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from? _Assessing Writing_, 16(2):81-96. Studies in Writing Assessment in New Zealand and Australia.
* Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_, pages 530-539, Gothenburg, Sweden. Association for Computational Linguistics.
* Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. CHI '22, New York, NY, USA. Association for Computing Machinery.
* Lee et al. (2023) Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esim Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. 2023. Evaluating human-language model interaction. _Transactions on Machine Learning Research_.
* Lee et al. (2024) Yukyang Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, and Kang Pilsung. 2024. Checkeval: Robust evaluation framework using large language model via checklist. In _First Workshop on Human-Centered Evaluation and Audiling of Language Models_.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chengqiang Zhu. 2023. QptEval: Nlg evaluation using gpt-4 with better human alignment. _arXiv preprint arXiv:2303.16634_.
* Mirowski et al. (2023) Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, CHI '23, New York, NY, USA. Association for Computing Machinery.
* Mysore et al. (2023)Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. 2023. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers. _arXiv preprint arXiv:2311.09180_.
* Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 

186-195, Honolulu, Hawaii. Association for Computational Linguistics.
* Sai et al. (2022) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems. 55(2).
* Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. _Advances in Neural Information Processing Systems_, 36.
* Schick et al. (2023) Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2023. PEER: A collaborative language model. In _The Eleventh International Conference on Learning Representations_.
* Son et al. (2024) Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. _arXiv preprint arXiv:2402.11548_.
* Stevens et al. (2012) Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David Buttler. 2012. Exploring topic coherence over many models and many topics. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 952-961, Jeju Island, Korea. Association for Computational Linguistics.
* Sun et al. (2022) Xiaofei Sun, Zijun Sun, Yuxian Meng, Jiwei Li, and Chun Fan. 2022. Summarize, outline, and elaborate: Long-text generation via hierarchical supervision from extractive summaries. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 6392-6402, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Tang et al. (2019) Hongyin Tang, Miao Li, and Beihong Jin. 2019. A topic augmented text generation model: Joint learning of semantics and structural features. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5090-5099, Hong Kong, China. Association for Computational Linguistics.
* van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021. Human evaluation of automatically generated text: Current trends and best practice guidelines. _Computer Speech & Language_, 67:101151.
* van der Lee et al. (2019) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel J. Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In _International Conference on Natural Language Generation_.
* Van Dijk (1977) Teun Adrianus Van Dijk. 1977. Text and context: Explorations in the semantics and pragmatrics of discourse.
* Wang et al. (2024) Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al. 2024. Weaver: Foundation models for creative writing. _arXiv preprint arXiv:2401.17268_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Weigle (2002) Sara Cushing Weigle. 2002. GPtscore: Evaluate as you desire. _Cambridge University Press_.
* Wolfe (1997) Edward W. Wolfe. 1997. The relationship between essay reading style and scoring proficiency in a psychometric scoring system. _Assessing Writing_, 4(1):83-106.
* Yang et al. (2023) Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023. DOC: Improving long story coherence with detailed outline control. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3378-3465, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2022) Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive repropting and revision. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4393-4479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Yao et al. (2024) Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungjun Kim, et al. 2024. Hyperclova x technical report. _arXiv preprint arXiv:2404.01954_.
* Zheng et al. (2023) Linmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. 2023. Judging llm-as-a- judgewg with mt-bench and chatbot arena. _ArXiv_, abs/2306.05685.
* Zhou et al. (2023) Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cottrell, and Mrinnaya Sachan. 2023. Recurrentgrpct: Interactive generation of (arbitrarily) long text. _Preprint_, arXiv:2305.13304.
* Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. _SIGIR_ 

Related Work

### Collaborative Writing with Language Models

Recent works that explore collaboration with LLMs during the writing process can be categorized into two aspects: 1) Outline Planning and Draft Generation, and 2) Recursive Re-prompting and Revision.

**Outline Planning and Draft Generation** involves incorporating the writer's intents and contextual information into LLM prompts to create intermediate drafts. Dramatron Mirowski et al. (2023) is a system for collaborative scriptwriting that automatically generates outlines with themes, characters, settings, flows, and dialogues. DOC Yang et al. (2023) improves the coherence of generating long stories by offering detailed control of their outlines, including analyses of generated outlines and suggestions for revisions to maintain consistent plot and style.

Building on these works, our WritingPath mimics the human writing process by structuring it into controllable outlines. While our approach shares similarities with DOC in terms of utilizing outlines, we diverge from focusing solely on story generation and propose a novel outline generation process that incorporates external knowledge through browsing. Our aim is to sophisticatedly control machine-generated text across a wide range of writing tasks.

- suggesting revisions in an iterative fashion to enhance the quality of written content. Re3 Yang et al. (2022) introduces a framework for maintaining the long-range coherence of draft generation. It operates separate rewriter and edit modules in its prompt chain to check and refine plot relevance and long-term factual consistency. PEER Schick et al. (2023) proposes a recursive revision framework based on the concept of self-training, where the model autonomously selects the editing operations for revision and provides explanations for the modifications it makes. RECURRENTGPT Zhou et al. (2023) utilizes a recursive, language-based mechanism to simulate LSTM Hochreiter and Schmidhuber (1997), enabling the generation of coherent and extended texts. While these works are relevant to collaborative writing with LMs, direct comparisons with our approach are unfeasible. These studies focus on specific tasks like story generation, requiring task-specific training and datasets, which are unavailable in Korean for our writing tasks.

Our WritingPath differs from previous works in its goals for utilizing LLMs in the writing process. Instead of relying on an ad-hoc recursive writing structure that may be inefficient, we establish a systematic writing plan that guides the generation process from the very beginning. Furthermore, we focus on free-form text generation rather than story generation and do not require separate training for writing, planning, or editing.

### Integrating External Information

Existing approaches have explored various methods to inject external knowledge into LLMs to improve their performance on text-generation tasks. For instance, Retrieval-Augmented Generation (RAG) Lewis et al. (2020), and Toolformer Schick et al. (2024) have developed techniques to connect LLMs with external search tools, enabling them to gather relevant information and generate more informative and accurate responses. However, despite these contributions to improving LLMs' access to information Asai et al. (2024), they inherently fall short of fully reflecting the diversity and complexity of the writing process Chakrabarty et al. (2023).

Our work distinguishes itself from previous approaches by focusing on emulating the modern writing planning process. With this structured approach, an LLM can efficiently produce high-quality text, significantly contributing to improving the control and quality of the generated text.

### Writing Evaluation

It is well-known that supervised metrics such as ROUGE and BLEU are ill-suited for evaluating natural language generation output, especially for open-ended writing tasks. Traditionally, such evaluation has depended on rubric-based human evaluation, which is a costly and time-consuming task Weigle (2002). Recent advancements in LLMs have led to the exploration of new paradigms that utilize LMs for evaluating LM-generated text Gralinski et al. (2019); Fu et al. (2023). However, to effectively assess free-form text, a more customized and interactive evaluation framework is needed.

We utilize CheckEval Lee et al. (2024), a fine-grained and explainable evaluation framework, to assess free-form text writing. By customizing a 

checklist with specific sub-questions for each writing aspect, we provide a more reliable and accurate means of evaluating writing quality.

## Appendix B Details of Dataset

We selected 20 blog posts for each domain7, resulting in 100 seed data points. For seed data construction, we generated metadata, including purpose, topic, keywords, and expected reader, based on the title and content of the blog posts. This metadata is the input to the WritingPath, helping the model understand the context of the post and generate relevant outlines and text. We created a test dataset of 1,100 instances per model under evaluation using the seed data. Each data point includes the outputs of each WritingPath step: an outline, additional information, an augmented outline, and the final text. With analysis experiments as well, we generated a total of 1,500 posts for each model, resulting in 4,500 instances in total. For human evaluation, we randomly sampled 10% of the outlines and texts and assessed their scores. The final texts were evaluated by human experts, and the dataset aligns the generated outputs from three models with the human scores.

Footnote 6: [https://blog.naver.com/](https://blog.naver.com/)

## Appendix C Details of Evaluation

### Compensation Details

Outline evaluators were compensated with a 6,000 KRW (\(\approx\) 4.2 USD) gift card for their 30-minute participation. And writing experts were compensated at a rate of 9,000 KRW (\(\approx\) 6.6 USD) per one-writing sample.

### Automatic Evaluation - Outline

* Logical alignment: Based on Chen and Eger (2023), we utilize Natural Language Inference (NLI) which examines whether the headers and subheaders within an outline logically connect, ensuring the structural integrity necessary for coherent argumentation8. Footnote 8: we utilize gpt-4-turbo for NLI evaluation
* Coherence: Through Topic Coherency metrics such as NPMI Stevens et al. (2012) and UCI Lau et al. (2014), this aspect assesses the thematic uniformity across the sections of outline, verifying a consistent narrative.
* Diversity: We measure the breadth of topics addressed by applying Topic Diversity metrics Dieng et al. (2020), aiming to ensure that the content of outline is comprehensive and varied.
* Repetition: Self-BLEU Zhu et al. (2018) is used to gauge the degree of redundancy within the outline, prioritizing efficiency in information presentation by minimizing repetition.

### Human Evaluation - Outline

The human evaluation criteria are based on aspects considered in previous studies on text coherence, relevance, and quality assessment Yang et al. (2022, 2023); Zhou et al. (2023); Ke et al. (2022). For both initial and augmented outlines, the human evaluation is performed on the following five aspects, using a 1-4 point scale:

* Cohesion: Evaluates whether the title and outline are semantically consistent.
* Natural Flow: Assesses whether the outline flows in a natural order.
* Diversity: Evaluates whether the outline consists of diverse topics.
* Redundancy: Assesses whether the outline avoids semantically redundant content.

Furthermore, we use two additional aspects for evaluating the augmented outline:

* Usefulness of Information: Assesses whether the augmented outline provides useful information beyond the initial outline.
* Improvement: Evaluates whether significant improvements have been made in the augmented outline compared to the initial outline, using a binary scale.

 



## Appendix D Further Analysis of Writing Quality

To further analyze the quality of the text generated through the complete WritingPath pipeline, we conducted a human evaluation based on the CheckEval framework. The results are presented in Table 2. The analysis by six writing experts showed that GPT-4 and HyperCLOVA X generally performed better than GPT-3.5 in terms of writing quality. HyperCLOVA X exhibited higher scores in specificity compared to other models, which is consistent with the findings reported in KMMLU Son et al. (2024) regarding the advantages of language-specific models. Detailed performance metrics across various domains and further LLM evaluations can be found in Table 5, 6. Furthermore, We consider seven key aspects (Section 3) for evaluating the quality of writing. CheckEval's binary responses for each aspect allow for identifying the specific factors contributing to the assessments. We found that logical fluency, coherence, consistency, and specificity significantly contribute to the improvement of text quality through the WritingPath (Figure 8).

During the evaluation of the writing quality, writing experts assigned binary overall quality ratings (1 for high quality, 0 for low quality) to the texts. We employed the Kendall tau correlation to examine the relationship between the overall binary ratings and the scores for each evaluation aspect. The analysis (Figure 9) revealed a significant correlation for all the aspects we designed. Interestingly, logical fluency, specificity, and coherence, which were found to be particularly important in determining the perceived quality of written content, are among the aspects that showed the most significant improvement through the WritingPath (Figure 8).

The progressive improvement in these aspects can be attributed to the effectiveness of using outlines. The initial outline (init) helps organize information more logically and coherently compared to using only metadata (meta), while the augmented outline (aug) further enhances the consistency and richness of the content. These findings highlight the importance of using outlines in the writing process and demonstrate how their gradual enhancement leads to better-structured, more coherent, and content-rich texts, ultimately improving the overall quality of the written output.

\begin{table}
\begin{tabular}{l|c c c c c c c|c} \hline \hline
**Model** & **Linguistic Fl.** & **Logical Fl.** & **Coh.** & **Cons.** & **Comple.** & **Spec.** & **Int.** & **Overall** \\  & binary & binary & binary & binary & binary & binary & binary & binary \\ \hline GPT-3.5 & 51.66 & 31.14 & 46.29 & 88.11 & 66.43 & 21.14 & 35.14 & 48.56 \\ \hline GPT-4 & 68.00 & 60.57 & 72.86 & 89.26 & 80.29 & 54.14 & 66.29 & 70.20 \\ \hline HyperCLOVA X & **89.71** & **84.46** & **91.14** & **98.06** & **92.57** & **74.00** & **80.00** & **87.13** \\ \hline \end{tabular}
\end{table}
Table 2: Human evaluation results for writing quality of final text (aug) across models.

Figure 8: Human evaluation results for writing quality (meta, init, aug) over various CheckEval aspects.

Figure 9: Kendall tau correlations between various writing aspects and overall text quality.





\begin{table}
\begin{tabular}{l|c c|c c c c c c} \hline
* & 3.950 & 0.542 \\  & Trends & Initial & 3.262 & 2.313 & 3.167 & 2.917 &
- & 3.975 & 3.000 & 0.542 \\  & Temporal & Averpeed & 3.588 & **3.218** & **3.576** & **3.842** & 3.688 & 0.308 \\  & Gradients & Initial & 2.958 & 2.915 & 2.842 & 2.417 &
- & 3.988 \\  & Gradient & Augmented & **3.202** & **2.428** & 2.417 &
- & 2.843 & 2.542 & 0.708 \\  & Cooking & Agglomerated & 2.718 & 2.213 & **3.468** & **2.883** & 2.542 & 0.488 \\  & IT & Initial & **3.468** & **2.927** & 2.752 & 2.792 &
- & 2.5 \\  & Automated & 3.600 & 2.750 & **3.708** & **3.290** & 2.795 & 0.542 \\ \hline _GPT 4_ & Binary & Initial & 3.375 & 2.350 & 3.383 & 3.367 &
- & 3.983 \\  & Structured & 3.446 & 2.908 & 3.088 & 3.833 & 3.208 & 0.833 \\  & Temporal & Avpeed & **3.452** & **3.608** & **3.702** & **3.842** & 3.333 & 0.750 \\  & Gradient & Initial & **3.726** & 3.125 & 2.831 & 2.917 &
- & 3.9 \\  & Cooking & Agglomerated & **3.202** & **3.708** & **3.803** & **3.800** & 3.208 & 0.667 \\  & Scaling & Agglomerated & 3.208 & **2.705** & **3.842** & **2.897** & 2.833 & 0.542 \\  & TF & Thesis & **3.000** & **2.917** & 3.200 & 3.042 &
- & 2.5 \\  & Incremental & Agenmed & **3.000** & **2.977** & 3.200 & 3.603 & 3.803 & 0.720 \\  & Memory & Agenmed & 3.735 & 3.847 & **3.002** & **3.803** & 3.803 & 0.833 \\  & Gradient & Initial & 3.500 & 3.125 & 2.830 & 3.042 &
- & 3.9 \\  & Cooking & Animated & 3.780 & 3.750 & 3.598 & 3.625 & 3.583 & 0.875 \\  & Weighted & 3.500 & 3.598 & 2.558 & 2.500 & 3.250 &
- & 3.9 \\  & Optimized & 3.380 & 3.598 & 3.702 & 3.702 & 3.200 & 0.720 \\  & Estimated & 3.380 & 3.615 & 3.702 & 3.702 & 3.200 & 0.720 \\  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\  & Integrated & 3.302 & **3.308** & **3.571** & **3.708** & 3.803 & 0.720 \\ \hline \end{tabular}
\end{table}
Table 4: Detailed outline human evaluation results.

Figure 10: LLM Evaluation

\begin{table}
\begin{tabular}{l|c|c|c c c} \hline
* & 0.638/0.208 & 0.480 & 48.01 \\  & Improved & Adjusted & 0.483 & **1.566/0.653** & **0.533** & **23.79** \\  & Trend & Initial & 0.575 & **1.546/0.648** & 0.708 & 21.56 \\  & Gradening & Initial & 0.575 & **1.546/0.648** & 0.670 & **13.21** \\  & Gradening & Initial & 0.569 & 0.570 & 0.571 & 26.55 \\  & Gradening & Initial & 0.569 & 0.570 & 0.571 & 26.55 \\  & Gradening & Invalid & 0.568 & **1.527/0.655** & 0.571 & 26.55 \\  & Gradening & Invalid & 0.568 & **1.527/0.655** & 0.571 & 26.54 \\  & IG & Expanded & 0.686 & **1.527/0.651** & 0.572 & 18.54 \\  & TG & Aggregated & 0.686 & **1.528/0.611** & 0.721 & 13.71 \\  & IT & Original & 0.649 & 0.725 & 0.522 & 25.67 \\  & \multirow{-2}{*}{
\begin{tabular}{} \end{tabular} } \clineine{2-2} _OFT 4_ & Bensery & Initial & 0.574 & **1.580**/0.643** & **0.560** \\  & Tweet & Initial & 0.577 & 1.584/0.635 & 0.585 & 18.50 \\  & Trend & Initial & 0.577 & **1.584**/0.635** & **0.560** & 17.61 \\  & Gradening & Invalid & 0.577 & 1.584/0.635 & 0.585 & 18.50 \\  & Gradening & Invalid & 0.615 & **1.529/0.635** & 0.685 & 16.63 \\  & Conditional & Initial & 0.573 & 0.579 & 0.576 & 23.53 \\  & Gradening & Invalid & 0.574 & 0.578 & 0.578 & 23.53 \\  & Total & 0.574 & 0.573 & 0.577 & **0.586** & 19.52 \\  & IT & Agenmed & 0.706 & **1.560**/**0.556** & 0.596 & **13.53** \\ \hline \end{tabular}
\end{table}
Table 3: Detailed outline automatic evaluation results.




\begin{table}
\begin{tabular}{c|l|r r r r r r r|r} \hline
**Model** & **Category** & **Linguistic FL** & **Logical Fl.** & **Coh.** & **Cons.** & **Complete.** & **Spec.** & **Int.** & **Overall** \\ \hline _GPT 3.5_ & Beauty & 96.00 & 75.42 & 90.00 & 94.17 & 89.58 & 52.50 & 63.06 & 80.10 \\  & Travel & 100.00 & 78.54 & 97.08 & 96.94 & 97.08 & 65.83 & 81.67 & 88.16 \\  & Gardening & 98.67 & 75.63 & 95.00 & 96.11 & 89.17 & 49.17 & 78.89 & 82.33 \\  & Cooking & 99.50 & 72.92 & 95.42 & 97.78 & 95.00 & 35.83 & 84.17 & 82.94 \\  & IT & 100.00 & 76.67 & 97.50 & 96.94 & 94.17 & 39.17 & 50.53 & 79.33 \\ \hline  & Total & 98.83 & 75.83 & 95.00 & 96.39 & 93.00 & 48.50 & 71.72 & 82.75 \\ \hline _GPP 4_ & Beauty & 99.17 & 89.79 & 97.50 & 99.44 & 99.17 & 84.58 & 98.61 & 95.47 \\  & Travel & 99.00 & 90.21 & 91.67 & 97.22 & 96.67 & 70.00 & 96.94 & 91.67 \\  & Gardeing & 99.67 & 90.00 & 94.17 & 98.33 & 100.00 & 74.17 & 97.78 & 93.44 \\  & Cooking & 99.67 & 89.58 & 93.75 & 98.61 & 97.92 & 63.33 & 96.67 & 91.36 \\  & IT & 99.17 & 88.96 & 88.33 & 96.11 & 99.17 & 41.67 & 76.11 & 84.22 \\ \hline  & Total & **99.33** & **89.71** & **93.08** & **97.94** & **98.58** & 66.75 & **93.22** & **91.23** \\ \hline _HyperCLOVA X_ & Beauty & 100.00 & 88.33 & 98.33 & 100.00 & 90.00 & 90.42 & 91.38 & 94.07 \\  & Travel & 99.50 & 83.75 & 90.42 & 97.50 & 91.25 & 79.58 & 92.77 & 90.68 \\  & Gardening & 99.33 & 88.13 & 93.33 & 98.61 & 95.00 & 70.42 & 84.16 & 89.85 \\  & Cockline & 98.67 & 82.29 & 88.75 & 96.67 & 90.42 & 87.92 & 91.11 & 90.83 \\  & IT & 98.50 & 76.04 & 81.25 & 92.50 & 87.08 & 55.00 & 52.50 & 77.55 \\ \hline  & Total & 99.20 & 83.71 & 90.42 & 97.06 & 90.75 & **76.67** & 82.38 & 88.60 \\ \hline \end{tabular}
\end{table}
Table 5: Detailed writing LLM evaluation results.





\begin{table}
\begin{tabular}{c|p{34.1pt} p{280.5pt}}
**Aspect** & **Subaspect** & **Descriptions** \\ \hline \hline  & Natural Expression & Does the given test read naturally without any unnatural rhythm or excessively emphasized parts? \\  & Text Length & Is the length of the text suitably for the purpose and is not excessively verbose or overly concise? \\  & Text Length & Is the vocabulary appropriate for the context, a novel complex, and suitable for the topic and reader? \\ Language Flussy & Vocabulary & Is the vocabulary appropriate for the context, an overly complex, and suitable for the topic and reader? \\  & Occulatory & Is the composition and sentence structure of the given text correct? \\  & Syntax & Is the spelling and punctuation of the given text correctly applied? \\  & Mechanic-Spelling, Punctual & Is the spelling and punctuation of the given text correctly applied? \\  & Occulatory & Is the spelling text have a clear and effective structure (layout)? \\  & Occulatory & Is the text free of repetitive or unnecessary context? \\  & Recrutive Context & Is the text free of repetitive or unnecessary context? \\  & Occulatory & Is the text, the sentences well connected and progressing manually and logically? \\ Logical Fluency & Acculatory & Is the text, the sentences well connected and progressing manually and logically? \\  & Inter-sentence Cobosion & Is the text with English \(q^{\prime}\) = \(q^{\prime}\) = \(q^{\prime}\) + \(



[MISSING_PAGE_POST]

 



### _WitingPath Prompt_

Figure 12: WritingPath Prompt for Each Stage (Step 1 and 2).





### _WittingPath Prompt_

Figure 13: WritingPath Prompt for Each Stage (Step 4 and 5).