WORDFLOW: Social Prompt Engineering for Large Language Models (2024)

Abstract: Large language models (LLMs) require wellcrafted prompts for effective use. Prompt engineering, the process of designing prompts, is challenging, particularly for non-experts who are less familiar with AI technologies. While researchers have proposed techniques and tools to assist LLM users in prompt design, these works primarily target AI application developers rather than non-experts. To address this research gap, we propose social prompt engineering, a novel paradigm that leverages social computing techniques to facilitate collaborative prompt design. To investigate social prompt engineering, we introduce WORDFLOW, an open-source and social text editor that enables everyday users to easily create, run, share, and discover LLM prompts. Additionally, by leveraging modern web technologies, WORDFLOW allows users to run LLMs locally and privately in their browsers. Two usage scenarios highlight how our tool’s incorporation of social prompt engineering can enhance laypeople’s interactions with LLMs. WORDFLOW is publicly accessible at https://poloclub.github.io/wordflow.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 42–50 August 11-16, 2024 ©2024 Association for Computational Linguistics
Two usage scenarios highlight how our tool’s incorporation of social prompt engineering can enhance laypeople’s interactions with LLMs. WORDFLOW is publicly accessible at https://poloclub.github.io/wordflow.

heading: 1 Introduction
text: Recently, there has been a surge in the popularity of large language models (LLMs) such as GPT4 (OpenAI, 2023a), Gemini (Team et al., 2023), and Llama 2 (Touvron et al., 2023). These pretrained artificial intelligence (AI) models demonstrate a diverse array of capabilities that are continually being discovered, including summarization, question-answering, creative writing, and translation (Bommasani et al., 2022). To instruct these general-purpose LLMs to perform specific tasks, users need to provide them with prompts—text instructions and examples of desired outputs (Brown
42
et al., 2020). These prompts serve as background contexts and guides for LLMs to generate text that aligns with users’ objectives. Prompting enables users to employ LLMs for various tasks with plain language; in fact, well-crafted prompts can make general-purpose LLMs outperform specialized AI models (Nori et al., 2023).
Designing effective prompts, known as prompt engineering, poses significant challenges for LLM users (Jiang et al., 2022). LLM users often rely on trial and error and employ unintuitive patterns, such as adding “think step by step” (Kojima et al., 2022) to their prompts, to successfully instruct LLMs. Prompt engineering, despite its name, is considered an art (Parameswaran et al., 2023) and is even compared to wizards learning “magic spells” (Willison et al., 2022). Prompt writers may not fully understand why certain prompts work, but they still add them to their “spell books.” Furthermore, prompting is especially challenging for non-AI-experts, who are often confused about getting started and lack sufficient guidance and training on LLMs and prompting (Zamfirescu-Pereira et al., 2023).
To help users prompt LLMs, researchers propose instruction tuning (Chung et al., 2022) and reinforcement learning from human feedback (Ouyang et al., 2022) to align a model’s output with users’ intent. Prompting techniques (Brown et al., 2020) are introduced to improve LLMs’ performance on complex tasks. Libraries and interactive tools have also been developed to streamline the prompt crafting process (e.g., Chase, 2022; Jiang et al., 2022). However, existing techniques and tools primarily cater to AI application developers who use LLMs to build AI applications (e.g., chatbot applications), overlooking non-expert users who use LLMs for everyday tasks (e.g., checking emails for grammar errors). To bridge this critical research gap, we propose social prompt engineering, a novel paradigm that leverages social computing techniques to facilitate collaborative prompt designs. We contribute:
• WORDFLOW, the first social and customizable text editor that empowers everyday users to create, run, share, and discover LLM prompts (Fig. 1). It features a direct manipulation text editing interface for applying LLM prompts to transform existing text, such as proofreading and translation, or generate new text, such as creative writing. Users can easily customize prompts and LLM settings, share prompts with the community, and copy commu-
nity prompts (§ 3). Two usage scenarios highlight how WORDFLOW and social prompt engineering can enhance users’ interactions with LLMs (§ 4). Finally, we discuss future research opportunities enabled by our system (§ 5). • An open-source1, web-based implementation that lowers the barrier for everyday users in designing effective prompts and applying LLMs to their daily tasks. By leveraging modern web technologies, such as WebGPU (MDN, 2023; team, 2023), our tool enables users to run cuttingedge LLMs locally without the need for dedicated backend servers or external LLM API services (§ 3.4). Additionally, we offer an opensource implementation to help future designers and researchers adopt WORDFLOW for exploring and developing future user interfaces for LLMs. To see a demo video of WORDFLOW, visit https://youtu.be/3dOcVuofGVo.
We hope our work will inspire the research and development of collaborative interfaces that help everyone more easily and effectively use LLMs.

heading: 2 Related Work
text: Addressing prompt engineering challenges. Researchers have proposed libraries such as LANGCHAIN (Chase, 2022), GUIDANCE (Lundberg et al., 2023), and OUTLINES (Willard and Louf, 2023) to help users write prompts programmatically and control the structure of an LLM’s output. By formulating prompting as programming, researchers propose techniques that help users edit (Fiannaca et al., 2023) and unit test prompts (Strobelt, 2023). COPROMPT (Feng et al., 2023) introduces a collaborative editor for multiple programmers to write prompts simultaneously. AI prototyping tools like PROMPTMAKER (Jiang et al., 2022), GOOGLE AI STUDIO (Google, 2023), OPENAI PLAYGROUND (OpenAI, 2023b), and PARTYROCK (Amazon, 2023) allow users to rapidly write and run prompts.
By leveraging visual programming techniques, tools such as AI CHAINS (Wu et al., 2022), PROMPT SAPPER (Cheng et al., 2023), and CHAINFORGE (Arawjo et al., 2023) enable AI application developers to visually design and test complex prompts. Similarly, PROMPTIDE (Strobelt et al., 2022), PROMPTAID (Mishra et al., 2023), and PROMPTERATOR (Sučik et al., 2023) employ mixed-initiative and interactive visualization tech-
1Code: https://github.com/poloclub/wordflow
niques to help LLM users brainstorm and refine prompts. These existing tools function as IDEs that help AI developers craft prompts that will later be integrated into other applications. In contrast, WORDFLOW aims to serve as a runtime interface for everyday users, who act as both the prompt engineers and direct users of their prompts, and may not be well-versed in AI technologies.
Social prompt engineering. Online communities, including Promptstacks (Promptstacks, 2023), ChatGPT Prompt Genius (Reddit, 2023), and ShareGPT (Eccleston and Tey, 2022), serve as platforms for prompt creators to share tips, collaborate, and stay updated on AI advancements. User prompts from social media have been scraped to create prompt datasets for AI model development (Wang et al., 2023). Online prompt marketplaces, such as PromptBase (PromptBase, 2023), PromptHero (PromptHero, 2023) and ChatX (ChatX, 2023), have emerged to allow users to buy and sell prompts for generative models. Midjourney’s Discord server (Holz, 2022) allows users to run and share prompts for text-to-image generative models, with dedicated sections for prompt critique and improvement (Oppenlaender, 2022). Building on the design of these communities, WORDFLOW provides an easy-to-use interface that unifies creating, running, sharing, and discovering LLM prompts. The most relevant related work is PROMPTSOURCE (Bach et al., 2022), an IDE for AI researchers and developers to write and share LLM prompts. PROMPTSOURCE targets AI experts using LLMs for natural language processing
tasks on datasets (such as data annotation), and it requires users to provide a dataset. In comparison, WORDFLOW targets everyday users using LLMs for daily tasks, such as grammar checking, without the need to provide any dataset.

heading: 3 System Design & Implementation
text: WORDFLOW is an interactive tool that empowers everyday users to easily create, run, share, and discover LLM prompts. It tightly integrates four views: the Editor View (§ 3.1), where users can write text, run LLM prompts, and inspect changes made by LLMs; the Personal Prompt Library (§ 3.2), offering a prompt manager for creating and editing prompts locally; the Community Prompt Hub (§ 3.2), enabling users to explore and search for the latest and popular prompts shared by the community; and the Setting Panel for configuring remote or local LLMs (§ 3.4).

heading: 3.1 Editor View
text: When users open WORDFLOW in their browser or its mobile and desktop progressive web app, they are presented with the Editor View (Fig. 1A). This view shows a familiar text editor interface with a Floating Toolbar anchored on the right. Users can type or paste text into the editor. The Floating Toolbar consists of three prompt buttons and a home button (shown on the right). Each prompt button is represented by an emoji icon and corresponds to a prompt template. Users can click the prompt button to run its prompt using the current paragraph
as the input text. If a user has selected some text, the selected text is used as the input for the prompt. Users can also click the home button to open a pop-up window that contains the Personal Prompt Library (§ 3.2, Fig. 2A), the Community Prompt Hub (§ 3.3, Fig. 1B), and the Setting Panel (Fig. 4).
Prompt input templating. In WORDFLOW, a prompt template includes pre-defined prefix text and a placeholder for the input text. For example, the prefix text can be “Improve the flow of the following text”. The input placeholder in the template serves as a variable that will be substituted with the selected text from the editor. Inspired by popular prompting tools such as LANGCHAIN and PROMPTMAKER, our tool supports basic prompt templating. Users can include a special string {{text}} in their prompt template to represent the input placeholder (Fig. 2B), which will be replaced with the selected text from the editor before running the prompt. If the user does not include the string {{text}} in the template, the input text will be appended to the prompt template.
Prompt output parsing. To run users’ prompts, WORDFLOW supports remote LLM API services, such as GPT 4 and Gemini API services provided by OpenAI and Google, as well as local opensource models, such as Llama 2 (Touvron et al., 2023) and Phi 2 (Abdin et al., 2023). Users can set their preferred models in the Setting Panel (Fig. 4). After receiving the output from the LLM API service or local model, the Editor View applies Myer’s diffing algorithm (Fraser, 2012) to compare the output text with the input text. It then highlights the changes made by the LLM (e.g., addition, replacement, and deletion) using different text background
colors (Fig. 1A). Users can click on the highlighted text to accept or reject the changes.
Inspired by LANGCHAIN, WORDFLOW allows users to add optional output parsing rules to a prompt by writing regular expression (regex) text (Fig. 3), which is useful for disregarding unrelated output text. For example, a user can prompt LLMs to structure the output in XML format (recommended by prompt engineering guidelines (Anthropic, 2023)), such as “Improve the flow of the following text. Put the rewritten text in an XML tag <output></output>”. The user can then add a regex pattern .*<output>(.*)</output>.* and a replacement rule $1 to parse the LLM’s output before it is displayed in the Editor View.

heading: 3.2 Personal Prompt Library
text: After clicking the home button , users can open the Personal Prompt Library to manage their local prompts (Fig. 2A). This view organizes each prompt as a Prompt Card, allowing users to search and sort prompts based on name, recency, and run count. To change the prompts in the Floating Toolbar (§ 3.1), users can simply drag a Prompt Card into one of the three prompt slots located in the bottom row, each corresponding to a prompt button in the Floating Toolbar. To add or edit a prompt, users can click on the button or a Prompt Card to open the Prompt Editor (Fig. 2B). The Prompt Editor comprises three forms: basic prompt information (Fig. 2B), optional advanced settings (Fig. 3), and optional sharing settings. In the basic prompt information section, users can configure the title, icon, and prompt template. The advanced settings allow more experienced users to set the LLM temperature, output parsing rules, and insertion rules (Fig. 3). To share a prompt with the community, users can provide a description, tags, and recommended LLM models in the sharing settings, and then click on the button.

heading: 3.3 Community Prompt Hub
text: The Community Prompt Hub enables users to browse and search for prompts shared by WORDFLOW users (Fig. 1B). Each community prompt is represented as a Prompt Card and is associated with at least one tag. Users can filter prompts by clicking on a tag and can also sort prompts based on recency and popularity (i.e., the number of times they have been run). By clicking on a Prompt Card, users can access the Prompt Viewer (Fig. 5) to examine detailed information provided by the prompt
creator, including the title, description, prompt template, and recommended LLM models. Finally, users can click on the button to include a copy of the community prompt in their Personal Prompt Library (§ 3.2), where they can run the prompt, make further refinements, and potentially share it again with the community.

heading: 3.4 Open-source Implementation
text: We implement WORDFLOW as a progressive web app using Web Components and LIT Element (Google, 2015). Users can use it as a mobile or desktop app by saving it as a Safari Web App or a Chrome app. WORD-
FLOW allows users to run LLMs through remote API services, such as GPT 4 provided by OpenAI, or directly run open-source LLMs, such as Gemma (Team et al., 2024), Llama 2, Phi 2, and TinyLlama, in their browser (Fig. 4). We use Web LLM (team, 2023) and WebGPU to implement ondevice LLM inference. In WORDFLOW, all local prompts are stored in the local persistent storage of the user’s browser. To enable users to share community prompts, we use Amazon API Gateway and DynamoDB as a backend. Additionally, we provide a Google Doc add-on (Fig. 6) that allows Google Doc users to directly use WORDFLOW within their editor. We open source WORDFLOW as a collection of reusable interactive components that can be easily adopted by future researchers and developers in their interactive LLM projects.

heading: 4 Usage Scenarios

heading: 4.1 Improving Technical Writing
text: As a recently graduated junior software developer, Wade has been struggling with writing API documentation and system architecture descriptions. Specifically, Wade is unfamiliar with explaining technical concepts in simple language that can be easily understood by different colleagues such as developers, UX designers, and program managers. One day, Wade came across a forum thread where developers were sharing LLM prompts that had helped them improve their technical documentation writing. Wade had never thought about using LLM to assist him in his writing before. Intrigued,
he clicked on a WORDFLOW prompt link shared in a popular comment on the thread. The link opened WORDFLOW in a new tab, displaying the Community Prompt Hub along with a pop-up showing a community prompt (Fig. 5). Wade found the prompt and its description to be suitable for his writing tasks, so he clicked on the button to copy this community prompt to his local library.
Wade decided to try out this prompt to improve his writing. He opened the Personal Prompt Library and dragged the newly added prompt into one of the Favorite Prompts slots (Fig. 2A), and the prompt appeared in the Floating Toolbar in the Editor View (Fig. 1A). Wade copied a paragraph from the API documentation that he was working on. However, before clicking on the prompt button in the Floating Toolbar, Wade suddenly remembered that his company prohibits employees from using LLM services (e.g., ChatGPT and Bard) with work materials, as a measure to safeguard trade secrets and sensitive information.
Upon reviewing the documentation of WORDFLOW, Wade discovered that WORDFLOW supports running LLMs locally in browsers without sending any data to third-party services (e.g., OpenAI and Google). Therefore, he configured the LLM model to Llama 2, a local LLM model, in the Setting Panel before running the prompt on his writing. Then, he observed the changes made by the LLM model, which were highlighted in the Editor View, and found the new paragraph to be much easier to read. After using this prompt for a few days, Wade shared the prompt link on his company’s mailing list, and more developers from his company began to use it to improve technical writing.

heading: 4.2 Customizing Translation Styles
text: Ember, a senior manager in a US financial firm, recently encountered difficulties in communicating with her Japanese counterparts due to the absence of a translator. The use of traditional translation software has sometimes caused confusion among her Japanese colleagues. For example, the software translated the English idiom “break the ice” to “氷 を砕く,” which means “destroy the ice” instead of her intended meaning of “relieving tension when people interact for the first time.”
Due to the recent popularity of LLMs, Ember decided to try using them to translate her documents from English to Japanese. As she writes in Google Docs, she explored the Google Doc Marketplace for an AI add-on and came across WORDFLOW. Upon installation, she opened the Community Prompt Hub (Fig. 1B) and selected the tag , which showed various popular translation prompts. She found a prompt titled “Translate English to Japanese.”
After adding this prompt to her library, she tried to run it with the input “break the ice”. However, WORDFLOW appended the incorrect translation “氷を砕く” to her document. Drawing from her previous experience interacting with ChatGPT, Ember decided to edit the prompt and provide additional instructions to guide the LLM model in considering her translation context. She opened the Editor View (Fig. 2B). and added a new sentence to the translation prompt: “My input text is used in US corporate communications” (Fig. 6 Right). Running the prompt again, WORDFLOW generated a more suitable translation “雰囲気を和らげる,” which means “ease the atmosphere” (Fig. 6 Left). Ember back-translated the translation to English using her other translation software and felt more confident in continuing to use this prompt for future translations. Finally, to help other people who need to translate English to Japanese in business settings, she shared her updated prompt with the community by clicking on the button (Fig. 2B).

heading: 5 Future Work & Conclusion
text: In this work, we present social prompt engineering, a new paradigm that leverages social computing techniques to facilitate collaborative prompt design. To realize social prompt engineering, we design and develop WORDFLOW, an open-source and social text editor empowering users to easily create, run, share, and discover LLM prompts. Two usage scenarios highlight social prompt engineering and WORDFLOW can assist everyday users in interacting with LLMs. Reflecting on our design and development of this system, we discuss future research directions to help everyone use LLMs.
• Usage log study. Using WORDFLOW as a research instrument, we plan to conduct a usage log study to evaluate social prompt engineering and investigate (1) the effectiveness of social prompt engineering in helping everyday users craft prompts, and (2) everyday users’ LLM use cases. We will examine the evolution of prompt editing and analyze community prompts to synthesize popular use cases and prompting patterns.
• Fitting into user workflows. Future tools like WORDFLOW can be seamlessly integrated into user workflows by being in situ and ubiquitous. With recent advancements in on-device machine learning, we see great potential for on-device LLMs, which allow users to avoid sending sensitive data to external services, reduce API costs, and use LLMs without network access.
• Enhancing engagement in social prompt engineering. There are great opportunities to enrich user interaction with LLMs using social computing techniques. To encourage user participation in prompt sharing, researchers can explore intrinsic motivations, such as designing an enjoyable social environment, and extrinsic motivations, such as virtual rewards and reputation systems.
• Promoting responsible AI. Social prompt engineering presents both opportunities and challenges for responsible AI. Platforms like WORDFLOW enable users to share prompting techniques to mitigate potential harms, but also run the risk of disseminating harmful prompts such as misinformation generators. In WORDFLOW, users can report harmful prompts, and we diligently monitor and moderate community prompts. Future researchers can explore social system designs that promote responsible prompting and develop methods to detect potentially harmful prompts.

heading: 6 Broader Impact
text: We propose social prompt engineering with good intentions—to help everyday users more easily and effectively use LLMs for their everyday tasks. In addition, we design and develop WORDFLOW, an open-source tool that is publicly accessible, to help everyday users easily create, run, share, and discover LLM prompts. However, bad actors might exploit our open platform to distribute prompts designed for harmful purposes. For example, they could share prompts that generate misinformation or content fostering divisive or extremist ideologies, exploiting WORDFLOW’s reach to influence vulnerable and inexperienced audiences. To address and mitigate these potential harms, WORDFLOW enables users to report harmful prompts. We also actively monitor and moderate community prompts. Given that social prompt engineering and social prompting systems are still in their early stages, further research is needed to understand and address their potential harms.

heading: Acknowledgements
text: This work was supported in part by an Apple Scholars in AI/ML PhD fellowship and J.P. Morgan PhD Fellowship. We are extremely grateful to Kaan Sancak, Jaemarie Solyst, Xinzhi Jiang for stress-testing our tool. We appreciate Tianqi Chen, Charlie Ruan, and Alex Cabrera for their suggestions and support for integrating on-device LLMs. We express our gratitude to Alex Bäuerle, Upol Ehsan, Muhammed Fatih Balin, Luis Morales-Navarro, Wesley Hanwen Deng, Young Wu, Giorgio Pini, Justin Blalock, Viraj Kacker, Seongmin Lee, Ben Hoover, Anthony Peng, Alec Helbling, Matthew Hull, Mansi Phute, Harsha Karanth, Pratham Mehta, Joanna Cheng, Lena Do, Smera Bhatia, Angelina Wang and Parul Pandey for their advocacy and valuable feedback.

heading: Future Work
text: Reflecting on our design and development of this system, we discuss future research directions to help everyone use LLMs. • Usage log study. Using WORDFLOW as a research instrument, we plan to conduct a usage log study to evaluate social prompt engineering and investigate (1) the effectiveness of social prompt engineering in helping everyday users craft prompts, and (2) everyday users’ LLM use cases. We will examine the evolution of prompt editing and analyze community prompts to synthesize popular use cases and prompting patterns. • Fitting into user workflows. Future tools like WORDFLOW can be seamlessly integrated into user workflows by being in situ and ubiquitous. With recent advancements in on-device machine learning, we see great potential for on-device LLMs, which allow users to avoid sending sensitive data to external services, reduce API costs, and use LLMs without network access. • Enhancing engagement in social prompt engineering. There are great opportunities to enrich user interaction with LLMs using social computing techniques. To encourage user participation in prompt sharing, researchers can explore intrinsic motivations, such as designing an enjoyable social environment, and extrinsic motivations, such as virtual rewards and reputation systems. • Promoting responsible AI. Social prompt engineering presents both opportunities and challenges for responsible AI. Platforms like WORDFLOW enable users to share prompting techniques to mitigate potential harms, but also run the risk of disseminating harmful prompts such as misinformation generators. In WORDFLOW, users can report harmful prompts, and we diligently monitor and moderate community prompts. Future researchers can explore social system designs that promote responsible prompting and develop methods to detect potentially harmful prompts