Interactive and Visual Prompt Engineering
for Ad-hoc Task Adaptation with Large Language Models
Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover,
Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush(c) prompt cart(a) template card(b) detail stripe

Fig. 1. Example of PromptIDE’s interface to explore variations of different prompts. Each variation is tested against up to twenty data
examples and represented as a template card (a). For each variation, rich detail can be tracked by using the detail stripes (b). If
performance and qualitative detail are convincing, a user can collect the prompt in the prompt cart (c).
Abstract— State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting
without the need for supervised training. This approach has gained popularity in recent years, and researchers have demonstrated
prompts that achieve strong accuracy on specific NLP tasks. However, finding a prompt for new tasks requires experimentation.
Different prompt templates with different wording choices lead to significant accuracy differences. PromptIDE allows users to experimentwith prompt variations, visualize prompt performance, and iteratively optimize prompts. We developed a workflow that allows users tofirst focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising
prompts using quantitative measures of the task. The tool then allows easy deployment of the newly created ad-hoc models. We
demonstrate the utility of PromptIDE (demo: http://prompt.vizhub.ai) and our workflow using several real-world use cases.
Index Terms—Natural language processing, language modeling, zero-shot models

1

I NTRODUCTION

Machine learning models for natural language processing (NLP) haveshown impressive results on benchmark tasks; however, translating thissuccess from model architects into specific applications for model users

H. Strobelt and B. Hoover are with IBM Research. A. Webson is with Brown
University. V. Sanh and A. Rush are with Huggingface. J. Beyer and H.
Pfister are with Harvard SEAS. A. Rush is with Cornell Tech.
Manuscript received xx xxx. 201x; accepted xx xxx. 201x. Date of Publication
xx xxx. 201x; date of current version xx xxx. 201x. For information on
obtaining reprints of this article, please send e-mail to: reprints@ieee.org.
Digital Object Identifier: xx.xxxx/TVCG.201x.xxxxxxx

remains a challenge. One challenge is that benchmarks typically assume
a supervised train-test workflow. The underlying task is carefully
designed top-down with annotated training data. However, a significant
portion of use-cases of NLP does not easily fit this workflow. For
example, consider a journalist, covering legal proceedings, who isinterested in finding all cited instances of a precedent [28], or a financial
analyst looking through past company financial statements to find casesof debt obligations [7]. Annotating enough data, training a model, andthen applying it to their task requires time and expertise that may not
be available for a user in this setting.
In recent years, an alternative bottom-up approach, known as prompting, has become popular for developing ad-hoc end-user tasks in
NLP [3, 29, 32]. To solve an ad-hoc task, the user provides, in nat-

2
2.1

R ELATED W ORK
Prompting as an Interface

The use of large language models as a replacement for supervised
learning was popularized by the GPT series of language models [29].
Prompting both in the zero-short and few-shot settings has been explored widely in NLP tasks [8, 15, 32, 43]. We consider only the case ofhuman-readable prompts to large models, which contrasts with methods such as prompt-tuning [16, 20], which learns a continuous prompt
embedding, and auto-prompting [33], which attempts to search for
prompts from scratch, both of which require a training step. For more
examples see a recent survey by Liu et al. [21]. Current prompt-basedmodels are primarily based on Transformers [38], which has become thede-facto model architecture for NLP models; however, nothing about
our visual analysis is specific to the model used or task considered, only
the prompting interface.
Prompting as a means to interact with generative models is prevalent
in online demos that often deploy Transformers as assistive agents.
For instance, OpenAI released a (closed-source) API with a simple
demo to interact with their proprietary GPT-3 model using text [25].1
Another notable example is Github CoPilot [10], which uses OpenAI’s
Codex [4], a GPT-like language model trained on code, to allow everyday programmers to turn natural language prompts into code throughan IDE like VSCode. With this model, a handful of well-crafted natural language instructions can code entirely functional browser-based
video games [23]. The flexibility and effectiveness of prompt-based
approaches encourage the use of prompting as the preferred way tointeract with powerful NLP models.
The flexibility of prompts as an interface also comes with a cost, as
downstream performance is closely tied to prompt wording. Writing
good prompts to extract the desired information from a model is usually
1 Many demos built around this API can be found at https://gptcrush.
com/

sportsRankingural language, a prompt template that describes the task along withtarget answer choices. For example, simply providing a prompt template such “Is the case {case} referenced in this text? {text}” could
alone provide a classification model for an ad-hoc task with no explicit
train and test data needed. This approach is possible through advancesin training large general-purpose models for language.
The promise of prompting is that it allows domain experts to solve
new tasks with only natural language inputs. However, while there areprompts that can achieve high accuracy on specific tasks, there is a large
amount of variance in the choice of the prompt template itself. Recent
papers have described how task accuracy is dependent on specifics ofprompt choices [27, 41, 45]. This leads to a brute-force procedure underwhich dozens of prompts are written, evaluated, and compared to findthe best fit for a task. In this sense, prompting transfers similar burdensof curating expert labels to prompt construction.
This work explores how interactive visualization can support prompt
construction for domain experts. Unlike aspects of model training, suchas hyperparameter tuning, prompting is not constrained to brute-force
exploration. Because prompt templates are written in natural language,
users can craft and customize them for their tasks of interest and refine
their answer choices based on the dataset. Outputs are legible to domain
experts who can observe the process to stop it early or adjust it basedon failures. They can rewrite prompts based on system observation
to find the best expressions for their task. The tool is agnostic to theunderlying model or datasets used but aims to support the expert intheir goals.
This work makes the following novel contributions. (1) PromptIDE
automates the creation and evaluation of many prompt templates simultaneously and supports different underlying models, tasks, and datasets.
(2) PromptIDE encourages a principled and repeatable workflow for
prompt engineering. Users are guided through the process, with opportunities for iterations at each step. (3) We demonstrate the utilityof PromptIDE and our workflow for several real-world use cases for
ad-hoc NLP models. We end the paper by exploring future challenges
and avenues for follow-up work.

relatedtext

Top-K (K=2)

wide
news

world

news

Generation
Answer Ranks
sc1 = f( sports, news )sc2 = f( world , wide

, news )Fig. 2. A language model can generate text by ranking all possible
events (tokens) at each time step based on their probability. Often, only
top-k tokens are considered as generation output. In PromptIDE, only
the answer ranks of defined answer choices are compared against eachother.

left to trial and error by the user, with few exceptions. One tool thatseeks to solve this problem is Prompts.ai [46], a visual tool that uses the
GPT-3 API to explore how a user-specified prompt template affects the
behavior of GPT-3 on individual examples, in conversations, and withdifferent generation parameters. However, its input space is limiting,allowing only one prompt template with limiting syntax to be tested
at a time. PromptSource [2] increases the power of the templating
language used to write general prompts. It also provides a platform forthe community to create, evaluate, and explore new prompts. PromptIDE extends this work, providing a principled workflow to automate
the time-intensive process of creating and evaluating many prompt
templates on different models, datasets, and tasks.
2.2

Visualization for NLP

Visualization tools to interact with language models have grown increasingly popular alongside the rise of the models themselves. These tools
can serve several functions: (1) to expose the internals of a particulararchitecture (e.g., Transformers, RNNs, LSTMs) to understand how it
encodes knowledge of the language [1, 6, 11, 12, 19, 24, 34, 39, 40]; (2)to explore and compare the behavior of a model’s internal distributions
during text generation [9, 35]; and (3) to understand how model behavior differs under controlled input or parameter changes [22, 26, 36].
PromptIDE is agnostic to the underlying language model and thus
serves the functions of (2) and (3).
Other visualization tools that treat NLP models as black boxes focus
on visualizations of output distributions and model performance withsingle custom input or static sets of inputs. For example, GLTR [9] focuses on visualizing output probability distributions to support humans
in detecting whether a provided text was generated by a model. LMdiff [35] extends these visualizations to support comparisons between
different language models on user-provided inputs and static corpora.
Neither of these provides an exploration of the input space. The Language Interpretability Tool (LIT) [36] is a comprehensive toolkit thatenables rapid exploration and error analysis for a model on a larger input dataset. However, LIT’s comprehensive analyses are not conducive
for rapid, iterative improvements of a prompt on general NLP tasks.
NLIZE [22] serves as a debugging tool that evaluates how a language
model outputs changes as a result of perturbations to its hidden staterather than its input. Unlike these existing tools, PromptIDE enables
exploration and evaluation of the infinitely large space of possible input
prompts.
3

M ODEL : P ROMPTING FOR NLP

Prompting is a paradigm for solving ad-hoc NLP tasks. We particularly
focus on zero-shot prompting that assumes we do not have access to
any training examples. Prompting is typically used in conjunction
with large pre-trained language models [3, 29]. These models arepowerful, but their size makes them difficult to train directly, whichfurther encourages this style of zero-shot prompting.
Prompting assumes access to a large language model (LM) pretrained on generic text. An LM is a probabilistic model over text.
Given an input text x, it gives the probability of output text y. The ideaof the prompting technique has been facilitated by recent improvementsin these models, primarily deriving from scaling Transformer neuralnetworks [38]. Recently, researchers have trained LMs that are directly
targeted for the end-use of prompting [31, 42]. These language models
can all be queried in a standard way. In this work, we utilize threedifferent LM queries (Fig. 2):
Generation - Sample a target output for a given input,ỹ ∼ p(y | x).
Ranking - Compare the rank score of different texts, where f is afunction based on p(y | x)( [3], details in Sect. 5.5),
f (y1 , x) < f (y2 , x).
Top-K - Find the k highest probability outputs from the model,
topky′ p(y = y′ | x).
We represent an NLP task as a table of examples, each associatedwith a fixed set of fields and a label. As a running example, we consider
the task of document topic classification. For this task, there is one field,
the article text, and one label, the topic of the article. The document
text might consist of,
Authorities have halted oil export flows from the main
pipeline in southern Iraq after intelligence showed a rebel
militia could strike infrastructure, an oil official said on
Saturday...
whereas the corresponding topic label would be World.
Contrast the standard ML approach for this task to prompting. A
standard approach would collect supervised labels for the task andtrain a model on these examples. In zero-shot prompting, we do nothave access to a training set. Prompting facilitates ad-hoc models by
converting each test example directly to a form natural language input
to which a large LM can respond. A user introduces a prompt template
and a set of answer choices. The prompt template describes how to
map the example fields to an input string x, whereas the answer choices
describe how to convert potentials outputs y back to labels for the task.
We follow the work of PromptSource [2], where researchers introduced a format for describing prompt templates.
In which section of a newspaper would the text appear?
{document}with answer choices given as a dictionary of labels:[World, Sports, Business, Science and Technology].
Utilizing the prompt template, we can construct an example prompt
x for model conditioning and answer choices y1 , . . . , yn . Each of the
choices can then be ranked under the model to provide an evaluation
score for the dataset.
We also support other task formats such as multiple-choice tasks
that allow different answers for each example depending on specificfields in the data set.
{question} Choose between the following answers:(A) { answerA } (B) { answerB } ...
Here the answer choices could either be the corresponding letters or
the answers themselves. We discuss this decision in Section 6.2.
Throughout, we assume a small set of labeled validation data forquantitative evaluation, which differs from the research on true zeroshot learning [27]. We note though that this is not enough data toattempt to automatically generate prompts directly. The main elements
of prompting can be summarized as:
M1 - Prompt Template. A user writes a prompt template consistingof a task description in natural language that utilizes the fields from thetask in a situated context. This leads to the construction of the input xthat is used for conditioning of the large LM.
M2 - Answer Choices. A user provides a dictionary of answer choices
paired with the original labels for a given task that offer different possible output wordings y to be considered by the model. The underlying
model uses ranking to determine which of these answer choices toselect. The original label paired with this answer choice is then theclassification choice selected.
M3 - Evaluation. A user can evaluate the current version of the system
under a known prompt for a set of validation data. This step will
provide a proxy score for how well the given wording of the prompt is
at capturing the underlying task.
4 G OALS AND TASKSWe held discussions with our NLP researchers on the team to determine
the functionality for a minimal viable prototype. We imagined how ourtool could enable our example personas journalist and analyst to workwith prompting. In the following, we summarize the insights of these
discussions.
While prompting is a promising approach, it is still too workintensive for many use cases. The problem is that performance ishighly dependent on the specific wording choices for templates (M1,
M2), which is reflected in a high variance in accuracy (M3). For example, previous work has shown that different choices of prompts often
lead to a more than 10-point spread in task accuracy between the best
and worse choices at stage M1, even though both were approved byhuman editors [31].
Brute force approaches for prompt search require a user to write alarge set of possible prompts and validate them empirically on a task.
However, these approaches are both computationally expensive andslow. An interactive tool should provide an approach for using fewer
resources and allow fast iterations for prompt engineering.
4.1 Goals
The high-level aim of PromptIDE is to provide a better approach for
prompt development by domain experts in terms of four targeted goals:
G1 - Support a broad set of ad-hoc NLP tasks. It is essential thatthe system is generic, as the user may not know beforehand the natureof the end-user task. The tool should present a single interface formultiple different potential downstream tasks. In this way, a tool is
not targeted specifically to model trainers but also to the end-user’s
application goals.
G2 - Faster and more informed prompt writing through feedback
from data. The process should let users target new language tasks
that arrive during their projects. A tool should enable the user todevelop prompts efficiently. It should also provide feedback on what
effect prompt variations have. The goal is to make prompt search lessautomated by giving the user the human-in-the-loop ability to edit andconstruct prompts based on their domain knowledge.
G3 - Ground prompt choices in quantitative measures. Prompt
customization replaces training for ad-hoc systems, but it is still criticalthat choices be grounded in task evaluation metrics. A key element ofinteraction in the system is that task scores be directly available in thetool itself.
G4 - Ease deployment of models to end uses. The tool should
provide a testbed for the wording and usage of the prompts, but for
actual usage, the prompts must be able to be run and used on actual
data cases. Our goal is for the user to be able to directly export the
prompts written in the system to a full system for use on the final task
itself.
4.2 Tasks
Given these goals, we identified a series of tasks that guided the development of PromptIDE:

M1
M2

Q1

Q2

M3
Q3

Fig. 3. PromptIDE UI is organized like a notebook with foldable sections
that follow the order of the main workflow but also allow quick iterations
within a section or between neighboring sections.

<M1(q), M2, M3> for q in Q1xQ2xQ3

Fig. 5. Prompt variation section. The user can quickly generate andprogressively test variations of prompts to identify promising candidates.
Each prompt variation is represented as a template card highlighting thevalues for template variables qx in unique colors and showing the numberof correct answers vs. data tested.

foldable sections (Fig. 3) that lead to the following workflow: a dataset
navigation section to select and browse data, a prompt variation section
to broadly explore prompt variations, a prompt refinement section to
help fine-tune a specific prompt, and a prompt testing section to explore
results of testing on a larger scale.
Fig. 4. Dataset navigation. Browsing through dataset samples anddataset schema lets the user get acquainted with the data. In this case,“label” refers to the index position within a list of potential labels.

Task 1 - Formulating and trying out prompts and prompt variations.
To allow users to quickly explore and run different prompt templates
for a specific NLP task, the tool provides an interactive interface forformulating and trying out many different prompts in a manner thatprovides feedback on a small set of real data examples. This interface
is agnostic to the NLP task. [G1, G2]
Task 2 - Encoding prediction details of the model. Even after finding
a good prompt template, it is critical to ensure that the answer choices
provided ensure that the prompt is useful and leads to successful results
on realistic NLP problems. The tool exposes the predictions of themodel beyond ranking to allow for other choices that could be selected.
[G2, G3]
Task 3 - Testing promising prompts on task performance. From
Task 1 and Task 2, there are many combinations of prompt templates
and answer choices to be considered, and each may be run on manydifferent examples. Testing provides insight to the user on how choices
for promising candidates lead to different resulting outcomes in a larger
data regime and informs them which elements of their design have beensuccessful. [G3]
Task 4 - Export prompt for concrete deployment. The end goal of
the system is to provide packaged prompts that can be used in realtasks after the exploration phase has concluded. For this to be useful,the final system must collapse the exploration steps and provide a full
prompt for deployment. [G4]5

D ESIGN

The visual interface of PromptIDE provides means to address the aforementioned tasks. With the text being the main carrier of information
for the tasks, we designate the most screen real estate to text and interaction with text while introducing visual encodings for abstraction
when useful.
At a high level, PromptIDE appears as a continuous notebook of four

5.1

The Four Sections of PromptIDE

The dataset navigation section (Fig. 4) enables browsing and selectionof a reference dataset needed for testing prompt templates against tasks
T1–T3 and respective goals G1–G3. It provides access to many standard NLP data using the huggingface datasets [18]. In addition, a user
can provide their own data as CSV or JSON files. While describing thedataset schema is syntactically sufficient, the tool enables browsing tohelp get a better understanding of the encoded semantics behind eachdata dimension. E.g., in Fig. 4, the dataset glue/sst2 (Stanford Sentiment Treebank v2) is selected, and the schema indicates the presenceof three fields: sentence, label, and idx. The naming of fields and therespective datatypes match the intuition of how a sentiment classification dataset could look like. But browsing shows that sentences can bevery short and of low quality.
The prompt variation section (Fig. 5 and Fig. 1) allows formulatinga prompt experiment for broad variations of prompt templates (M1 in
Sect. 3) using up to three template variables q1 , q2 , q3 , and spanningtheir combinatorial space Q1 × Q2 × Q3. The user can formulate a
prompt template (M1) using dataset fields (e.g., {{text}}), template
variables qx , and plain template text. The list of answer choices (M2)
can be formulated as a static or dynamic list (e.g., World | | | Sports | | |Business). The correct answer (M3) can be dynamically retrieved fromthe dataset and the answer choices (e.g., answer choices[label]).
After defining the space of all variations < M1(q), M2, M3 > forq ∈ Q1 × Q2 × Q3, the model can now be asked to predict answers for
all variations on a small set of data items. As soon as the experimentis started, each variation of M1(q) is represented as a template card,
highlighting the template variables qx by a preassigned color (q1 in
red, q2 in blue, q3 in gold) and showing the sum of correctly evaluatedsamples as a bar chart. Then, progressively, the full set of variations istested against the set of data items, adding results for one data item at atime. For each step in the progression, the order of template cards is
updated, keeping the stack sorted by decreasing performance against thead-hoc task. At any time, when the user has gathered sufficient evidenceof what could be promising candidates, they can stop the progression
and re-iterate before the experiment would have been finished. This
procedure of iteratively formulating prompt variations and trying themout on a small dataset addresses task 1 (G1, G2).

Fig. 6. Prompt refinement section. To incorporate small optimizations for promising prompts, the user can test against a small subset of data
frequently. For each data item, the evaluation chip indicates the prediction, the ground truth, if they match (green or gray), and the normalized
distribution of probabilities as a bar chart on top.

The prompt refinement section (Fig. 6) takes one of the template
variations and enables quick iterations for fine-tuning this template.
Using only one variant M1 allows the user to try on one batch of data
items at interactive rates (T1, G1, G2). For each iteration, a performanceoverview is shown in an evaluation chip that indicates if the task wasevaluated successfully for a data item (green background) or not (grey).
It shows the predicted answer in black and the correct answer in green.
If they match, only one is shown. On top of each evaluation chip, a bar
chart indicates the relative probability of all possible answers sorted byrank and normalized to maximum probability.
The green bar highlights the ground truth, and the leftmost bar
indicates the current prediction. This encoding allows, e.g., insights if
a wrongful prediction was close to being a coin flip (similar height formost left bar and green bar) or if a correct prediction was a good choice(significant difference between the leftmost bar in green and the secondbar from the left), targeting task T2. Besides running quick iterations
using a small data regime, the user can trigger an experiment of testing
against a larger test set (T3).
The results of this testing are shown in the prompt testing section(Fig. 7) of PromptIDE. After the model completes the testing against alarger dataset, the results are presented such that a user might be able
to answer the questions: How well did my prompt perform against thetask (T3, G3)? What did the model confuse using my prompt (T2, T3,
G3)? How could I potentially tweak the answer choices (T3, G3)?
For all three questions, PromptIDE provides a visual encoding that
can help find an answer. A stacked bar chart indicates the share ofcorrect predictions vs. incorrect ones to indicate prompt performanceat the highest abstraction level (Fig. 7(a)). If the answer choices allow,
a confusion matrix shows across class scores ((Fig. 7(b)) and if the
in-class performance was good (large values on the diagonal axis). If
the answer choices are dynamic but still form groups, the tool shows
the top ten most abundant ground truth groups in the confusion matrix.
For datasets where the answer choices do not fall into groups, nothingis shown.
Finally, to help answer the question about potential answer choice
tweaks, the tool records the top five ranked generation tokens for eachdata item independently. Then, for each group of answer choices, thesetokens are accumulated, and the number of appearances in the top fiveis recorded. Additionally, the average rank they had per answer groupis calculated (value between 1 and 5). The list of the most frequent andhighest-ranked answer tokens is shown in Fig. 7(c), with tokens sorted
by decreasing appearance count. The green background indicates thebest average rank in the group (which does not need to be the mostappearing item). The use case in Sect. 6.1 gives an example of thepractical use of this feature. Similar to the confusion matrix, nothing isshown if the answer choices do not form groups.
5.2 Visual Encoding and Interactions
Across the foldable sections, a user can investigate more detail about the
prompted data items using the detail stripes (Fig. 1). Each detail stripe
consists of an upper part highlighting the answer options, the predicted
answer (on top), and the ground truth answer (encoded in green). The
panels below show details about the prompted text, the ground truth
answer, and what the model has generated. Upon request, each ranked
result can also show the probability of the respective answer option.
Detail stripes can be shown when tracking a specific template variation (Fig. 5) in the prompt variation section. They can be unfolded
beneath the performance chips in the prompt refinement section (Fig. 6).
They show detail about the respective subsets when the user clicks on
the performance bars or the cells of the confusion matrix in the prompt
testing section (Fig. 7). Detail stripes target task T2 (G2).
To collect and store promising prompts, PromptIDE provides a
shopping cart (Fig. 8). From the cart, templates can be sent to theprompt variation section or the prompt refinement section. Templates
can be exported from the cart to deploy them seamlessly with the toolingprovided in the PromptIDE repository (T4, G4). To add prompts, auser can trigger buttons from the template cards in the prompt variation
section (Fig. 5) or the refinement section (Fig. 6). If a prompt has
been evaluated against the larger testing set, the performance result
will be automatically added to the corresponding item in the cart forcomparison. Furthermore, a read-only PromptSource shopping cart
(Fig. 6) shows templates that have been created for a specific dataset
by the global community. These prompts could serve as inspiration forstarting a user’s own prompt idea.
5.3 Example Interactive Workflow
A prototypical interaction workflow using PromptIDE starts with opening the data browser section to investigate the schema and concrete
examples of own or globally available data. The user clicks on one itemthat serves as a good exemplar and is shown for reference in the prompt

(a)

(b)

(c)

Fig. 7. Prompt testing section. Showing results of testing against a mid-sized subset of data to get quantitative measures to help answer questions:
How well did my prompt perform against the task (a)? What did the model confuse using my prompt (b)? How could I potentially tweak the answer
choices (c)?

Fig. 9. Early version of prompt variation testing. The interface is lesspowerful than the final version: only one template variable q1 that caneither be prefixed or appended to the prompt. Testing is only against
one dataset item. Each answer option has its own color, which was notconsidered useful by early users.

Fig. 8. Shopping carts allow users to collect their own prompt variants(left) or browse through prompts made available by the BigScience
promptsource community (right).

variations and prompt refinement sections. The user then opens theprompt variation section and writes down a prompt template using data
and template variables. Alternatively, they open the prompt source cart
and scan for examples that could serve as inspiration, copy them over,
and add template variables. They run the experiment and stop it earlybecause one prompt variation seems to perform very well. They use
the shortcut to copy the specific prompt over to the refinement section,
where, through multiple small edits and along multiple data portions,the prompt seems to be performing well. During this, the users observe
the performance chips and occasionally the detail stripes. Later, theuser triggers the larger-scale testing to see if they over-optimized theprompt for a local data range. After a short period of time, they notea high confusion between labels A and D. Inspired by the most common top five predictions, they iterate over their answer choices andrun the test again, resulting in better performance. The prompt (andmaybe some intermediate steps) are saved in the shopping cart. The
best-performing prompt is exported to a JSON file. The user can now
run the newly created ad-hoc model (original LM plus prompts in the
JSON file) with a simple input-output interface or as a batch processing
script on new data for their customers.
5.4 Design Iterations and Rationale
While consulting with our co-authors, who are NLP domain experts,
we went through multiple design iterations on different parts of the tool.
In this section, we highlight some of them and provide a more in-depth
design rationale for certain parts of PromptIDE.

The overall design as a notebook with foldable sections was inspiredby the popular use of Jupyter notebooks in the NLP community. It
allows occupying the screen with different views while keeping theviews connected in a natural order. This allows the user to build amental map that is established by scrolling up and down. If, instead,
we had mapped the steps of the interactive workflow to independentviews, the user might not be able to build this mental model due tothe constant context switches. To assist with navigating to a specificsubsection, the menu bar acts as a table of content that scroll-animatesto the respective sections.
Our use of progressive visualization methods [37] was driven bytechnical design considerations. We started thinking about what themodes of interaction during a progression would be. We decided
to use progressive updates for the prompt variation testing because
early stopping in this phase of the exploration has proven useful. For
example, testing on data with just one label class may reveal local
effects relatively quickly, in which case the user can stop the test. On
the other hand, we did not use the early stopping interaction during
prompt testing on a larger dataset. The goal for this part of the workflow
is to observe more global effects by running the test to completion.
Early stopping might increase the chance of observing a subset ofthe data that has local effects, negating the intention of larger-scaletesting. Furthermore, the progression in the prompt variation section iscontrolled by the client and stops if the tool is closed. The larger-scaletesting is controlled by the server and continues running even when
PromptIDE is being closed on the client-side.
A fundamental technical design decision was which language to use
for formulating the prompts with data and template variables (for T1).
The three options for our decision were: (1) invent a new language, (2)use a templating language, or (3) use a general programming language.

M1

M2

M3
(c)
Substitute "Technology" with "Science"
in M2
(b)

(a)

(d)

Fig. 10. Use Case Document Classification for a news dataset. Details in Sect. 6.1

We quickly decided against inventing our own language that would
have to be explained and maintained. The choice for using a templating language (Jinja) over a general programming language (Python)
was based on the observation that PromptSource [31] used the samelanguage. This made the PromptSource parsing work immediately
available to us and also allowed us to build the read-only shopping cartwith prompt examples gathered by the partner project. Another concern
was that a general programming language could be a prime target formalicious server attacks once the tool is released to the public.
The detail stripes underwent several design iterations. An earlyversion assigned a color to each answer choice (Fig. 9). But our domain
experts did not express interest in being able to track ranks of answer
choices across data items and found it more distracting than useful. The
switch to a simpler color encoding was also enforced by the required
capability to allow dynamic answer options that change per data item
and repeat a few times. This would have required an impractically large
number of categorical colors.
5.5

Implementation

Our PromptIDE prototype consists of a backend in Python that communicates with a frontend written in Typescript and Svelte. We use theopenly available pre-trained T0-3B large language model [31] provided
through the huggingface platform 2 . For long-running queries, thebackend provides a custom-built queuing and execution system thatkeeps the memory footprint for the model low. In our implementation,
answer options are ranked by the decreasing average log-likelihood:i<la( ∑ log(pia ))/la ,i=0with pia being the probability for the i-th token of answer a and la being
the answer’s token length.
The demo system is available at http://prompt.vizhub.ai. We
will make the source code available upon acceptance of the paper.
The open-source version allows easy use of custom data that is either
provided as a CSV file or by the Huggingface dataset interface.
2 https://huggingface.co/bigscience/T0_3B

6

U SE C ASES

We illustrate how we can use PromptIDE to interactively prompt a diversity of NLP tasks, compare these prompts, save them and export themoutside of PromptIDE. We experiment with a range of standard tasks in
NLP, including document classification, reading comprehension, andnatural language inference. The tool enables seamless development ofprompts for a wide variety of tasks and formats [G1] while quickly providing feedback on prompts patterns that generalize to many instances
[G2].
6.1

Document Classification

The most common end-user task for NLP [17] is document classification for an end-domain. The task of document classification is todetermine the label of a document from a fixed set. It can be used in adhoc models for filtering a large set of documents or collecting statistics
about a large collection. Domain expertise is critical in classification
since the specific wording may lead to different results.
As an example use case we consider a prototypical version of thistask with the goal of classifying the topic of a document. The AG News
dataset [44] is often used for benchmarking this task and consists of text
documents and labels that indicate which topic they are associated with(labels are canonically listed as “World, Sports, Business, Sci/Tech”).
This task (introduced in Sect. 3) is representative of an ad-hoc classification task that a user might consider for prompt development with alanguage model [G1].
We can first explore this dataset through the dataset navigation
section and then initialize the process through the prompt variation
section. This section allows the user to write several different prompt
templates [T1] as well as answer templates for these prompts [T2].
Fig. 1 (a) shows some examples of these templates and the way theyuse fields from the underlying data set and template variables q1 and q2.
We select one of the better-performing prompts to investigate further inthe prompt refinement section.
Figure 10(a) shows the output of the selected prompt for some data,
and we observe that there is some confusion about the labels. Upon
detailed inspection (see example in Fig. 10(b)), we observe that thelabels can be ambiguous even under human evaluation. So, we send
the prompt to larger-scale testing [T3,G3].

The testing reveals the specific problem. The ground truth label“Technology” is not a great choice and gets confused with other phrases(Fig. 10(c)). Upon inspection of the Top 5 Rank Predictions, we observethat the token “Science” is a very frequent wording for this groundtruth, even more than the term “Technology” itself. We use this insightto refine the answer options in the refinement section by substituting“Technology” with “Science”. In this case, the wording of the answer
changed, but other feedback would lead to changes in the wording of
the prompt itself.
We run the testing again. After receiving the results, we see that themodification increased performance substantially (Fig. 10(d)). We can
now go to the shopping cart and export the new model [T4,G4].
6.2

Multiple-Choice Answering

The task of reading comprehension is to answer a question from acomplex document. It can be used for ad-hoc models to find documentsthat provide answers to specific questions. The RACE dataset [14] isa multiple-choice reading comprehension dataset built from English
examinations used for benchmarking this task. For a given text extract
and a question about that text, the model has to choose the correct
answer among four possibilities. Unlike document classification, thepossibilities are different for each sample.
We explore prompts that introduce four answer choices, associatingthem with letters (A, B, C, and D): “Possible answers:”, “Choose
between A, B, C and D:”, or nothing. Figure 11 shows that “Choose
between A, B, C, and D:” consistently gives worse results than thetwo other variations, independently of how the input is introduced (q1).
We discard this variation, and following [42], we try out prompts that
explicitly state the instruction at the beginning of the prompt. We
note that even though the performance remains the same for most of
the prompt variations, some variations degrade the performance (forinstance “Please select the correct answer among all of the options.”).
We select one of these prompts for refining and test it on 100 examples. From the confusion matrix and the most common top 5 predictions (Figure 12), we notice that the model is often predicting “E”
even though it is not among the answer choices and that it tends topredict “C”, as it is always the second most frequent prediction when
it is not the correct label. This observation hints at potential class imbalances in the training mixture of the underlying model and warrants
more investigation. We find that the training mixture contains a variety
of multiple-choice question answering datasets that contain from 3(COS-E [30]) to 8 (QASC [13]) answer choices, which leads to thehigh frequency of labels A, B, and C.
The interactive nature of PromptIDE makes it easy to develop a
lot of prompts, save them and export them to a different environment.
Once the best prompt has been identified, the end-user can use themoutside of PromptIDE, for instance, to deploy a prompted language
model in production [G4]. The JSON export format makes the wholeinteractive development extremely versatile and compatible with mostof the standard codebase.
6.3

Sentence Similarity

The task of natural language inference is to determine the semantic
relationship between two similar sentences. An ad-hoc domain expert
would use a similar task to check whether documents agree with or contradict specific target statements. Recognizing Textual Entailment [5] isan inference dataset where a model is asked to assume that one piece oftext (the “premise”) is true and to classify whether another piece of text
(the “hypothesis”) must also be true, i.e., whether the premise “entails”
the hypothesis. For example, if the premise is Steve Jobs was attacked
by Sculley and other Apple executives for not delivering enough hot
new products and resigned from the company a few weeks later. andthe hypothesis states that Steve Jobs worked for Apple., then the data islabeled as “true” or “entailment”.
Past work [41] has used prompting for this task but found that it wasdifficult to find a prompt wording that performed robustly on different
examples. Starting from the prompts in [41] we can use PromptIDE for
estimating the models’ robustness to different wordings in the templates.

Fig. 11. Mapping dynamic answer choices to simple outputs such as
letters can help the model by simplifying its output space.

Fig. 12. The label “C” is systematically the second most frequent prediction of the model, which hints at a potential class imbalance in the
training set of the underlying model.

In particular, the prompt variable variants (q1, q2, q3) allow us to easily
control and test wording variations.
To start finding good prompts for RTE, we copy and paste the main
instructive variations of [5] into q1. In previewing 20 examples, we
see T0-3B’s performance across prompts ranges from a respectable
70% to 50% (Fig. 13a). We add the best and the worst prompts tocarts for further investigation under the “Refine Prompt” section. Fromthe confusion matrix (Fig. 13b), see that most mistakes are models
under-predicting entailment for sentence pairs where the premises do,in fact, entail their hypotheses (i.e., ground truth is “Yes”).
Another noteworthy finding from the “Most Common Top 5 Predictions” is that, for the entailment class, the model generates “No”
with a higher rank more frequently than “Yes” (Fig. 13c), yet it also
generates “yes” and “True” with higher rank more frequently than “no”
and “False” (Fig. 13d). Similar to Sect. 6.1, we could simply change
the desired target words for a performance boost. But we can also
study the effect of explicitly providing the answer choices in the input sequence by appending “True or false?” to every template, whichfurther improves performance for all templates. However, explicitly
providing the answer choices in input sequences does not always control model behaviors in accord with human intuition. Prepending theanswer choices at the beginning of input sequences does not confer any
performance boost.
Finally, PromptIDE is also prime for studying adversarial conditions
such as irrelevant and misleading prompts. For example, using themisleading-extreme prompts from [41], Fig. 14a shows that “is this

(a)

(b)

(c)

No > Yes
yes > notrue > false(d)Fig. 13. Use case Natural Language Inference on RTE dataset [5]. Details in Sect. 6.3

...

...
(a) with "?"(b) without "?"Fig. 14. Using PromptIDE to study adversarial conditions by using misleading-extreme prompts from [41]. Details in Sect. 6.3

grammatically correct” outperforms just instructive prompts such as“does this imply that”. However, if we remove the question marks in the
global template, then the misleading and irrelevant prompts consistently
underperform the instructive ones (Fig. 14b), suggesting that models
may be using question marks as some kind of heuristic feature.
7

variables we should support (q1, q2, q3) vs how many can we handle
with combinatorical explosion of variants.
Another lesson learned was that the more complex charts tend to beignored whilst the simple charts with complex algorithmic ideas got
much faster appreciated and accepted. E.g., the most common “Aha!”
moment was caused by top 5 tokens idea in the prompt testing section.

E ARLY F EEDBACK AND L ESSONS L EARNED

After internal deployment within the prompting expert group we received initial qualitative feedback from two colleagues. When being
asked about most helpful PromptIDE features, they answered: “visualizing the best verbalizer in the confusion matrix form and the Most
Common Top 5 Predictions format. Furthermore, it’s really helpful topreview the prompt variations and their performances and use refine tocopy to the Prompt Refine section.”
Main criticism came from the limitation of prompt variant variables:“In agnews dataset, when I add q3 and q4, the Prompt variable variants
field doesn’t have q4. . . . we could have some indicator of maximum
prompts allowed.” Interestingly, there where also recommendations
about UI improvements: “I am confused about the use of red underlines. . . I would prefer to use something other than red because red indicatessomething is wrong.” or simplifications like “I think using some ‘i’popup icon for storing some information (with more descriptions) ismuch better than outright displaying technical texts like (frequency >5 | avrg rank < 5).”The lesson we learned from working on this project are manifoldwith some of them are already indicated in Sect. 5.4. The surprise to us
was how much constructive feedback we got about the user interface
during the development and after deployment, often started with an
apology about being a non expert in UI. We recommend to establishingearly on that all UI feedback is very welcome and not a overstepping incompetences.
A constant fight for compromising between flexibility vs simplicity
became a major task. E.g., determining how many prompt variation

8 C ONCLUSIONS AND F UTURE W ORK
We present PromptIDE, a system for domain experts to customize models for ad-hoc tasks without requiring training expertise. The system
adopts the prompting framework that has recently been developed for
NLP tasks while developing an interactive visualization environment
for customizing prompts for new tasks. The approach extends beyond
brute-force prompt trial-and-error to facilitate the exploration of prompt
language and the development of new prompt templates and answer
choices. The system is open-source http://prompt.vizhub.ai/
and can work with any available language model backend.
As the methodology of prompting develops, there are many areasof extension for PromptIDE. Currently, the tool supports tasks with aknown set of choices, but there are many NLP tasks where the correctlabel may be a free-form response. These could be incorporated throughsupport for new metrics such as BLEU in the prompt refinement section.
PromptIDE also assumes that the user is able to deduce how to update
prompts based on task metrics. Ideally, the system could provide direct
advice on how to change prompts or highlight text spans that would
lead to better results through methods like gradient saliency. Thesemethods are currently too computationally expensive to run on large
promptable models, but will likely improve with more research.
9 ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers for their constructive feedback and helpful comments. This work was partially fundedthrough NSF grant IIS-1901030 and the MIT-IBM Watson AI Lab.

R EFERENCES[1] J. Alammar. Interfaces for explaining transformer language models. https://jalammar.github.io/explaining-transformers/,
2020.
[2] S. H. Bach, V. Sanh, Z.-X. Yong, A. Webson, C. Raffel, N. V. Nayak,
A. Sharma, T. Kim, M. S. Bari, T. Fevry, et al. Promptsource: An integrated
development environment and repository for natural language prompts.
arXiv preprint arXiv:2202.01279, 2022.
[3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,
A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. HerbertVoss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler,
J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever,and D. Amodei. Language models are few-shot learners. In H. Larochelle,
M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds., Advances in Neural
Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual, 2020.
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language
models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[5] I. Dagan, O. Glickman, and B. Magnini. The pascal recognising textual
entailment challenge. In Machine Learning Challenges Workshop, pp.
177–190. Springer, 2006.
[6] J. F. DeRose, J. Wang, and M. Berger. Attention flows: Analyzing andcomparing attention mechanisms in language models. IEEE Transactions
on Visualization and Computer Graphics, 27(2):1160–1170, 2020.
[7] I. El Maarouf, Y. Mansar, V. Mouilleron, and D. Valsamou-Stanislawski.
The finsim 2020 shared task: Learning semantic representations for thefinancial domain. In Proceedings of the Second Workshop on Financial
Technology and Natural Language Processing, pp. 81–86, 2020.
[8] T. Gao, A. Fisch, and D. Chen. Making pre-trained language models
better few-shot learners. In Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 3816–3830. Association for Computational Linguistics, Online, Aug.
2021. doi: 10.18653/v1/2021.acl-long.295[9] S. Gehrmann, H. Strobelt, and A. Rush. GLTR: Statistical detection andvisualization of generated text. In Proceedings of the 57th Annual Meetingof the Association for Computational Linguistics: System Demonstrations,pp. 111–116. Association for Computational Linguistics, Florence, Italy,
July 2019. doi: 10.18653/v1/P19-3019[10] Github. Github copilot · your ai pair programmer. https://copilot.
github.com/.
[11] B. Hoover, H. Strobelt, and S. Gehrmann. exBERT: A Visual Analysis
Tool to Explore Learned Representations in Transformer Models. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics: System Demonstrations, pp. 187–196. Association for Computational Linguistics, Online, July 2020. doi: 10.18653/v1/2020.acl-demos.
22[12] T. Jaunet, C. Kervadec, R. Vuillemot, G. Antipov, M. Baccouche, andC. Wolf. Visqa: X-raying vision and language reasoning in transformers.
IEEE Transactions on Visualization and Computer Graphics, 28(1):976–
986, 2021.
[13] T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal.
Qasc: A dataset for question answering via sentence composition.
arXiv:1910.11473v2, 2020.
[14] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the2017 Conference on Empirical Methods in Natural Language Processing,pp. 785–794. Association for Computational Linguistics, Copenhagen,Denmark, Sept. 2017. doi: 10.18653/v1/D17-1082[15] T. Le Scao and A. Rush. How many data points is a prompt worth? In
Proceedings of the 2021 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pp. 2627–2636. Association for Computational Linguistics, Online,
June 2021. doi: 10.18653/v1/2021.naacl-main.208[16] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameterefficient prompt tuning. In M. Moens, X. Huang, L. Specia, and S. W.
Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana,
Dominican Republic, 7-11 November, 2021, pp. 3045–3059. Association

for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.
243[17] Q. Lhoest, A. V. del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil,
J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Sasko, G. Chhablani, B. Malik, S. Brandeis, T. L. Scao, V. Sanh, C. Xu, N. Patry,
A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière,L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas,
A. M. Rush, and T. Wolf. Datasets: A community library for natural
language processing. In H. Adel and S. Shi, eds., Proceedings of the2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2021, Online and Punta Cana,
Dominican Republic, 7-11 November, 2021, pp. 175–184. Association for
Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-demo.21[18] Q. Lhoest, A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen,
S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško,
G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry,
A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière,L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas,
A. Rush, and T. Wolf. Datasets: A community library for natural language
processing. In Proceedings of the 2021 Conference on Empirical Methods
in Natural Language Processing: System Demonstrations, pp. 175–184.
Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, Nov. 2021. doi: 10.18653/v1/2021.emnlp-demo.21[19] R. Li, W. Xiao, L. Wang, H. Jang, and G. Carenini. T3-vis: visual analytic
for training and fine-tuning transformers in NLP. In Proceedings of the2021 Conference on Empirical Methods in Natural Language Processing:
System Demonstrations, pp. 220–230. Association for Computational
Linguistics, Online and Punta Cana, Dominican Republic, Nov. 2021. doi:
10.18653/v1/2021.emnlp-demo.26[20] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts forgeneration. In C. Zong, F. Xia, W. Li, and R. Navigli, eds., Proceedings of
the 59th Annual Meeting of the Association for Computational Linguistics
and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August
1-6, 2021, pp. 4582–4597. Association for Computational Linguistics,
2021. doi: 10.18653/v1/2021.acl-long.353[21] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train,
prompt, and predict: A systematic survey of prompting methods in natural
language processing. CoRR, abs/2107.13586, 2021.
[22] S. Liu, Z. Li, T. Li, V. Srikumar, V. Pascucci, and P.-T. Bremer. Nlize: A
perturbation-driven visual interrogation tool for analyzing and interpreting
natural language inference models. IEEE transactions on visualizationand computer graphics, 25(1):651–660, 2018.
[23] A. Mayne. Building games and apps entirely through natural language
using openai’s code-davinci model. Andrew Mayne Blog, Mar 2022.
[24] Y. Ming, S. Cao, R. Zhang, Z. Li, Y. Chen, Y. Song, and H. Qu. Understanding hidden memories of recurrent neural networks. In 2017 IEEE
Conference on Visual Analytics Science and Technology (VAST), pp. 13–24.
IEEE, 2017.
[25] Open AI.
OpenAI playground.
https://beta.openai.com/
playground.
[26] A. Pearce.
What have language models learned?
VISxAI
Workshop 2021, https://pair.withgoogle.com/explorables/
fill-in-the-blank/, Jul 2021.
[27] E. Perez, D. Kiela, and K. Cho. True few-shot learning with language
models. NeurIPS, 2021.
[28] O. Popescu and C. Strapparava. Natural language processing meets journalism. In Proceedings of the 2017 EMNLP Workshop. Copenhagen,Denmark: Association for Computational Linguistics.–2017, 2017.
[29] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.
Language models are unsupervised multitask learners. OpenAI blog,
1(8):9, 2019.
[30] N. F. Rajani, B. McCann, C. Xiong, and R. Socher. Explain yourself!
leveraging language models for commonsense reasoning. In Proceedingsof the 2019 Conference of the Association for Computational Linguistics(ACL2019), 2019.
[31] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai,
A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, M. Dey, M. S. Bari, C. Xu,
U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak,
D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X.
Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma,
A. Santilli, T. Fevry, J. A. Fries, R. Teehan, S. Biderman, L. Gao, T. Bers,
T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shottask generalization, 2021.
[32] T. Schick and H. Schütze. It’s not just size that matters: Small language
models are also few-shot learners. In Proceedings of the 2021 Conferenceof the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pp. 2339–2352. Association
for Computational Linguistics, Online, June 2021. doi: 10.18653/v1/2021.
naacl-main.185[33] T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, and S. Singh. Autoprompt:
Eliciting knowledge from language models with automatically generated
prompts. In B. Webber, T. Cohn, Y. He, and Y. Liu, eds., Proceedings of the2020 Conference on Empirical Methods in Natural Language Processing,
EMNLP 2020, Online, November 16-20, 2020, pp. 4222–4235. Association
for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.
346[34] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush. Lstmvis: A toolfor visual analysis of hidden state dynamics in recurrent neural networks.
IEEE transactions on visualization and computer graphics, 24(1):667–676,
2017.
[35] H. Strobelt, B. Hoover, A. Satyanaryan, and S. Gehrmann. LMdiff: A
visual diff tool to compare language models. In Proceedings of the 2021
Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 96–105. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, Nov. 2021. doi: 10.
18653/v1/2021.emnlp-demo.12[36] I. Tenney, J. Wexler, J. Bastings, T. Bolukbasi, A. Coenen, S. Gehrmann,
E. Jiang, M. Pushkarna, C. Radebaugh, E. Reif, and A. Yuan. The language
interpretability tool: Extensible, interactive visualizations and analysis for
NLP models. Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing: System Demonstrations, pp. 107–118,
2020.
[37] C. Turkay, N. Pezzotti, C. Binnig, H. Strobelt, B. Hammer, D. A. Keim,
J.-D. Fekete, T. Palpanas, Y. Wang, and F. Rusu. Progressive data science:
Potential and challenges, 2018. doi: 10.48550/ARXIV.1812.08032[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural
information processing systems, 30, 2017.
[39] J. Vig. A multiscale visualization of attention in the transformer model.
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37–42. Association for
Computational Linguistics, Florence, Italy, July 2019. doi: 10.18653/v1/
P19-3007[40] Z. J. Wang, R. Turko, and D. H. Chau. Dodrio: Exploring transformer
models with interactive visualization. In Proceedings of the 59th Annual
Meeting of the Association for Computational Linguistics and the 11th
International Joint Conference on Natural Language Processing: System
Demonstrations, pp. 132–141. Association for Computational Linguistics,
Online, Aug. 2021. doi: 10.18653/v1/2021.acl-demo.16[41] A. Webson and E. Pavlick. Do prompt-based models really understandthe meaning of their prompts? ArXiv, abs/2109.01247, 2021.
[42] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M.
Dai, and Q. V. Le. Finetuned language models are zero-shot learners.
CoRR, abs/2109.01652, 2021.
[43] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou.
Chain of thought prompting elicits reasoning in large language models.
arXiv preprint arXiv:2201.11903, 2022.
[44] X. Zhang, J. J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
[45] T. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. CoRR,
abs/2102.09690, 2021.
[46] S. Zhidkov. Advanced GPT-3 playground. https://prompts.ai/.