AutoTrial: Prompting Language Models for Clinical Trial Design
Authors: Zifeng Wang
Cao Xiao
Jimeng Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Drug Development
Clinical Trial
Large Language Model
TL;DR: We are the first to leverage LLMs for drafting clinical trial protocol design.

Abstract: Clinical trials are critical for drug development. Constructing the appropriate eligibility criteria (i.e., the inclusion/exclusion criteria for patient recruitment) is essential for the trial's success. Proper design of clinical trial protocols should consider similar precedent trials and their eligibility criteria to ensure sufficient patient coverage. In this paper, we present a method named AutoTrial to aid the design of clinical eligibility criteria using language models. It allows (1) controllable generation under instructions via a hybrid of discrete and neural prompting, (2) scalable knowledge incorporation via in-context learning, and (3) explicit reasoning chains to provide rationales for understanding the outputs. Experiments on over 70K clinical trials verify that AutoTrial generates high-quality criteria texts that are fluent and coherent and with high accuracy in capturing the relevant clinical concepts to the target trial. It is noteworthy that our method, with a much smaller parameter size, gains around 60% winning rate against the GPT-3.5 baselines via human evaluations.

AutoTrial:
Prompting Language Models for Clinical Trial Design
Zifeng Wang
Cao Xiao GE Healthcare Seattle, WA, USA cao xiao@ge. com
Jimeng Sun UIUC
UIUC
Urbana, IL, USA zif zifengw2@i www.iiiiiiiiiii..iioiiiiiimiiiee edu
Urbana, IL, USA jimengeillinois edu
Abstract
using LLMs, with the solution focusing on the fol- lowing aspects.
Clinical trials are critical for drug development. Constructing the appropriate eligibility criteria (i..., the inclusiom/exclusion criteria for patient recruitment) is essential for the trial's success. Proper design of clinical trial protocols should consider similar precedent trials and their eli- gibility criteria to ensure sufficient patient cov- erage. In this paper, we present a method named AutoTrial to aid the design of clini- cal eligibility criteria using language models. It allows (1) controllable generation under in- structions via hybrid discrete neural prompting, (2) scalable knowledge incorpora- tion via in-context learning, and (3) explicit reasoning chains to provide rationales for un- derstanding the outputs. Experiments over 70K clinical trials verify that AutoTrial gen- erates high-quality criteria texts that fluent and coherent and with high accuracy in captur- ing the relevant climical concepts to larget trial. It is noteworthy that our method, with much smaller parameter size, gains around 60% winning rate against the GPT-3.5 baselines human evaluations.
Comprehending instructions. The LLM will be prompted with key information about trial, such the target conditions and treatments, and the additional instruction generate the criteria for participant recruitment. This process requires the employed LLM to comprehend the input and adapt the input instruction to generate precise eligibility criteria that meet the specified objec- tives. also necessitates the domain knowledge about clinical trials stored in LLMs.
Referring to prior studies. A thorough liter- ature search is important for human experts to design clinical trials (Chew, 2019). Similarly, the employed LLMS should be able leverage the context information, such as retrieved eligibil- ity criteria from prior successful studies, as the reference to generate better trial design.
Rationalizing the generation. LLMs should offer the rationale behind the generated criteria, which is important for clinical experts to under- stand and adopt the generation results in practice.
1 Introduction
Generative large language models (LLMs) are drawing attention due to their ability to create co- herent and human-like text documents. Clinical trial design documents written the planning stage of the drug development process, which is crucial for the success of the trial. However, it can challenging even for experienced professionals: around 57% trial protocols have at least one sub- stantial amendment in eligibility criteria (CSDD, 2016). The suboptimal trial design may cause in- sufficient recruitment, severe adverse events, or insignificant efficacy, thus inducing huge financial losses and time waste. Each amendment will fur- ther cause millions of dollars in loss and months of delays.
In this paper, we propose to augment clinical trial design using LLMs, motivated by the evidence that LLMs can act as implicit knowledge bases (Petroni et al., 2019; Taylor et al., 2022). Our method is equipped with instruction tuning for trial proto- col design and explicit supervision for producing grounding rationales. This is enabled with the fol- lowing technical features:
Instruction prompting for adapting expert in- structions. Instruction tuning for granular con- trol over the generated criteria to follow diverse user intentions.
In this paper, we propose to generate the cligi- bility criteria for clinical trials in natural language
Scalable and efficient knowledge expansion. A combination 1) external memory for dense re-

---

triever and 2) internal memory for neural prompt- ing, which is amenable updating the model incrementally as new data are available.
are the first develop LLMs focusing on trial de- sign through a mixture of techniques, including instruction tuning, wridence-grounded generation, and supervised learning for rationale generation.
Explicit supervision for generating grounding rationales. Adaption of LLMs with a reasoning capability through a supervised paradigm, mak- ing the generated criteria more transparent and interpretable.
2.2 Clinical Trial Design
The clinical trial design is research topic for the NLP community, and there only few works related to clinical trial design, either focusing on trial feature embedding or trial design evaluation. For trial feature embedding, Marshall et al. (2017) extracted text pieces that describe the key trial char- acteristics as a summary report. More recently, Wang and Sun (2022) developed self-supervised document model for dense retrieval for clinical tri- als. For trial design evaluation, Kim et al. (2021) manually adjusted criteria broaden patient ac- crual and assess the influence of criteria, while Liu et al. (2021) utilized Shapley scores (Lundberg and Lee, 2017) to estimate the change of hazard ratio of the included oncology patients when removing each criterion. Despite these efforts, existing work focuses clinical trial design automation.
Our AutoTrial is the first model that utilizes LLMs for automating clinical trial design. It repre- sents an important step toward using AI to facilitate clinical trial design. The rest of the paper is orga- nized as follows: In $2, we review related work. In $3, we dive into the proposed method in detail. In $4, we present the experiment results. It is notewor- thy that AutoTrial is proven to generate accurate criteria (with precision 0.91, recall 0.92, F1 0.91, and Jaccard score of 0.84 in clinical accuracy eval- uation) while almost all baselines get less than 0.5 in these metrics. Moreover, our method reaches around 60% winning rate against GPT-3.5 in trial design tasks via human evaluation. Finally, in $5, we conclude and discuss future work.
Method
2 Related Work
AutoTrial utilizes decoder-based architecture for generating target criterion based on input trial synopsis and manual instructions. The training process consists of two stages: pretraining and finetuning.
2.1 Large Language Models
Large language models pre-trained on web-scale text data exhibit extraordinary emergent capabil- ity in diverse set of natural language processing (NLP) tasks (Kaplan et al., 2020; Brown et al., 2020). It was recently witnessed that LLMs can be further tuned to align with human preferences through instruction tuning (Chung et al., 2022; Wang et al., 2022) and reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022: Yuan et al., 2023; Dong et al., 2023).
During the pretraining stage, the model is trained on large corpus of trial documents in order to learn to reason through multiple steps and mimic the retrieved input criteria exemplars.
In the finetuning stage, the model is trained to generate the target criterion according to the in- put instructions. For example, an instruction <age> that urges the model to populate the crite- rion describing the participant's 's age requirement.
Despite the remarkable capabilities of large lan- guage models (LLMs) trained on general text cor- pus, they often face challenges when generating highly domain-specific tasks unless they undergo additional tuning. Research has shown that even "small" 300M LM, with instruction tuning, can outperform LLMs with over 100B parameters (Ya- sunaga et al., 2022). This finding encourages the efforts to develop customized expert LLMs by performing instruction tuning on domain-specific datasets, c.g., clinical notes and scientific publi- cations (Singhal et al., 2022). In this work, we https://ppatform.
It is noteworthy that the model can extended to new instructions and trial exemplars without retraining. The flowchart is depicted in Fig. 1. We will elaborate on the details of the training and inference procedures of AutoTrial next.
3.1 Problem Setup
The generation model represented by the func- tion f. and generates a target criterion ye based on input x = {x,, Xx,
Fangine gpt-3. 3-0-3.55tumm S-turbo--sa0 e301: openai. watpaai..coo/ooooo bttdooss/mmeel//pp/./..
x,,x, x, Here, x, denotes trial

---

Pretraining
Generation
[title disease treatment]
Reasoning multiple steps
setup
trial setup
+ exemplar
Model
rationale
[Ine5] ye
Xxx Xy
X5 multi-step
Xet
exemplar entity1
exemplar entity2
exemplar @ entity3
|inc1 @ inc2 @
...]
[inc3 inc4 ...]
reasoning
Finetuning
Model
Model
Model
entity
bmi
xy
generate
encode E,
multiple candidates
multi-step reasoning
prompiymen
setup
exemplar
Model
[inc3 Inca _1
[incl inc2 @ ... @ criterton1
X
[inc1 inc2 @ @ criterion1'
multi-step reasoning
entity
criterion
body mass indox
[inc1 @ inc2 @ ... @ criterion1"
[[ncc inc2 @ G. age age above 18)
between 18.5 and
clustering ranking by ppl
kg/mm2 Y: criterion
x
X
N
[inct @ inc? @ @ ciiterion
Retrieved In-context Exemplars
y;
y:
The workflow of the proposed AutoTrial. Step I: pre-train unlabeled trial documents with prompts to mimic the multi-step reasoning. Step Il: finetune the model to generate criteria under instructions. Step III: generate diverse target criteria by instructions with large-scale sampling plus clustering and ranking.
Figure I:
setups, which is a concatenation of the trial title, condition, and treatment, as the elements illustrated by Fig. I. x, denotes the discrete prompt describ- ing the objective criterion, e.g.. "bmi" prompts the model to generate the criterion for body mass in- dex of the participants. X, denotes the exemplars retrieved from relevant trials built for the in-context learning of LLMs. To this end, formulate X = x[,,,, which contains the reasoning steps x e.g., chain of criteria that leads to the target criteria, the targeting instruction x e.g., the objective to be set for recruiting patients, and the target criterion x that describes the requirement based on the instruction.
3.2.1 Discrete Prompt
The discrete prompt is motivated by the prospect of in-context learning of LLMs (Wei et al., 2022), as the reasoning ability LLMs can enhanced via the input-output exemplars, e.g., the concatenation of a series of criteria x the target instruction x and the target criteria x.. We formulate the discrete prompts with specialized tokens:
we wrap the introduction of trial setups, including title, disease, and treatment, using specialized tokens like <title>, <disease>, and <treatment>, The setup offers the basic context of a trial.
1. Trial Setup:
The model also controlled by continuous prompt hn, which is specific each type of in- struction, e.g., the targeting entity that the criterion should contain. The model is trained to generate criteria y with multi-step reasoning: generating relevant criteria one by one and ultimately yield- ing the target criterion. Therefore, the generation process is expressed in Eq. (1),
2. In-context Exemplar: we curate the exemplar that resembles the multi-step reasoning procedure: the model first generates a series of intermediate rationales that lead the final answer. Concretely, the exemplar x, {x( x5, x retrieved from the external knowledge store and demonstrate as the template for the model outputs. x are many eli- gibility criteria wrapped by <inc> or <exc> indicat- ing inclusion or exclusion criteria. x" describes the instruction wrapped by <statement>, e.g., tell the model to generate criterion describing the age re- quirement by "<statement> age". x is the target criterion wrapped by <target>, e.g., "<target> age is above 18 yrs old".
y f(x,, Xey X,, h,).
(I)
Referring to the exemplar Xe, the model outputs y = Yt @ Ye. where y; is the reasoning steps and denotes the target criterion.
3.2 Hybrid Prompting
3. Textual Instruction: following the exemplar, x, enforces the model obey the instruction, wrapped by <statement> such as "<statement> gender".
We opt to employ hybrid of discrete and neural prompting to endow the model with the ability to generate criteria based on specific instructions.

---

The exemplars are stored in an external knowl- edge store providing an open-book reference that the model can refer to during the generation. is built on the training data - ((x,y:)) that is amenable to edit, add or delete during the course of training and generating needless of re- training the model. We utilize neural encoder T named Vec (Wang and Sun, 2022) that encodes trial setups x, to dense embeddings, as h, (xx)) Fd that carry rich semantics of the trials. Consequently, the knowledge store is given
the relation "NYHA (III. IV)" from the criteria "NYHA class is above II". However, the parser may not be able to extract relevant instructions from all available trial documents. We hence propose to train our method in two stages: first pretraining on large set of unlabeled trial documents and then finetuning on the processed dataset of instruction- criteria pairs. This approach allows to make the most of the available data and facilitate the model performance. Pretraining. We create pretraining dataset
by Eq. (2).
Cpre {(xs, Xes Yt, Vt, ye):)?
(4)
= ((h., xe) (x,y) E EC}
(2)
where the model f is urged to generate y yr Dyc in Eq. (1). The inputs comprise the trial setup X, and the exemplar x, which is also composed of multiple criteria. Drawing the inspiration from (Taylor er al., 2022), we decide to include prompts and special tokens in the pretraining stage. Specifi- cally, we explicitly emphasize the step-by-step rea- soning task by inserting the separate tokens <inc> and <exc> into x, and yı. and the model is super- vised to generate the intermediate rationales and yield the target criterion.
with the embeddings serving as the keys and the exemplars as the values. Here, the rector-based search engine can be implemented for efficient ex- emplar retrieval on the fly.
3.2.2 Neural Prompt
Consider the embeded input tokens Xel (x1,...,.77 as Hzi {h....., h)) E Rixd we prepend neural prompts to Ha get the prompted input H²1 = hp H Formally, we create a set of instruction indices I, the i-th instruction x, is parameterized by Eq. (3),
Our method is built based on decoder-based CLM (e.g., GPT2 (Radford et al., 2019)) where the decoder predicts y autoregressively. Denote the learned decoding distribution as po(-), the ob-
hp (i.:).
where E, R x is the trainable embedding ma-
jective is the maximum log-likelihood estimation given by Eq. (5).
trix. [i, indicates looking up the i-th row the matrix; MLP: Rd' + d projects the embedded instruction to the same dimension as H.
LMLE Σ log po (5)
The neural prompting is modular, meaning that it can be easily modified to incorporate additional in- structions I' by simply extending the index set I {I, and the embedding matrix E, EE, E,) for those instructions. When the model finetuned on new data, we can only update the instruction embedding E, while the rest of the model remains frozen. This allows the model to effectively learn to generate based on a broader range of instructions while minimizing the risk of catastrophic forget- ting, i.e., the performance degradation on previous data.
where yel are tokens in y before the I-th token; L the total number of tokens in the target y. Finetuning. After pretraining, the model is fine- tuned on the dataset C, and taught to follow the instruction when generating criteria. The inputs and outputs are described in Eq. (1). In addition to the MLE loss in Eq. (5), we apply a contrastive loss LCL (Su et al., 2022) to enhance the model representation learning, as in Eq. (6).
L
La = Lx (L
Σ
Σ
3.3 Multi-stage Training
(6)
bij=lj
As described in $3.1, have dataset contain- ing pairs of input instructions (denoted as x,)) and corresponding criteria (denoted as yc). We extract clinical relations from the raw criteria to formu- late the training and testing data, e.g., extracting
max (0. s shhy hy,) + s shh hy
where hy is the embedding of token yt, P is the pre- defined margin, s(-) is the cosine similarity func- tion. The finetuning loss combines the objectives

---

Table The statistics the used trial data.
well those with void eligibility criteria. We ex- tracted 75,977 clinical trials and applied Clinical Trial Parser (FAIR, 2022) to extract 751,258 med- ical relations from the eligibility criteria of these trials. The train-test split is shown in Table 1. For each trial, we sampled one criterion as the target and several others as input exemplars, resulting in 2,528 231 unique training samples out of 400K tri- als the pretraining data. The validation and test trials were excluded from the pretraining data.
Train
Valid
Test
# trials
54,703
6,079
15,195
# inclusion
153,169
17.145
42,269
# exclusion
128,310
14,581
35,247
Avg inc length
121.0
120.4
118.5
Avg exe length
148.3
153.0
145.2
from Eqs. (5) and (6) as given by Eq. (7),
LrT = LMLL LCL.
(7)
4.2 Evaluation Strategy
Automatic Evaluation. To evaluate the quality of the output criteria, which are expressed natural language, we employ metrics from the NLG lit- erature, including CIDEr (Vedantam et al., 2015), ROUGE-L (Lin, 2004), METEOR (Lavie and Agar- wal, 2007), and BLEU (Papineni et al., 2002). These metries allow to assess the fluency and coherence of the generated criteria quantitatively. We evaluate all the methods at the criteria level and trial level. At the criteria level, the model will generate each criterion separately, using the concatenated trial setup texts and the first three tokens of the targeting criteria as input. At the trial level, the model will take the concatenated trial setup texts as input and generate all of the criteria for the trial once.
Note that the decoding distribution in the finetuning stage is (olyy<s,, x, hp) that differs from the one used in the pretraining shown in Eq. (5).
3.4 Generation
Denote the vocabulary by V. we conduct top-k sampling repeatedly to acquire diverse candidates Y (5 by Eq. (8),
y1 ~ po e0, x, hp), s.t. i1 E ((k)),
(8)
where ((k)) is subset of V that maximizes Eyev(h Polyly<i, x, hp), and [D(k.)] k2 We further adopt clustering and ranking to select sam- ples from the generated candidates. We first encode y by Trial2Vec to dense embeddings hy and ap- ply k-means clustering with hig clusters. We then compute the perplexity (ppl) of each output ÿg. and pick the sample with the minimum ppl in each clus- ter to form the final candidate set with ky samples. An example of the input and output of AutoTrial can be found in Table 5 in the uppendix.
Clinical Accuracy. To evaluate the clinical ac- curacy of the generated criteria, we run Clinical Trial Parser (FAIR, 2022) on the generated crite- ria to extract the medical relations and compare them with the relations extracted from the corre- sponding ground-truth criteria. We evaluate the overlapping of two relation sets by the precision, recall, F1 Fl-score, and Jaccard similarity. Human Evaluation. We perform manual evalua- tion to compare the generated clinical trial design from our method with the generated by general LLM. We enlisted the expertise domain experts to assess and choose the superior output between our method and the LLM's output for a given trial synopsis. This allowed us to collect feedback and calculate the winning rate.
4 Experiment
We conduct extensive experiments to evaluate AutoTrial in the following aspects:
The overall quality in terms of criteria-level and trial-level trial protocol generation.
The capability of the continual update for new trials and instructions.
4.3 Implementations
The ablation analysis for the components of the proposed method.
To the best of our knowledge, there were exist- ing algorithms for automatic trial design generation. We thus propose to compare AutoTrial with dif- ferent NL. models: finetuning (FT), prefix-tuning (PT) (Li and Liang, 2021), catrivval-uugmenee gen- eration (RAG) (Lewis et al., 2020), and contrastive
4.1 Dataset
We collected clinical trial documents from Clin- icalTrials.gov (NIH, 2023) and filtered out those without valid interventions, diseases, or titles, as

---

Table Automatic evaluation of eligibility criteria generation results on the test set on the trial-level, i.c. compare the concatenated inclusion/exclusion criteria of trial with the corresponding groundtruth. BI is short for BLEU-I.
Trial level
Inclusion ROUGE-L
Trial level
Exclusion
Method/Scores
B1
METEOR
CIDEr
B1
METEOR
ROUGE-L
CIDEr
GPT2-FT
9.2
23.6
7.5
0.10
7.2
8.3
2.7
0.02
GPT2-RAG
16.4
25.2
9.9
0.09
9.7
19.4
7.4
0.09
GPT2-PT
20.0
19.2
22.8
0.17
16.5
15.0
12.6
0.14
GPT2-SimCTG
9.8
24.8
10.6
0.11
9.4
11,4
5.5
0.06
T5-FT
21.9
29.8
18.3
0.18
13.2
13.9
8.0
0.07
TS-RAG
22.7
28.9
16.7
0.15
14,9
11.8
7.5
0.06
TS-PT
43.2
21.0
23.3
0.09
17.8
23.4
11.1
0.19
T5-SimCTG
22.1
30,3
17.6
0.17
15.8
13.1
7.7
0.06
AutoTrial
58.7
40.8
40.6
0.24
54.4
36.3
35.3
0.33
Table 3: Automatic evaluation of eligibility criteria generation results on the test set the criteria-level, i.e.,
compare the generated inclusion/cxclusion criteria with the groundtruth one by one. B1 is short for BLEU-1.
Criteria level Inclusion
Criteria level Exclusion
Method/Scores
B1
METEOR
ROUGE-L
CIDEr
B1
METEOR
ROUGE-L
CIDEr
GPT2-FT
9.0
18.1
27.0
0.54
18.2
16.4
21.4
0.47
GPT2-RAG
19.9
17.7
28.0
0.50
13.8
17.3
22.1
0.52
GPT2-PT
10.1
13.9
18.3
0.23
14.1
12.3
10.9
0.13
GPT2-SimCTG
32.3
16.6
32.3
0.68
29.1
14.5
23.0
0.68
TS-FT
12.8
6.8
9.4
0.11
11.2
3.9
5.1
0.04
TS-RAG
14.7
11.3
14.2
0.30
12.1
5.3
6.8
0.09
T5-PT
22.3
9.8
15.8
0.26
10.4
15.5
9.3
0.18
T5-SimCTG
20.3
11.5
11.6
0.33
12.4
10.3
10.5
0.17
ArtoTrial
39.7
24.3
35.3
0.79
38.4
24.2
30.0
0.68
learning + contrastive search (SimCTG) (Su al., 2022). We choose GPT2 (Radford al., 2019) and T5 (Raffel et al., 2020) as the backbones. We also compare with GPT-3.5 to verify if general LLMs can generate reasonable eligibility criteria (Ouyang et al., 2022)
inclusion criteria generation generally better than for exclusion criteria generation. We conjecture that inclusion criteria are presented prior to exclu- sion criteria in the training data, which may lead to the truncation of the latter due to the model's ss maximum acceptable length. Besides, errors may accumulate when generating criteria in autore- gressive manner. AutoTrial mitigates the order issue credited to the hybrid prompting.
For our method, we leverage GPT-2 (Radford al., 2019) as the backbone model. We set the maximum context length as 1,024. the pertaining stage, we train the backbone model with batch size of 64, learning rate 5e-5, weight decay lc-4, and 5 epochs. In the instruction tuning stage, we train the model with batch size of 16, learning rate 5e-5, weight decay le-5, and 10 epochs.
Clinical accuracy. We present the clinical accu- racy evaluation results in Table As aligned with the automatic evaluation results, AutoTrial per- forms better at criteria generation, with bigger performance gap. For example, AutoTrial is the only method that yields recall above 0.5 (w/ 0.91), FI above 0.6 (w/ 0.91), and Jaccard scores above 0.4 (w/ 0.84) in inclusion criteria generation. It wins over baselines with prominent margin in exclusion criteria generation. These results demon- strate that our method can generate criteria accu- rately aligned with the provided instructions.
4.4 Exp 1: Generation Quality
Text quality. Table shows automatic evalu- ation scores with the evaluation done at the trial level. The results show that AutoTrial demon- strates superior performance over the baselines in all four metrics (BLEU-1, METEOR, ROUGE-L, and CIDEr) for both inclusion and exclusion critc- ria. We draw similar conclusions from Table with the evaluation at the criteria level.
We also observe that most methods obtain decent precision and AutoTrial has the best performance. implics a low hallucination rate in our method's generated text because most generated relations
One notable finding is that the performance for

---

100 trial groups divided by the targeting disease. Base- Table Clinical accuracy evaluation results cligibil- ity criteria generation results the test set. P, R, F1, Jac are short for precision, recall, Fl score, micro-Jaccard score, respectively.
30
METSOR
BLEV
20
20
Type
Method/Score
P
Jac
GPT2-FT
0.74
0.35
0.47
0.31
50
GPT2-RAG
0.81
0.41
0.54
0.37
1.5
GPT2-PT
0.75
0.45
0.56
0.39
40
ROUGe
GPT2-SimCTG
0.89
0.40
30
1.0
Inclusion
T5-FT
0.77
0.10
0.17
0.09
20
0.5
T5-RAG
0.82
0.13
0.12
TS-PT
0.74
0.17
0.27
0.16
T5-SimCTG
0.68
0.04
0.08
0.04
AutoTTrill
0.91
0.92
0.91
0.84
Figure 3: In-depth analysis of generation quality across
GPT2-FT
0.69
0.21
0.33
0.20
GPT2-RAG
0.59
0.26
0.36
0.22
lines are based on GPT-2.
GPT2-PT
0.36
0.25
0.30
0.17
GPT2-SimCTG
0,80
0.23
0.36
0.22
Exclusion
T5-FT
0.24
0.03
0.06
0.03
4.5
Exp 2: In-depth Analysis
TS-RAG
0,24
TS-PT
0.18
0.25
0.21
0.12
Performance Divergence. We divided the raw test trials by their target diseases, leading to 100 non- overlapping sets, with each set sharing the same target disease. We then evaluated the generated texts within each subset. We created box plots of the obtained scores (evaluated on the combination of inclusion and exclusion criteria) in Fig.
T5-SimCTG
0.16
0.01
0.02
0.01
Auto'Trial
0.85
0.89
0.87
0.76
Humen Evel: Inclusiion Crioara
Human Eval: Exclusion Oriadi
Poloo Ern Warning It5s
Werinn
0.4
0.4
0.2
Our results indicate that AutoTrial exhibits su- perior performance across all metrics, It achieves the highest median performance and has a more stable score distribution, with both high upper bound and lower bound for all metrics. Among the baseline methods, SimCTG performs the best on three metrics, with the exception of METEOR. However, should noted that its worst-case per- formance was typically much lower than that most other methods. We also zoom to show the performances of trials targeting the top eight most frequent diseases/condittons in Fig. 6, where AutoTrial consistently wins over all basclines.
0.0
0.0
GPT-3.5-Turbo
GPT.3.5-Turbo
Figure 2: Human evaluations of the winning rate of AutoTrial against GPT-3.5 when GPT-3.5 does zero- shot generation (no exemplar), 1-shot, and 5-shot in- context learning.
are also concretely mentioned in the groundtruth eligibility criteria. However, the baselines perform much worse regarding the recall and Jaccard scores. It indicates that AutoTrial is advantageous in the high coverage of targeting clinical relations in the generated criteria.
Qualitative Analysis. We present several qualita- tive results of our model in Table 5. The model inputs have two parts: manual input and automati- cally built input, where the manual input concate- nated with the automatic input and passed to the model. Users set up the basic trial information and can opt to offer different instructions for generating criteria. As observed in the first four rows of Table 5, the outputs vary when provided with different instructions for the same trial. Furthermore, can be observed that the generated outputs are fluent, coherent, and closely resemble the referential man- ually written criteria.
Human Evaluation. The human evaluation re- sults are available Fig. 2, where we identify that our method significantly outperforms the GPT-3.5 baselines, i.e., in over 60% cases, the output cri- teria from AutoTrial are considered better than the from GPT-3.5. This again emphasizes the op- portunity of developing expert LLMs that surpass general LLMs at much less cost. It is also interest- ing that 5-shot GPT-3.5 worse than the 1-shot and zero-shot ones. We conjecture that GPT-3.5 is impacted by the context that contains irrelevant exemplars when generating for the targeting trials.

---

10-
Inclusion
exclusion
22
M.
BLEU
Nethod
-20
13
Retran
100
Incremental
433
1.0
33
80005
ROUCCE
3.6
Methed
Nethod Rozar
0.4
2.
Aetrinn
We present the results Fig. 4. The Incremental model demonstrates the capability of mitigating catastrophic forgetting when extended to new data. However, the gap between the two vari- ants expands over time. The Incremental model decays to the level of the best baseline after be- MAD MGA
A7
MAh RAG Prompt
AT
5/b MER AR BAG Fronpe
Figure 4: Comparison between two variants of AutoTrial: Re-train and Incremental. The Re-train variant trained on all subsets. while the Incremental variant updates its knowledge only new subsets. The scatter plot also includes the performance of four base- lines, which are trained all data.
Figure 5: Ablation experiments of AutoTrial when removing one module. AT: the original version; w/o MSR: without the multi-step reasoning supervision; w/o RAG: without the retrieval-ane -aageeee--egeeeeeedd generation; w/o Prompt: without the neural prompting.
reach trade-off between utility and cost.
4.6 Exp 3: Incremental Learning
4.7
Exp 4: Ablation Study
One major merit of AutoTrial is continually up- date its internal and external memory without the need of retraining on all collected data. To demon- strate the capability of AutoTrial in continuously updating its knowledge, we designed two vari- ants of our method: Re-train and Incremental. These variants were trained on four subsets of the training set: (C1,C2, C3, Ca), with the instruc- tion types being equally assigned and mutually ex- clusive in each subset. The models encountered the subsets sequentially, with the Re-train model learning by combining all previously seen subsets, e.g., it would be trained and evaluated on {C1,C)) C2] when Cg is revealed. In essence, Re-train is the theoretic upper bound for all incremental learning methods. In contrast, the Incremental model is also evaluated C2) but it would be trained on C2 only when is revealed. Additionally, during training, the Incremental model only updates the neural prompting while freezing all other parame- ters.
We conducted ablation study (shown in Fig. 5) to compare the original version of AutoTrial with its variants when removing certain components: the multi-step reasoning supervision (w/o MSR), the retrieval-augmenter generation (w/o RAG), and the neural prompting (w/o Prompt). The results show that both RAG and Prompt have significant impact on the final performance. MSR performs similarly inclusion criteria compared the orig- inal version but has worse results on exclusion cri- teria. Despite this, MSR is ultimately retained in the final model as produces more balanced re- sults among inclusion and exclusion criteria and also provides insight into the model's 's reasoning path, making it more interpretable.
5 Conclusion
In summary, this paper presents AutoTrial that uses language models to aid in the design of clini- cal trial protocols. Our method is able to generate high-quality criteria texts that are fluent, coherent, and clinically accurate, by using a combination of controllable generation, scalable knowledge in- corporation, and multi-step reasoning. This can potentially reduce the risk of clinical trial failure by ensuring that trials are properly designed and have sufficient power to evaluate proposed therapies.
ing updated on the fourth subset when the total number of instructions is 4x more than in the first subset. We hence suggest incrementally updating AutoTrial until the new instructions reach around 3x more than the last fully retrained checkpoint to
Limitations
The proposed method, AutoTrial, is a valuable tool for designing clinical trials by providing con-

---

trollable generation under instructions, scalable knowledge incorporation, and multi-step reasoning, However, it is important to note that one limitation of the method is that is dependent on the qual- ity of data used to train the language model. the clinical trial database used to train the model contains biases or inaccuracies, these limitations may be present in the generated criteria texts. To ensure the quality of the generated criteria texts, it is crucial use high-quality, accurate, and up-to- date data to train the language model, which can be achieved by regularly updating the clinical trial databases used for training.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang. Kashun Shum, and Tong Zhang. 2023. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv preprint urXiv:2304.0667.
FAIR. 2022. Library for converting clinical trial eligi- bility criteria to machine-readable format.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford. Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883
Jae Hyun Kim, Casey Ta, Cong Liu, Cynthia Sung. Alex M Butler, Latoya A Stewart, Lyudmila Ena, James R Rogers, Junghwan Lee, Anna Ostropolets, et al. 2021. Towards clinical data-driven cligibil- ity criteria optimization for interventional covid- 19 clinical trials. Journal of the American Medical In- formatics Association, 28(1):14 22.
Additionally, the method may not be able to ac- count for unexpected or rare side effects or issues that may occur during the trial, which may impact the safety and efficacy of the proposed treatment. It is important to note that AutoTrial should be considered a supportive tool for designing clinical trials and the final decision should always be made by human clinicians. The tool can aid in identifying relevant trials and generating high-quality criteria texts, but ultimately, it is the responsibility of the clinician to evaluate the overall design and safety of the trial, taking into account the unique charac- teristics and needs of the trial population. The tool should be used as an aid in the design process, but not as a replacement for the expertise and judgment of human clinicians.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Wonkshop on Statistical Machine Trans- lation, StatMT '07, page 228 231, USA. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge seaelldgge-itetiveeeeeeee NLP tasks. Advances Neural Information Processing Systems, 33:9459- 9474.
A Akknowledgeents
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning: Optimizing coutinuous prompts for generation. Annual Meeting of the Association for Computational Linguistics, pages 4582-45997.
This work was supported by NSF award SCH- 2205289, SCH-2014438, and 183808..
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-111
Ruishan Liu, Shemra Rizzo, Samuel Whipple, Navdeep Pal, Arturo Lopez Pineda. Michael Lu, Brandon Amieri, Ying Lu, William Capra. Ryan Copping, et al. 2021. Evaluating eligibility criteria of oncology trials using weal-world data and ai. Nature, 592(7855)x629- 633.
Boon-How Chew. 2019. Planning and conducting clini- cal research: the whole process. Cureus, 1 11(2).
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instructionfiietunee language models. arXiv preprint arXiv:2210. 14416
Iain J Marshall, Joël Kuiper, Edward Banner, and By- ron C Wallace. 2017. Automating biomedical evi- dence synthesis: Robotreviewer. In Proceedings of the conference. Association for Computational Lin- guistics. Meeting, volume 2017, page 7.
Tufts CSDD. 2016. Amendments reduce number of patients, but at high cost, longer study times. Tech- nical Report 1, Tufts Center for the Study of Drug Development, Tufts University.
NIH. 2023. ClinicalTrials.gvv.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744
language models. In Advances in Neural Information Processing Systems.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning. Percy S Liang, and Jure Leskovec. 2022. Deep bidirectional language-knowled graph pretraining. Advances in Neural Information Processing Systems, 35:37309- 37323.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BELU: method for automatic eval- uation of machine translation. In Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. 2023. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint 23X5v2230..05302.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- odge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing, pages 2463-2473.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res.. 21(140):1-67.
Karan Singhal, Shekoofeh Azizi, Tao Tu, Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewwss Stephen Pfohl, al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.1333:
Yixuan Su, Tian Lan, Yan Wang. Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06417.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic, 2022. Galactica: A large language model for science, arXiv preprint arXiv:2211 1.00085.
Ramakrishna Vedantam, C Lawrence Ziticck and Devi Parikh. 2015. Cider: Consensus-based image descrip- tion evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4566-4575
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith. Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-Instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arxiv:2212.100 10560.
Zifeng Wang and Jimeng Sun. 2022. Trial2vec: Zero- shot clinical trial document similarity search using self-supervision. In Findings of EMNLP.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large

---

AutoTrial
PT
RAG
SimCTG
50
40
BLEU
30
20
10
30
METEOR
20
10
40
30
ROUGE
20
10
1.5
1.0
CIDE
0.5
Healthy
Prostate Cancer
Obesity
Diabetes
Muttiple Myeloma
Breast Cancer
Hypertension
Coronary Disease
Condition
Figure 6: The generation quality across the trials targeting to the top-8 most frequent discases/comdittion..

---

Table 5: Qualitative generation results of AutoTrial for criteria generation under instructions. Manual Input: the context textual inputs offered by users, where trial setups x, are shared for the same trial and instructions x, specific each criteria; Automatically Built Input: the reference criteria automatically retrieved and built the input x, for AutoTrial; Output: results generated by AutoTrial; Groundtruth: the corresponding criteria written by human clinicians in original trial documents.
Manual Input and Automatically Built Input will be concatenated as the final input. yellow highlights the instruction tokens; green highlights the setup tokens;
blue highlights the reference tokens; red highlights the targe! tokens. Reference texts are truncated in the middle due the limited space.
Manual Input
Automatically Built Imput
Output
Groundtruth
<instr> <bmis </instr>
<titles
<ref> <inc> subjects with bmi 20-
<ines> <inc>
Body
Body mass index (BMI)
Single-dose and Multiple-dsse Study
kg/m2.... ... <exc>
gastrointesti-
index
19
within the runge 19-35
Evaluate Pharmacokinetics and
discases: active uleer, gastrointestinal rectal bleeding,
35 kg/m',
inclusive
kg/m² (inclusive). BMI
Pharmaedyynamics DBPRI08 Tablets
inflammatory
</ines>
weight (kg) height' (mP)
Type Diabetes Mellitus Patients
bowel syndrome, biliary duct obstruc- active gastritis
<discase>> Type Diahetes Mellitus
controlled by medication, etc. </ref>
<treatment> DBPRIOS tablets
<instr> <nyhu> </instr>
<title>
A
<ref> <inc> subjects with bmi of 20-
<exes> <exc>
Heart Serious dysthythmius, obvious left ventricu- lar dysfunction, New York Heart Association (NYHA) functional class III or IV
Single-dosc and Multiple-osse Study
45 kg/m2 <inc> subjects with 7% s hbalc & 10% who have been diagnosed with r2dm at least 8 woeks <exc> se- vere gastruinlestina discases: active ul- cer, gastrointesttinl or rectal bleeding. active inflammatory howel syndrume, biliary duct obstruction, active gustritis that is not controlled by medication, elc.
failure (NYHA class III and IV) <lexes>
to Evaluate the Pharmacokinneties and
Pharmacodynamics of DBPRIOS Tabkts
in Type Diabetes Mellitus Patients
<discases Type Diabctes Mellitus
<treatment> DBPR 108 tablets
</ref>
<instr> <sbp> </instr> <titles
<ref> <inc> subjects with bmi 20- kg/m2 ... <exc> severe gastrointesti-
Subject hypotension (systolic bluod pressure mmHg) hypertension (systolic hlood pressure 2 140 mmlg dias-
Uncontrolled hyperten- systolic pressure
Single-dose and Multiple-dose Study
Evaluate Pharmacukinetics and
diseases: uleer, gastrointestinal
2 mmflg diastolic
Phamacodynamics of DBPRI08 Tablets
reetal bleoding, active inflammatory bowel syndrome, biliary obsiric- tion, active gastritis not controlled by medication, etc. <heel
pressure 2 100 mml
Type Diabetes Mellitus Patients
<disease>> Type Diabetes Mellitus
<treatment> DBPR tablets
tolic blood pressure mmHg) at screening </exes>
A Single-dosc and Multiple-osse Study
<instr> sogt> <finstr> <title>
<ref> <inc> subjects with bmi of 20- 45 kg/m2 <exc>> severe gastrointesti- nal discases: active uleer, gastrointestinal or rectal bleeding, active inflammatory bowel syndrome, biliary duct obstrue- tion, active gastritis that not controlled by medication, etc. </ref>
Paticnts who had QTc interval 2
<exes> <oxc>
Patients who have the sccond or third degree atrioventricular block, long Q-T syndnume, QTc=500 ms without cardiac pacemaker
to Evaluate the Pharmacokineties and
450 ms in males or > 470
Pharmacndynaricc of DBPR108 Tablas in Type Diabetes Mellitus Patients
ms in females </exes>
<disease>> Type 2 Diaheles Mellitus
<treatment> DBPRI0S 108 tablets
<instr> <wiff__cipeeeeeee ceteeeexecttncy> <fissr>>
<ref> <ine> female putients with repro- ductive potential
<incs> <inc>
Life
Life expectancy of
<title>
Gefitinib Combined With
have aegative within hours prior to start of study medication. all female patients of childhearing poten- tial, and all male patients, ... clin-
pectancy </ine>>
weeks
least weeks
Chemotherapy Antiangiogensis in Pa-
scrum pregnancy
tients With Bim Deletion Low EGFR
Mutation Abundance <discasc> Non-
small-cell Lung Cancer <creetments
Gefitinih, pemetrexed gemcitabine plus carhoplatin, hevacizumah
brain metastases ical signs or symptoms brain metas- tases radiological evaluation manda- tory). </ref>
<instr>> <age> </nssr>> <title>
Nutri-
<ref> <inc> bmi 18.5 29.9 kg/m2 <exc> self-reported malabsorption (c.g. difficulty digesting absurbing nutri- from food, potentially leading bloating, cramping gas) <hell>
<ines> <inc>
Aged 60 older </ines>>
Age 60 years older
ent Synergy in Beef and Stimulation of
Protein Synthesis in Elderly Klissasee
Healthy <treatments
3
cooked, 85% lean ground heef, 20 grams beef protein isolate


Reviews:
[{"review_id": "xWz8sukBka", "reviewer": "Reviewer_AGTQ", "summary": "", "questions": "It would be interesting to see further clarification about the choice of evaluation metrics from the authors. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Clarity\n>  the authors should be more careful in their word choice\n\nWe acknowledge that the presentation of this paper can be further improved. For the issue mentioned by the reviewer, we rephrase the parts describing the challenges for AI for trial design with the following three main points:\n\n- **Comprehending instructions**: LLMs need to comprehend the input trial synopsis and adapt to expert inputs to generate precise eligibility criteria.\n- **Referring to prior studies**: LLMs need to be able to refer to prior studies for better generation.\n- **Rationalizing the generation**: LLMs need to offer the rationale behind the generated criteria so as to be trusted by human experts in practice.\n\n### Q2 Experiment setups & Technical details\n\n> missing a number of key details in Section 3\n\nIn Section 3, we mainly describe the pipeline of AutoTrial. We improve the paper presentation to make it more clear for reading and reproducing. \n\n- We add the example of the input components in the problem setup (Section 4.1) for better understanding.\n- We add the implementation details of our method in Experiment section 4.3, including the maximum context length, batch size, etc\n\nWe also make a comprehensive polish of the method section for better clarify in the new version.\n\n\n\n### Q3 Choice of evaluation metrics\n\n> It would be interesting to see further clarification about the choice of evaluation metrics \n\nIn Section 4.2, we describe the three evaluation strategies: automatic evaluation, clinical evaluation, and human evaluation. We believe the first and the last are both standard evaluation metrics for general text generation tasks. Automatic evaluation involves a series of text overlapping measurements like CIDEr, ROUGE, etc., that computes how similar the generated criteria are to the target criteria written by human experts. Human evaluation involves the manual check of how well the generated criteria approximate the target criteria as a remedy to the automatic evaluation. We further add a clinical accuracy evaluation to check if the generated criteria describe the same target as the target criteria under the clinical context, e.g., if both criteria describe the same requirement for age and maybe in different ways. It is important because automatic evaluation may overestimate the textual descriptions while neglecting the semantic meaning of the criteria.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates novelty through the introduction of a new task: the automation of clinical trial design through the creation of a language model. This has the potential to aid clinical trial design, thus contributing to the advancement of drug development by increasing the likelihood of successful trials. The motivation for clinical trial protocol design, which is a challenging task, is effectively communicated. Overall, the paper is readable and understandable.  In terms of assessment, the authors evaluated their proposed model, considering a comprehensive array of baseline models and employing various metrics to portray its effectiveness. Given the novelty of this task, it is worth considering whether the chosen metrics provide the most optimal evaluation for this specific context, although seemed reasonable.\n\n\n", "reasons_to_reject": "While the authors have motivated the need for a clinical trial automation process, the authors should be more careful in their word choice. A significant example to this end: it was mentioned in the first paragraph of the introduction, \"one area where generative LLMs have shown significant potential is in clinical trial protocol design\" without any reference which contradicts the paper's main claim that they are \"first to develop LLMs focusing on trial design\". The authors also pointed out that the models often lack the capability of \"ability to adapt expert instructions\" in the introduction without any reference or evidence to it. It is crucial to demonstrate how the generic models fail or at least provide a reference to this claim.\n\nThe authors are missing a number of key details in Section 3 , making it difficult to reproduce the paper's results. An example of this is in the pre-training of AutoTrial,  the authors used title, disease, treatment, etc. as the input (i.e. trial setup), however, did not mention the maximum length considered for the input. Generally, clinical trial documents are lengthy documents, and it is of utmost importance in this scenario.  While the importance of the \"maximum acceptable length\" parameter is acknowledged in Section 4.4, attributing potential performance discrepancies between exclusion and inclusion criteria to truncation, no discussion is presented on the length selected. Additionally, the paper lacks in-depth details about the model used, describing it merely as a \"decoder-based causal language modeling architecture.\"\n\nThe problem setup in Section 3.1 is not as clearly demonstrated as it could be. It would be highly beneficial if the different input attributes, such as 'targeting instruction,' 'reasoning steps,' and 'textual description of the given instruction,' were illustrated with a sample input. Given that the paper's primary focus revolves around 'the criteria of trials,' a list of potential criteria (e.g., age, gender, BMI, etc) for the eligibility section (inclusion/exclusion) would greatly aid the reader in comprehending the core concepts.\n\nIn summary, while the authors have effectively emphasized the need for clinical trial automation, they should revise the article providing missing details, particularly regarding model parameters, model specifications, and sample input demonstrations, which would greatly enhance the clarity and reproducibility of the paper.\n\n\n\n", "questions_to_authors": "It would be interesting to see further clarification about the choice of evaluation metrics from the authors. ", "paper_topic_and_main_contributions": "The paper introduces a novel task: the automation of clinical trial design, with a specific focus on eligibility criteria. The objective is to potentially aid the design of clinical trial protocols. The authors systematically address this task by defining the problem, creating a dedicated dataset, introducing a model termed \"AutoTrial,\" and rigorously assessing its performance.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a few grammatical mistakes and reference issues. For example, in section 3.3 the authors referred to the dataset as section 3.1 which will 4.1. There is a spelling mistake in the title of section 4.3 ( 'mehtods'). There are styling issues throughout the paper while referring to different sections."}}, {"review_id": "0heuaY41cg", "reviewer": "Reviewer_ZRCV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Retrieval component performance\n\n> the coverage of the external knowledge base may not be great enough ... \n\nWe agree that retrieval performance is a one of the key factors that impact the performance of retrieval-augmented generation models. In this paper, we add the whole clinicaltrial.gov database as the knowledge store to ensure a comprehensive coverage of all types of clinical trials. In practice, it is usually beneficial to add more available knowledge sources, for example, from scientific publications. It is also useful to clean, filter, and summarize the retrieved knowledge to make sure a higher recall of the needed evidence. All these strategies we would like to leave as the future work to explore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work shows the retrieved criteria from relevant trails as the exemplars could be used to improve the quality of generation. Ane the in-context exemplar serves as a good template for the model's multi-step reasoning outputs.\n2. A discrete and neural prompting approach is proposed to better solve the clinical trial design problem and the method outperforms many strong baselines. \n3. Ablation has been shown to justify the importance of RAG and Prompt module, and the MSR is shown to contribute in the model interpretability.", "reasons_to_reject": "In the clinical trial setting, the coverage of the external knowledge base may not be great enough for all the possible queries. If the retrieval confidence is not good enough, how to facilitate the in-context learning abilities of language models?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method named AutoTrial to aid the design of clinical eligibility criteria using language models. The method contains three main technical features: instruction prompt tuning for controllable generation, scalable and efficient knowledge incorporation via in-context learning and multi-step reasoning. Extensive experiments show that this method outperforms many strong baselines by a large margin, and the generated criteria texts are fluent, coherent, clinically accurate and of high-quality.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "sec 4.3 baseline methods"}, "extra_text": {}}, {"review_id": "olmWcRRdVR", "reviewer": "Reviewer_JAD5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Presentation\n\n> this paper is quite dense and hard to follow\n\nWe have made a comprehensive edits of the method section to make sure a better readability. Specifically, we follow the suggestions from Reviewer #AGTQ to add more examples for the introduced components in the prompt.\n\n\n\n### Q2 Data and code availability\n\nAll the data we use are available to download from the CT.GOV database from https://aact.ctti-clinicaltrials.org/data_dictionary. We will release the preprocessed data and the model code in the later stage.\n\n\n\n### Q3 Implementation details\n\n>  no details about the implementation and experimental setup\n\nWe have added the implementation details of our method in Section 4.3 of the new version for better reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- important medical application\n- good performance\n- several analyses are performed to give insight into the results", "reasons_to_reject": "- The paper is quite dense and hard to follow (for instance, the neural prompt section is quite difficult for readers who aren't prompting experts)\n- It's not clear if the data and code will be made available\n- There are no details about the implementation and experimental setup: nb of runs, hyperparameters,...", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies generative language models for designing clinical trials, focusing on inclusive and exclusive criteria generation. The authors propose a novel prompting-based framework that they train from scratch using existing clinical trial announcements.  The model outputs both clinical trial criteria and the reasoning steps used to generate the criteria.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reasons to accept, as noted by the reviewers, are mainly in the areas of novelty and originality and include:

- Novelty through the introduction of a new task.
- Motivation for the application is effectively communicated. 
- The authors evaluated their proposed model using a comprehensive array of baseline models and various metrics to portray its effectiveness (including human evaluation).
- The method outperforms many strong baselines.
- The ablation study provides important information on the contribution of different components.

The areas for improvement are mainly in terms of clarity and the need to add missing details, especially in Section 3:
- Missing a number of key details in Section 3, which describes the pipeline. In their rebuttal, the authors commit to "add the example of the input components in the problem setup (Section 4.1) for better understanding and add the implementation details of our method in Experiment section 4.3, including the maximum context length, batch size, etc"
-  The coverage of the external knowledge base may not be great enough for all the possible queries.
- Certain sections may be hard to follow (for instance, the neural prompt section is quite difficult for readers who aren't prompting experts)
- It's not clear if the data and code will be made available (the authors note that the data is available now and that the code will be made available in the future).

These can be addressed through revision of the writing and inclusion of further detail, which the authors have committed to provide.

One specific limitation raised by reviewer AGTQ that the authors do not address in their response is as follows: 'it was mentioned in the first paragraph of the introduction, "one area where generative LLMs have shown significant potential is in clinical trial protocol design" without any reference which contradicts the paper's main claim that they are "first to develop LLMs focusing on trial design". ' In my view, it is crucial to appropriately address this point in order to convey the novelty of the contribution presented in the current paper.

The authors should pay careful attention to implementing the suggested improvements to ensure a clearer and more accessible presentation of their work.
Decision: Accept-Main