impact of sample selection on in-context learning for entity extraction from scientific writing
Authors: Necva Bölücü
Maciej Rybinski
Stephen Wan
Venue: EMNLP 2023 Findings (2023)
Keywords: GPT-3.5
in-context learning
sample selection
entity
scientific

Abstract: Prompt-based usage of Large Language Models (LLMs) is an increasingly popular way to tackle many well-known natural language problems. This trend is due, in part, to the appeal of the In-Context Learning (ICL) prompt set-up, in which a few selected training examples are provided along with the inference request. ICL, a type of few-shot learning, is especially attractive for natural language processing (NLP) tasks defined for specialised domains, such as entity extraction from scientific documents, where the annotation is very costly due to expertise requirements for the annotators. In this paper, we present a comprehensive analysis of in-context sample selection methods for entity extraction from scientific documents using GPT-3.5 and compare these results against a fully supervised transformer-based baseline. Our results indicate that the effectiveness of the in-context sample selection methods is heavily domain-dependent, but the improvements are more notable for problems with a larger number of entity types. More in-depth analysis shows that ICL is more effective for low-resource set-ups of scientific information extraction

Impact of Sample Selection on In-Context Learning for Entity Extraction from Scientific Writing
Necva Bölücü, Maciej Rybinski, Stephen Wan CSIRO Data61
Inecva.bolucu: iosiauil.....iiee.e.ooobnniiii stephen. wan)@csiro.au
Abstract
2023), including entity extraction from scientific documents (Dunn al., 2022), and also for lever- aging reported scientific knowledge in downstream data science applications (Sorin et al., 2023; Vert, 2023). These models, such as GPT-3 (Brown et al., 2020) and LLAMA (Touvron et al., 2023), with bil- lions of parameters and pre-trained on vast amounts of data, have showcased impressive capabilities to tackle tasks in zero- or few-shot learning by leveraging in-context learning (ICL) (Radford al., 2019; Brown et al., 2020).
Prompt-based usage of Large Language Mod- els (LLMs) is an increasingly popular way to tackle many well-known natural language prob- lems. This trend is due, in part, the ap- peal of the In- mn-Coteext Learning (ICL) prompt set-up, in which few selected training exam- ples are provided along with the inference re- quest. ICL, a type of feo-shot learning, is espe- cially attructive for natural language processing (NLP) tasks defined for specialised domains, such as entity extraction from scientific docu- ments, where the annotation is very costly due to expertise requirements for the annotators. In this paper, we present comprehensive analy- sis of in-context sample selection methods for entity extraction from scientific documents us- ing GPT-3.5 and compare these results against a fully supervised transformer-baat based baseline. Our results indicate that the effectiveness of the in-context sample selection methods is heavily domain-der dependent, but the improvements are more notable for problems with larger number of entity types. More in-depth analysis shows that ICL is more effective for low- oowrresourc set- ups of scientific information extraction.'
In ICL, models are provided with natural lan- guage prompt consisting of three components: a format, set of training samples (input-lahe) pairs demonstrations), and test sentence. LLM outputs the predictions for a given test input without updat- ing its parameters. The main advantage of ICL is its ability to use the pre-existing knowledge of the language model and generalise from small num- ber of context- speecffcc samples. However, ICL has been shown to be sensitive the provided samples and randomly selected samples have been shown introduce significant instability and uncertainty to the predictions (Lu et al., 2021; Chen et al., 2022; Agrawal et al., 2022). This issue can be alleviated by optimising the selection of the in-context sam- ples (Liu et al., 2021; Sorensen et al., 2022; Gonen et al., 2022).
1 Introduction
Extracting relevant information from scientific doc- uments plays a crucial role in improving methods for organising, indexing, and querying the vast amount of cxisting literature (Nasar et al., 2018; Weston et al., 2019; Hong et al., 2021). However, annotating datasets for scientific information ex- traction (IE) is a laborious and costly process that requires the expertise of human experts and the development of annotation guidelines.
ICL sample selection methods can be divided into 2 categories: (1) the methods for choos- ing samples from the train set (e.g., the KATE method (Liu et al., 2021)), and (2) finding the best prompts by generating samples (e.g., the Perplexity method (Gonen et al., 2022), SG-ICL (Kim et al., 2022)). These methods can significantly reduce the need for extensive human annotation and allow LLMs to adapt to various domains and tasks.
In recent years, large language models (LLMs) have demonstrated remarkable performance on var- ious natural language processing (NLP) tasks (Wei et al., 2022; Hegselmann et al., 2023; Ma et al.,
We rely on the survey of ICL (Dong et al., 2022) and delimit the methods for sample selection, from the inference stage of ICL. Our aim is to provide a comprehensive analysis of these methods for se-
'The ende is publicly available at adalin16/1e 6/ICLEEE.
com/

---

lecting samples from the train as part of ICL for Entity Extraction from scientific documents. Most of the methods have been applied with prompt gen- eration (i.e., to select the best generated sample). Here, we use the methods only for sample selection from the training set of the dataset for entity extrac- tion from scientific documents and compare their effectiveness for this problem. We also propose the use of the Influence method (Koh and Liang, 2017) in an oracle setting, to provide a best-case scenario to compare against. We investigate the in-context sample selection methods (see $3) and evaluate the methods adapted for entity extraction problem on 5 entity extraction datasets: ADE, MeasEval, SciERC, STEM-ECR, and WLPC, each covering a different scientific subdomain or text modality (see $4.1 for dataset overview).
are very limited (Gutiérrez et al., 2022; Wan et al., 2023). Gutiérrez et al. (2022) evaluate the perfor- mance of ICL on biomedical tasks, Named En- tity Recognition (NER) and Relation Extraction (RE). In addition, Wan et al. (2023) apply an entity- aware demonstration using the kNN sample selec- tion method (Liu et al., 2021) for RE.
To the best of our knowledge, our work is one of the first attempts for IE from scientific documents that present a comprehensive analysis of in-context sample selection methods for the problem with detailed analysis.
3 Methods
In this section, we describe the ICL sample selec- tion methods for entity extraction from scientific documents. First, we describe the ICL approach in Section 3.1 and then introduce the sample selection methods in Section 3.2.
Our experiments show that while fully super- vised finetuned PLMs are still the gold standard when training data can be sourced, choosing the right samples for ICL can go long way in im- proving the effectiveness of ICL for scientific en- tity extraction (see $5.1). Our experiments demon- strate an improvement potential of 7.56% on av- erage across all experiments, when comparing the oracle method (the Influence method) to the ran- dom sample selection baseline, and 5.26% when using the best-performing method in test setting (KATE). Moreover, our evaluations show that our main conclusions hold in simulated low-resource setting (see $5.2). Finally, our extensive exper- synthesise some prescriptive advice for other NLP researchers and practitioners tackling scientific entity extraction (see $ 5.5).
3.1 In-context Learning
Given an LLM, ICL can be used to solve the en- tity extraction problem for D = (X,Y), where X are the sentences the entities for each sentence. The prompt P con- sists of k the number of samples for the few-shot learning, samples (T) (selected from the train set or generated; this work, we focus only on the for- mer) with gold entities (T(sprann ,etriin) is the 7th sample) with format (7) and test sentence (s:) (P= I T + sfest) (see Appendix B). Prediction is done by selecting the entities with the highest probability for each sentence in the test set.
Wi,..', w,,) and Y are
iments allow
3.2 Sample Selection Methods
2 Related Work
We follow the survey in-context learning (Dong et al., 2022) and choose the following methods to use for sample selection for ICL entity extraction from scientific documents.
By increasing the size of both the model and the corpus, LLMs have demonstrated the capability of ICL, which uses pre-trained language models for new tasks without relying gradient-base training (Brown et al., 2020). In various tasks, such as inference (ibid), machine translation (Agrawal al., 2022), question answering (Huang et al., 2023; Shi et al., 2023), table-to-texi generation (Liu et al., 2021) and semantic parsing (An et al., 2023), the ICL use of LLMs mentioned by Brown et al. (2020) has been shown to be on par with supervised baselines in terms of effectiveness.
KATE (Knn-Augmented in-conText Example se- se- lection) is a kNN-based method to select ls sam- ples which are close test sample based on tence embeddings and distance metrics (Euclidean or Cosine Similarity). We follow KATE to select samples from the train set of datasets for each sen- tence in the test set.
Perplexity is metric to evaluate the perfor- mance of language models by calculating the prob- ability distribution of the next token given the con- tent provided by the preceding tokens. The metric
Other studies have found, however, that ICL does not always lead to better results than fine- tuning. Previous studies investigating ICL for IE

---

ADE
MeasEval
SciERC
STEM-ECR
WLPC
# Sentences
3,076
542
1,861
942
8,581
Train set
Tokens
65,244
18,642
45,412
20,801
108,047
Entities
7,768
882
5,568
4,560
25,229
Sentences
769
155
275
118
2,859
Dev set
# Tokens
16,715
6,069
6,521
2,697
36,490
# Entities
1,993
278
808
605
9,207
# Sentences
427
294
551
118
2,861
# Tokens
8,755
10,068
13,401
2,470
37,371
Test set
Entities
1,069
499
1,681
559
9,707
Avg
15.30
9.16
19.28
18.59
6.82
Avg
131.75
171.56
151.35
146.76
75.28
# Entity types
2
18
Table I: Statistical details of datasets. Avg e is the average length of entities and Avg is the average length of sentences.
provides insights into the unexpectedness of a sen- tence in the context of given language model. Go- nen et al. (2022) use perplexity scores of prompts select the best prompt, rather than selecting amples from the dataset, and synthetically gener- ated prompts through paraphrasing with GPT-3 and back-translation Unlike Gonen et al. (2022), experiments we focus on selecting in-context sam- ples from the training set instead of selecting the better prompt. As the sample selection method, we calculate the perplexity of cach train sentence us- ing language model (LM) and take the le samples from the train set with the lowest perplexity, which means the sentence more likely and consistent with the patterns it has learned from the training data of LM. Unlike the other in-context sample selection methods (Random, KATE, etc.), the se- lection of the samples is independent of the test sentences (i.e.. the same samples from the train set are characterised by lower perplexity, indepen- dently from the test sample presented alongside).
model analysis (Cook and Weisberg, 1982; Chatter- jee and Hadi, 1986; Hampel et al., 1986). Koh and Liang (2017) adapt the functions for machine learn- ing (ML) understand model behaviour, debug models, detect dataset errors, and create adversarial training samples. The aim of the functions is calculate the influence of a training sample train on a test sample s gtstt formulated as the change in loss on teat if the training sample train were removed from training. This yields the influence of grain to solve the task for fest
The influence method is used in the literature to detect errors in the dataset and to create ad- versarial training samples (Koh and Liang, 2017). We adapted Influence as a method to study poten- tial performance gains for ICL sample selection because it scores the contribution of a sample to the training process. Similar to in-context sample selection methods, we select h samples from the train set that have higher influence sentences from the test set by using the baseline finetuned RoBERTa model (see Section 4.2) as the model calculate the loss the experiments. Since the fluence method's practical applicability limited (it uses test labels to select the ICL samples via the loss), we use it as a best-case (or oracle) base- line, where the sample ranking is based on training utility, rather than vocabulary similarity signal.
BM25 is a bag-of-words retrieval model that ranks relevant samples (sentences) appearing in each train set by relevance to given test sam- ple (Schutze et al., 2008; Robertson et al., 2009). Similar retrieval-based methods for augmenta- tion of the input with similar samples from the train set (Xu et al., 2021; Wang et al., 2022b), we select most relevant samples from the train set (so, those with higher BM25 scores) for each test sentence in the experiments.
Experiments
4.1 Datasets
We evaluate the sample selection methods in ICL for entity extraction from scientific documents. We use datasets from the different subdomains:
Influence functions (Koh and Liang. 2017) were originally used statistics for the context of linear

---

ADE (Gurulingappa et al., 2012): a subset of MEDLINE case reports describing adverse effects arising from drug use.
Finetuned RoBERTa baseline To compare the sample selection methods in ICL against a sensible baseline, we trained an entity extraction model on the datasets using RoBERTa (Liu et al., 2019) PLM (RoBERTa-base) We formulate the fully tuned task as token-level labelling using the BIO tags.
MeasEval (Harper et al., 2021): a dataset col- lected from scientific documents from 10 dif- ferent subjects and annotated for 4 entity types (Quantity, Measured Property, Measured En- tity, Qualifier). Since the other entities are dependent (e.g., triggered or nested) on quan- tity entities, we use only Quantity entity type in our experiments.
Zero-Shot For zero-shot setup, we formulate prompts using only format (I; see Appendix B) and test sentences from the test sets for each dataset.
Random Sampling In this approach, we ran- domly select & in-context samples from the train for every test sentence.
SciERC (Luan et al., 2018): an extension of Semßival 2017 Task 10 (Semlival 17) (Augen- stein et al., 2017) and SemEval 2018 Task (SemEval 18) (Buscaldi et al., 2017) datasets. The dataset contains 500 abstracts of Artifi- cial Intelligence (AI) papers with 6 scientific entity types*.
4.3 Experimental Setup
Baseline RoBERTA PLM is finetuned utilising Hugging Face? (Wolf et al., 2020) library. The hyperparameters used in the finetuning PLM are the batch size of 32, max length of 128, the learn- ing rate of le-5, and 15 epoch of training, and experiments are done on single NVIDIA Quadro RTX 5000 GPU. We train the model five times with different random seeds and report the mean and standard deviation of the results to account for the training variance of the model.
STEM-ECR' (D'Souza et al., 2020): dataset containing abstracts from the same subjects of MeasEval dataset for scientific en- tity extraction, classification, and resolution. Although there are 7 entity types, we follow the baseline study (D' Souza et al., 2020) and use 1 of them: Data, Material, Method, and Process.
For the baseline, zero-shot and random sampling, and ICL sample selection experiments, we build the system using the FasyInstruct* (Ou et al., 2023) framework to instruct LLMs for entity extraction from scientific documents with defined entity ex- traction prompts and entities of the datasets. In the experiments for ICL sample selection, we use a maximum 20 in-context samples due to the GPT-3
WLPC (Kulkarni et al., 2018): a dataset col- lected from wet lab protocols for biology and chemistry experiments providing entity, rela- tion, and event annotations for wet lab proto- cols.
token limit and 100 sentences from each test set because of the cost of GPT-3.5 usage. The experiment is repeated five times on the test set to calculate the average score and corresponding standard deviation for random sampling (see detailed results in Appendix D). For the KATE, we use [CLS] token embeddings of the RoBERTa PLM and OpenAl embedding API leext-embedding-adda--ee
Statistical details of datasets are given in Table 1,
4.2
Baseline Methods
In our experiments, we compare ICL sample selec- tion methods with a finetuned pre-trained language model, RoBERTa, zero-shot lcarning in which no samples are used for the GPT-3.5 prompt, and ran- dom sampling in which samples are randomly se- lected for the prompt.
to obtain sentence embeddings. We treat the embedding generation method (RoBERTa vs. GPT) as another hyperpa- rameter (much like the number of samples k). We calculate the distance between embeddings using the Euclidean and cosine similarity metrics for each test sentence and select similar k sentences based on the distance scores in KATE. We calculate the
hhttps://githht com/harrperco/MMaasEall hhttp://nlp.cs washington, edu/sciIE/ 'We use Other as the shortened form of OtherScientificTermii the rest of the paper
htt uni-hannover.. de/dataset/ s tem-eccrvi-e
"We thus leave out Task, Object, and Results entity types, since those are almost always nested within the other scientitic entity types.
'https://hugginfface http://www..om./iini///a/aa/ttte

---

Method
ADE
MvasEval
SciERC
STEM-ECR
WLPC
Baseline models
RoBERTa
90.4240.13
50.68
68.52-1.30
69.70_3.46
28.36411.25
Zero-shot 71.29
19.65
17.86
28.89
31.64
Random 71.5610.33
22.19+1455
29.27
26.85+1.26
32.20+1.22
In-context sample selecting methods
KATE
83.11
22.75
29.97
30.78 $
45.021 $
Perplexity 79.13
21.43
31.31
26.57
30.46 i
BM25
77.28
24.72
35.96
25.61
44.14 i
Influence 86.35
27.13
36.47
27.81 1
45.41
Table 2:
Main results for methods selecting in-context samples. The best results are given in bold. The best results of the in-context sample selection method are given in underline. i denotes statistical significance level y = 0.05 compared the supervised RoBERTa baseline and & denotes statistical significance level of p 0.05 compared to the random sampling.
The entity-level Macro Fj score of datasets on the full test set are for ADE g9.00 MeasEval 65.62
SciERC 11. STEM- 66.43 12, and WLPC 40.51
Method
ADE
EvasEval
SciERC
Stem-ECR
WLPC
RoBERTafuu'
90.42+0.13
50.6843.93
68.52+1300
69.7043.46
28.36+11.25
Baseline models
RoBERTagi
14.32=71.09
19.20=12.90
10.16+0.30
15.4245.78
10.57
Zero-shot
71.29
19.65
17.86
28.89
31.64
Randomsa 66.53
21.32
25.61
21.38 +1.8
28, 28.46
In-contxt sample selecting methods
KATE%1
69.06t t
24.48
26.78t
26.49
28.97
Perplexity%1
68.83
22.23
26.42
25.84 :
26.05
BM25%1
72.66 $
23.39f 3
31.33t $
24.24 }
36.73
Influenceg1 73.68 $
24.21 t
32.49 f
25.01 t
34.24
Table Main results for methods of selecting in-context samples using %1 train set. The best results are given in bold. The best results of the in-context sample selection method given in underline.
i denotes statistical significance level of 0.05 compared to the supervised RoBERTa baseline (RoBERTag)) and denotes statistical significance level of 0.05 compared to the random sampling (Randomy:) for low-resource scenario.
perplexity of the samples from the train set by us- ing the RoBERTa PLM (using the method outlined in (Salazar et al., 2019)) and select samples with the lowest perplexity for all test sets of the datasets in the Perplexity method. For BM25, we utilise rank-bm25' library with default parameters (term frequency saturation k of 1.5, document length normalisation of 0.75, and constant for negative IDF of a sentence in the data of 0.25). We use the finetuned RoBERTa to select & samples, as de- fined in the study of Jain et al. (2022), for each test sentence in the Influence method.
Statistical significance The statistical signifi- cance of differences in macro F, score is evaluated with an approximate randomisation test (Chinchor, 1992) with 99, 999 iterations and significance level 0.05 for sample selection methods (KATE, Perplexity, BM25, and Influence) and supervised RoBERTa baseline model and the random sam- pling (c.g., influence RoBERTa and influence -> random sampling). For significance testing, we used the results yielding the median entity-Level Macro F; score for the supervised RoBERTa base- line model and the random sampling (so, a run close to the mean value reported in the tables).
As the evaluation metric, we use entity-level
Macro F, score.

---

5 Results and Discussion
to low-resource problems without annotated data (zero-shot) or with very small train sets (few-shot using selected samples).
5.1 Main Findings for Selecting In-context Samples
Our main experimental results given in Table 2 for randomly selected 100 sentences from each of the test sets of the datasets (see Section 4.1) for en- tity extraction. Detailed experiments with various k samples in ICL can be found in Appendix D.
5.3 Test Set
To understand the impact of the test set in the ex- periments, we used 3 different randomly sampled test sets. We present the results for the ADE and WLPC datasets (see Appendix C for statistical de- tails of test sets), where ICL methods perform com- petitively with the fully supervised baseline. The results can be found in Table and for ADE and WLPC, respectively. can be seen that the first test set of the WLPC dataset is challenging for the baseline model, finetuned RoBERTa. However, in-context sample selection methods, with the ex- ception of Perplexity, appear to be less affected by the test set composition and yield similar results across different test sets.
Before drilling down into the in-context sample selection methods, we note that the baseline model, RoBERTa, outperforms the ICL for entity extrac- tion from scientific documents across all datasets except WLPC, similar the study of Gutiérrez al. (2022) conducted on Biomedical IE. We get the highest entity-level Macro Fj score among sample-selcction methods for all datasets using the Influence method. Additionally, the performance of sample selection methods is low for the Mease- val, SciERC, and STEM-ECR datasets, and the gap between the results of finetuned RoBERTa baseline and the Influence method is very large for these datasets. This difference in performance may be due to the difficulty of the datasets (Sci- ERC, STEM-ECR) and the differences between train and test sets of the datasets (Measeval) (see Appendix A for detailed analysis).
Method Baseline models RoBERTa 90.42 13
Set
Set
Set 3
92.15.0.01
88.68_0.25
Zero-shot
71.29
72.87
72.24
Random
72.23 74.5610.33
75.8313.15
In-context sample selecting methods
KATE
83.11
84.47
82.65
Perplexity
79.13
77.31
77.72
BM25
77.28
78.89
77.76
Influence 86.35
85.43
84.21
The Influence method performs comparably with the RoBERTa model for the ADE dataset. More- over, despite the complexity of the WLPC dataset with 18 entity types, it is surprising that the effec- tiveness of shoo and ICL is better than that of the finetuned RoBERTa model. We hypothe- sise that this might be due to the method selecting samples from the correct minority classes. Inter- estingly, the textual similarity signal is almost as good, as the results of both BM25 and KATE are almost as good.
Table 4: Results for different test sets for ADE dataset.
The best results are given in bold. The best results of the in-context sample selection method are given underline.
Method
Set 1
Set 2
Set 3
Baseline models RoBERTa
28.36-11.20
35.93
26.42
Zero-shot
31.64
37.32
37.30
Random
32. 32.20+1.22
35. 35.17
30.77+3.17
In-context sample selecting methods
KATE
45.02
46,86
42.47
Perplexity
30.46
34.96
38.38
5.2 Low-Resource Scenario
BM25
44.14
41.44
43.20
To understand how important the size of the train- ing set is for fully supervised finetuning of the base- line PLM model, RoBERTa, and sample selection methods for ICL, we run the experiments with 1 1% of the train set to simulate a low-resource scenario. The results can be found in Table 3. Although there is decrease in the results of ICL for all datasets, it is much less drastic than for the supervised models, which not surprising. It is well known that sufficient amount of annotated data is needed to finetune PLM. Therefore, the robustness of ICL methods is a valuable finding that can be applied
Influence 45.41
41.39
43.18
Table 5: Results for different test sets for WLPC dataset.
The best results are given in bold. The best results of the in-context sample selection method are given in underline.
5.4 Error Analysis
In Table 6, we give the entity-tpe-wiise entity- level Macro F, score for the datasets for each ICL method and baseline models. The detailed error analysis of the Influence method - our oracle

---

Baseline Models
In-context Sample Selection Methods
Datasct
Enfity
RoBERIa
Zero-shot
Random
KATE
Perplexity
BM25
Influence
Adverse-Rifees
86.29
60.13
62.61
79.79
72.61
69.22
84.16
ADE
Drug
95.48
82.45
86.12
86.43
85.65
85.34
88.54
McasEval
Quantity
49.55
19.65
22.15
22.75
21.43
24.72
27.13
Generic
71.43
5.23
18.42
18.67
18.32
26.61
27.11
Material
71.88
6,34
10.21
17.64
17.55
20.67
19.45
Method
74.14
45.52
52.18
52.65
61.26
62.15
63.08
SciERC
Metric
76.19
0.00
15.43
15.66
16.58
18.12
17.37
Other
66,30
24.23
55.62
46.82
50.01
52.43
55.33
Task
60.69
12.17
23.67
23.47
30.15
35.84
36.46
Data
73.71
29.12
29.33
32.52
27.61
23.34
29.45
Material
89.21
36,98
23.18
31.45
32.18
31.44
37.12
STEM-ECR
Method
51.61
22.23
21.12
29.45
19.16
22.34
22.27
Process
76.17
31,56
24.41
29.68
28.28
27.89
31.11
Action
35.84
69.24
60.11
72.67
61.23
71.13
Amount
26,51
53.27
$3.25
57.22
39.52
66.23
55.28
Concentration
47.62
36.11
50.32
37.28
46.45
46.11
Device
37.04
26.18
16.65
28.26
6.24
43.15
42.78
Generic-Mcasure
33.33
0.00
30.88
0.00
0.00
Location
18.18
21.53
29.35
38.43
20.00
49.45
45.22
Measure- Type
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Mention
46.15
0.00
0.00
0.00
0.00
0.00
Method
26.92
15.64
15.21
32.45
8.15
29.41
15.52
WLPC
Modifier
0.00
26.18
20.62
36.18
14.32
28.32
37.45
Numerical
21.89
0.00
36.54
0.00
35.42
0.00
43.37
Reagent
0.00
46.18
40.23
58.46
42.05
53.42
62.33
Scal
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Size
46.15
0.00
66.96
0.00
29.32
60.24
0.00
Speed
0.00
60.45
0.00
50.29
33.33
60.27
70.45
Temperature
42.31
80.34
80.41
92.10
67.21
71.23
86.18
Time
32,94
67.42
71.18
67.37
72.41
67.18
67.43
pH
66.67
0.00
0,00
0.00
0.00
100
100
Table 6: Entity-type-eee results of each in-context sample selection method and baseline models.
method shows that there are 2 types of errors in the predictions: (1) correct entity type wrong entity span, where the model predicts an entity with correct entity type that is not annotated in the dataset, (2) wrong entity type wrong entity span, where the model predicts an entity with wrong entity type. The visualisation of the sam- ple 15 sentences for error analysis can be found in Appendix E.
this study) as Quantity entities, c.g., Qualifier as Quantity (a more specific example: 'total counts per gram' predicted Quantity, instead of the cor- rect entity type Qualifier). Another conclusion from the error analysis for the Mcaseval dataset is that GPT-3.5 tends to predict entity spans that are longer than the gold ones (e.g., gold: '11%' predicted: 'axis 2 =11%%'..
Results from the SciERC dataset show that ICL with sample selection methods struggles in the pre- diction of less frequent entity types (Generic, Ma- terial, Metric, Task) compared to entity types with higher frequency. In particular, Other is the most frequent entity type in the dataset and GPT-3.5 of- ten extracts correct span and mislabels it as Other entity type. In addition, the average sentence length of SciERC is higher than the other datasets. How- ever, the number of entities is less than the other datasets, and the Influence method tends to retrieve samples with more entities than the whole dataset. This results in extracting entities that are not actu- ally entities in the dataset.
For the ADE dataset, all models perform bet- ter for the Drug entity type. The reason may be the shorter entity length (Adverse-Effect: 18.85, Drug: 10.27) and small vocabulary (Adverse-Effect: 2,786, Drug: 1,290), although the frequency of Adverse-Effect is higher than Drug in the train set and also in the selected sam- ples in each in-context sample selection method, Unlike other datasets, we also encounter predic- tions with entity types that are not present in the ADE datasct (c.g., Discase, Number, Route).
For the MeasEval dataset, the most common error is the mislabeling of spans corresponding to other entity types (Measured Property, Mea- sured Entity, and Qualifier, which are left out in
For the STEM-ECK dataset, the Influence method is able to extract the correct spans. How-

---

ever, it has difficulty in accurately labelling the spans because the dataset imbalanced. The fre- quency of the Material and Process entity types is higher, which leads the Influence method to se- lect samples with these entities and consequently label the extracted entities with these entity types. Finally, the WLPC dataset very dense in terms of entities in the sentences, despite the sentence length. Since the dataset is imbalanced (the entity types Action, Reagent, Amount, and Location occur more frequently than others), the Influence method retrieves samples covering these entities and, as result, extracts mainly these entities. Moreover, the dataset is composed of instructional text and the Action entity is mostly a verb in the sentence, which easy to extract and correctly label.
language model for the entity extraction problem (but, again, points the issue of needing a decent amount of training data to eventually train a few- shot model). The BM25 method, however, is very simple and effective for each of the datasets, with- out relying on any finetuned model (or any training, for that matter) for ICL sample selection.
Using these methods in selecting samples from a very limited training set (see Section 5.2) and test- ing on different test sets (see Section 5.3) shows that the methods are more robust compared to the baseline model, finetuned RoBERTa. In particular, our experiments in a simulated low-resource setting show that RoBERTa tuned with just 1% of the train set can be used effectively improve ICL sample selection (e.g., via the KATE method), while per- forming very poorly on the actual prediction lask. is very valuable learning applicable to subdo- mains without annotated data or with very limited annotated datasets.
5.5 Discussion
In practical applications, one may not have enough annotated data finetune PLM for a task. In such cases, might be required to use ICL for the prob- lem. Therefore, we explore the performance of the sample-selection methods which can be more effective in this case. First, we note that the ran- dom sampling method given in baseline methods is also competitive, especially in the low-resource scenario (see Section 5.2).
When we analyse the main results (see Table 2) and the results of the low-resource scenario (sec Table 3), find that KATE performs better in a data-poor set-up where the number of samples is severely limited. This shows that KATE has a re- markable ability to order a suboptimal subset of in- context samples. This suggests that KATE derives meaningful insights from limited data, making it a valuable method when data scarcity is challenge. Also, BM25 offers effective and efficient mech- anism for sample selection that can be utilised in true few-shot setup.
Among the sample selection methods, we ob- tain the best results for ADE and WLPC with sentences coming from the [CLS] token of fine- tuned RoBERTa (finetuned using the train set of datasets), for the SciERC, STEM-ECR, and Mea- sEval datasets, we obtain the best results with Ope- nAI embeddings for the KATE method. This may be due to the insufficient training set for these tasks since we use the embeddings from finetuned RoBERTa (which is also used as the baseline model in the study). On the other hand, using OpenAl em- beddings in sample selection, despite being costly, avoids the pitfall of needing enough annotated train- ing data to train supervised model in order to be able to select samples for ICL (although, admit- tedly, even very under-trained PLM appear to be effective for sample selection; see further in this section).
Another observation is that the Influence method, a classic technique from statistics, proves highly effective in selecting samples from a larger pool of samples. The method evaluates the impact of training sample by assessing its effect on loss. typically the loss of test samples. While it is oracle method, its high effectiveness highlights a performance gap between a loss-based signal and cample-similarigg -aaaee signal. We believe that bridging this gap is a challenge worth exploring in future research into ICL sample selection methods. However, it should be noted, that the effectiveness of Influence decreases in extreme few-shot setup, possibly due to high training variance caused by very small number of instances. This, in turn, high- lights the robustness of KATE and BM25. BM25, as a heyword-matchie
We calculated the perplexity of sentences us- ing pre-trained and finetuned RoBERTa language models for the Perplexity method, and we obtained better results using the finetuned RoBERTa, which highlights the benefits of domain- adaptaaioon of a
method, does not require training (we used default hyperparameters in all experiments). KATE can fall back on a PLM's

---

ability to create text embeddings to overcome the training data scareity, instead of relying on the loss signal produced with the under-trained layers of the model (i.e., the classification head).
effectiveness for related NLP problems (Lu et al., 2021; Rubin et al., 2021). Moreover, & is a hy- perparameter in few-shot learning that depends on the sample selection method and the dataset. We tested directly on the test set without using a val- idation set. Finally, we did not apply contextual calibration (Zhao et al., 2021) for entity extraction, which has been shown to improve the performance of contextual learning for NLP tasks, and kept this as future work.
6 Conclusion
In this paper, we explore the in-context sample selection methods for ICL entity extraction from scientific documents. Since entity extraction is crucial step in IE from scientific documents, we analyse the methods in detail using several datasets from different subdomains, and with different en- tity types. The experimental results show that the baseline model, finetuned RoBERTa, still achieves the best results for this problem on 4 of 5 datasets. However, the in-context sample selection methods appear to be more robust to the train set data avail- ability and achieve similar results to using a full train set when only small annotated training set is used for the problem, yielding significantly better results than the baseline model in this low-resource setup.
Ethics Statement
The datasets used in our experiments are publicly available. Both these datasets are focused on pro- cessing (publicly available) scientific literature, thus constituting a low-risk setting.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis. Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. context Examples Selection for Machine Translation. arXiv preprint arXiv: 12.02437.
Our work aims to extract entity spans using LLM with ICL. We focus on simple in-context sample se- lection methods based on similarity, perplexity, rel- evancy, and influence, and use GPT-3.5 as LLM in ICL. However, there are several alternative LLMs pre-trained on different domains, that could be more aligned with the task of scientific entity ex- traction. As future work, we hope to add a compar- ative dimension to our work by using these LLMs, since the ICL. behaviour of L.Mss can change de- pending on their scale and pretraining. We also plan to explore the performance of the in-context sample ordering methods (Lu et al., 2021), which are shown impact the ICL effectiveness as well.
Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng. Weizhu Chen, and Jian-Guang Lou. 2023. Skill-Based Few-Shot Selection for In-Context Learning. arXiv preprint arXiv:2305.142 14210.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lak- shmi Vikraman, and Andrew McCallum. 2017. Se- meval 2017 task 10: Scienceie-extractio keyphrases and relations from scientific publications. arXiv preprint arXiv:1704.02853.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 1877-77-1
Limitations
We investigate the impact of the ICL selection meth- ods for entity extraction from scientific domains. Although we tested several methods on various datasets from different subdomains, due to the high cost of LLM models, we limited our experiments to small subset of test sets and used only GPT- 3.5. Moreover, the methods, KATE, Perplexity, and Influence (an oracle method), require finetuned models for better performance in selecting samples did not
Davide Buscaldi, Anne-Kathrin Schumann, Behrang Qasemizadch, Haïfa Zargayouna, and Thierry Charnois. 2017. Semeval-2018 task 7: Semantic rela- tion extraction and classitication in scientific papers. In International Workshop on Semantic Evaluation (SemEval-2018), pages 688.
Samprit Chatterjee and Ali S Hadi. 1986. Influential observations. high leverage points, and outliers in linear regression. Statistical science, pages 379-393.
from the annotated dataset. In addition, investigate which instruction is most appropriate. We also did not directly investigate the ordering of the selected samples, also shown to have impact of
Jiawei Chen, Qing Liu, Hongyu Lin, Xianpei Han, and Le Sun. 2022. Few-shot named entity recog- nition with self-describing networks, arXiv preprint arXiv:2203.1 12252.

---

Nancy Chinchor. 1992. The statistical significance of the muc-4 results. In Proceedings of the 4th Confer- ence on Message Understanding, MUC 1992, pages 30-50.
data with large language models. In International Conference on Artificial Intelligence and Statistics. pages 5549-5581. PMLR.
Zhi Hong, Logan Ward, Kyle Chard, Ben Blaiszik, and Ian Foster. 2021. Challenges and advances infor- mation extraction from scientific literature: a review. JOM, 73(11):3383 -4000.
R Dennis Cook and Sanford Weisberg. 1982. Criticism and influence analysis in regression. Sociological methodology, 13:313-361.
Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Ce- cile Paris. 2019. Using similarity measures to select pretraining data for NER. arXiv preprint arXiv:7904.00555
Zixian Huang. Jiaying Zhou, Gengyang Xiao, and Gong Cheng. 2023, Enhancing In- in-Context Learning with Answer Feedback for Multi-Span Question Answer- ing, arXiv preprint arXiv:2306.04500
Qingxiu Dong. Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun. Jingjing Xu, and Zhi- fang Sui. 2022. A Survey for In-context Learning. arXiv preprint arXiv:2301.00234
Sarthak Jain, Varun Manjunatha, Byron C Wallace, and Ani Nenkova. 2022. Influence Functions for Sequence Tagging Models. arXiv preprint 10X1::210.141177.
Jennifer D'Szu Anett Hoppe, Arthur Brack, Mo- hamad Yaser Jaradeh, Sören Auer, and Ralph Ewerth, 2020. The STEM-ECR dataset: grounding scientific entity references in STEM scholarly content to au- thoritative encyclopedic and lexicographic sources. arXiv preprint urXiv:2003.01006.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Tacuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022, Self- Generated In- In-Context Learning: Leveraging Auto- regressive Language Models a Demonstration Gen- erator. arXiv preprint arXiv:2206.0800 08082.
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894 PMLR.
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. 2022. Struc- tured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint arXiv:22/2.0525::
Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu Machiraju. 2018. An annotated corpus for machine reading of instructions in wet lab protocols. arXiv preprint arXiv:1805.0012
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke 7eutlmoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What Makes Good In-Context Examples GPT-3? arXiv preprint 2rXiv:2101.06k 100.066044
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin HofiaannAAptius. and
Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of biomedical informatics, 45(5):885-895
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint 190.v19907.1169
Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Wash- ington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about gpt-3 in-context learning for biomedical ie? think again. arXiv preprint
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.0872 08778..
arXiv:2203008110.
Frank Hampel. Elvezio M Ronchetti, Peter Rousseeuw, and Werner A Stahel. 1986. Robust statistics: the approach based influence functions. functionn Iitey-Interscience; New York.
Yi Luan, Luheng He. Mari Ostendorf, and Han- naneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scien- tific knowledge graph construction. arXiv preprint arXiv: htl::1808.00000 096002
Corey Harper, Jessica Cox, Curt Kohler, Antony Scerri, Ron Daniel Jr, and Paul Groth. 2021. Semeval-2021 counts and measure-
task 8: Mcaseval-extractee ments and their related contexts. In Proceedings of the 15th International Workshop on Semantic Evalu- ation (SemEval-2021) pages 306-316.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.088. 303.0855.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023. Tabllm: Few-shot classification of tabular
Zara Nasar, Syed Waqar Jaffry, and Muhammad Kamran Malik, 2018. Information extraction from scientific articles: a survey. Scientometrics, 117:1931-199

---

Yixin Ou, Ningyu Zhang, Shengyu Mao, Runnan Fang. Yinuo Jiang, Ziwen Xu, Xiaolong Weng, Lei Li, Shuofei Qiao, and Huajun Chen. 2023. EasyIn- struct: An Easy-to-use Framework Instruct Large Language Models. https:/ /ggi wtw.ggttth......ooooooo wtt.c/oo//........e //nnlp/ EasyInstruct.
Chengwen Wang, Qingxiu Dong, Xiaochen Wang. Haitao Wang, and Zhifang Sui. 2022a. Statistical Dataset Evaluation: Reliability, Difficulty, and Valid- ity, arXiv preprint 2xxv:2222..9227.
Shuohang Wang, Yichong Xu, Yuwei Fang. Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022b. Training data is more valuable than you think: simple and effective method by retrieving from training data. arXiv preprint arXiv:2203.0877
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11900
Stephen Robertson, Hugo Zaragoza, al. 2009. The probabilistic relevance framework: BM25 and be- yond. Foundations and Trends in Information Re- trieval, 334)3333-38.
Leigh Weston, Vahe Tshitoyan, John Dagdelen, Olga Kononova, Amalic Trewartha, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Named entity recognition and normalization applied to large- scale information extraction from the materials sci- ence literature. Journal of chemical information and modeling, 59(9):3692-3702
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning retrieve prompts for in-context learning. arXiv preprint arxiv:2112.08633
Julian Salazar, Davis Liang. Toan Q Nguyen, and Katrin Kirchhoff. 2019. Masked language model scoring. arXiv preprint arXiv: 14659.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrutions, pages 38-45.
Hinrich Schutze, Christopher Manning, and Prab- hakar Raghavan. 2008. Introduction to information retrieval. Cambridge University Press.
Fobo Shi, Peijun Qing, Dong Yang, Nan Wang. Youbo Lei, Haonan Lu, and Xiaodong Lin. 2023. Prompt Space Optimizing Few-shot Reasoning Suc- cess with Large Language Models. arXiv preprint arXiv:2306.03799
Yichong Xu, Chenguang Zhu, Shuohang Wang. Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. 2021. Human parity on commonsenseqa: Aug- menting self-attention with external attention. arXiv preprint ar.0v:2112.03254.
Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An theooret approach to prompt engineer- ing without ground truth labels. arXiv preprint arXiv:2203.11364
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR
Vera Sorin, Yiftach Barash, Eli Konen, and Eyal Klang. 2023. Large language models for oncological appli- cations. Journal of Cancer Research and Clinical Oncology, pages 1-4,
A Dataset Details
To understand the performance of the methods the datasets, we calculated the difficulty of the datasets and the similarity between the train and test sets of datasets, As difficulty metries, we use 2 metrics: Entity Ambugity Degree (EAD), and Text Complexity (TC) (Wang et al., 2022a). We also use Target Vocabulary Covered (TVC) as similarity metric (Dai et al., 2019). The details are given in Table 7.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.139) 13971.
Sean-Philippe Vert. 2023. How will generative Al dis- rupt data science drug discovery? Nature Biotech- nology, pages 1-2.
EAD captures observable variation in the infor- mation complexity of datasets and our findings show that the SciERC and STEM-ECR datasets have the highest degree of ambiguity, implying that is more difficult for models to predict correct
Zhen Wan, Fei Cheng. Zhuoyuan Mao. Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation ex- traction using large language models, arXiv preprint arXiv:2305.02105.

---

You are a hishty intel and apcursle Named-ant jecacnlerecNEey syakum, You saka that given passage cleesity rno ralloing preceined emity ypee: Generic, Material Method, Mobric, OtherSclenditc
Prompt Template
selected samplos
The anslyow is called "Amorphh".
New Query
Sentence
Entity
Input
standard dosasets)
Large Language Model
Our excerimentt show that inga modela illearily cutperfomn forn EM traaalation modila
Muthod, W. log trear modela), (E: Mathod: W. IM translation modals)
Output
thes papper we propose complementerr
spproach holps usors process
Genertc, complomentary approact). Mothod, coooss models)
JE: Genert,, W. anayyzed (E: Method, W. "Amorph")
Figure I: Illustration in-context learning for entity extraction.
Difficulty
Similarity
Dataset
Test set
# Entities
Avg
Avg
Set 1
260
15.12
133.06
Dataset
EAD
TC
TVC
ADE
Set 2
247
14.71
129.7
ADE
0.42
30.72
81.51
Set 3
227
15.88
123.4
MeasEval
0.32
9.13
53.68
Set 1
462
7.85
72.72
WLPC
Set 2
457
8.39
68.62
SciERRC
2.26
41.09
68.12
Set 3
8.51
80.83
STEM-ECR
2.07
61.06
66.12
WLPC
1.51
35.79
69.32
Table 8: Statistical details of test sets used in Section 5.3.
Avg e the average length entities Avg s the average length of sentences.
Table Difficulty and similarity scores of datasets,
E Visualization of Entities
entity types for ICL methods. It can also be seen that the TC values of the SciERC and STEM-ECR datasets higher than those of the other datasets. In addition the difficulty metrics, the TVC simi- larity metric calculates the similarity of the tokens the training and test datasets and shows that the MeasEval test set is less similar to the train set compared to the other datasets.
The visualization of errors made by the Influence method with gold entities for 15 sentences are given in Table 10, 11, 12, 13 and 14 for ADE, Mea- sEval, SciERC, STEM-ECR, and WLPC datasets, respectively. We use different colours except green to highlight the entity types and we high- light the wrong entity type even if the extracted entity is correct, and the wrong extracted or wrong labeled entity with green, in the prediction of Influ- ence method.
Prompt Template
For the experiments, we use the prompt format (l) of the EasyInstruct framework defined the Named Entity Extraction (NER) task. The prompt used in zero-shot and few-shot learning is given in Figure with the illustration of ICL for entity extraction.
C Test Set Details
Test set details used Section 5.3 are given in Table 8
In-Context Learning Experiments
The experimental results with various & samples in
ICL conducted for 100 sentences can found in Table 9.

---

Method
ADE
MeasEval
SciERC
STEM-ECR
WLPC
Random
Sampling
1-shot
70.62+1.23
1..9
21.1040.01
21.27+00.3
27.1841.22
3-shot
72.24
18.5111.66
22.34 13:25
25.87
29.3411.22
5-shot
74.56110.3
20.61
23.8341.09
25.6711.24
29.77
10-shot
73.63
18.30
26.69
26.85
32.20
20-shot
72.5217.30
22.49 11.45
29.27 10.73
26.83
29.57 11.18
KATE
1-shot
71.44
20.65
23.94
26.21
32.34
3-shot
77.76
20.29
24.97
23.89
43.33
5-shot
81.45
21.76
27.56
26.22
37.06
10-shot
83.11
22.75
29.97
26.60
40.68
20-shot
77.31
22.55
29.84
30.78
45.02
Perplexity
1-shot
72.45
20.98
17.42
26.26
20.03
3-shot
75.12
19.73
22.12
26.57
30.46
5-shot
79.13
20.58
27.28
23.15
24.84
10-shot
78.52
21.08
31.31
22.86
24.53
20-shot
76.51
21.43
28.79
24.11
21.13
BM25
1-shot
75.40
21,43
24.42
24.55
35.01
3-shot
75.94
21.37
28.46
25.61
38.35
5-shot
74.99
23.24
31.90
24.01
39.66
10-shot
77.28
23.76
35.99
23.69
42.09
20-shot
76.74
24.72
35.96
24.94
44.14
Influence
1-shot
72.13
24.45
21.15
18.54
31.47
3-shot
78.67
15.52
24.18
24.18
35.53
5-shot
86.35
27.13
30.78
27.81
40.36
10-shot 83.36
26.74
36.47
26.43
45.41
20-shot
78.23
25.42
35.11
25.15
41.18
Table 9: ICL experiments with different & in-context samples. The best results for each in-context sample selection method are given in bold.

---

Table 10: Selected sentences from the test set with gold and predicted entities for the ADE dataset. AE is the abbreviation of Adverse- -frse--fffettt entity type.
sulphusslanzin Drog
salphasslazzne prog
in
socondary to cathimazale prop with roambinant human grandl granulacyte ocyte calony stimulaing facter G- CSP
prcpiancyy aBB
with recomimant buman graullogie colony stimal ammusiing foctor Distrrrea Dreg
function
human temanngn A .(b a distinct phenogype exists the sufety CP pregnuncy
penicilamine meeu Truug
phenotype exists and salety of
impaired ronal AB
beginming on 17th day of a course
17th 17th of course
is found almoss exclusively childheod nephentic syndrome
is found almost exchniielly in childiood nepliratic syndiriee
dantrlenn Drue
performed patient with seronegative theumatoid arthritis who developed hypogammaggh www.linemiii...ommm Ar: in couose ...
who developed by www.gniniiiiiiaaaaa. in
impoired renal function
therayy for man Hodglin 's Jymphhoma patlsaastttaeaa..
mppily respomstics to intorenouus dantrolone.
bone manow supresssoo, bone mirrow suppesssion two cardio pittents with trisomy 211
pasient with an
while rocei D panicillaminee Drog
bepaiitoxicitty AR while receiving
intravernus Sooste
Bronthiolits ottitemns with organizing poeumomin after nimotimab theropy for noo Hodgkin 's (ymphona
lymphuummoppllly A. and Iepaitiis beginzing
chemolerasp in breast cancer patient with
chematherapy in Ireast cancerpaaeeeeooo pasess
[ymptaddsppptlly As and Depuiitis as
rapidly respunsiv to
thempy for acme myyoardia infarstion pr
depressed prec patsents.
in uwo candise palients with uisomy 21
Uharagy for acute my anyocandiall infarction.
Trancient tranddone prop -imduced hypomanic gymptoms, A occumed three depretsed
polnomayy toxicity AR is msally a dramatic condition
induced polnoney tocicly 4.8 is usally denuanie conditinn
cartimarole pres
rheumatood antrriik aoree
iheumatiod artheitis muneme developed eviilence of
tedlphooshhammde Druz human lenotogen.a A (b
Two patients with dicumatoid ariiiis deve developed evidlence of Inputotosiclyy A
inducad namroleyeicc malignant ppant syndrome,
induced neurolegsic maligmaat yyndroee, a
Brunciolltss obliienns with organizing encumonia alter siuxsimab pres
inlluced plotosciiiill seactioos A.
indaced plotosenarivity reactions A.
cental nevvous tonicitty
contai naramms toxicit
occured
secomdary
capocitabine Dvag
complicating high dose iodeamidee proup
complicatinng high dose irndaaiide D Drop
capecitabine
indueed bone marros uppressionn..
fever
Fever
of Agramnloyytnn gassis AE and neurropemic sepels A.
of sgramnlocyton tosis a.5 and noutropenss supsls 4.
performed patient with delinize seronegative
Massive sabaisciaal loustoona, a after alteplaso Druu
Maroive suhifascial hematoma, i after aktplass roup
Transient traoodone induced pommmiic sy sppoomnnn.
gyclpphophuriile.om prug
According to the literature. chiomihhell Dro
consellition demssttt A.
S7 Inflence developed comssellation llatioo demmatiss A
chimmmuuillg...
unsociate with
ansociated with
Two cares of incqulistiin Dig
Two mequitzzin prgp
metoeley wenneleprmmiie wamideepppppp... prp
S11 Influence Fuhminant metoclegramisee pras
induced
paieens with
Peripheral neuropatty A/d
Peripheral neuropaty E
-nnhhedd
According the literature,
Geneitubine Dovog Doog
Lehaal amaria A
Captopni p
Captopnil pruu
Letbal amoria
conchule
conclude
develoope
Pulminem nt
St Inthouse
52- Infhnence
Influence
S4 Infheence
55 Infhoence
86 Influence
5B Infloenee
89 Inrhnencs
S10 Influence
S12 Influence
S13 Influence
S14 Influence
S15 Influonce
S1 Gold
52 Gold
$3 Gold
S4 Gold
53 Gold
S6 Gold
57 Gold
88 Gold
59 Gold
S10 Gold
S11 Gold
S12 Gold
S13 Gold
S14 Gold
S15 Gold

---

before they encoumter Rhez.
before they esceenter Rhon.
0.15 g CO2 m lacoumeddquusn quauunuuuuuuuuuuuuuuu ....
10 electron
electrom
-666 kan s-1 Liquancc and the effeetive thenual velucity along the LOS to the is 12.5 km s- -lqunncity (Wood et al. 20005.
6..gue u km.s and effactive thermal velocity along the LOS to the stars is 12 12.nntt km (Whod et 2005).
g CD2 m-2 h- boleccmmmtdd
and latiniee 78 'Qunaiity from simulations RI I-R 88 ('aalle 1) are shown in the upper punel of Fig. ax a function of 10 K
overa lange carpox of C code was that 89% 89% woggttt of the programs studied contaimed at least one' Qunniitt dependennce cluster componed of 10
of the progroms had dependencc elaster that consamed over hall% Qullsrtaantit of the programm
We alto found evidence of super peer-laa large dlusters: 478 of the programs had dependence clatte thas comsumed over half% half% grg.gngnniin of the program
the plastic zone at the tip of the sub- Ml--titaallly londed cnck was sectional and observed using Grssmisiion optical microscopp
cole.
donoted signiticut portions of code.
hetween "C and 109 Ciuumnitty
a 1700 1700att
will illostttte ont tests of Liouville's theorem using data for olectrons with an eneggy Fi 90 le o quanity and a pitch angle is 170 quantity
White the values are bosed equizox simo labions foond secsonal diffeeennss to insigaitiintt gonenting tomporatire: chonges of <11 Kquansty
10% 10%tuua
Miguentity while heterotrophic component D. (iQuuaiitty
Table 11: Selected sentences from the test set with gold and predicted entities for the MeasEval dataset.
fromn sinuullttoonn Intiionionn RI -RS (Table 1) are shown upper panel of 12 as fumetion of
After fraeture at 20 C, the plantee zone of the sub-oriicalll londed crack was soctimted and observed using transmassion optical microseopp.
Here the rubber or thermoplostic pontickes are typically about 0.1-5 prequaante in dianeter with a volume finction of about 5- 2
Here the nuhhher or thermoplasii purticles are typically (0115 pum sumgsenticc in diameter with a volume fraction of about 2020 %.
but the competition between these effects results in the fracture energy being inde independenn: of the tust temperature between 55 'C and 109 CC
While the valnes are haxed on equiloo simu sinulations. we found sccsomal differences to be insigni fiaiiiittt generating tomporatae changes of 100KK
The average of program enalied 20KI oC qaoneia these clusters more than 10% (0iguunnti bly doneted signittaant
of the programs studied contiined at least use depeddenesss seseddence.co cluseer composed of
We will illustrate our lests of Linuville' 's theoeem using data for electrons with an energy E is 90 ke Visonnttt and a pitch ungle
50iligventtru while huterstrophic component
This scemario may also explain other Apectodinium 2619.6 and and 26 imgraanti (although Section 4.1).
Fig. 5 shows the avenge slice size dos dosintiin when using the Jover (onguanntty settings compared to the bighox.
The average size of the programs stalied was 20KL CLLLOC "OLLoORRRgaane Quantity 50 these clusters of more than 10% (0%
This sccaarlo may also explain the other peokss Apsztodinum at 2619. uonetttt and 2614,7 llqaontit
but the compettion between these effects results in the fracture energy being indapendeen of the tedt temperature
Fig. 5 shows the avangg slice sine des intioononn when using the lanee two scttings exmpared to the tighest.
In all illguannitss programs, the Low seting yiclds larger slices cumpared to the High sotting
In prograns-gguao. quaeevity tthe Low setting yields larger slices compared to the High sottigg
The model proflees were convolved speciral resolutte k- 17. 775000
17,5000
larger fraction
fraction
The mode) proñlees conviived specitall resilation R
We also found evidence of super-lagge cluster: 40%
over a hagg cuppes C codee was tlai. 89% 2P%
78 Qus
g CD2 he Icoerniuutdd
volucity of the ISM with respoct to Earth is
velucity of ISM with respect Eanhh is
10-5 misur uontity and latitude
After fracture at 20 Corennntty
D. Rigunnitty
10 5 mharg
...
-
-
$1 Inthnence
82- Influence
53 Infhnence
S4 Influence
55 Influence
56 Infhuence
57 Infhuence
S8 Influence
$9 Influence
Influence
511 Influence
$12 Influence
$13 Inffuence
S14- Influence
S15: Influrnce
51 Gold
52 Gold
83 Goold
84 -oold
55 Gold
56 Gold
$7 Gold
58 Goold
$9 Gold
$10 Gold
S11- Gold
$12- Gold
Gold
S14 Gold
$15 Gold

---

Table 12: Selected sentences from the test set with gold and predicted entities for the SciERC dataset, O is the abbreviation of Other.
annotated training corpus wuppliedeeeeiii
requeko compleie
memoraee traising coppos Maseerial
generaie the final miruut
object slape
objective functions
R-gram medellsy
objectt siager
rul: applicalionncce
generuec output.
when complete
objective functiono
withoou relying on any
chunges or when
o
surfare mesho minimmie
aboul
rule application chod
Spp Japanele ohammteer tringso
jud shod
in traditional
speeeh sy myooc withoot relying on any
prion Iaton Ledgey
prion
dictioneyy lookupp oeceeeers and
an
approgeiits ditabust moril
this information added to the legment and is used to
staliin of requeste
speech systak
to verily the practical feasibility of our algosithm
Japunese charactor strings
sy syntacis knowledgeo
gynactic know ledgerr
lack of smichtrc
conducted numeecou simulation to vatily pracical Feuniiiltt of out algonibmmeenaa
ar tha
recurs couuestructionn au resuls and snforce lbeit comi wilb a
enforee cons consistency
and
al appropriate
transprent objeatso by moving observer.
information to segment
that can model
moving
umess of (eaturth in rubost PCA.-ttodd
dhem when the
dittionr lookap
sevema kiinds of dichonaries segment and
address the
hand cralted rales ot that encode busic
in mius PCA
Labeled danara is replaced by few hand crafeed miles that enconde basic
that can model
motily them the
pareer af Iita
based Seraiive deformation choel
objestor
Stgglt
kiinds of diet regegeent
Requettis can also instruct the sysemm ld nobiy
Ieformation
The theoreticalll jasiites effectre of featurs
stges
ie two
reseurch diretiont
oark
weakly super sppervised vised dependency purse Poak
bosed a weakly supervised dependenncy
inteliggun agest ap chood
mobile Geserss for
propuses new directionn address
When segmento is found be an NF iemso
collection of salient image feanursk The request
Seleggesenss
Each part is collection of salient inage Teatures
this paper, we explore what can about
said ahout The resul (bcoreticaaly jusilles libe effectiv effectiveesss explore
When segmentis found tn an htmoo
Amorph
NB iemmso
NE feemi
We conducted mumeroos wamulationnn
Amurgh O
This Work propeess new
ThyyGeeeri crri improve oa sa
is called
Our approachh Genevi is hased
Labeled data is replaced by few
recogniass
recoganes
improve the
Requeston instruet
The analyaerc is called
The analy mallyeen
Pinst, #GGnneri
Our approache
The Pheyeeemeen erte
is basel on
The request wark
Amerplost
Amerphou
In this
Finst.
S1 Inftuence
82 Influence
83 Infheence
84 Infhnence
85 Influence
56 Infheence
56 Influence
56 Influence
89 Influence
S11 Influence
S12 Lnffusnce
S13 Influence
S14 Influence
818 Influence
S1 Gold
52 Gold
53 Gold
Gold
S5 Gold
S7 Gold
58 Gold
89. Gold
S10 Gold
S10 Gold
Gold
Gold
S14 Gold
S15 Gold
S12

---

aquecus cell 1oom empprrettre poott omiit-yer multi multi-raation
in non aguesms Na cell foom temperature
such as octaaeedral and square py pyramidal coordnattonn www.diattinn..... D
solar minimum
Uhe proposed algocilmms
al solur minimum patta
which are normally
groposed algoriimms Pratese
locatittiiin lield leeld trials rocca
exhibiis different gromeetical arrngements Pa soch as octishedrs and square pytanidal coordimction PalteePProooe Proveey
Table 13: Selected sentences from the test set with gold and predicted entities for STEM-ECEES ECR datasct.
coll--aattonn field tials
(1:44:)) Datt
bosa prodicted PrOaEA
Dast
which are normally
from hPSC lines cnterial
in Enilt--ycae mutti- mutit-lcctionn
of purticle pare were found on messured moceer value of Loughaess ph
the nightside lonosghenee Sia near the teminator region
of D:A blend coeppositionsstttio ateeral
CVD meiloed
of D blend compo pitiiosiinnnssoo of Material
with subssquent mathol thod
the measured value of toughne Sw tha
bour the teminator region
the per formonneee
of the
heal netwerk, aateeeeeeen changed Proerss diseused Mf aettrria
has predicted, P Proceus
demomarnies encellemt electrnie perfarance pata
into hop hopiincso- like colls colls Material
hopatocyte like calls tevial
Nodes' Pranso prevailed when network, St changed discosed of FAc
electrode serfomatnce
with subsequent
be obtained MPSC of aterria
Satun't magnet www.ooppcccccc of aferraa
pate
from 30 to 3.30m
30 130m (30m
in
a single-cell erpopreser vation Pr Precee for NiPSCSs in su spspessioe Me terval
goometrical aetangentt compenants Dete
variyy the pario menformnnce
of the resultenn trangeencc gatoty salotyyosatttenn
can be ohtained
elose monitrring iing Proeees of the cositrent tonagenic gasotyes Easotypsaattco
Chrisollie symbols ctanbollaaaatii staterinl
across Satun's mig wwg.stooouus.....mmmmmm Materal
a wide ragepatte
demomstntes excellem
CDC procestar
mensarch paaa
is free of Chrnoofill symbools....t
CDC process P Prcceceee on
in
were found on
mains muinsnining the nightsiee iome iomopheree sphrreereeeee i
extessive experiments Proceas ast carsed out on several data sels-ata to
whole hrain mussures
wide range Darke
We aptiiindd sing lell---ea cryopreson ergppeeerenntinn Air fhot for hnPS Sad oe torteeraa hal
differentiate paatiiitppooeeee Proecoo
differentiato proveee
ware activityy it p. occes scoss
over
The dual-la layer carbon lilm Material preprred roccen uxing
pafferellpptte
several dain
of IPSC
of purtille size poera
Punhermoce, nea--homoggeon papulittio aFSC ontorial
whole
efficency Data
over
The presened contra formul aiton mone a is frve of
onhalt sabsittted NaFso2 M tovial
prepured using
maintaining--
have potentia to
potontial to
provided
provided
cahunces effic effcienny p Poweess
close monitoring occas
cohals- sabseiuted NaF NaFx
exhibiis P Prnccse
The presented contavarian formul formulation ationpzatt
extensive experimenis wth od carrted
Diffecences the level activity Daata
Funhermore, near homoggggggg tomogemaus rxpllations
when
Disrihated souree Incalization toed
colancesp rececu
Howeves no significant eleects pesettreent vOCKAA
However, no signilicant praceea
this ion flow Matercel contribuses to
The dual- carbon lilmaratersa
commrluttes
Nodes role- shall prevalled Proeeeee ma
in the lovel pote
Data
PA P-ppoiiffe IPS walls salls Materval
FAP spocific cells sell-tarersn
Ditrituted snuroe Incaliatton,
The metal complesee peoesssssnnno Meterva!
The metal complexes Ma tetmal
We optimi zod wadiieddreeeee recess
Hense we recommend
Henxe we resommend
Herein we report that
Ameeiiing p. Fox
Aneeaingg P Pronecs
Herein, report
Diffrrasea Data
this lon flow
-..
Influence
82. Influence
83 Influence
S4 Inlluence
85 Influence
S6 Inflvence
S7- Inltnenee
SR Intueecc
Iniluence
S10- Influatce
S11 Influetce
S12 Inlleence
$13 Infuencc
Gold $14 Inlluence
S15 Influacce
S1 Gold
52. Gold
83 Gold
54 Cold
55 Gold
56. Gold
Gold
S8 Gold
S9 Gold
S10 Gold
S11 Cold
$12 Gold
$13 Gold
815 Gold

---

Action
DAA stention
Modh Fer
rostipend Action
Table 14: Selected sentences from the test set with gold and predicted entities for the WLPC dataser.
Mrdd Feet
RAmount
(bc Deriee
Dre Desiee
Mused Clycagen pt inagent
gently of oud fice
tomm
and vinus avaembige charact characterizaiinn www.eeriiaiiiiiiin
DNA Rengent
the
TM For High Quality Genenic DNA Extraction From Grau- Plan--osiive Bactenia
EDTA
Action invertigg Action
S0pl meennt
Pte to For High Quaitty Genomic DNA Extrccionn From Grai- Gran--oosiii Positive Bacteria
for 15 minulesy
at 60"C Tomperuu
the
for 2 anautles
for minutess
for minatess
times to preppiine ppinne Actioo
3pl mmnun
for 1S minutesy
60'Cy
tomm
Elatronn microscopp for ViUUS thteee Bid Vilus assemblage charedeer
procipiinn ot saad
10M0 Syroed for minunely
Mossure
For 5-10 minutery at
WalofRespentt
add adliinntonn
Precipitation Solution pore
DO NOT DISCARD
for 5-10 minurcs
14,0 14,00kgggggeee
liquid plregeee
and
14, MMInge
Moddy fer
for vinus inlentiiictiionn won.intionnnn
Lquu Ge
55 SS-C
Glyesgen
100p1 Precippi Soluzion Dengee
momummmen dimss to
and
stenile oads
lncabate testom the sample at 55-60' Cremp
recovvrry at on
x
x
periodically
10 cemmer
the
tbe
super wat.tttttttton action
at
Mdd collect Aesiom
higher than 60CC.
sampl tomplepp no and
color nolorbounmggeaii garni Spinn
ant
collect
the supernatant
mecnn
ibb sample
sample
the sample
the campleppe.
cre voice
tube periodically
nubes Des site
for min cnlorbotmaggeo gento Spin,
Thore are the ords of interesk
Do not heat higher than 60 C. nal heat Bus
lime
incressed fine
450pl
thee
ibc
Election microcup sppy
cells
S10 Inffuence Canrituge actiom
Poor Aotvon
Diaard Actiom
Disard Aesions
These the
OamPPopp
Add Action
Add Action
Incabale
Incabate
Incabate
Add Auston
Add Restom
Coutiitgee
Invert InvertAction
Invert Ancion
Lnvert Aucionn
AntetAattion Andion
uncrened
mine
Pour
For
$1 Inthennc
52 Influence
$3 Inthenne
84 Infhnence
Infheence
S6 Infheenc
S7 Infhence
S8 Infhsence
59 Infhoence
$11 Influence
Influence
513 Influence
S14 Inffuence
S15 Infuuence
51 Gold
$2 Gold
$3 Gold
$4 Gold
S6 Gold
57 Gold
58 Gold
S10 Goldd
S12 Gold
814 Gold
S15 Gold


Reviews:
[{"review_id": "EHImRBEjMb", "reviewer": "Reviewer_JH68", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviews. \n\n- **Why only one LLM has been used rather than using other instruction based models such as FLanT5, LLAMA and the Scientific LLM i.e., Galactica?**\n\nThank you for the comment. We agree that a comparison of different LLM would be interesting, but due to our limited resources we only used GPT-3.5 in our experiments and stated this as a limitation of the study. \n\n- **Description of Prompt templates is not provided.**\n\nDescription of prompt templates is provided in Section 3.1, Appendix B, and in Figure 1 of Appendix B. We are going to provide code repository for the experiments (on acceptance), which will also allow another way to access the prompt template. We would be happy to add an example of a full prompt to Appendix B in the camera-ready version of the paper, if the reviewers deem it beneficial to the paper. \n\n- **Why authors finetuned Roberta on scientific information extraction task as the supervised baseline. There are some more advanced scientific pre-trained language models (e.g., SPECTER [1] and SciNCL [2]) which are not examined. Because RoBerta is pretrained on wikipedia dataset.**\n\nOur intention was to provide >>a<< baseline well established in entity extraction tasks. The paper is focused on instance selection for ICL, not on comparing different PLMs in fully supervised settings. This being said, we agree that choice of a strong baseline is important to provide appropriate context. In this sense, we believe that RoBERTa works reasonably well. Our follow-up experiment involving RoBERTa and SciNCL shows that RoBERTa outperforms SciNCL on 3/5 datasets with full training data and 2/5 datasets under the limited training data regime. In any case, using a science-specific PLM, would have changed nothing in our main conclusions, SciNCL is better/worse than ICL in the same cases (dataset/data availability) as RoBERTa. While we believe that adding a science-specific model (such as SciNCL) does not add much to the discussion of ICL results, we’d be open to adding this baseline to the final version of the paper to demonstrate that the comparison between full tuning and ICL is not unique to RoBERTa.  \n\n- **Usage of external information for in-context learning technique is another interesting paradigm. So, the author could use S2ORC [3] as the required external source of information.**\n\nThank you very much for the suggestion. The aim of the study is providing a comprehensive analysis of ICL methods to choose samples from the train sets as formulated in the literature [1,2,3,4]. Therefore, however interesting, the use of external information for in-context learning (or response grounding) is out of the scope of this paper. \n\n- **SciREX[4] could be another dataset to conduct these experiments**\n\nThank you for pointing out the dataset. In our experiments, we selected 5 datasets with different number of entities and subdomains to provide a comprehensive analysis of the ICL methods by covering different scientific subdomains. We believe that our selection is broad enough to justify the findings, and that missing the SciREX dataset does not compromise the soundness of our evaluation. Nonetheless, we’d be happy to include the SciREX dataset in our future work.\n\n- **The results reported in the Tables are not analyzed properly. Authors could analyze the results rather than reporting it in tabular form.**\n\nWe are not quite sure what is improper in the analysis of the results presented in the manuscript. If there is a specific valuable insight that can be gathered from the tables, but is omitted in the discussion, we’d appreciate if the Reviewer specified it.  \n\n- **Significance tests are missing. To show the proposed strategy is beneficial.**\n\nThank you for reminding us about the significance tests. We did the tests but inadvertently forgot to include the results of the significance tests in the paper. We will add the results to the camera-ready version. Most of the ‘clear’ differences (between our approach, influence, and random baseline/trained supervised RoBERTa baseline) in the Tables 2 and 3 are also significant (we ran a pseudo randomisation test with an 0.05 confidence level), so our key conclusions stand. As for the results that already are in the manuscript (although it’s not quite the same thing), results reported in Tables 4 and 5 (as well as the variance reported on the random baseline) shed some light on the variance/confidence intervals. \n\n- **Anonymous source code repo is not provided.**\n\nAs we stated in the paper, we will publish our code in the camera-ready version. We believe this would alleviate the reproducibility issues pointed out by the Reviewer. \n\n**REFERENCES**\n\n[1] Radford, Alec, et al. \"Language models are unsupervised multitask learners.\" OpenAI blog 1.8 (2019): 9. \n\n[2] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901. \n\n[3] Liu, Jiachang, et al. \"What Makes Good In-Context Examples for GPT-3?.\" arXiv preprint arXiv:2101.06804 (2021). \n\n[4] Min, Sewon, et al. \"Rethinking the role of demonstrations: What makes in-context learning work?.\" arXiv preprint arXiv:2202.12837 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper is well written with Good motivation.\n\n2. Novelty of the work is strong enough.\n\n2. Empirical results demonstrate that the in-context learning technique is really helpful without developing any other supervised setup.", "reasons_to_reject": "1.Why only one LLM has been used rather than using other instruction based models such as  FLanT5, LLAMA and the Scientific LLM i.e., Galactica?\n\n2. Description of Prompt templates is not provided. \n\n3. Why authors finetuned Roberta on scientific information extraction task as the supervised baseline. There are some more advanced scientific pre-trained language models (e.g., SPECTER [1] and SciNCL [2]) which are not examined. Because RoBerta is pretrained on wikipedia dataset.\n\n4. Usage of external information for in-context learning technique is another interesting paradigm. So, the author could use S2ORC [3] as the required external source of information. \n\n5. SciREX[4] could be another dataset to conduct these experiments \n\n6. The results reported in the Tables are not analyzed properly. Authors could analyze the results rather than reporting it in tabular form.\n\n7. Significance tests are missing. To show the proposed strategy is beneficial.\n\n8. Anonymous source code repo is not provided.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work  presents a comprehensive analysis of in-context sample selection methods for entity extraction from scientific documents using GPT-3.5 and compare these results against a fully supervised transformer-based baseline.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1.  SPECTER: Document-level Representation Learning using Citation-informed Transformers. ACL'20.\n\n2.  Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. EMNLP'22.\n\n3. S2ORC: The Semantic Scholar Open Research Corpus. ACL’20.\n\n4. SciREX: A Challenge Dataset for Document-Level Information Extraction"}}, {"review_id": "54qymPVCMl", "reviewer": "Reviewer_v6zx", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviews.\n\n- **The paper only analyzes existing approaches for in-context sample selection for entity extraction from scientific documents. Findings are interesting but no new methods are proposed.** \n\nThe influence method is used in the literature to detect errors in the dataset and to create adversarial training samples [1]. We adapted Influence as an ICL method for sample selection because it involves the scoring of a training sample, which allows for a straightforward application. To the best of our knowledge, we are the first to use Influence as an ICL method for sample selection. We will clarify this point in the camera-ready version. \n\n- **The findings from the paper are very similar to https://arxiv.org/abs/2203.08410. Finetuned LMs achieve superior performance in specialized domains (such as biomedical) than LLMs with in-context learning. This paper extends that study to further analyze different in-context sample selection strategies on scientific documents.** \n\nWe use a somewhat similar experimental setup, but our findings are quite different. Our results show that ICL yields better results in training-data-poor setup. We agree that the focus is also different (we do focus on different ICL strategies and comparing them), but we can’t see how this broader point makes for a reason to reject our submission.   \n\n- **It would strengthen the paper if more tasks over scientific documents (e.g. relation extraction, classification etc.) are considered.** \n\nWe agree that extending the analysis of ICL techniques to other tasks would be interesting, but we can’t see how one could fit another 2-3 tasks, and consequently another 10+ datasets, and corresponding discussions into a single conference paper. We are happy to take this suggestion onboard to guide our future work. \n\n- **The paper doesn't compare against other few-shot entity extraction approaches. It will be nice to compare different approaches for low-resource settings.**\n\n We position our work within the fine-tuning vs ICL discussion, but, while a search for an alternative low-resource solution lies out of the scope of this work, we agree that another few-shot baseline could be beneficial to contextualise our results. If the Reviewer pointed us to an established baseline, they’d be interested in seeing here, we’d be happy to include it in the camera-ready version of the paper. \n\n- **It only limits to GPT3.5 and does not study the impact of selecting the appropriate instruction and order of in-context examples. It will be interesting to see how this behavior generalizes to other LLMs and settings.**\n\nWe agree with this comment, but due to our limited resources we only used GPT-3.5 in our experiments and state this as a limitation. On the point about order of the sample: while we do not tackle this problem explicitly, the ICL methods do yield an ordering of the samples. In fact, it probably bears heavily on the results in extreme few-shot settings (i.e., at 1% of training data for the smallest datasets), where KATE outperforms Influence. We’d be happy to elaborate on this point in the camera-ready version of the paper.  \n\n - **Yet another direction that many recent works explore is using LLMs as annotators. That could provide annotated data to finetune smaller LMs.** \n\nThank you for the comment. Our focus here is on ICL sample selection, but using LLMs as annotators is an interesting idea. \n\n**REFERENCES** \n\n[1] Koh, Pang Wei, and Percy Liang. \"Understanding black-box predictions via influence functions.\" International conference on machine learning. PMLR, 2017. \n\n[2] Gutierrez, Bernal Jimenez, et al. \"Thinking about gpt-3 in-context learning for biomedical ie? think again.\" arXiv preprint arXiv:2203.08410 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Some of the strengths of the paper are:\n\n1. The paper provides a comprehensive analysis of in-context sample selection methods for entity extraction from scientific documents using GPT-3.5.\n2. It compares the performance of in-context sample selection methods (low annotation budget) with finetuned LMs (large annotation budget), and shows the finetuned LMs still outperform LLMs with in-context learning.\n3. The paper provides detailed analyzes and discussion of how different in-context sample selection methods affect performance. These findings are important to improve methods for extracting important information from scientific documents. ", "reasons_to_reject": "Some of the weaknesses of the paper are:\n\n1. The paper only analyzes existing approaches for in-context sample selection for entity extraction from scientific documents. Findings are interesting but no new methods are proposed. \n2. The findings from the paper are very similar to https://arxiv.org/abs/2203.08410. Finetuned LMs achieve superior performance in specialized domains (such as biomedical) than LLMs with in-context learning. This paper extends that study to further analyze different in-context sample selection strategies on scientific documents. \n3. It would strengthen the paper if more tasks over scientific documents (e.g. relation extraction, classification etc.) are considered.\n4. The paper doesn't compare against other few-shot entity extraction approaches. It will be nice to compare different approaches for low-resource settings. \n5. It only limits to GPT3.5 and does not study the impact of selecting the appropriate instruction and order of in-context examples. It will be interesting to see how this behavior generalizes to other LLMs and settings.\n6. Yet another direction that many recent works explore is using LLMs as annotators. That could provide annotated data to finetune smaller LMs. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates different in-context sample selection methods for in-context learning for entity extraction from scientific documents, where annotating large datasets is expensive. It provides a comprehensive evaluation of different selection methods across different sub-domains and entity types. It finds that fine-tuned RoBERTa still outperforms different in-context sample selection methods. In-context sample selection methods, however, are robust when limited annotated data is available.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ABM1twC52F", "reviewer": "Reviewer_89Vn", "summary": "", "questions": "In Section 5.3, the description in the text and the tables do not seem to match.  You said \"... in-context sample selection methods, with the exception of KATE, appear to be less affected by the test set composition and yield similar results across different test sets.\"\nHowever Table 5 does not seem to suggest so? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Generally speaking, the results still fall far behind the strong baseline.**\n\nThank you for the comment. The main question of this study is to investigate the potential of GPT-3.5 in ICL learning for the task of entity extraction of scientific documents, without claiming that ICL is a more effective method than a fully supervised model for this task. We formulate this conclusion in Section 6. \n\n - **GPT output is known to be unstable. It is unclear how consistent the evaluation results can be, with multiple runs.** \n\n In our experiments, we conducted random sampling experiments multiple times (Table 2 for the main results and Table 3 for the low-resource scenario). As can be seen, the standard deviation of the random sampling method is lower than the supervised baseline model (finetuned RoBERTa). We have also run the other ICL methods several times as preliminary work for the study and have obtained similar outcomes for them (see Tables 4 and 5).  We believe that these results, while not a complete analysis of the variance, indicate that the outcomes are reasonably stable (but the results are sensitive to sample selection/order). They don’t, however, answer the questions about the stability of the model over time, as results can change with updates of a closed model. It would be interesting to see a broad reproducibility study in some months/years. \n\n- **The reasons/hypotheses why different selection methods have different results are not discussed in details, although the results are exhibited in detailed ways.** \n\nWe are happy to discuss some of our hypotheses in the final version of the paper. Some of these include:  \n\n* KATE works better when the choice of instances is very limited (data-poor set-up/low-resource scenario) - this leads us to believe that the method is quite good at ordering a suboptimal subset of in-context samples.  \n\n* In our study, the experimental results showed that Influence is more effective in selecting samples from a larger pool of candidates. The method is originally a classic technique from robust statistics that computes the impact of a training sample by assessing its effect on loss (here, loss on the test sample).  It turns out that this signal based on sample utility is effective for ICL sample selection, possibly more effective than similarity-based signal (e.g., KATE, BM25). However, in extreme few-shot setting it appears that the loss-based signal of Influence degrades more (possibly due to high training variance with such a low number of instances) when compared to other methods (where, for example, KATE possibly falls back on PLM embedding ‘skill‘). Nonetheless, Influence still yields effectiveness significantly better than that of a random selection ICL and the supervised model, even with a candidate pool as small as a handful of instances (see experiments on MeasEval or STEM-ECR at 1% training data). Additionally, when the candidate pool becomes slightly larger (i.e., in other experiments @1% of training data), Influence yields the highest (or second highest, on one occasion) effectiveness of all ICL sample selection methods. \n\n**REFERENCES** \n\n[1] Dong, Qingxiu, et al. \"A survey for in-context learning.\" arXiv preprint arXiv:2301.00234 (2022). \n\n[2] Koh, Pang Wei, and Percy Liang. \"Understanding black-box predictions via influence functions.\" International conference on machine learning. PMLR, 2017. \n\n**Questions For The Authors:**\n- **In Section 5.3, the description in the text and the tables do not seem to match. You said \"... in-context sample selection methods, with the exception of KATE, appear to be less affected by the test set composition and yield similar results across different test sets.\" However Table 5 does not seem to suggest so?** \n\nThat is right, the sentence should read ‘(...) with the exception of Perplexity (...)’. Thank you for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper demonstrates methods of ICL sample selection. ICL is a new and fast developing field, and we should encourage publications in this domain.\n2. It tries a variety sample selection methods, on 5 datasets. For each method, there are different choices of text embeddings too. Other researchers could adopt some of the ideas, not only for entity extraction tasks, but also for other NLP tasks involving ICL.\n3. Some findings and analyses are interesting and insightful.\n", "reasons_to_reject": "1. Generally speaking, the results still fall far behind the strong baseline.\n2. GPT output is known to be unstable. It is unclear how consistent the evaluation results can be, with multiple runs.\n3. The reasons/hypotheses why different selection methods have different results are not discussed in details, although the results are exhibited in detailed ways.", "questions_to_authors": "In Section 5.3, the description in the text and the tables do not seem to match.  You said \"... in-context sample selection methods, with the exception of KATE, appear to be less affected by the test set composition and yield similar results across different test sets.\"\nHowever Table 5 does not seem to suggest so? ", "paper_topic_and_main_contributions": "The paper presents an analysis of In-Context Learning (ICL) sample selection methods for entity extraction from scientific documents.\nA few sampling methods are used to select in-context samples from training set, namely KATE, Perplexity, BM25 and Influence. Baseline models include a finetuned RoBERTa model, as well as two LLM-based methods using zero-shot and random sampling, respectively.\n\nThe finetuned RoBERTa still achieves the best results for this problem on 4 of 5 datasets (except WLPC). However, the in-context sample selection methods appear to be more robust when there are not sufficient training samples. The paper also demonstrates that the effectiveness of in-context sample selection methods is domain-dependent, but improvements (over zero-short or random sampling) are more notable for problems with a larger number of entity types. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates methods for in-context-learning (ICL) sample selection for entity extraction from scientific documents and compares it against a fully supervised baseline. The reviewers acknowledge that the motivation for the work is important and that the experiments demonstrate the key contribution, that ICL is useful for low-resource settings even though a fully-supervised solution still outperforms ICL (which is to be expected). 

The majority of the issues identified by the authors are suggestions for further research (e.g. considering more NLP tasks) or changes to the experimental setup that don't affect the conclusions of the work (e.g. using a stronger supervised baseline). The remaining issues seem to have been successfully addressed by the authors.
Decision: Accept-Findings