Title: Assessing the quality of information extraction
Venue: NeurIPS 2024
Accepted: no
Keywords: information extraction

large language models

quality evaluation

name entity recognition

needle in a haystack test

schema.org
Abstract: Advances in large language models have notably enhanced the efficiency of information extraction from unstructured and semi-structured data sources. As these technologies become integral to various applications, establishing an objective measure for the quality of information extraction becomes imperative. However, the scarcity of labeled data presents significant challenges to this endeavor. In this paper, we introduce an automatic framework to assess the quality of the information extraction/retrieval and its completeness. The framework focuses on information extraction in the form of entity and its properties. We discuss how to handle the input/output size limitations of the large language models and analyze their performance when extracting the information. In particular, we introduce scores to evaluate the quality of the extraction and provide an extensive discussion on how to interpret them.

1

Abstract

Advances in large language models have notably enhanced the efficiency of information extraction from unstructured and semi-structured data sources. As these technologies become integral to various applications, establishing an objective measure for the quality of information extraction becomes imperative. However, the scarcity of labeled data presents significant challenges to this endeavor. In this paper, we introduce an automatic framework to assess the quality of the information extraction/retrieval and its completeness. The framework focuses on information extraction in the form of entity and its properties. We discuss how to handle the input/output size limitations of the large language models and analyze their performance when extracting the information. In particular, we introduce scores to evaluate the quality of the extraction and provide an extensive discussion on how to interpret them.

1 Introduction

In the domain of natural language processing (NLP), information extraction (IE) stands as a critical task, transforming unstructured or semi-structured data into a structured format conducive to indexing, exploration, and further analysis. The increasing amount of data across digital platforms underscores the urgency for sophisticated IE techniques that can parse through volumes of information with precision. An extensive survey about IE is provided by [1], where the authors highlight the complexity of processing and analyzing text to derive meaningful information, given the heterogeneity and volume of such data.

Large language models (LLMs) have revolutionized IE by introducing generative methods for structuring knowledge from text. LLMs excel across diverse domains without extensive task-specific training. A survey by [9] details the progress of LLMs on IE tasks. Here, the authors address specific aspects of information extraction, including entity recognition, relation extraction, event detection, and universal IE. They review the existing models and their efficiency on a comprehensive collection of annotated benchmarks. Nonetheless, the challenge of quantitatively assessing the quality and completeness of extracted information persists, particularly in the absence of labeled datasets for benchmarking. Before conducting the experiments introduced in this paper, we perform IE on a vast corpus of business documents utilizing LLMs. While the extraction process is beyond the scope of this paper, some details about the extraction are given in Section 3.

To measure the quality of extraction, we propose an evaluation framework that relies on artificially generated complex information which is infused into the document to test the efficiency of LLMs in IE tasks. This paper introduces an iterative extraction process and a novel score, MINEA (Multiple Infused Needle Extraction Accuracy), to address the critical need for objective quality assessment measures. By inserting artificial information ("needles") into the data, the proposed method creates a synthetic ground truth for evaluation, enabling the measurement of extraction quality in various specific domains even without manually labeled data. The empirical analysis demonstrates the utility of MINEA for evaluating LLM-based IE in scenarios where ground truth is unavailable. By

automating the quality assessment of information extraction, the framework could reduce the need 39

40

for manual review by experts, saving time and resources and thus enhance the efficiency and accuracy

41

of information extraction from large volumes of unstructured data.

The paper is organized as follows: Section 2 presents a related work that inspired us when developing 42 our IE quality assessment method; Section 3 sketch a way in which structured information is obtained 43 using LLMs; Section 4 deals with shortcomings arising when treating long contexts by LLMs; finally 44 Section 5 introduces the novel method to access the quality of IE and provide the reader with practical 45 tips; Sections 4 and 5 are supplemented by numerical studies. The data used in these studies are an 46 internal set of documents related to a business case in the healthcare industry. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

2 Related work

Acommon practice in many specialized IE tasks is that well-trained experts review what was extracted and provide ground truth as done in [5]. Such an approach is relatively reliable, however, it is manual and very time-consuming.

In [4] they suggest summary score without reference (SUSWIR), a score to evaluate the quality of text summaries without the need for human annotations. The SUSWIR score can be used for IE tasks where the extracted information is viewed as a compression of original data. The score compares the original text with its summary. From its nature, it is very useful when comparing the outputs of extraction tasks among themselves, i.e., the best extraction/summary has the highest score value. On the other hand, its ability to provide an objective absolute evaluation of a single extraction is disadvantaged because the desirable output is not known.

Recently, an effort to eliminate the requirement for human involvement relies on LLMs. These prove themselves as highly cost-effective data creators, either by labeling unlabeled data or generating data given the labels, see [7]. Therefore they may substitute human experts providing the ground truth by doing their work in an automatic way.

Needle In A Haystack (NIAH) 1 evaluation is a tool designed to evaluate the performance of LLMs in retrieval across different sizes of context. Short targeted information, the 'needle', is inserted into a large, more complex text body, the 'haystack'. The goal is to test an LLM's ability to find and make use of this piece of information.

Our method builds on LLMs acting as data creators, but instead of annotating the complete data, it only automatizes the process of creating the needle. I.e., given an original text, an LLM generates the needle. The needle then substitutes the ground truth.

3 Capturing the structure

The form of needles depends on a form of data, on structure capturing the information and on the task being solved. The needles can be short paragraphs of text, account records, graph nodes as you extract information from continuous text, table, graph, respectively. The structured arrangement of information is beneficial for consecutive processing and analysis. It helps to highlight relationships among distinct information pieces. There are countless ways to impose a structure on unstructured data in order to capture the relevant information. To demonstrate our methodology for measuring the quality of information extraction, we specify a particular structure and tailor the needles to it.

3.1 Schema

To impose a structure on the data, we adopt the idea of schema markup [3] which is used to 79 communicate the content of a web page to the search tool. The schema markup is in the form of 80 structured data and can be viewed as a compression of the essential information. The structure 81 is defined by Schema.org 2 vocabulary which is a set of entity types, each associated with a set of 82 properties and hierarchically arranged. Figure 1 shows an example of structured information inspired 83 by Schema.org. It describes three entities of types 'Insight', 'Person' and 'Organization'. Each 84

Similarly, we extract and compress the relevant information contained in data using an LLM. 88 Schema.org presents a clear basis for the categorization of various entities contained in data. In the 89 rest of the paper, by schema we mean a predetermined set of types, such as {'Person', 'Project', 90 'Product', 'Legislation', 'Event', 'OpportunityArea', 'Insight', 'Substance', 'Thing', 'BioChemEn91 tity', 'MedicalCondition'}, together with their properties. The schema is set at the beginning and 92 the information to be extracted depends on it. Therefore the schema has to be tailored to a particular 93 scope of the (proprietary) knowledge and application. If a more complex or uncommon entity needs 94 to be captured, it is natural and very easy to extend the set of core types by more detailed descriptive 95 and custom vocabulary. E.g., 'Insight' and 'OpportunityArea' are not native Schema.org types, but 96 we will use them in our study. The usage of suitably tailored schema is beneficial for specialized 97 applications since it narrows the information to the relevant core and hence potentially improves the 98 overall performance. On the other hand, the usage of schemata is not restrictive as the scope can be 99 always extended by using a broader set of types. 100

101

102

103

104

105

106

107

108

109

110

111

112

3.2 The role of LLMs

LLMs are rather effective in the creation of structured data, cf. [9]. Using dedicated prompts, we get a structured text file describing entities found in the documents and matching types of predefined schema. The predefined schema (types and properties) is given to an LLM within the prompt. The LLM is asked to analyse the document, identify an information relevant to the mentioned types of entities and populate the schema with this information. It is asked to be attentive to nested entities, maintain consistency and uniqueness of extracted entities. Indeed, LLM is not prohibited from extracting entities whose types do not appear in the predefined schema. It is worthy to note, that LLMs are known to inherit biases present in their training data. If not carefully managed, these biases could lead to unfair or inaccurate information extraction, impacting decision-making processes.

Besides the information extraction task, LLMs can be used to suggest suitable Schema.org types for a particular document. An example together with a prompt is shown in Appendix B1.

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

4 Length aspects

When focusing on the quality of IE performed by an LLM, several limitations that LLM presents in terms of the length of data to be extracted from must be considered. Each LLM has a maximal content limit it can process, both on the input and the output. The limit on the output is typically much more strict. When trying to use the maximal possible input another issue may appear - the Lost in the middle phenomenon [8] says that the ability of LLMs to retrieve information from a long context declines and that the attention focuses on the beginning and the end of the context while it tends to attenuate information in the middle.

To demonstrate shortcomings arising from these limitations numerically we use gpt-4-1106-preview model. 3 The model is limited by 4095 tokens on the output and by 128000 tokens on the input (context window limit). The following sections present two major LLM limitations we have to consider before performing IE, namely length restrictions in Section 4.1 and Lost in the middle problem in Section 4.2.

4.1 Length restrictions

Long data are difficult to process because of the restrictions posed by the maximum amount of:

(O) output tokens: The restriction on output tokens means that there is some maximal length of data from which most entities can be efficiently extracted. If the length of the text exceeds this maximum, there would be no tokens for extra entities.

(I) input tokens: Maximal size of context window (input) prohibits the extraction of data exceeding the specific token limit.

Another difficulty regarding the output is the tendency of LLMs to generate rather brief responses which do not use the allowed maximal number of tokens. This unwillingness of models can be circumvented by prompting. Even so, the limited number of output tokens is typically too low and prevents effective extraction from long texts.

With a more sophisticated approach, the restriction (O) becomes irrelevant and only the restriction (I) will apply. The issue imposed by (O) is overcome by splitting the source document into smaller pieces which are extracted independently. A significant drawback is that the extracted information can be easily duplicated - extracted independently from multiple text pieces. Iterating the calls to the LLM with instruction to continue with already started extraction, i.e., continuing with the extraction in a single thread, helps to extract more information and avoid duplication. As we insist on continuation, more and more information is added and the extraction is more thorough, at least to some point - this will be addressed in detail in Section 5.1. Further, a lower number of duplicates is found due to the extraction history, i.e., all information extracted until present, which is kept within the thread.

The combination of both improvements - text splitting and iterated calls, has proven itself to perform the best. We split the document into distinct text pieces which we extract sequentially. Extraction from each text piece is carried out by several iterated LLM calls while taking into account the extraction history from previously extracted text pieces. Once the sum of the lengths of the text pieces and the extraction history exceeds the context window limit, i.e., restriction (I) applies, a new independent extraction starts. A single structured output, per document or once (I) is applied, is created by appending all entities identified from each text piece.

4.2 Lost in the middle

In the case of long documents, whose extraction consumes almost the whole context window, LLMs are giving more inconsistent results and we can observe a presence of the Lost in the middle phenomenon, see [8]. We extract information from several long documents from our business case which are each split into 15 pieces and its processing consumes almost the whole context window. We add the sixteenth piece identical to one of the fifteen that are already extracted and measure a redundancy score, for details see Appendix A. Each column of Table 1 then states the redundancy of the newly extracted information with the information that was already extracted from the same piece of the text before. The table presents mean values per four distinct documents. We can notice that

for the parts 'in the middle' the proportion of redundantly extracted entities (entities with the same 162 'name' attribute) is higher than for those at the beginning and the end. 163

5 Quality of extraction

Once the information is extracted from data into a structured form defined by the chosen schema, e.g., Figure 1, the quality of such extraction is important to evaluate. In practice, it is very rare to be equipped with ground truth and its human generation requires vast expertise in the scope of data and a ridiculous amount of time. Therefore we adopt methods from [4]. They examine semantic similarity, relevance, redundancy, and bias and compound these into a single score called SUSWIR, for details see Appendix A. The score and its subparts are very useful when comparing distinct extractions among themselves, e.g., we can use it to find an optimal number of iterated LLM calls. Unfortunately, the score does not represent an absolute way of evaluation. It does not provide a complete insight into the task - some information (= entities) can be missing, misclassified or their properties not filled in correctly. To come up with a robust and general solution we generalize the NIAH test, which is commonly used to measure the ability of LLMs to process long documents, cf. [6].

5.1 Iterated LLM calls

Since the first LLM extraction is typically not exhaustive, iterating the extraction process helps with the completeness of extraction. To improve the quality of extraction, we ask LLM to process the document again and search for other entities which were not extracted yet. A question arises: What is the optimal number of iterations? It is desirable to stop when additional LLM call will return no or only a few new entities. The answer however depends heavily on the text being extracted and on the chosen schema. Below, we present a small comparative study regarding the contribution of iterated extraction to its quality. We interpret the extracted structured data, e.g., Figure 3, as a summary of the original text document. To measure the quality of the summary we adopt the scores from [4] (a convex combination of these scores creates the overall SUSWIR metric), namely semantic similarity , relevance , and redundancy avoidance . We use a modified bias avoidance score from [4] and add two new scores, relevance spread , and incompleteness score . See Appendix A for more details.

Consider document which length is approximately 12k chars. Table 2 compares the content of the document with extracted information created iteratively by succeeding LLM calls. Each iteration enriches the extracted information, but the benefit decreases. From the third iteration, i.e., after four LLM calls, the majority of scores in Table 2 are either getting worse or stagnating (the arrows following the score name indicate the direction in which the score improves). It is obvious that shorter and longer text will require less or more iterations to extract majority of information without reducing its semantic and factually relevant meaning, respectively. Further, the risk that the LLM will suffer from hallucinations increases as we observe a growth of bias. In the rest of the paper we use three iterations to extract documents of approximate length 12k chars within all extractions (if not stated otherwise).

5.2 Test the quality

This section introduces a robust and versatile score to objectively measure the quality of IE. Assuming the structure is imposed by some schema, see Section 3.1, we would like to measure the IE quality as

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

a portion of successfully extracted entities, i.e., the accuracy of name entity recognition (NER) task taking into account even the context captured by entity properties. Unfortunately, such an experiment is unfeasible without labeled data. As a consequence, it is unfeasible in many specialized tasks because of the absence of suitable labeled data unseen by LLM models. This can be the case with very recent datasets as well as proprietary datasets. To overcome this issue we use inspiration by NIAH test to build up an automatic and general procedure to access the quality of IE tasks.

5.2.1 Needles

A 'needle' in our context represents an entity. It is created according to the chosen schema, i.e., a list of types we want to extract from the document. We use an LLM to generate a short paragraph introducing a new original (not appearing in the document) entity, but still relevant to the scope of the document, for an example see Figure 2, and for more details on generation process see Appendix B2. This artificial paragraph, the needle, is then placed into the document body at random (taking into the account natural units within the text as sentences, paragraphs, etc. if applicable). Moreover, the needle is accompanied with several properties, namely we assign to the needle a name, short description and keywords, see Figure 2. This additional properties are assigned to the needle by the LLM.

5.2.2 Multiple infused needle extraction accuracy

To measure the quality of extraction we propose a multiple infused needle extraction accuracy (MINEA) score. Its computation combines the approach of NIAH evaluation and NER task. We scatter several needles at random over the text document body (such that the inserted needles fill 10 to 30% of the enriched text) and measure how many of them were successfully extracted. Since we know what exactly was inserted, we know what should be extracted. Then we can objectively measure the quality of extraction on these new entities and moreover, we can compare extracted information from the document with and without needles. Table 3 shows extraction accuracy - MINEA score - total and per schema type - measured on a vast corpus of business documents with predefined schema consisting of types 'BioChemEntity', 'Event', 'Insight', 'Legislation', 'MedicalCondition', 'OpportunityArea', 'Person', 'Product', 'Project', 'Substance' and 'Thing'.

5.2.3 Identification of needles

Matching the generated needles with extracted entities imposes a challenge and mostly depends on the formulation of needles. If the needles are too complex or too vague, the straightforward identification changes into a serious problem. For this reason, we equip the needles with additional properties which are then used to compare the needles with extracted entities and to decide whether the needles were extracted successfully or not.

We present several alternative ways how to measure whether the extraction of a needle is successful:

n an entity with a name perfectly matching the needle name is found; ns the needle name is found among the extracted information;

237 238 239

k an entity with some number of keywords perfectly matching the needle keywords is found, the number is determined by the threshold parameter determining the percentage of keywords to be matched;

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

llm an entity matching the needle according to LLM is found. 240

̸

Note that other conditions can be constructed, e.g., based on the short description instead of keywords, etc. Table 4 shows whether the conditions are fulfilled in the example illustrated by Figures 2 and 3. Namely, the condition n is not satisfied ('AI Clan Meeting' = 'AI Meeting', 'Graph Index' = 'GRIX'). Condition ns is satisfied only for needle representing an entity of type 'Event' ('AI Clan Meeting' can be found in the extracted information). There are three keywords out of the six assigned to the needle representing the entity of type 'Event' which match the keywords of an extracted entity, hence k0.5 is, and k0.6 , k0.7 are not satisfied (there is an entity within the extracted information with 50% of keywords being the same as the keywords of the needle). In the case of the second needle, there are four such keywords, therefore k0.5 and k0.6 are satisfied. Finally, both needles are identified within the extracted information by an LLM.

̸

Table 5 shows scores (ratios of successfully extracted entities) based on the above criteria in the case of our business documents. The types of inserted needles are 'BioChemEntity', 'Country', 'Event', 'Insight', 'Legislation', 'Person', 'Product', 'Project' and 'Substance'. Matching the needle and entity name usually does not perform well if the name is prone to modification (e.g., person name with and without title), or if the entity is easy to be misclassified (an entity of type 'Country' was often extracted as 'Place' whose name did not match the country name). Searching for a needle name in all extracted information gives very accurate results if the entities are well characterized by their name (compare for example types 'Person' and 'Legislation' with type 'Insight' where the name is not a natural attribute). Matching the needle and entity keywords depends on the threshold parameter - with a lower proportion of keywords that have to match the score value increases and the reliability of the entity identification decreases. An LLM performs well the entity identification and it is an important criterion in the case of more creative types such as 'Insight'. Finally, the MINEA score for each type is taken as the maximum of the scores (the values are highlighted).

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

5.2.4 Model comparison

MINEA score can be used to compare the performance of distinct LLMs, see Table 6. A corpus of documents is infused by needles representing entities whose types match the schema introduced in Section 5.2.2. Three OpenAI LLMs 4 are used to extract a relevant information under the same setting (the same model parameters such as temperature, the same number of iterations, the same prompting, etc.). Model gpt-3.5-turbo is outperformed by gpt-4-turbo by almost 15% and gpt-4-turbo is outperformed by gpt-4o model by another 12%. Note that the achieved accuracy is lower than presented in Table 3, since only one iteration instead of three was performed in order to reduce the computational time.

Conclusions

In this paper, we focused on quality evaluation of information extraction (IE) performed by large language models (LLMs). First, we delved into the technical limitations of LLMs complicating the extraction of information from a long context. To extract reasonable information from data it is needed to take into the account features such as context window limits, iterated extractions, extraction history recording and Lost in the middle phenomenon. Once the extraction is performed, assessing its quality is essential. However in many customized tasks, a truly objective method is missing, because of the lack of labeled data fitting the scope of the application. The versatile method presented in this paper overcomes the issue by adjustment of the data by insertion of an artificial information, a needle, into it. The artificial information created to this purpose is application and data-specific, but the method itself is applicable generally across the field of IE. By controlling the generation process of the needles, we created a synthetic ground truth that enables us to absolutely measure the extraction quality even when no labeled data is available. We introduced a MINEA score to measure the quality of extraction. The key part is a decision rule on whether a needle was successfully extracted or not. MINEA possibly combines several decision rules into one final score. Our empirical analysis of the MINEA score on a specialized dataset demonstrated its utility for evaluation of LLM-based IE tasks when ground truth is unavailable.

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

References

Kiran Adnan and Rehan Akbar. Limitations of information extraction methods and techniques for heterogeneous unstructured big data. International Journal of Engineering Business Management , 11:1847979019890771, 2019.

Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , pages 65-72, 2005.

Matthew Edgar. Schema and structured data markup. In Tech SEO Guide: A Reference Guide for Developers and Marketers Involved in Technical SEO , pages 67-78. Springer, 2023.

Abdullah Al Foysal and Ronald Böck. Who Needs External References?-Text Summarization Evaluation Using Original Documents. AI , 4(4):970-995, 2023.

Neil Jethani, Simon Jones, Nicholas Genes, Vincent J Major, Ian S Jaffe, Anthony B Cardillo, Noah Heilenbach, Nadia Fazal Ali, Luke J Bonanni, Andrew J Clayburn, et al. Evaluating ChatGPT in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores. 2023.

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss. arXiv preprint arXiv:2402.10790v2 , 2024.

Dong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen W White, and Sujay Kumar Jauhar. Making large language models better data creators. arXiv preprint arXiv:2310.20111 , 2023.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 , 2023.

Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. Large language models for generative information extraction: A survey. arXiv preprint arXiv:2312.17617 , 2023.

Appendix A

To measure the quality of the summary we adopt the methods from [4]: semantic similarity combines latent semantic similarity and cosine similarity; relevance is measured using METEOR score, see [2], without chunk penalty; redundancy avoidance compares extracted entities among themselves using a threshold parameter - entities with a higher cosine similarity are assumed to be redundant; redundancy avoidance can be focused on a single particular property of entities (we use 'name' as this pivotal property).

We modify the bias avoidance score from [4] to be J ∗ ( A,B ) = | A ∩ B | | B | , where A represents the entities in the original text document and we normalize by a number of entities that were extracted, | B | . The score controls how much information in the structured file is not present in the original text, i.e., a potential hallucination of an LLM.

We add two new scores: the relevance spread is the standard deviation of relevance over the text pieces to which the document is split and normalized by the mean value, its higher values indicate that the extraction from distinct text pieces is unbalanced; the incompleteness score just measures the proportion of entities with incomplete information (at least one property value missing or unfilled), e.g., the entity 'AI Enthusiast' in Figure 1 has an unknown 'birthDate'.

Appendix B

Except for the IE task, LLMs are used in several subtasks within the paper, namely to determine schema types appearing in the document, to create a suitable needles fitting contextually to the

336

337

338

339

340

341

342

343

344

345

document and to identify whether a needle was extracted or not. In the following, we provide the reader with prompts and examples of these subtasks.

B1 Discovering a schema

Figure 4 shows a prompt to obtain the Schema.org types from the attached text - Wikipedia article about IE. 5 An LLM is asked to assign relevance to the types to distinguish the most important ones.

Figure 5 shows the entity types that were deduced from the text, together with their relevance and reasoning for why they were chosen. The most relevant types are those directly mentioned - 'Article', as the webpage content itself is represented as an article, 'SoftwareApplication', and 'WebSite' (all with maximal relevance). The least relevant identified types are generic - 'Thing', as a parent type of many directly mentioned types, and 'LearningResource', as a categorization of the article style.

SYSTEM

Youareskilledproofreader.Analyzetheattachedtextandentitiesit

containscarefullyand determine allschema.orgtypesbeingpresent.

Output just a list of allTYPESfound.Providerating of relevance(1-10)

foreachidentifiedtypeandurltowebsitewherethetypeisdefined.

*Attachedtext:

https://en.wikipedia.org/wiki/lnformation_extraction

ASSISTANT

Based onthe attached text,thefollowing schema.org types canbeidentified:

1. **Article**:This type is clearly represented through the mention of "Information extraction"as an article within the content.Relevance:10-[Article](https://schema.org/Article)

2.**EducationalOrganization**(duetoreferencestouniversitiesandresearchinstitutes):Giventheacademic

citations andreferences,organizationsinvolved ineducation andresearchareimplicitlypresent.Relevance:8-

[EducationalOrganization](https://schema.org/EducationalOrganization)

3.**Person**(authorsof thereferenced works):Thereferencessectionmentions multiple authors,indicating their presence ina structured form.Relevance:9-[Person](https://schema.org/Person)

4.**Event**(MessageUnderstanding Conferences):The text mentions specific conferences, which are events. Relevance:8-[Event](https://schema.org/Event)

5.**SoftwareApplication**(GATE,Apache OpenNLP,DBpedia Spotlight,etc.):The section on free or open-source softwareindicatestheseentities.Relevance:10-[SoftwareApplication](https://schema.org/SoftwareApplication) 6.**CreativeWork**:Giventhereferencestopublishedpapers,books,and academicoutputs,this typeis represented.Relevance:9-[CreativeWork](https://schema.org/CreativeWork)

Wikipedia andreferencestootherwebservices.Relevance:10-[WebSite](https://schema.org/WebSite)

8.**Organization**(inreferencestoDARPA,Reuters,etc.):Various organizations arementioned throughout thetext. Relevance:8-[Organization](https://schema.org/Organization)

9. **Thing**(as a generic type for topics covered, e.g.,"natural language processing", "information extraction"): Given (https://schema.org/Thing)

10.**LearningResource**(implied bytheeducational and instructional natureof thetext):Considering thetext's aim toinform and educate about Information Extraction.Relevance:7-[LearningResource] (https://schema.org/LearningResource)

11.**DataFeed**(considering thereferencetonews datafeedsforIEapplications):Specificallymentioned inthe

contextof informationextractionfrom newsfeeds.Relevance:8-[DataFeed](https://schema.org/DataFeed)

12.**ProgrammingLanguage**(whenreferring to usage of Java,Python,etc.in software and toolkits):Implicit

through mentioning specific programming languages used in the context of NLP and IE.Relevance:9-

[ProgrammingLanguage](https://schema.org/ProgrammingLanguage)

346

347

348

349

350

B2 Creating needles

A needle, i.e., a text paragraph fitting thematically to the document, but being new and unique to it, is generated by an LLM using the prompt in Figure 6. The prompt specifies the type of entity that the needle should represent. Multiple needles of the same type can be obtained easily within a single LLM call.

Figure 7 shows ten needles representing the entities of type 'Person' generated based on a Wikipedia 351 article about IE. In the next step properties such as a name, description and keywords can be generated 352 by an LLM. 353

SYSTEM

You havebeencreated totest attentiveness and mindfulness ofproof

readers.

Yourtaskisto suggestapieceof informationtobeinserted to a

document theywill carefullyexamine.Theinformationneedstobe

completelyNEWandUNIQUE,notappearinginthedocument,butFIT

toit THEMATICALLY.Aproof readerwill pass thetestonly,if he

manageto discoverthe newly added text.

particular entity type. The type you are asked to create a piece of text

about is'Person'.Stickonlytothistype!!You will beseverely

penalized for each created entity classified to a wrong schema.org

type.

Suggest10selfsufficientinformationpiecestobeinserted.Use

DIFFERENTformatandstyleforeachpieceinordertoconfusethe

proof readers.Avoid patterns! E.g., placing the name of the entity

always at the beginning is a typical pattern behavior. DO NOT

HIGHLIGHT the name of the entity.

*Type: Person

*Output:texts to insert

*Document to insert to:

https://en.wikipedia.org/wiki/Information_extraction

354

.

ASSISTANT 1.ArnoldJenningsisarenownedexpertinthefield of InformationExtraction andhaspublishedseveral groundbreakingresearchpapers onthe topic inprestigiousjournals.

2.In1995,ProfessorElenaMartinezpioneered a newapproachtoNamed EntityRecognition,revolutionizing theway entities areidentified innatural languageprocessing.

upcomingInternational ConferenceonNLPandMachineLearning.

4.VanessaRodriguez,arising starintheworld of InformationExtraction,recently developed a cutting-edge algorithmthatsignificantlyenhancestheaccuracyofeventextractionfromtexts.

5.ThecollaborationbetweenDr.Michael NguyenandDr.SophiaLeeresulted inthe creationof apowerful toolfor RelationshipExtraction,whichhasbeenwidely acclaimed intheNLPcommunity.

6.ProfessorSamuelBrownreceived theprestigiousExcellenceinInformationExtractionAwardforhis groundbreakingresearchonCoreferenceResolutionincomplextextual data.

7.Inherlatestresearchproject,Dr.LilyMartinezsuccessfullyappliedInformationExtractiontechniquestoanalyze andextractvaluableinsightsfromalargedatasetofsocialmediaposts.

8.The innovativeworkof Dr.OliverWalkerinTemplateFillinghaspaved the wayformore efficient extraction of structuredinformationfromunstructureddocuments.

9.ChristinaLee,aleadingresearcherinthefieldofNamedEntityRecognition,hasdevelopedastate-of-the-art systemthatcanaccuratelydetectandclassifyvarioustypesofentitiesintextualdata.

10.ProfessorNathanTurner'scontributionstoSemi-structured InformationExtractionhavebeeninstrumental in advancingthecapabilitiesofautomatedsystemsinprocessingandinterpretingdiversetypesofdatasources.

B3 Identifying needles 355

The quality of extraction is evaluated based on the proportion of successfully extracted needles. An 356 LLM can be used to decide whether the needle was extracted or not using the prompt presented in 357 Figure 8. 358

SYSTEM

You are an expert in named entity recognition (NER).

Givenalistofentitiesyour goal istodecideforeachof them whether

inthe attached structured list is anelementdirectlymatchingtoit (the

elementand theentityhavetobeonetoonecorrespondence).Each

entityhasatype,name,description andkeywords.Takecare about

these attributes when making the decision.

For EACH entityin the list you are supposed state a simpleYES or NO

andinthecaseofpositiveansweroutputthematchingelement,inthe

caseofnegative answeroutputbymeaningtheclosestelement.

Rule:numberofdecisions=numberofentities

You will be strictly penalized for doing decisions upon entities which

arenotmentioned!!!

List ofentities:{list_ents}

Attached structured list:{extracted}

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the development of an automatic framework to assess the quality of information extraction (IE), which is the main contribution of the paper. This is supported by the introduction of the MINEA score and the discussion on handling input/output size limitations of large language models (LLMs).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations related to the complexity or vagueness of the needles, dependence on the chosen schema and criteria for needle identification (Section 5). Further the paper focuses on limitations of LMMs in IE tasks such as input/output size constraints, lost in the middle phenomenon, bias and hallucinations (Section 4).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results that require formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides detailed descriptions of the experimental setup, including the use of LLMs for IE and the creation of synthetic ground truth data. This is detailed in Sections 3 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The paper does not provide open access to the data and code due to the proprietary nature of the business documents used in the experiments. However, it provides detailed instructions on how to replicate the methodology.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies the use of LLMs, the schema used for structuring data, and the process of generating needles for evaluation. These details are provided in Sections 3, 4 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not include experiments that require statistical significance testing or error bars. The experiments in Sections 4 and 5 present mean values of reasonably large samples. The experiments are not repeated, each of them is carried once on a set of

distinct documents containing a large amount of entities. In Section 5, a vast set of unique needles (with repeating types) is used to infuse the documents.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The paper does not provide detailed information on the compute resources used for the experiments. The requirements such as time of execution are determined especially by used LLMs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics, ensuring that the methods and data used do not violate ethical guidelines. The proprietary data used is handled with confidentiality and integrity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper is primarily concerned with the technical methodology, the introduction of the MINEA score, and the empirical analysis of the framework's performance. The potential positive impacts are mentioned in Introduction: by automating the quality assessment of information extraction, the framework could reduce the need for manual review by experts, saving time and resources and thus enhance the efficiency and accuracy of information extraction from large volumes of unstructured data. The negative aspects of using LLMs for IE tasks such as inherited bias and potential hallucinations are mentioned especially in Sections 4.2 (Lost in the middle problem) and 5.1 (bias avoidance score).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release any data or models that pose a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing models are properly referenced and credit to their creators is given. These are either LLMs or metrics such as SUSWIR and METEOR (Section 5 and Appendix A).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets that require documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects that would require IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper studies the evaluation of information extraction, particularly LLM-based IE, in scenarios where human-annotated data is unavailable.\n* The proposed evaluation framework relies on the `Needle in a haystack` evaluation. That is, an LLM is first used to generate a piece of information (needle) given the original text; then, the needle is infused into the document, and the quality of IE is assessed by whether the needle can be successfully extracted. \n* In addition to this evaluation framework, the authors also discussed several aspects to be considered when using LLM-based IE for processing long documents.\n\nSTRENGTHS:\nAn interesting application of `Needle in a haystack` evaluation in information extraction.\n\nWEAKNESSES:\n* The writing quality is not great, and several areas require further clarification\n\t* The current paper structure is confusing; not sure what role Sections 3 and 4 play in this paper, e.g., whether the authors were proposing a new LLM-based IE approach\n\t* I suggest providing a formal definition of IE studied in this paper because it is very confusing to know what information is extracted. For example, in the abstract, `entity and its properties` is mentioned; in Section 3, `short paragraphs of text` seem to be the information extracted `from the continuous text`; also see Q2 \n* The main contribution of the paper is an automatic framework to assess the quality of the IE; however, the authors didn't conduct any experiments to demonstrate the effectiveness of the proposed framework (e.g., whether the evaluation results correlate with human judgments); the other main limitation is the authors evaluate the quality of extraction based on the proportion of successfully extracted needles but totally ignore the correctness of extracted information (precision)\n* The experiments are conducted on private datasets with only several toy examples described in the paper; it will be very difficult for others to reproduce the results. I would suggest conducting experiments at least on some document IE datasets, for example, from news or biomedical domains.\n\nQUESTIONS:\n1. Line 7: Information retrieval is mentioned once but not explained anywhere else; suggest clarifying its meaning in this paper\n2. Figure 1: should the value of these fields (e.g., name, description, keywords) be directly copied from the original text? what do these numbers (9) (8) in the `keywords` field mean? \n3. Table 1: what higher redundancy scores (more duplicated entities) mean? How do these results tell the `Lost in the Middle` phenomenon? \n4. What LLMs do you use to generate the needles and for identifying needles? How does this affect the extraction models to be tested? e.g., is the model more likely to achieve better performance if the needle is created by the same model? does the model achieve better scores if the same model is used for evaluating whether the needle is found (`llm` column in Table 4)?\n5. Table 3: what does `chosen schema` mean? Do you mean the LLM is not instructed to recognize entities belonging to these categories, but they are still recognized?\n6. What evidence (empirical results) can support the claim that 'the combination of both improvements --- text splitting and iterated calls, has proven itself to perform the best (line 146)'?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the quality evaluation of information extraction (IE) performed by large language models (LLMs). It discusses the methods to handle the input/output size limitations of the LLMs and their performance in IE. It also introduces additional scores to evaluate the extraction quality and discusses how to interpret them.\n\nSTRENGTHS:\n1. This paper analyses the technical limitations of LLMs complicating the extraction of information from a long context.\n2. This paper presents to insert a needle into the data to evaluate the performance of IE without labeled data.\n\nWEAKNESSES:\n1. The analysis of the performance of LLMs in IE is not new and has various analysis, such as in the following papers:\n\n> [1] Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness (Li et al., 2023)\n\n> [2] Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors (Han et al., 2023)\n\n> [3] When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks (Peng et al., 2023)\n\nAmong the papers, the authors in [3] also analysed LLMs' limitations in long context understanding, which is similar to the conclusion of this paper. \n\n2. This paper lacks a thorough literature review in LLM for IE as well as new evaluation formats, such as [1, 2, 3] and the following paper:\n\n> [4] Evaluating Generative Language Models in Information Extraction as Subjective Question Correction (Fan et al., LREC-COLING 2024)\n\n3. This paper only focuses on the NER task but lacks the other IE tasks, e.g. relation extraction and event extraction. Additional experiments are required to test the generalisability of the method. The number of samples tested is also limited (see \"# entities used for evaluation\" in Table 3).\n\nQUESTIONS:\nSee \"Weaknesses\".\n\nLIMITATIONS:\nSee \"Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework to capture information extraction quality in the absence of humanly labelled and curated datasets. It explains how an approach on how to include the schema, and the role and limitations of LLM's (specifically gpt-4-1106-preview).\n\nExperiments are done (I guess), by \"extracting information\" from long business documents originating from the healthcare sector. Several scores are presented according to the SUSWIR metrics. It delves into the \"lost in the middle\" phenomenon. It introduces the MINEA score, a newly proposed metric.\n\nSTRENGTHS:\nIt tries to address a relevant problem in the field (curated benchmark data is hard to come by).\n\nWEAKNESSES:\n- The paper is from the start extremely vague and misses concrete statements and explanations about the work done. The contributions are unclear, the data is essentially undefined, for most of the work what exactly is being done is simply unclear.\n\t\n- Even the task of \"Information Extraction\" is not concretely described in a way that is reproducible.\n\t\n- Line 7-8: \"The framework focuses on information extraction in the form of entity and its \n\tproperties\". \n\t\n- Table 1: it is completely lost upon me what is being presented here.\n\n-  \"We extract information from several long documents from our business case\". What are these documents? What are they originating from?\n\n-  The scores mentioned are \"redundancy\". How is this measured? What do these scores represent? Is lower or higher better? Even these basic questions are not answered. All of this in the appendix (where it shouldnt be), and the further tables are not better.\n\n- The work is very dry. There are no figures that explain or examplify what the problem is, or how this framework is supposed to fit.\n\t\n- The related work section is short and doesn't address the original point (evaluation in absense of benchmark data).\n\n- It is unclear to me how this work should contribute in any form to evaluation in the absence of benchmark data.\n\t\n- The introduced MINEA score is \"explained\", but not examplified or mathematically defined.\n\t\n- All examples are screenshots of data in JSON format rather than helpful explanations.\n\nQUESTIONS:\n- What do you see as \"Information Extraction\" in this work?\n\n- What are the concrete contributions of this work\n\t\n- What is MINEA? And how can it be helpful towards assessing IE?\n\nLIMITATIONS:\nNo. The paper does not concretely address the limitations of this metric. There are no good, bad examples provided.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper proposes an automated framework for evaluating the quality of IE tasks using LLMs. The framework introduces a scoring method called MINEA, which creates evaluation criteria by injecting artificial data ("needles") into documents. The paper also discusses how to deal with the limitations of LLMs when processing large amounts of data, and introduces an iterative extraction process to improve the completeness of the extraction and reduce repetition.

STRENGTHS:
s1. The introduction of the MINEA score is somewhat innovative.

s2. The paper is clear explanations of the proposed framework.

WEAKNESSES:
w1. Lack of Originality: The originality of the paper is insufficient. Related work has already mentioned using the "needle" method to evaluate the information extraction capabilities of LLMs. While this paper adds the use of large models to help create the needles, the contribution is still lacking.

w2. Insufficient Experimental Description: The description of the experimental setup is missing, including the experimental environment, data sources, and dataset sizes. However, the paper spends too much space on toy examples.

w3. Unreliable Conclusions on Length Limitations: For the experiments on the input and output length limitations of models, the paper only tested one model, making the conclusions unreliable.

QUESTIONS:
q1. Please clarify the advantages of this method as the paper does not explain them clearly.

q2. Please provide more details on the specific iterative process and its implementation.

q3. Could you elaborate on the experimental setup, including whether the experiments were conducted multiple times and the reliability of the results?

LIMITATIONS:
L1. The paper should provide a comparison to existing work to highlight the improvements.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12