Challenges in End-to-End Policy Extraction from Climate Action Plans (2024)

Abstract: Gray policy literature such as climate action plans (CAPs) provide an information-rich resource with potential to inform analysis and decision-making. However, these corpora are currently underutilized due to the substantial manual effort and expertise required to sift through long and detailed documents. Automatically structuring relevant information using information extraction (IE) would be useful for assisting policy scientists in synthesizing vast gray policy corpora to identify relevant entities, concepts and themes. LLMs have demonstrated strong performance on IE tasks in the few-shot setting, but it is unclear whether these gains transfer to gray policy literature which differs significantly to traditional benchmark datasets in several aspects, such as format of information content, length of documents, and inconsistency of document structure. We perform a case study on end-to-end IE with California CAPs, inspecting the performance of state-of-the-art tools for: (1) extracting content from CAPs into structured markup segments; (2) few-shot IE with LLMs; and (3) the utility of extracted entities for downstream analyses. We identify challenges at several points of the end-to-end IE pipeline for CAPs, and we provide recommendations for open problems centered around representing rich non-textual elements, document structure, flexible annotation schemes, and global information. Tackling these challenges would make it possible to realize the potential of LLMs for IE with gray policy literature.

heading: 1 Introduction
text: Gray policy literature — non-commercial and nonacademic documents which can include white papers, technical reports, and working papers — is an information-rich resource that is generally difficult to navigate due to the volume and diversity of format (Pandita and Singh, 2011; Lawrence et al., 2015; Turner et al., 2005). Paid for by public funds,
these documents are usually freely available and often the most timely resource on policy issues (Rothstein and Hopewell, 2009). Lawrence et al. (2015) found that half of surveyed policymakers would be more likely to use gray policy literature if information were easier to find and access.
Information extraction (IE) tasks in the NLP space are designed to make it possible to efficiently sift through such information, but gray policy literature poses several challenges for traditional IE. They are distributed as long PDFs with inconsistent document structure, such that relevant sections cannot be easily automatically extracted. They are designed to be visually appealing with crucial information organized in rich non-textual elements such as tables and graphics (Turner et al., 2005). In contrast, the standard datasets that IE is designed to perform well on take the form of short, plain text documents from domains such as webtext or news articles (Riedel et al., 2010; Roth and Yih, 2004).
In general, IE models that are trained on these standard datasets can be adapted to new domains by finetuning with annotated examples, but this may not be feasible for gray policy documents (Gururangan et al., 2020). Collecting a large number of manually annotated examples for a static set of entities can be prohibitively expensive with gray policy literature due to fast-paced and diverse developments in the field. Over standard datasets, large language models (LLMs) have demonstrated strong performance in entity and relation extraction in the few-shot setting (Yuan et al., 2022; Wan et al., 2023; Wadhwa et al., 2023).
But, it is unclear to what extent LLMs can be used to extract information from gray policy literature in the few-shot setting. In this work, we present a case study of few-shot IE with LLMs over climate policy text. We specifically consider climate action plans (CAPs) from the state of California. CAPs are distributed as long PDFs, sharing many of the challenging properties of gray policy
156
literature including information-dense non-textual elements and inconsistent structure. If possible, extracting rich, structured representations of policies with minimal annotation effort would be useful for many applications such as auditing emissions reductions, searching for relevant adaptation strategies for a specific climate hazards, or aggregating local government actions to the state or federal level.
We evaluate few-shot IE performance of LLMs for an end-to-end setup, where the input is a CAP in PDF form, and the output is a set of policy entities and relations (Figure 1). We inspect to what extent existing PDF parsers are able to preserve crucial policy information (raw text recognition), recognize textual and non-textual elements (element recognition), and extract a document structure that would be useful to a domain expert to segment the document (structured segmentation). Then, using the parsed segments of the CAP, we analyze how well entities can be extracted from informationdense non-textual elements (intra-element extraction), and more generally the relation extraction performance of LLMs in a few-shot setting (segmentlevel extraction). Finally, we experiment with modes of useful representations of the extracted entities and relations (extraction utility).
Based on our analysis over the CAPs, we pro-
vide recommendations for future directions in NLP that would improve IE with gray policy literature. We propose: (1) more flexible annotation schemes to account for inconsistencies in how entities are expressed across documents; (2) better representations of non-textual elements in the context; (3) methods that extract vague, imprecise, and subjective entity types pervasive in gray policy literature; and (4) the use of rich document structures.

heading: 2 Climate Policy Extraction
text: CAPs generally contain sections describing proposed policies and an inventory of emissions produced by the jurisdiction. Climate policies can be classified as adaptation to climate changes or emissions reduction measures. An example CAP for the city of Encinitas, California can be found here.
California municipal and county CAPs exhibit significant variability in structure but typically contain a similar set of sections. These include front matter, a discussion of the regulatory framework, a description of local and regional climate projections, and, critically for our purposes, chapters on specific and detailed mitigation policies, associated emissions reductions, and sometimes also chapters on implementation and on adaptation policies. Climate policies can be broadly categorized as either mitigation policies which reduce greenhouse gas
emissions or adaptation policies which build community resilience to the impacts of climate change. The average length of a CAP is approximately 62k tokens.1 CAPs are distributed by jurisdictions as PDFs.
There are at least three user groups could derive significant value from extracting policies from CAPs. Local sustainability officers seek to understand the policies of similar jurisdictions and how their own policies compare. State agencies wish to monitor the progress of CAP implementation and to aggregate policy commitments to the state and federal levels. Academic researchers seek to understand CAP characteristics, credibility, evolution over time, and effectiveness in shaping policy. To date, efforts to systematically extract policies have been expensive and have progressed sporadically (Berrang-Ford et al., 2021; Goonesekera and Olazabal, 2022)

heading: 3 Task Definition
text: Given a CAP containing policy information, we want to perform relation extraction, where the entities are policy names p1, . . . , pj and a closed set of policy attributes a1, . . . , ai. We consider a single relation type of reference between a policy pk and an attribute, indicating that the attribute describes the policy pk. Each attribute can refer to at most one policy. For each policy pk, we expect that the referential attributes a1, . . . , ai construct a sufficiently informative representation to perform some downstream analysis.

heading: 4 Method
text: Given a set of CAPs in the form of PDF documents, we first parse the PDFs into a structured markup format (e.g. HTML, Markdown, JSON) using a PDF parser. The resulting file would contain various elements such as headers, paragraphs, tables, and lists. As a part of the raw file content, the parser produces some document structure to mark sections and sub-sections or in some cases the table of contents. We can formalize the document structure as a graph of section headers and segments containing the section content, where headers are linked to corresponding segments, and headers can be subsumed by other headers to reflect the hierarchy induced by the PDF parser.
Since the average CAP in our dataset is 170 pages, it is necessary to divide CAPs into coherent,
1As tokenized by the GPT-3.5-turbo-0125 model.
topically focused segments. Given a fixed level in the document structure hierarchy, the CAP can be segmented according to the structure. For example, a CAP can be broken down into a series of segments, each corresponding to the content of a sub-section. This results in a corpus of structured markup segments to perform relation extraction over.
We perform zero and few-shot relation extraction by prompting autoregressive LLMs. We draw in-context examples from the set of annotated segments. We use a two-step entity-extraction procedure to extract relations, where the second prompt is dependent on the model output of the first. For each segment, we first prompt the LLM to produce the set of policy names that appear in the segment. Then, for each policy name pk we prompt the LLM to produce the attributes a1, . . . al referring to the policy name.
The relation extraction task is linearized as is conventional with IE using LLMs (Paolini et al., 2021). Extracted entities are expressed as a JSON mapping between entity types and mention spans. Spans that do not appear in the segment are discarded to reduce the effect of model hallucination.

heading: 5 Related Work
text: In contrast to the understudied climate policy domain, there have been extensive studies about the performance of PDF parsing tools for documents with complex layouts over scientific domains (Ramakrishnan et al., 2012; Bast and Korzen, 2017; Meuschke et al., 2023). PDF parsers generally perform well on scientific text (Bast and Korzen, 2017), but isolating performance on non-textual elements Meuschke et al. (2023) show that PDF parsers struggle to extract tables more than all other content elements. (Deng et al., 2024) has found that processing tables as images using multi-modal LLMs can be more effective than parsing the tables into text.
For few-shot IE using LLMs, there have been mixed results over biomedical and clinical text domains (Hu et al., 2023; Li and Zhang, 2023; Jimenez Gutierrez et al., 2022; Li and Groth, 2023). There has been some limited work to perform fewshot IE in the climate policy domain: Buster et al. (2024) extract features of wind energy systems from PDF ordinances using LLMs. In our work, we systematically evaluate the end-to-end IE pipeline at both the extraction and PDF parsing stage for a
more general climate policy taxonomy. A key limitation in using LLMs for IE over long, complex documents is limited context length. While there has been work to build models that can process long contexts (Beltagy et al., 2020), memory and attention constraints result in parts of the context being ignored. With scientific documents, Dagdelen et al. (2024) found that failures occurred when the number of tokens exceeded the model context window. Accordingly, we focus on making use of CAP structure to produce segments that fit within the context window of LLMs.
Historically, NLP tools have been used for climate policy text to identify salient topics using clustering or topic modeling (Brinkley and Stahmer, 2021). There has been some work to classify policy type or targets (e.g. pledge net-zero vs. emissions reduction) (Sachdeva et al., 2022; Biesbroek et al., 2020; Juhasz et al., 2024).
More recently, there have been multiple largescale initiatives to extract structured representations of policies. Sewerin et al. (2023) spent over 600 hours to annotate 412 documents with 42 policy instrument and design types. Similarly, BerrangFord et al. (2021) have annotated climate hazards and adaptation efforts in 1,682 articles with the assistance of 126 researchers. Accordingly, in our work, we study the capacity of LLMs to assist and reduce the effort required to collect and maintain information resources about the state of climate policy.

heading: 6 Experiments and Analysis
text: To perform an evaluation of state-of-the-art endto-end IE, we annotate the documents at multiple levels of granularity: raw PDFs, structured markup CAPs, filtered CAP segments, and elements of each segment.

heading: 6.1 Dataset
text: We collect a dataset of 227 publicly available CAPs scraped from California city and county government websites published between 2006 and 2022 (Boswell and Greve, 2023). This dataset is used for each of the annotation tasks.
To verify that core policy information can be retained in PDF parsing, we annotated descriptions of climate policies at up to five levels of granularity for 17 raw PDF CAPs from San Diego County (16 municipal and one county CAP). On average, the most concise descriptions of a policy were on 7.5
words (e.g. "Promote Installation of Commercial and Industrial Photovoltaic Systems"), and the most granular descriptions were 48.9 words (e.g. "Implement and enforce Title 18, Chapter 18.30, Section 18.30.130 of the Carlsbad Municipal Code, mandating solar photovoltaic energy generation systems on existing non-residential buildings undergoing major renovations."). We collected 1,183 policy entities.
To evaluate relation extraction performance, we annotate richer representations of policies with policy mentions and corresponding attribute mentions over a sample of parsed, structured markup segmented CAPs. Over 65 segments, we marked 102 climate policies and 838 attributes, with an average segment length of 401 words. Based on existing climate policy taxonomies (Boswell et al., 2019), we developed a minimal closed set of 11 policy attributes. Frequent subjective and ambiguous cases resulted in inter-annotator agreement Fleiss’ Kappa 0.39 and Krippendorf’s Alpha 0.41.
Segments were annotated by six in-house undergraduate annotators with backgrounds in public policy and computer science. All annotators used INCEpTION (Klie et al., 2018).
In addition to segments, we inspect the utility of document structure produced by the PDF parsers. For each parser, we extract a hierarchical list of sections and sub-sections. This is either explicitly generated by the parser, or induced by the header tags produced in the parsed output. Given only the ordered document structure induced by a PDF parser, we prompted an annotator to mark a subset of section headers that suggest the section likely contains policy information. In this annotation task, we determine the extent to which induced structure can be used to narrow the space of candidate segments. For the 17 San Diego CAPs, we annotated the structure produced by four PDF parsers: Nougat, Marker, GROBID, and Adobe Extract.

heading: 6.2 Extracting Document Content and Structure
text: We experiment with common parsers to extract and structure text from PDF documents:
Nougat (Blecher et al., 2023) does not rely on an external OCR engine. Instead, it uses a visual encoder, an mBART decoder, and a tokenizer specialized in scientific text. The parsed output is in a markup language that supports headers, which are used for segmen-
tation, and LaTeX tables.
Marker is a widely-used pipeline of deep-learning models including a Tesseract OCR engine to extract text, detect page layout, and convert to markdown.2 Marker supports hierarchical headers in the parsed output, which we use for segmentation.
GROBID (Lopez, 2009) structures PDFs into an XML/TEI encoded document using maximum chain Markov models and linear-chain CRF. GROBID also extracts the table of contents, which we use for segmentation.
Adobe Extract API uses Adobe Sensei ML to extract paragraphs, lists, headings, tables.3 We convert the output to HTML format. We are able to extract the table of contents using Adobe Extract, and segment the documents according to varying levels of depth of the table of contents.
To evaluate raw text and element recognition, we use Levenshtein distance (Levenshtein et al., 1966), or the number of character insertions, deletions substitutions necessary to transform a contiguous span from an extracted segment into a reference piece of text. We measure the recall of a set of parsed segments using a threshold of 10 edits. Among matching extracted segments, we normalize distance over the length of the reference text.
Raw Text Recognition: In Table 1, we inspect to what extent the PDF parsers preserve the policy descriptions and uncover an underlying document structure that would make it feasible to extract the
2https://github.com/VikParuchuri/marker 3Adobe PDF Extract API Technical Brief
policy segments under annotation resource constraints.
First, we find that for most PDF parsers, core policy information is retained after parsing the PDF into structured markup formats. We estimate how much of the core policy information is dropped or heavily distorted in the parsing process by comparing the annotated policy descriptions and the CAP segments using the fuzzysearch library4.
Element-wise Recognition: We also evaluate the PDF parsers for element recognition. Critical elements in CAPs include tables, paragraphs, lists, and headers. For a random sample of 10 CAPs, we identify a policy-rich segment and annotate critical elements in the segment from the raw PDF. We can measure recall of these elements in the structured markup form of the CAP.
In Table 2, we observe that table and list elements are typically more challenging to exract in CAPs than purely textual elements like headers and paragraphs. In Table 1, we observe that most policy text segments can be matched in the PDF parser output to with the exception of GROBID. Almost all PDF parsers struggle to recognize tables in CAPs. In contrast with scientific articles, Blecher et al. (2023) report table recall 50 points higher on open-access ArXiv articles than CAPs, Poor parsing performance on tables is an important bottleneck for policy extraction over CAPs, since tables are often the most information-dense elements of the document.

heading: 6.3 Segment Filtering
text: The PDF parsers convert the documents to a structured markup format, which includes ordered lists of section headers and segments containing the
4https://github.com/taleinat/fuzzysearch
section content. In practice, inference can be expensive over potentially hundreds of sections in the document. Additionally, inference over less relevant sections such as front matter, policy landscape, and climate history and projection sections may yield false positives in the policy extraction task as these sections often contain boilerplate discussions of policies not specific to the relevant jurisdiction (Scott et al., 2022). Given only the high-level document structure produced by the PDF parser (i.e., section headers, table of contents), we filter the subset of policy-rich segments. After annotating 25 document structures, we pass BERT-based representations of the section headers through a 2-layer feed-forward neural network with a binary classification head indicating policy informativeness. We perform this structured segmentation using the parsed document structure to identify a subset of segments that contain policy information.
Structured Segmentation: We measure the utility of the structure that the parser extracts. In Table 1, we find that after annotating a sufficient substructure, Adobe suffers no loss in policy information, while reducing the amount of segments to process by 58%. The structure produced by Nougat and Marker is generally longer, often containing extraneous elements, since the structure elements include any header in the documents. This may result in annotator mistakes and consequently a small loss in policy information.

heading: 6.4 Relation Extraction
text: Given a corpus of segments from the CAPs, we perform inference in zero and few-shot settings. To select in-context examples we use the k nearest neighbors from the target example based on cosine distances of Sentence-BERT representations (Reimers and Gurevych, 2019) to the target context. Using 10-fold cross-validation over the set of
segments annotated for both policies and attributes, all results are computed over 6 random seeds.
At inference time, generated outputs are parsed as a JSON object. We assess GPT-3.5-turbo-0125 using the function-calling feature of the API to constrain the output to a JSON format. We use a function-calling finetuned 7B Llama2 model as well.5 This results in a set of policies, where each policy name is linked to a set of attributes. We evaluate these widely-used models to measure both segment-level extraction and intra-element extraction performance. In post-processing, predicted spans that do not appear in the context are dropped. We compute standard metrics (P, R, F1) for linearized, typed relation and entity tuples. We use a relaxed string matching setup between reference and generated spans similar to previous work with generative models that do not produce standardized outputs (Wadhwa et al., 2023).
Segment-level Extraction: We can observe in Table 3 that when gold policy names are given, the model performance is strong for both Llama2
5https://huggingface.co/Trelis/Llama-2-7b-chat-hffunction-calling-v2
and GPT-3.5-turbo. Without gold policy names, the performance is extremely poor, especially for RE. Including multiple in-context examples in the prompt is necessary for reasonable performance. Upon manual inspection of model output, we observe that this is partially a result of the ambiguous nature of annotation. For example, a policy may be broadly previewed in an introduction section with abstract terms and concretely enumerated as a list of measures in the appendix of the CAP. With global context and knowledge about how CAPs are typically structured, an annotator can correctly ignore abstract mentions of policies. The model, however, is limited to only a single segment.
The model performance does not seem to improve with higher quality examples. We experiment with varying the size of the exemplar pool from which we select in-context examples in Figure 2. We find that for both models, annotating more than 10 exemplars does not seem to improve overall F1 performance significantly. In general, higher quality exemplars improves precision, but has little effect on model recall. One explanation for this is that the ICL examples have erratic levels of policy abstraction, so that similar contexts are not necessarily more useful for policy recall.
Intra-element Extraction: We inspect performance of models over policy-rich document elements such as tables and lists. We annotate an additional 20 segments that contain table and list elements and report zero-shot performance in Table 4 for policy name extraction. For all table and text formats, policy name extraction performance is poorer over segments containing non-textual elements than segments that contain only paragraph elements. Upon manual analysis of the model output, we observe that models can easily identify policies from well-formed tables (i.e. there are no breaks in columns, cells are merged consistently).
We can observe instances of hallucination in Table 5. In the first example, the model hallucinates a policy called “Severe Storm Preparedness Measures” intended to target the climate hazard described in the content. While this can be avoided altogether by enforcing that extracted spans occur in the context, models would be most prone to policy hallucination if the segment maps to a section that does not contain policies. In the second example, we can observe that well-behaved tables with contiguous rows and columns can also be interpreted incorrectly. For example, the model incorrectly
policy name: “Peak Demand Reduction” description: [“Collaborate with SCE and encourage businesses to enroll in Energy Efficiency and . . . ”], responsible-entity: [“SCE”, “CVAG”, “Los Angeles County Energy Enterprise Management . . . ”], cost: [“$2000”], sector: [“Commercial Buildings”]
Table 5: Example annotations from GPT-3.5-turbo. Among the entity types and extracted spans, we observe hallucinated policy names and attributes when there are no ground-truth policies in the context (top) or when the model fails to localize row content from tables (bottom).
reports two management entities: “CVAG”, “Los Angeles County Energy Enterprise Management Information System (EEMIS)” that appear in the following row of the table.

heading: 7 Downstream Extraction Utility
text: A question of interest to policy researchers is how CAP focus has shifted from mitigation to adaptation as faith in global mitigation efforts has declined (Hoesung Lee and José Romero (eds.), 2023). We classify a policy as an adaptation policy using a heuristic string match over the referring attributes with the regular expression “heat|precip|flood|fire|sea level”. A keyword search with this regular expression over the entire document could extract mentions of hazards in introductory sections that are not associated with concrete policies.
Using a GPT-3.5-turbo model, we extracted 47,006 climate policies from 227 jurisdictions. 4.6 percent of the extracted policies mention the five hazards in their “climate-hazard” attribute. The low percentage is expected as CAPs have traditionally focused on greenhouse gas emission reductions rather than adaptation.
Linking the extracted set of policies to county and municipal characteristics reveals spatial variability in the fraction of adaptation policies (Figure 3) with a slight indication that coastal jurisdictions are more focused on adaptation. The share of adaptation policies has increased over the past 16 years. There is a significant positive linear relationship between the fraction of adaptation policies
and the logarithm of community per capita income. This may indicate that wealthier communities show more interest in safeguarding their assets, though further investigation is required to rule out potential confounding factors, such as distance to the coast.

heading: 8 Recommendations
text: To improve end-to-end IE over gray policy literature, we propose several directions for future work based on our analysis of California CAPs.
There is a need for more flexible annotation schemes for inconsistently formatted documents. The same entities will frequently appear at different levels of abstraction or detail between documents and within a document. We need mechanisms to specify which mentions to extract. This is distinct from fine-grained entity typing annotation where there are no vertical coreferences between types.
IE systems need better representation of nontextual elements such as tables and rich graphics, and for semantic representations of the text to be built from information derived from all modalities. In a text-only modality, it may be useful to build representations that localize information content according to reading order.
Vague, imprecise, and subjective entity types are crucial for gray policy literature, and current methods to refine the set of extracted entities such as providing ICL examples or detailed instructions may be insufficient. In the case of CAPs, interpretation of policy mentions often require global document information or domain expertise.
IE systems should also be designed to leverage
rich document structure. In the case of CAPs, for example, we observed that that the table of contents in isolation is informative enough for annotators to infer which sections are relevant.

heading: Acknowledgements
text: Thanks to Adrienne Greve and Mike Boswell for valuable feedback on this work. We additionally thank our annotators: McKenna Carlson, Nida Firdaws, Elvin Li, Wushuang Li, Matthew Le, and Calvin Nguyen.

heading: A Extended Related Work
text: In part, a core challenge in entity and relation extraction with generative models is inconsistent output format (Jiang et al., 2024). Previous work has shown that manual annotation of model output can reveal that unannotated spans are a major source of errors, and that soft matching of spans can make evaluation more precise (Wadhwa et al., 2023; Han et al., 2023). Katz et al. (2023) has shown that constraining the output to a JSON format can also improve the consistency of LLM outputs – a finding that we make use of in this work.

heading: B Dataset and Additional Results

heading: Future Work
text: To improve end-to-end IE over gray policy literature, we propose several directions for future work based on our analysis of California CAPs. There is a need for more flexible annotation schemes for inconsistently formatted documents. The same entities will frequently appear at different levels of abstraction or detail between documents and within a document. We need mechanisms to specify which mentions to extract. This is distinct from fine-grained entity typing annotation where there are no vertical coreferences between types. IE systems need better representation of nontextual elements such as tables and rich graphics, and for semantic representations of the text to be built from information derived from all modalities. In a text-only modality, it may be useful to build representations that localize information content according to reading order. Vague, imprecise, and subjective entity types are crucial for gray policy literature, and current methods to refine the set of extracted entities such as providing ICL examples or detailed instructions may be insufficient. In the case of CAPs, interpretation of policy mentions often require global document information or domain expertise. IE systems should also be designed to leverage
rich document structure. In the case of CAPs, for example, we observed that that the table of contents in isolation is informative enough for annotators to infer which sections are relevant