PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming (2024)

Abstract: Relation extraction aims to classify the relationships between two entities into pre-defined categories. While previous research has mainly focused on sentence-level relation extraction, recent studies have expanded the scope to document-level relation extraction. Traditional relation extraction methods heavily rely on human-annotated training data, which is timeconsuming and labor-intensive. To mitigate the need for manual annotation, recent weaklysupervised approaches have been developed for sentence-level relation extraction while limited work has been done on document-level relation extraction. Weakly-supervised documentlevel relation extraction faces significant challenges due to an imbalanced number "no relation" instances and the failure of directly probing pretrained large language models for document relation extraction. To address these challenges, we propose PromptRE, a novel weaklysupervised document-level relation extraction method that combines prompting-based techniques with data programming. Furthermore, PromptRE incorporates the label distribution and entity types as prior knowledge to improve the performance. By leveraging the strengths of both prompting and data programming, PromptRE achieves improved performance in relation classification and effectively handles the "no relation" problem. Experimental results on ReDocRED, a benchmark dataset for document-level relation extraction, demonstrate the superiority of PromptRE over baseline approaches.

text: Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 132–145 August 16, 2024 ©2024 Association for Computational Linguistics
Relation extraction aims to classify the relationships between two entities into pre-defined categories. While previous research has mainly focused on sentence-level relation extraction, recent studies have expanded the scope to document-level relation extraction. Traditional relation extraction methods heavily rely on human-annotated training data, which is timeconsuming and labor-intensive. To mitigate the need for manual annotation, recent weaklysupervised approaches have been developed for sentence-level relation extraction while limited work has been done on document-level relation extraction. Weakly-supervised documentlevel relation extraction faces significant challenges due to an imbalanced number "no relation" instances and the failure of directly probing pretrained large language models for document relation extraction. To address these challenges, we propose PromptRE, a novel weaklysupervised document-level relation extraction method that combines prompting-based techniques with data programming. Furthermore, PromptRE incorporates the label distribution and entity types as prior knowledge to improve the performance. By leveraging the strengths of both prompting and data programming, PromptRE achieves improved performance in relation classification and effectively handles the "no relation" problem. Experimental results on ReDocRED, a benchmark dataset for document-level relation extraction, demonstrate the superiority of PromptRE over baseline approaches.

heading: 1 Introduction
text: Relation extraction is a crucial task in natural language processing that aims to classify the relationships between two entities (e.g., Pacific Fair and Queensland) into pre-defined categories (e.g., located in). It has various downstream applications such as question answering (Veena et al.,
2017) and knowledge graph construction (Distiawan et al., 2019).
While previous research has mainly focused on relation extraction within a single sentence, recent studies have expanded the scope to document-level relation extraction (Yao et al., 2019). Traditional relation extraction methods (Tan et al., 2022a; Ma et al., 2023) heavily rely on human annotation for training data, which is time-consuming and laborintensive. To mitigate the need for manual annotation, recent weakly-supervised approaches (Sainz et al., 2021; Yang and Agrawal, 2023) have been developed for relation extraction with minimal or no manual annotation. For example, Qu et al. (2018) extracted textual patterns from seed examples and used those patterns as weak supervisions for re-
132
lation extraction. Sainz et al. (2021) represented each relation class using a label verbalizer and then solving the relation extraction task by a textual entailment model. Wang et al. (2022a) analyzed an "extremely unlabeled" scenario where each relation type had only one instance, reducing the training set to about five thousand labeled relation triplets. However, these methods were primarily designed and evaluated for sentence-level relation extraction, which limits their generalizability to document-level relation extraction datasets like ReDocRED (Tan et al., 2022b), where the presence of a substantial number of "No Relation" or NA classes poses additional challenges.
To address this limitation, we study the problem of weakly-supervised document-level relation extraction. Recent large language models (LLMs) have achieved great success in a wide range of natural language processing tasks (Brown et al., 2020; Touvron et al., 2023). We investigate the ability of the pretrained large language models on the document-level relation extraction task. We focus on three pretrained large language models: UnifiedQA (Khashabi et al., 2020, 2022), LlaMA, LlaMA2 (Taori et al., 2023; Touvron et al., 2023), and ChatGPT (Ouyang et al., 2022).
UnifiedQA is a T5 model (Raffel et al., 2020) pretrained on four different question-answering settings: extractive, abstractive, multiple-choice, and yes/no questions. UnifiedQA performs comparably to specialized state-of-the-art models on most relation extraction datasets. ChatGPT, developed by OpenAI, is a powerful generative large language model known for its impressive generalization capabilities. However, the closed-source nature of the ChatGPT model limits its accessibility for downstream applications. We utilize the text output of ChatGPT without accessing its internal embedding space or doing any model fine-tuning. LlaMA is a collection of foundation language models ranging from 7B to 65B parameters trained on only publicly available datasets. After fine-tuning on an instruction-following dataset (Taori et al., 2023), LlaMA and LlaMA2 are able to produce reasonable responses to the input instructions. We use LLaMA7B and LLaMA2-7B, which has a good balance between model performance and efficiency.
We propose PromptRE, a novel weaklysupervised document-level relation extraction method that combines prompting-based techniques with data programming (Figure 1). Given a known type-relation distribution, we first investigate var-
ious ways of prompting the pretrained large language models for relation classification. We then investigate different ways to select the most confident outputs using data programming, a technique that combines multiple sources of weak supervision. By leveraging the strengths of both prompting and data programming, we achieve improved performance in relation classification and effectively handle the "no relation" problem. Furthermore, we leverage ChatGPT as a summarizer to extract relevant information about the entities of interest. This plays a crucial role especially when dealing with lengthy documents which contain unnecessary extra information. To the best of our knowledge, we are the first to propose weakly-supervised document-level relation extraction. Our contributions are summarized as follows:
1. We propose the first weakly-supervised method, PromptRE, for the document-level relation extraction task.
2. PromptRE is a novel method that combines various types of prompting outputs with data programming. PromptRE further incorporates the label distribution and entity types as prior knowledge to improve performance.
3. Extensive experiments on the ReDocRED dataset demonstrate the capability of PromptRE over baseline methods. Ablations provide a comprehensive study on weakly-supervised inference ability. Multiple case studies show the incompleteness of existing document relation extraction datasets.

heading: 2 Related Work

heading: 2.1 Document-level Relation Extraction
text: Document-level relation extraction is a crucial task in natural language processing, as more than 40.7% of relations require multiple sentences to extract (Yao et al., 2019). Consider an example of document-level relation extraction in Figure 2. The task is to identify the relationship between a pair of entities ("Pacific Fair" and "Queensland") in the input document. Each of the entities has two mentions in the text (denoted by superscripts). To infer their relationship, it is evident that the mentionmention pair involving the first mention of each entity provides the most valuable information for extracting the relationship between them.
Pacific Fair1 is a major shopping centre in Broadbeach Water on the Gold Coast, Queensland1, Australia. It was Queensland2’s largest regional shopping centre until 2006. Pacific Fair2 was developed by Hooker Retail Developments and opened in 1977 on what was swampland with 96 specialty stores and two anchor tenants.
Figure 2: Sample document relation extraction task from DocRED (Yao et al., 2019). The red text indicates the head entity, and the blue text indicates the tail entity. Here, the head is related to the tail by "P131: located in the administrative territorial entity".
Compared to sentence-level relation extraction, document-level relation extraction requires reasoning over multiple sentences which requires neural models to model long-range information. Additionally, entities may contain multiple mentions, which could include irrelevant information. However, this also allows for more information to model the relationship between entity-entity pairs.
Pretrained language models, such as BERTbased models (Xu et al., 2021), have demonstrated significant success in document-level relation extraction. For example, BERT-based methods have employed techniques like hierarchical inference networks (Tang et al., 2020), improved co-reference reasoning (Ye et al., 2020), and adaptive thresholding. Additionally, graphical neural networks (GNNs) (Zeng et al., 2020) have also been utilized for modeling document-level relation extraction. GNNs are used for feature learning on a coreference graph (Sahu et al., 2019), edgeoriented learning techniques (Christopoulou et al., 2019), utilizing attention mechanisms (Guo et al., 2019), and applying iterative refinement strategies for aggregating multi-hop information (Nan et al., 2020). Moreover, several works have proposed new loss functions to tackle the class-imbalance problem in document-level relation extraction (Zhou et al., 2021; Tan et al., 2022a).
However, previous research on document-level relation extraction has relied heavily on human annotation for generating training data, which can be a time-consuming and labor-intensive process. Limited work has been conducted on documentlevel relation extraction methods that do not require human annotation.

heading: 2.2 Weakly-Supervised Relation Extraction
text: Weakly supervised methods have been extensively explored for relation extraction (Jiang, 2009; Huang and Wang, 2017; Qu et al., 2018; Wang et al., 2018; Li et al., 2018). For example, Huang and Wang (2017) utilized residual connections and convolutional neural networks (CNNs) to select relevant candidates to enhance supervised relation classification. Qu et al. (2018) extracted textual patterns from seed examples to provide additional supervision. Phi et al. (2018) introduced a rankingbased approach for seed selection, improving bootstrapping and distantly supervised relation extraction. Sainz et al. (2021) proposed representing each relation class using a label verbalizer and addressing the relation extraction task with a textual entailment model. Wang et al. (2022a) analyzed an "extremely unlabeled" scenario where each relation type had only one instance and reduced the training set to a smaller number of labeled relation triplets (but still contained more than 5000 training triplets).
However, these methods were either primarily designed and evaluated for sentence-level relation extraction or still require many labels, which limits their generalizability to our weakly-supervised document-level relation extraction task.

heading: 3 Methodology
text: We propose PromptRE, a weakly supervised document-level relation extraction method that combines large language model prompting with data programming. An illustration of the overall framework of PromptRE is shown in Figure 3.

heading: 3.1 Problem Definition
text: In our task formulation, we consider a document D consisting of M sentences (s1, s2, ..., sM ) and N entities (e1, e2, ..., eN ). Given this document D, a specified entity pair (ehead, etail), and a set of positive entity-entity relations (r1, r2, ..., rk), the objective is to predict a set of relations (r̂1, r̂2, ..., r̂p) between the pair of entities based on the information extracted from the document. Note that each entity can have multiple occurrences within the document D.

heading: 3.2 Entity-Oriented Document Preprocessing
text: One challenge in document-level relation extraction is the long context. Models need to be able to find and focus on the information specific to the

heading: Is there a relationship between Pacific Fair and Queensland?
text: given pair of entities. In our PromptRE pipeline, we leverage the power of ChatGPT to solve this problem. For example, given an entity "Pacific Fair" and the document shown in Figure 2, we ask ChatGPT "Based on the given paragraph, summarize the information about "Pacific Fair" \n Pacific Fair is a major shopping center in Broadbeach Waters...".
ChatGPT will generate a natural language summary of the information about the entity in the paragraph. We concatenate the summary of head and tail entities to form the text description of the two entities (denoted as <context>). In later stages, this summary is utilized as the context in place of the original document for relation prediction.

heading: 3.3 Relation Prediction Via Prompting
text: For predicting the relation class, we explore two approaches: Relation Specific Prompting and Open Ended Prompting. For both approaches, we query all non-identity entity-entity pairs in the documents. That is, if one document has ne entities, we query ne(ne − 1) times.
Relation-Specific Prompting We prompt the large language models over all possible relation classes, for all possible entity pairs. For each relation class, we hand-craft a yes-no question. For example, for the relation class, "instance of", our hand-crafted version of the question is "Is "Pacific Fair" an instance of "Queensland" <context> ?". To quantify the certainty of the large language model, we obtain a prediction score LogitSR by subtracting the logit
of the ’no’ output from the logit of the ’yes’ output. This logit score LogitSR is calculated over each relation class and normalized to obtain a predicted relation class distribution.
Open-Ended Prompting With the open-ended approach, we only prompt the large language models once for each entity pair with the question "What’s the relationship between "Pacific Fair" and "Queensland" <context> ?". From there, we obtain the entity pair embedding as follows:
EOE =
∑|embed| i=1 LLM(POE)embed
|embed|
of the large language model output. Note that EOE ∈ RNhidden , where Nhidden is of hidden dimension size from the large language model.1
In addition, we encode each of the relation classes with a relation embedding Erels ∈ RNrels×Nhidden where Nrels is the total number of relation classes. We use cosine similarity between the entity pair embedding EOE and the relation embeddings Erels to compute a score over each relation, which is then normalized to obtain a predicted class relation distribution.
Utilizing the Type Distribution In our problem setup, we assume that we know the distribution of the relation classes given the types. We argue that this assumption, while strong, is reasonable. For
1Nhidden is 1024 for UnifiedQA and 4096 for LlaMA and LlaMA2.
example, it makes sense that no "Person - Person" entity pair could have the relation of "country of citizenship", as it does not make logical sense. It would be much more reasonable if the entity types were "location - person". We assume that this implicit knowledge is provided by the expert on the domain on which this framework is applied, and therefore we add the relevant relation distribution to the predicted probabilities of the raw scores, given the entity type pairs. In our experiments, we estimate this relation distribution from the ReDocRED dataset. More details can be found in Appendix A.
Multi-Label Prediction Since each document has multiple possible labels, we take only the top p percentile of confident predictions over all of the valid relation classes. Note that in this step, we do not consider the No-Relation class.

heading: 3.4 Addressing the No-Relation Issue
text: Although we can extract potential relations via relation prompting from the previous section, we face the issue of false positives in relation prediction due to the large number of No-Relation classes. To address this, we design PromptRE to choose only the most confident relation predictions.
Relation Existence Prompting To obtain the model prediction for the existence of a relation in the input text, we prompt the model with the following prompt: "Is there a relationship between "Pacific Fair" and "Queensland" <context> ?" To quantify the certainty of the large language model, we obtain a prediction score LogitRE by subtracting the logit of the ’no’ output from the logit of the ’yes’ output. This score is used to preserve only the most confident model predictions.
Data Programming (DP) We combine multiple sources of weak supervision to select highly confident predictions from the previous step of relation prompting. Data programming is a framework to create denoised pseudo-labels from multiple sources of weak supervision from labeling functions (Ratner et al., 2016, 2019).
A labeling function (LF) is a noisy heuristic that takes in data and assigns labels to unlabelled data or abstains from making a prediction. For example, f(text) = return SPAM if "http" in text else ABSTAIN is a labeling function for spam detection.
At a high level, we frame the problem as dependency graph Gsource where each labeling function λi is dependently conditioned on the true label Y . In our case, we assume conditional independence of all λi|Y . For this case, the dependency graphs will have observable cliques O = {λi, i ∈ nlf} ⊂ C, where nlf is the number of labeling functions.
From here, we can analyze the covariance matrix of an observable subset of the cliques in Gsource, leading to a matrix completion approach for recovering estimated accuracies µ (used in the final label model to predict P (Y |λ)).
We assume that µ = E(ψ(C)) where ψ(C) is vector of indicator random variables for all combinations of all but one of the labels emitted by each variable in clique C.
The norm of the covariance of observed LFs cliquesO and separator set S cliques Cov(ψ(O)∪ ψ(S)) can be used to recover µ.
Cov(ψ(O) ∪ ψ(S)) = Σ = [ ΣO ΣOS ΣTOS ΣS ] (1)
Its inverse is:
K = Σ−1 = [ KO KOS KTOS KS ] (2)
Applying block matrix inversion, we get:
KO = Σ −1 O + cΣ −1 O ΣOSΣ T OSΣ −1 O
c = (ΣS − ΣTOSΣ−1O ΣOS) Let z = √ zΣ−1O ΣOS , then
KO = Σ −1 O + zz T
Solving for z can directly recover µ via Algorithm 1 in Ratner et al. (2019).
Reducing NA Predictions via Weak Supervision To address the "No relation" issue, we attempt to combine multiple sources of weak supervision through data programming to obtain a stronger prediction. We consider three sources of weak supervision below.
1. The first source is the logit of relation-existence prompting LogitRE . A higher logit indicates a better likelihood of a relationship between the pair of entities. Additionally, by rephrasing the prompt in different ways, we obtain different views of the model opinion on the existence of a relationship. Other paraphrases could include
2. The second source is the average logit of relation-specific prompting LogitSR. The motivation is that if the entity pair has a low average logit for every relation-specific prompt, then it is not relevant to any of the relation classes and there is likely no relationship between the pair of entities.
3. The third source is the average cosine similarity between the entity pair embedding EOE and the relation embedding ERels. Similar to the previous motivation for relation-specific prompting, if an entity pair embedding is very dissimilar from every relation embeddings, then there is likely no relationship between the pair of entities.
To summarize, we combine the three sources of weak supervision as input to the data programming model. Then, we take the argmax from the probabilistic predictions of the data programming model and it as a mask to ensure that only the most probable predictions remain. Following the approach of Ratner et al. (2019), we also fit a logistic regression model on X = EOE and label model predictions Ŷ ∼ P (Y |λ) in order to smooth the decision boundaries.

heading: 4 Experiments

heading: 4.1 Dataset
text: To evaluate our methodology, we use ReDocRED (Tan et al., 2022b), an open-access, document-level relation extraction dataset that improves upon the popular DocRED dataset (Yao et al., 2019) by
resolving incompleteness, addressing logical inconsistencies, and correcting coreferential errors. Table 1 shows the amount of training data available for all data splits as well as the ChatGPTparaphrased entity-relevant text summary. Note that we primarily use the Dev set of ReDocRED for our experiments for computational practicality.

heading: 4.2 Experimental Settings
text: For the large language models for relation extraction, we compared UnifiedQA (Khashabi et al., 2020, 2022) (both 3b and large versions) and Alpaca-lora2–a reproduction of the Stanford Alpaca LlaMA model (Taori et al., 2023; Touvron et al., 2023; Wang et al., 2022b) using LoRa (Hu et al., 2021).
Some experiments can only be run with a subset of these models. For example, logitsSR is highly expensive to compute as it requires (n2e − ne) × Nrels, so we only run the UnifiedQA-Large for this score computation. For all other score computations, we may use all models: UnifiedQA-3b, UnifiedQA-3b, and LLama-7b as the base models. Additionally, we also perform weak supervision experiments without logitsSR due to its high cost (See App. C for more details). In our experiments, we use precision, recall, and F1 scores as the evaluation metrics for the performance comparison. More details about these evaluation metrics can be found in Appendix B.

heading: 4.3 Results
text: Table 2 shows the main results of our experiments. We observed that the logit performance of prompting every entity-entity pair with the relevant relation prompt does not perform as well as using the cosine similarity of the open-ended QA embeddings and the prompt embeddings. We suspect that this may be due to several reasons, including the lack of regularization of the score output. Additionally, it is possible that using cosine similarity allows the model to capture a more semantically meaningful snapshot of its response, rather than just a single scalar value.
As expected, using the ground truth NA labels leads to a large improvement over relaxing the assumption. It demonstrates the difficulty in determining the existence of relations in documents under weak supervision and points out an exciting direction for future research.
2https://github.com/tloen/alpaca-lora
Effect of Language Model Size The performance comparison between different model sizes is shown in Table 2. One observation is that the UnifiedQA-large model performs better than the UnifiedQA-3b model across all the metrics. Khashabi et al. (2022) observed similar results between the large and 3b models, so this is not entirely unexpected. Additionally, it is interesting to see that even the LLaMA-7b model, the largest
model we consider, provides no significant benefit over using the UnifiedQA-large model. This implies that prompting by itself does not work well for document-level relation extraction, as we see that the relations predicted are generally not of the same format as the true relation classes (see Section 4.4).
Effect of Entity-Oriented Document Preprocessing In this ablation study, We investigate the effects of using ChatGPT for entity-oriented document preprocessing (inspired by (Yu et al., 2022)). From Table 3, we see that using the ChatGPT-generated entity-oriented document summaries yields better performance compared to using the original document as input for relation extraction. However, this suggests that further research could potentially avoid expensive calls to the private model without sacrificing too much performance.
Effect of Relation Type Distribution We further investigate the effect of relation type distribution on relation class prediction. As shown in Table 4, the type distributions are strong signals as prior knowledge to enhance the performance of weakly-supervised relation extraction. We observe that using type distribution only without any other weak supervision achieves almost half of the performance of PromptRE in Table 2. Our results from Table 2 further show that the combination of
prompting + type distribution performs the best.

heading: 4.4 Case Studies
text: We analyze some example outputs from the predictions of the LLaMA-7b model, as shown in Table 5. We see that practically, the LlaMA model output is biased towards much longer and more detailed text than is required for the relation prediction problem. This could explain why the embeddings between the answers and the relation text would be difficult to correlate, leading to worse performance. Furthermore, the third example shows an instance of an indirect relation. It is true that Ross Patterson Alger was born in Canada and moved to another part of Canada. This is a common failure case with the responses–the relation prediction is too specific to the original text. The final two examples indicate a weakness in the dataset. As with any relation extraction dataset, ReDocRED is not complete, and the large language model was able to pick up on two relations not in the ground truth labeled set–"received a bachelor’s degree from" alderman" respectively.

heading: 5 Conclusion
text: In this paper, we investigate several methods to integrate prompting and data programming for relation classification and evaluate our model on ReDocRED. Results show that our best results yield around 10.2 F1 on the development set, a promising result for almost no supervision. Since this is a novel application, further research is required to investigate strategies for improvement. Some ideas include the following.
The NA Issue: The large number of "no relations" continues to be an issue for less than supervised methods for document relation extraction on existing datasets. Further work should focus on more efficient and accurate ways to mine distant labels to address this issue. One major roadblock that coincides with NA is the lack of complete labels in the dataset as shown in the case studies. Future work could improve on existing document relation extraction datasets accordingly.
Extending to the Few-Shot Case: It is usually possible to query human experts for a few examples of the required classification task. Researching ways to take maximum advantage of a small set of labels would also be highly practical, and would not require much extra effort on the annotators. This could also tune a model to better address the "No Relation" issue.
Final Thoughts: We find that weakly supervised document-level relation extraction is a uniquely difficult problem due to the incomplete labels in popular datasets, and we propose PromptRE to attempt to solve it by combining prompting and data programming. We show the effect of tuning different experimental setups, including model size, entity-oriented summarization, and the effect of our relation-type distribution assumption. Case studies support the finding that existing document-level relation extraction datasets may be severely lacking in label completeness. Although the results are a considerable margin from contemporary supervised methods, we hope that this work can serve as a stepping stone in this novel area of less-thansupervised document relation extraction.
Limitations
Although we investigated multiple different LLMs and parameters and the type relation distribution for relation prediction as well as addressing the false positives, the performance we attained is still limited compared to supervised methods on the same task. Additionally, relation prediction is dependent on the prompt choice, as we see from the open-ended prompts performing better than asking specific relationships. Data Programming is also dependent on high-quality sources of weak supervision, as we see from the improvement in performance when not considering the logits in Table 4. Effectively mapping the output of language models to the concrete label space without training remains a hard problem for future work to tackle.
Ethical Statement
Based on the methodology we have currently employed, we do not foresee any significant ethical concerns. All the documents and models utilized in our study were obtained from open-source domains, ensuring a transparent and accessible source of information. Additionally, PromptRE requires no LLM training, eliminating the risk of model drift. Additionally, the task of relation extraction is a widely recognized and well-studied problem across various natural language processing applications.
However, it is crucial to acknowledge a minor factor, namely the presence of potential hidden biases within the pretrained language models used in our analysis. These biases may stem from the data on which the models were trained, which could have inadvertently introduced implicit human biases. While our usage of these pretrained language models enables us to identify relationships between arbitrary entities, it is conceivable that biases may emerge if one were to explore sensitive relation classes and entities.

heading: A Parameter Settings
text: All models were run on an NVIDIA A6000 with 48 gigabytes of VRAM. Still, around 10 days were required to fully run the experiments. For particularly expensive computations, like LogitsSR, only the fastest model–UnifiedQA-large–could be feasibly run.
All models were downloaded from Huggingface (Wolf et al., 2019). We used the default setup of the pretrained models and did not do further finetuning. All the step mentioned in the methodology section works on the output of the pretrained models.
Supervised results DREEEAM (Ma et al., 2023) and KD-DocRE (Tan et al., 2022a) were taken from the original source papers.

heading: B Evaluation Metrics
text: To keep in tradition with existing document relation extraction work, we report both F1 and Ign_F1 as computed by the official metrics from ReDocRED. F1 refers to micro-averaged F1 score that combines precision P and recall R
F1 = 2PR
P +R
P = length of correct (h,t,rel) preds
length of all (h,t,rel) preds
R = length of correct (h,t,rel) preds
length of correct (h,t,rel)
Where (h,t,rel) denotes a tuple of the predicted head, tail, and relation. Ign_F1 is computed similarly to above but ignores the samples in the DocRED’s distantly supervised training set. (Note that we do not use any distantly labeled data).

heading: C Effect of Relation-Specific Prompts
text: In this ablation study, we investigate the usefulness of LogitsSR on both the relation class prediction part as well as the addressing of the NA issue. Because each entity-entity pair has to prompt with all relations, it is quite expensive to perform. Thus, we only perform experiments with the fastest model we consider–UnifiedQA large. From Table 2, we see that by itself, using the logits do not perform as well as embedding similarity.
For the case of reducing NAs, we actually do not include it as a source of weak supervision in the data programming framework due to its inefficiency. However, if we did include it, we would see that performance drops as well, as shown in Table 6.
Ross Patterson Alger ( August 20 , 1920 – January
16 , 1992 ) was a politician in the Canadian
province of Alberta , who served as mayor of
Calgary from 1977 to 1980 . Born in Prelate ,
Saskatchewan , he moved to Alberta with his family
in 1930s . He received a bachelor of commerce
degree from the University of Alberta in 1942 . He
served with the Royal Canadian Air Force during
World War II . After the war , he received an
MBA from the University of Toronto . He settled
in Calgary and started a career in accounting .
In 1958 , he was a public school board trustee
, and later became the chairman . From 1971 to
1974 , he was an alderman on Calgary City Council
. In 1974 , he ran for mayor losing to Rod
Sykes . He was elected mayor in 1977 and served
one term until 1980 . During Alger ’s term ,
notable accomplishments include the construction
of the Ctrain ’s first leg , the bid for the
XV Olympic Winter Games , and planning for the
Olympic coliseum . His brother was Harry Alger .
Alger died of cancer in 1992 , which had first
been diagnosed in 1985 .
Figure 4: Original document for Case Study 1.

heading: D Relation Distribution Calculation
text: We test our assumptions of the relation type distribution. Specifically, we how the performance changes with more or less expert annotated documents. The results are shown in Table 7. Recall that in total, we have 500 documents, so 1% of all documents represent only 5 annotated documents. This reinforces our assumption that creating this relation/type distribution is not exorbitantly expensive. Furthermore, this computation is only an estimate of the actual input that domain experts would provide, so it is possible that real world performance would be better or worse depending on the distribution of true types and relations.
Mess of Blues is an album by Jeff Healey . It
was released in 2008 less than two weeks after his
death and just three weeks shy of his 42nd birthday
. Four of the album ’s tracks were recorded live
in front of audiences , two of the live tracks at
the Islington Academy in London , and the other
two live tracks at Jeff Healey ’s Roadhouse in
Toronto . The other six tracks were recorded at
Studio 92 in Canada by Norm Barker and Richard
Uglow . The whole album features the band which
normally accompanied Jeff at his club , Jeff Healey
’s Roadhouse . The song " Mess of Blues " , which
appears on the album was written by Doc Pomus and
Mort Shuman and was originally recorded by Elvis
Presley .
Figure 5: Original document for Case Study 2.

heading: E Additional Case Studies
text: We analyze a few more examples of the model output in Table 8 with the original document in Figure 5. While the first 3 labeled examples make intuitive sense and are correct, the lack of label completeness is evident in the latter 3 examples. For example, in the last example, Richard Uglow is the recorder of part of the album, but the pair "Richard Uglow" and "Mess of Blues" is marked as "No relation" in the dataset. This type of weakly labeled situation is a particularly difficult problem for weakly-supervised methods like ours because, without training, it’s very hard to inform the model with subtle details regarding the relation label space.

heading: Future Work
text: Future work could improve on existing document relation extraction datasets accordingly. Extending to the Few-Shot Case: It is usually possible to query human experts for a few examples of the required classification task. Researching ways to take maximum advantage of a small set of labels would also be highly practical, and would not require much extra effort on the annotators. This could also tune a model to better address the "No Relation" issue. Final Thoughts: We find that weakly supervised document-level relation extraction is a uniquely difficult problem due to the incomplete labels in popular datasets, and we propose PromptRE to attempt to solve it by combining prompting and data programming. We show the effect of tuning different experimental setups, including model size, entity-oriented summarization, and the effect of our relation-type distribution assumption. Case studies support the finding that existing document-level relation extraction datasets may be severely lacking in label completeness. Although the results are a considerable margin from contemporary supervised methods, we hope that this work can serve as a stepping stone in this novel area of less-thansupervised document relation extraction. Limitations
Although we investigated multiple different LLMs and parameters and the type relation distribution for relation prediction as well as addressing the false positives, the performance we attained is still limited compared to supervised methods on the same task. Additionally, relation prediction is dependent on the prompt choice, as we see from the open-ended prompts performing better than asking specific relationships. Data Programming is also dependent on high-quality sources of weak supervision, as we see from the improvement in performance when not considering the logits in Table 4. Effectively mapping the output of language models to the concrete label space without training remains a hard problem for future work to tackle