# Modeling Semantic-Aware Prompt-Based Argument Extractor in Documents
## 2 Related Work  
### Pre-Trained Language Models for Event Extraction  
In recent years, pre-trained language models (PLMs) have achieved remarkable success across a wide range of natural language processing (NLP) tasks, including event extraction. Among these, BERT (Bidirectional Encoder Representations from Transformers) [12] and GPT (Generative Pre-trained Transformer) [13, 14] represent two influential paradigms: bidirectional masked language modeling and unidirectional autoregressive generation, respectively. BERT-based models have significantly advanced event extraction tasks by leveraging deep bidirectional contextual representations. For instance, [PERSON] et al. [15] proposed a hybrid framework that employs separate encoders for event detection and argument extraction, effectively mitigating feature interference between tasks and enhancing overall performance. Additionally, [PERSON] et al. [16] introduced a joint document-level event extraction approach using a Token-Token Bidirectional Event Completed Graph (TT-BECG), which captures intricate token-level relationships and improves the extraction of complex event structures. These studies demonstrate the robustness and adaptability of BERT-based models in handling diverse event extraction scenarios. On the other hand, the GPT series--particularly GPT-3 [14], GPT-3.5, and GPT-4--have shown remarkable performance in generative and reasoning-heavy tasks via few-shot or zero-shot prompting. GPT-3.5 serves as an improved intermediate step between GPT-3 and GPT-4, with enhanced reasoning ability and reduced hallucinations. [PERSON] et al. [17] explored the feasibility of using ChatGPT-3.5 for event extraction and found that, while ChatGPT can perform the task, its performance is approximately 51% of that of specialized models in complex scenarios, highlighting challenges in robustness and prompt sensitivity. Conversely, [PERSON] et al. [18] proposed ChatIE, a two-stage framework that transforms zero-shot information extraction into a multi-turn question-answering problem, enabling ChatGPT to achieve impressive results and even surpass some fully supervised models on certain datasets. These findings suggest that, while GPT-based models show promise in event extraction, especially in low-resource settings, careful prompt design and task decomposition are crucial for optimal performance. From a mathematical perspective, both BERT and GPT are based on the Transformer architecture [19], but they differ fundamentally in their pre-training objectives and attention mechanisms. BERT is trained using a masked language modeling (MLM) objective. Givenan input sequence \(X=[x_{1},x_{2},\ldots,x_{n}]\), the model randomly masks tokens and learns to predict them based on bidirectional context:  
\[L_{\text{MLM}}=-\sum_{i\in\mathcal{M}}\log P(x_{i}|X_{\setminus i}) \tag{1}\]  
where \(\mathcal{M}\) is the set of masked positions and \(X_{\setminus i}\) denotes the masked sequence. This bidirectional encoding enables better reasoning over global context. Its attention mechanism can be expressed as follows:  
\[\text{Attention}(Q,K,V)=\text{softmax}\Bigg{(}\frac{QK^{\top}}{\sqrt{d_{k}}} \Bigg{)}V \tag{2}\]  
In contrast, GPT uses a causal language modeling (CLM) objective:  
\[L_{\text{CLM}}=-\sum_{t=1}^{n}\log P(x_{t}|x_{1},x_{2},\ldots,x_{t-1}) \tag{3}\]  
and employs a causal mask to block forward attention:  
\[\text{Attention}_{\text{GPT}}(Q,K,V)=\text{softmax}\Bigg{(}\frac{QK^{\top}}{ \sqrt{d_{k}}}+M\Bigg{)}V \tag{4}\]  
where \(M\) is a mask matrix enforcing the left-to-right constraint. This unidirectional nature is advantageous for generative tasks but limits GPT's performance in extraction settings where bidirectional context is essential. Beyond BERT and GPT, other advanced PLMs have been developed with stronger long-document modeling capabilities, which are also relevant for document-level event argument extraction (D-EAE).