## 4 Case Study
### Performance Evaluation of Information Extraction Compared to Other Models
#### 4.3.4 Performance Evaluation of Information Extraction by GPT-4 and Other Models  
The final score of the data depends on the comprehensive assessment of exact matching and cosine similarity. Based on existing research on the accuracy of information extraction models and text similarity standards [49; 50; 51], we set the determination criterion as \(S_{f}\geq 0.85\). That is, when the structured extraction score reaches or exceeds 0.85, the \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Requirement Category** & **Subject** & **Attribute** & **Object** & **Action** & **Function** & **Value** & **Principle** & **Condition** \\ \hline Functional requirement & 0.25 & 0.15 & 0.15 & 0.10 & 0.25 & - & - & 0.10 \\ Interface requirement & 0.30 & - & 0.30 & 0.30 & - & - & - & 0.10 \\ Performance requirement & 0.30 & 0.30 & - & - & - & 0.30 & - & 0.10 \\ Physical requirement & 0.20 & 0.20 & 0.20 & 0.10 & - & 0.20 & - & 0.10 \\ Design constraint & 0.30 & 0.20 & - & 0.10 & - & - & 0.30 & 0.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Weight allocation for different requirement categories. extracted result is deemed correct. Based on this criterion, we conducted an evaluation on a test set of 105 data entries. The experimental results indicate that the GPT-4 model correctly extracted 89 entries, including 35 functional requirements, 9 interface requirements, 10 performance requirements, 19 physical requirements, and 16 design constraints, achieving an overall extraction accuracy of 84.76%, according to the evaluation criterion. In comparison, the extraction accuracy of the GPT-3.5-turbo model was 79.05%, while both the BERT and RoBERTa models achieved the same extraction accuracy of 59.05%. The extraction accuracy for different requirement categories across the models is presented in Table 10.  
The results of this study demonstrate that GPT-4 significantly outperforms traditional pre-trained fine-tuned models, slightly surpassing the generative pre-trained model GPT-3.5-turbo as well. One of the primary reasons for this disparity is the limitation in the amount of training data in our experiment. With only 411 labeled data points, models like BERT and RoBERTa, which rely on supervised fine-tuning, struggle to develop adequate generalization capability [52; 53; 54]. According to the definitions of the five requirement categories in Section 3.1, Table 1, pre-trained fine-tuned models often encounter significant challenges when processing requirement texts such as interface requirements and design constraints. This is because these texts involve multiple levels of dependency and nested semantics. In these cases, BERT and RoBERTa often fail to fully capture the semantic context, leading to incomplete extraction results or erroneous word segmentation. For example, as shown in Table 11, when extracting the phrase \"H3.5-Turbo\" (relevant systems) from the sentence \"H3.5-Turbo\" (\"The automatic fare collection system should implement interfaces with relevant systems\"), BERT mistakenly extracts \"H3.5-Turbo\", which is not a valid term. These issues further affect the final scores of BERT and RoBERTa, resulting in a decline in the extraction accuracy of the models. In contrast, GPT-4 and GPT-3.5-turbo demonstrate superior performance due to their adoption of a different learning paradigm. Unlike BERT and RoBERTa, both GPT-4 and GPT-3.5-turbo do not rely on large amounts of labeled data for fine-tuning; instead, they employ few-shot learning [44], where a small number of example data points are provided in the prompt to adapt to new tasks. By embedding the information extraction template in the prompt design, both GPT-4 and GPT-3.5-turbo can effectively learn the structure of the requirement text and align their outputs accordingly.