## 4 Case Study
### Performance Evaluation of Information Extraction Compared to Other Models
#### 4.3.3 Evaluation System Construction  
To evaluate the accuracy of different models in information extraction tasks, research team members manually annotated a randomly sampled 20% subset of the test data. Cross-validation was conducted to ensure the high reliability of the manually annotated extraction results. Given the significant differences in core elements across various requirement categories, this study establishes a unified matching standard applicable to GPT-4, GPT-3.5-turbo, BERT, and RoBERTa, based on the information extraction template outlined in Section 3.2. The evaluation system integrates exact matching and cosine similarity [47] computation to ensure the accuracy and rationality of the structured data extraction. For nodes consisting of subject, attribute, and object elements, these nodes represent key entities in the requirements. Any deviation in these nodes may result in the introduction of incorrect nodes during the subsequent graph construction process. To mitigate this issue, we employ a strict string-matching approach, where every term extracted from the results must exactly match the manually labeled terms; otherwise, it is considered a mismatched match. When multiple subjects, objects, or attributes are present within the requirement text, the number of correctly matched elements is used as the score for that item, and the matching accuracy is calculated using the following formula:  
\[S_{\text{EM},i}=\frac{\sum_{j=1}^{N_{i}}\mathbb{I}\big{(}\hat{y}_{i,j}=y_{i,j }\big{)}}{N_{i}} \tag{1}\]  
where \(S_{\text{EM},i}\) represents the exact matching score for element \(i\); \(N_{i}\) denotes the total number of instances of element \(i\) in the manually annotated data; \(\hat{y}_{i,j}\) represents the \(j\)-th value predicted by the model for element \(i\); \(\mathbb{I}(\cdot)\) is an indicator function, which equals 1 if \(\hat{y}_{i,j}=y_{i,j}\), and 0 otherwise. For edge elements and textual components, such as actions, functions, values, principles, and conditions, which are characterized by lengthy text content and rich semantic information, relying solely on exact matching is insufficient for comprehensively assessing the semantic similarity between the model's extracted results and the manually annotated data. Cosine similarity calculation can serve as a metric for evaluating the similarity between the two. Therefore, this study employs a cosine similarity computation method based on BERT word embeddings (bert-base-chinese) for matching [48]. In actual requirement entries, the same data may contain multiple elements of the same category. For instance, during the structured information extraction of functional requirements, multiple functional items such as Function 1, Function 2,... Function N may be extracted. Before computing cosine similarity, all text elements of the same category are first merged and concatenated to form a unified textual representation. The overall similarity is then calculated by comparing this aggregated representation with the manually annotated elements. \begin{table}
\begin{tabular}{c c} \hline
**Parameter** & **Value** \\ \hline Pretrained model & bert-base-chinese, hfl/rbt6 \\ Batch size & 8 \\ Dropout & 0.3 \\ Epochs & 30 \\ Learning rate & \(5\times 10^{-4}\) \\ Optimizer & AdamW \\ \hline \end{tabular}
\end{table} 
Table 8: Pre-trained deep learning model parameter configuration. Let \(v_{i,m}\) denote the semantic vector representation of element \(i\) extracted by the model and \(v_{i,h}\) denote the semantic vector representation of element \(i\) extracted from the manual annotation.