# A FAIR Resource Recommender System for Smart Open Scientific Inquiries
## 3 The Deep Web Communication Protocol
### Message Structure and Data Model
### Evaluation Metrics

Our evaluation framework assesses FAIRFind's performance along two main dimensions: Exact Match Accuracy and Data Retrieval Success. These collectively measure the system's ability to navigate, identify, and interact with deep web sources indexed through DWCP representations. #### 5.2.1 Exact Match Accuracy

This group of metrics captures whether the retrieval pipeline successfully aligns with ground truth access paths at different levels of resolution. For each natural language query \(q\in\mathcal{Q}_{\text{eval}}\), we compute binary indicators reflecting the correctness of predicted access components. First, we assess whether the top-ranked site \(s^{*}\), as returned by the semantic retrieval system, matches the ground truth site associated with the query. This is formalized as the Site Match Rate:

\[\text{match}_{\text{site}}(q)= \begin{cases}1&\text{if }s^{*}=\text{site\_url}_{q}\\ 0&\text{otherwise}\end{cases}  \tag{8}\]

The Site Match Rate over all queries is computed as

\[\text{Site Match Rate}=\frac{1}{|\mathcal{Q}_{\text{eval}}|}\sum_{q\in \mathcal{Q}_{\text{eval}}}\text{match}_{\text{site}}(q) \tag{9}\]

where \(|\mathcal{Q}_{\text{eval}}|\) is the number of queries in the evaluation set, and \(\text{match}_{\text{site}}(q)\) is defined in Equation (8). \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt}} \hline \hline
**Simple Query** & **Complex Query** \\ \hline \multirow{4}{*}{Find entries in the PR2 database where Level is “Alveolata” and Class is “Dinoflagellates” with Version 5.0, focusing on taxonomic groups annotated by experts.} & I am looking for detailed information on proteins with unique dynamic properties, such as those with chameleon subsequences or Dual Personality Fragments, as analyzed through molecular dynamics simulations. Specifically, I want to explore how these proteins interact with other biological molecules and their domain limits. \\ \hline Find samples with Data Type as “Metagenome” or “Natural Organic Matter” from the National Microbiome Data Collaborative database, focusing on studies related to the Earth Microbiome Project and the 1000 Soils Research Campaign. & I am looking for microbiome data samples that involve metagenomic analysis or natural organic matter studies, particularly those associated with large collaborative projects like the Earth Microbiome Project or soil research initiatives. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Examples of simple and complex query pairs used in the benchmark set. Next, we check whether the retrieved schema documents points to the correct page url, where the structured data are hosted. The corresponding Page Match Rate is defined as

\[\text{match}_{\text{page}}(q)= \begin{cases}1&\text{if predicted page}=\text{page}_{\text{url}}_{q}\\ 0&\text{otherwise}\end{cases}  \tag{10}\]

\[\text{Page Match Rate}=\frac{1}{|\mathcal{Q}_{\text{eval}}|}\sum_{q\in\mathcal{ Q}_{\text{eval}}}\text{match}_{\text{page}}(q) \tag{11}\]

Beyond locating the page, the system must also extract and interpret the correct access pathway. We compute the Access Path Match Rate based on whether the retrieved DWCP access path block matches the reference in both structure and semantics:

\[\text{match}_{\text{access}}(q)= \begin{cases}1&\text{if predicted access}_{\text{path}}=\text{access}_{\text{path}}_{q}\\ 0&\text{otherwise}\end{cases}  \tag{12}\]

\[\text{Access Path Match Rate}=\frac{1}{|\mathcal{Q}_{\text{eval}}|}\sum_{q\in \mathcal{Q}_{\text{eval}}}\text{match}_{\text{access}}(q) \tag{13}\]

Alongside these binary correctness indicators, we also measure the total time required to resolve each query. The Average Retrieval Time, denoted as \(t_{q}\), includes all steps: semantic site ranking, schema re-ranking, access path processing, and content fetching (including any browser-based interaction or LLM inference). This latency metric provides insight into system responsiveness.