# A FAIR Resource Recommender System for Smart Open Scientific Inquiries
## 3 The Deep Web Communication Protocol
### Message Structure and Data Model
#### 5.2.2 Data Retrieval Success

While exact matches reflect alignment with the ground truth, the practical goal is to return structured, usable data. We therefore define a success criterion--Success@1 --which captures whether the query ultimately produced a valid tabular dataset, regardless of the precise access path. Formally, we say that a query succeeds if the final output \(\mathcal{D}_{q}\) contains valid tabular data, regardless of whether the exact retrieval path followed the ground truth:

\[\text{success}(



selecting and summarizing form elements, generating representative values for form fields, and semantically aligning user queries with access paths. All remaining components of the system--vector-based retrieval, HTML rendering, and execution logic--remained fixed. This design isolates the influence of the language model on accuracy and latency. ### Execution Pipeline

Each user query is processed through a multi-stage pipeline involving semantic retrieval, schema-level access resolution, and structured data extraction. The end-to-end interaction includes document embedding, site and schema re-ranking, form interaction (if applicable), and output parsing. The overall procedure is summarized in Algorithm 2. This function is applied to every query \(q\in\mathcal{Q}_{\text{eval}}\) and generates a tuple containing evaluation metrics and runtime, which is subsequently logged for further analysis. This process is executed for all queries in the benchmark set. The resulting logs form the basis for performance comparisons across language models, access types, and query complexities. ```
Input: Natural language query \(q\), ground truth \(g_{i}=(\text{site\_url}_{i},\text{page\_url}_{i},\text{access\_path}_{i})\) Output: Evaluation result\((q)\) and response time \(t_{q}\)
1 Compute embedding \(\phi(q)\) using the sentence encoder;
2 Retrieve top site candidates \(S=\{s_{1},s_{2},\ldots\}\) based on vector similarity over HTML chunks;
3 Select top-ranked site \(s^{*}\in S\) and retrieve all schema-level DWCP records \(\mathcal{D}_{s^{*}}\);
4 Re-rank \(\mathcal{D}_{s^{*}}\) by computing \(\text{sim}(\phi(q),\phi(d_{j}))\) for each schema \(d_{j}\);
5 Select best schema document \(d^{*}\) and extract access\(\_\)path\({}^{*}\);
6 Compare access\(\_\)path\({}^{*}\) with \(g_{i}\)-access\(\_\)path to evaluate path accuracy;
7 Simulate access interaction using process\(\_\)access\(\_\)path\((\text{access\_path}^{*},\text{page\_url},q)\);
8 If successful, extract resulting tabular data;
9 Record query execution time \(t_{q}\);
10 Compute and return: \(\text{result}(q)=(\text{match}_{\text{site}},\text{match}_{\text{page}}, \text{match}_{\text{access}},\text{success})\) return\(\text{result}(q)\), \(t_{q}\)
```

**Algorithm 2**Evaluation Pipeline for Semantic Query Execution

## 6 Results

We evaluated FAIRFind on a curated benchmark of 50 scientific data sources linked from FAIRsharing, using both simple and complex queries. The results are organized by the two primary evaluation axes defined earlier. ### Exact Match Performance

Figure 3 presents the exact match performance across four open-source language models. We evaluated three types of retrieval correctness--site match rate, page match rate, and access path match rate--each capturing a distinct level of alignment between user intent and system prediction in the deep web indexing pipeline. Among the models evaluated, LLaMA 3.3 achieves the highest site match rate at 0.673, reflecting its superior ability to identify the correct scientific repository based on the semantics of the input query. DeepSeek-R1 follows closely with a match rate of 0.633, while Qwen3 and Gemma both score 0.571. These results suggest that while all models demonstrate a reasonable understanding of query-to-domain alignment, LLaMA 3.3 benefits from stronger grounding in scientific terminology and document structures. In the task of identifying the correct data interface page, LLaMA 3.3 again leads with a page match rate of 0.531.