# A FAIR Resource Recommender System for Smart Open Scientific Inquiries
## 3 The Deep Web Communication Protocol
### Message Structure and Data Model
A realistic simulation of user interaction is made possible by the model's inference of realistic input values for each field based on the chosen form. The system simulates the entire interaction as a user would by using XPath-based automation to fill out and submit the form within the browser context. The HTML page returned, whether via GET or POST, is parsed for structured content. If a valid table is found, the headers and sample rows are extracted accordingly. In cases where no usable form is detected, the system scans for static HTML tables as a fallback. If found, it still constructs a DWCP object with access type as an html-table. Once the structured data are extracted, they are wrapped into a DWCP-compliant JSON object. This object includes all necessary metadata, such as headers, sample rows, access method, and any form-specific access path information. The goal is to ensure that the interaction is reproducible and the resource is programmatically accessible. Each DWCP object \(o_{ij}\) is then embedded into a high-dimensional semantic vector space via an encoder function \(\phi\). The resulting embedding \(\phi(o_{ij})\in\mathbb{R}^{d}\) is stored in the metadata index \(\mathcal{M}\) alongside its corresponding DWCP object, supporting future retrieval based on similarity. ### Semantic Query Engine

The goal of the query engine is to allow users to retrieve structured scientific data using natural language queries, without requiring knowledge of the exact source database, access method, or form structure. Our approach uses a multi-stage retrieval and reasoning system built on top of the DWCP index. The system combines semantic search over embedded metadata with LLM-based interpretation and form execution. It consists of the following core stages: (1) site-level ranking, (2) access path retrieval, (3) dynamic interaction, and (4) structured output rendering. #### 4.5.1 Vectorized Document Indexing

To support fast and flexible retrieval, each DWCP record--whether representing a form interaction, downloadable file, or static table--is split into two types of documents:

* HTML Content Chunks: Extracted from the rendered page's visible content. * Schema Blocks: Representations of DWCP access path metadata. These documents are preprocessed into fixed-length text chunks using a sliding window mechanism. Each chunk is embedded using a SentenceTransformer model, mxbaimembed-large-v1 and stored in a ChromaDB vector store. This model was selected based on its strong performance-to-size trade-off, as ranked on the MTEB leaderboard [45]. This dual representation of documents supports hierarchical retrieval from relevant websites down to specific access pathways. #### 4.5.2 Stage-One Retrieval: Site Ranking

Given a user query \(q\), the system first searches over HTML content to rank the most relevant websites. For each site, we retrieve top-\(k\) matching content chunks \(\{h_{i}\}\) and compute the average distance to the query embedding:\[\text{TopSites}(q)=\arg\min_{\text{site}}\frac{1}{k}\sum_{i=1}^{k}\text{dist}(\phi(q ),\phi(h_{i})) \tag{3}\]

where \(\phi(\cdot)\) is the embedding function that maps text to a vector space, and \(\text{dist}(\cdot\cdot\cdot)\) is the cosine distance function. This yields a ranked list \(S=\{s_{1},s_{2},\dots\}\) of FAIRsharing sites likely to contain relevant data. #### 4.5.3 Stage-Two Retrieval: Access Path Matching

After selecting the top-ranked site \(s^{*}\), the engine restricts its search to schema-level documents under that site. DWCP metadata documents \(d_{j}\) are retrieved using

\[\text{TopSchemas}(q,s^{*})=\arg\min_{d_{j}\in\text{SCHEMA}(s^{*})}\text{ dist}(\phi(q),\phi(d_{j})) \tag{4}\]

where SCHEMA\((s^{*})\) is the set of all schema-level DWCP documents associated with site \(s^{*}\), and \(d_{j}\) is the embedding of document \(j\). Each retrieved schema document corresponds to a page URL and a JSON-formatted DWCP access path.