Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Authors: Nabeel Seedat
Mihaela van der Schaar
Venue: ICML 2025 poster (2025)
Keywords: schema matching
healthcare
Large Language Models
data-centric AI
Primary area: applications->health_medicine
TL;DR: self-improving compositional language model programs for schema matching across heterogenous data sources

Abstract: Schema matching -- the task of finding matches between attributes across disparate data sources with different tables and hierarchies -- is critical for creating interoperable machine learning (ML)-ready data. Addressing this fundamental data-centric problem has wide implications, especially in domains like healthcare, finance and e-commerce --- but also has the potential to benefit ML models more generally, by increasing the data available for ML model training. However, schema matching is a challenging ML task due to structural/hierarchical and semantic heterogeneity between different schemas. Previous ML approaches to automate schema matching have either required significant labeled data for model training, which is often unrealistic or suffer from poor zero-shot performance. To this end, we propose Matchmaker -  a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker also self-improves in a zero-shot manner without the need for labeled demonstrations via a novel optimization approach, which constructs synthetic in-context demonstrations to guide the language model's reasoning process.  Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches, highlighting its potential to accelerate data integration and interoperability of ML-ready data.

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Nabeel Seedat 12 Mihaela van der Schaar'
Abstract
1. Introduction
The success of machine learning (ML) models hinges on a critical yet often overlooked challenge: access to large, inte- grated and interoperable datasets (Jain et al., 2020; Gupta et al., 2021; Renggli et al., 2021; Sambasivan et al., 2021). Although well-structured and uniform datasets like those on Kaggle are commonly assumed as the norm, such data is rare luxury in practice. In real-world scenarios, tabular data often exists in heterogeneous and disparate databases with diverse formats, schemas, and terminologies, requiring har- monization to make the data "ML.-redy"" and interoperable. The heterogeneity of databases presents three critical issues for ML: (1) data harmonization and integration are arduous tasks. Hence, researchers often limit the features/covariatte used for model training to smaller, often common, set of features (Avati et al., 2021: Si et al., 2021; Rajkomar et al., 2018), thereby limiting the potential performance of their ML models; (2) even if all the features are used, the lack of data interoperability means limited external validation of MI. models (Balch al., 2023; Lehne al., 2019; Williams et al., 2022; Tiwari et al., 2020; Colubri et al., 2019; Goetz et al., 2024; Seedat et al., 2020), which can undermine the credibility and utility of the ML models; and (3) missed op- portunities for insights on larger harmonized datasets (e.g., larger patient populations), which may not be apparent when analyzing data sources independently.
Schema matching - the task of finding matches be- tween attributes across disparate data sources with different tables and hierarchies- - is critical for cre- ating interoperable machine learning (ML)-reayy data. Addressing this fundamental data-centric problem has wide implications, especially in do- mains like healthcare, finance and e-commerce but also has the potential to benefit ML models more generally, by increasing the data available for ML model training. However, schema match- ing is a challenging ML task due to structural/hier- archical and semantic heterogeneity between dif- ferent schemas. Previous ML approaches auto- mate schema matching have either required sig- nificant labeled data for model training, which is often unrealistic or suffer from poor zero-shot per- formance. To this end, we propose Matchmaker a compositional language model program for schema matching, comprised of candidate genera- tion, refinement and confidence scoring. Match- maker also self-in oef-iierovee in zero-shot manner without the need for labeled demonstrations via novel optimization approach, which constructs synthetic in-context demonstrations to guide the language model's reasoning process. Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outper- forms previous ML-bbaed approaches, highlight- ing its potential to accelerate data integration and interoperability of ML-ready data.
Schema matching is critical first step in data harmoniza- tion, aiming to establish correspondences between attributes (i.e., features/covariates) measured across different data sources. Once matched, these correspondences can help harmonize data from disparate sources into a cohesive, MI.-ready format. To understand the concept of schema matching, let us unpack the components of schema. A schema defines how data is organized in database, com- prising different tables (collections of related data entries) and columns (also known as "attributes" or "features") ") that represent specific data fields. Importantly, schemas go be- yond simple tabular data commonly found in CSV files, as they capture the hicrarchical structure and relationships be- tween different tables and their attributes. For example, in healthcare, schemas from different hospitals may have vary- ing tables and attributes representing patient information,
'Department of Applied Mathematics and Theoretical Physics, University of Cambridge FFouddationa Machine Learning Re- search, Thomson Reuters. Correspondence to: Nabeel Seedat <ns741 @ aak.ac.uk>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Source Schema
Target Schema
OESERAATION
petcee 4I
ACT,ATIE AVTNT
VISI DF
VEn DETLOCCUTENCCE
:
partette doc 2020
MEAAL MIMMEI
Search challenge:
Information Mismatch (No Match possible)
PROVICER
Viai WALLECCREe.co volus
Search challange:
Search challongo:
25 Tabtes in Scheme
37 Tables in Schoma
Table identification
Attribbute idlnttffcation
(Database
(Structural Semantic hetoregeneity)
Differences schemas: Database & Structural hetarageneity
heterogeneity))
Figure 1: Example showing the complexity of schema matching due to the multi-faceted challenges: Database heterogeneity (green arrows): Identifying the correct target table, each schema has different number of tables, the corresponding information may be distributed differently across tables in each schema. Structural heterogeneity (green arrows): Once the appropriate table found, matching attributes is complicated by differences in schema architectures, hierarchies, granularity. Textual heterogeneity (green arrows): Ambiguity in matching when attributes have the same names but different meanings, or vice versa. Information mismatch (red arrows): Some attributes in one schema may lack corresponding match in the other schema, adding to the complexity of the matching process.
lab measurements, diagnoses and treatments, with complex relationships and hierarchies connecting the tables. Conse- quently, schema matching involves analyzing the context of attributes within the schema hierarchy to establish mean- ingful mappings that preserve the intended semantics and relationships. It goes beyond simple one-to-one column matching, considering not only the attribute itself but also the hierarchical structure and relationships between tables defined by the schema. Notably, schema matching does not assume access to data, relying on only attribute names, descriptions and metadata (e.g., in healthcare, patient data cannot be queried or accessed directly due to privacy con- cerns or regulations (Zhang et al., 2021)))
tempted to address this, they have poor zero-shot perfor- mance and poor scalability in terms of the number of LLM calls. These limitations have hindered the adoption of ML for schema matching, meaning schema matching is still a largely manual and time-consuming task. To highlight the need for automated and improved ML schema matching, in the healthcare domain, it took 500 hours for two experts to map the schemas between the MIMIC database and the OMOP common data model (Paris al., 2021), demonstrat- ing the substantial and non-trivial effort required.
Despite the need, schema matching is a challenging ML task, as shown in Fig. 1, as without access to the raw data, schema matching methods must rely only on the attribute names and other metadata to infer correspondences between attributes across schemas. This requires reasoning about various chal- lenges, namely: Semantic heterogeneity: ambiguous potential mappings, where attributes across schemas might have the same name but different meanings, or different names but the same meaning. Structural heterogene- ity: schemas that have varied architectures, hicrarchies, and representational granularity. Database heterogeneity: differences in the number and organization of tables across schemas. e.g. source schema table information may be rep- resented across multiple target schema tables. Hence, it is non- con-triival to identify the appropriate table for an attribute. Information mismatch: Information may be contained in one schema, but not in another schema. Hence, reason- ing about "no possible match" " is as important as reasoning about possible match.
The importance and value of schema matching cannot be overstated, as integrating data from various data sources such as different regions, organizations vital in healthcare but also in finance and e-commerce (Sheetrit et al., 2024; Zhang et al., 2021; El Haddadi et al., 2024). Schema matching is also generally valuable to any- one working on ML, as a step toward increasing the training and validation data available to the ML community. e.g, in healthcare, integrating data from multiple hospitals can lead to more comprehensive datasets to train more performant ML prognostic models. Similarly, in e-commerce, combin- ing diverse customer data from various platforms can enable more accurate ML models built on customer data.
applications is
Unfortunately, prior ML approaches for "automated" schema matching often require extensive labeled data (Li al., 2020; Zhang et al., 2021), which is often costly and time-consuming acquire, making these methods imprac- tical for real-world use. Although LLM-based methods (Narayan et al., 2022; Mirchandani et al., 2023) have at-
These issues make schema matching challeng- ing task that cannot be solved by simple methods such as semantic similarity alone (see Fig. 2).
2

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
To this end, we intro- duce Matchmaker, self-improving com-
0.7
To reduce the need for labels, LLMs have been applied to schema matching (Zhang et al., 2023a; Narayan et al., 2022; Zhang et al., 2023b). However, methods like LLM-DP us- ing pre-trained LLMs (Zhang et al., 2023a; Narayan et al., 2022) have demonstrated poor zero-shot performance (see Sec. 5). Performance improvements were obtained with human-labeled examples of +:500 examples, from which in- context examples are selected. However, reliance on human labeling is often unrealistic, limiting applicability. Interest- ingly, even LLMs such as Jellyfish (Zhang et al., 2023b), which are fine-tuned for schema matching on task datasets, have shown poor matching performance. Beyond match- ing performance, both LLM and supervised methods (e.g. SMAT (Zhang et al., 2021)), formulate schema matching as binary classification task over the full Cartesian product of source and target schema attributes. e.g. for each pair of source- target attributes, the LLM is prompted to provide label of Yes/No for the match (i.e. Is attribute A related to Attribute B? yes/no). The result is poor scalability, which is computationally expensive for large schemas and costly due to the large number of LLM calls, hindering real-world applicability. We compare LLM calls in Appendix D.2.
0.8
Aceuracy
0.5
Matchmrker (Ours)
0.4
positional language
0.3
Semantic matching
model program for schema matching.
0:2
0.1
Top-k predictions
Matchmaker lever- ages the reasoning capabilities of LLMs via compositional language model
Figure 2: Result showing semantic similarity alone cannot solve schema matching, with low acc@k, compared to Matchmaker.
program with multi-stage LLM calls that comprise candidate generation, refinement, and confidence scoring (see Appendix for examples of this process). Matchmaker also self-improves without labeled data (zero-shot), via novel optimization process using synthetic in-context examples for the different stages of the language model program. Matchmaker makes the following contributions:
Contributions: ! We address recent calls to develop (Bal-
ML methods for data harmonizaiiovinteeoerbbiiii agopalan et al., 2024; Gilbert et al., 2024). ② We introduce a novel formulation of schema matching information retrieval rather than binary classification. Matchmaker, a novel compositional language model pro- gram to address the complexities of schema matching. 4 We introduce a scoring mechanism allowing for human-in. the-loop deferral not possible with prior methods. 5 We introduce novel optimization mechanism allowing Match- maker to self-improve in zero-shot manner via synthetic in-context examples that guide Matchmaker's 's reasoning process. ④ We empirically demonstrate that Matchmaker outperforms different schema matching baselines on real- world schema matching benchmarks, along with showing the value of our self-improvement mechanism and how Matchmaker can be used with a human-in human-in-the-e loop..
3 We propose
The closest work to ours is ReMatch (Sheetrit et al., 2024), which uses retrieval to find semantically similar candidate matches, thus reducing the search space. It then prompts an LLM to match a source schema attribute with retrieved target schema candidates. However, ReMatch relies solely on semantic matching, which we empirically demonstrate in Sec. 5 does not suffice for real-world schemas. Our ap- proach, Matchmaker, diverges from ReMatch along three dimensions (see Table 4): (1) System: ReMatch uses single LLM call, while Matchmaker decomposes the task into multi-stage compositional LLM program with itera- tive reasoning steps. (2) Candidate generation: ReMatch relies solely on semantic retrieval, while Matchmaker in- corporates diverse candidate generation sources, including retrieval for semantic candidates and an LLM-driven contex- tual reasoning candidates. (3) Optimization: ReMatch has fixed/static LLM prompt template, while Matchmaker is an LLM program where we dynamically optimize the prompts via synthetic in-context examples.
2. Related Work
This work engages with literature on schema matching (see Fig. 3) and contributes to data-centric AI.
Schema matching. Previous ML-based schema matching approaches have shown promise, but suffer from limita- tions that hinder their practical applicability. Early works (Mudgal et al., 2018; Shraga et al., 2020; Li et al., 2020) computed similarity scores between schemas (Do & Rahm, 2002; Gal, 2011), but focused on the simpler entity match- ing task (matching items within columns) rather than the more complex schema matching problem. Recent methods like SMAT (Zhang et al., 2021) use attention to tackle full schema matching. However, they require extensive labeled matches (over 50%), rendering them impractical for real- world settings where labeled data is scarce or expensive to obtain, often requiring domain experts to annotate.
Data- Centric AI. Data-centric AI is an area of growing importance in the ML community, aiming to systematically improve data quality for ML (Zha al., 2023; Whang et al., 2023; Seedat et al., 2023c) via methods such as sample selec- tion and valuation (Seedat et al., 2023b; 2022; 2023a; Jiang et al., 2023) of pre-existing integrated datasets. This work addresses a fundamental upstream problem: schema match- ing, which enables the creation of harmonized datasets. Con- sequently, schema matching is a contribution to data-centric Al by tackling a critical issue that precedes and supports existing approaches to enhance data quality for ML.
3

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
3. Schema Matching
Ast Ag and its contextual information dgs Dg. the rank- ing function assigns score to each candidate attribute Asj E C. S At and its contextual information dej E Dz:
3.1. Preliminaries.
Consider the schema matching task, where the goal is to map attributes from source schema (S,) to a target schema (S,). Each schema S is defined
r((Asis (a)), (Atj, > r((Asi, dai), (Ark des)) + Aej is more relevant to Asi than Ark-
collection of tables T (Ti; T2, Tm. Rach table T; contains set of attributes A, = {An, A ). Additionally, each ta- ble 73 is associated with metadata m, describing the purpose and content of the table. Similarly, each attribute Ai is as- sociated with a description dıy. which includes information describing the attribute, its data type and relational context offering key contextual information to aid in the matching process. The schema matching task, defined below, aims to find matches between attributes across different schemas, accounting for structural hierarchies, interrelationships and constraints. Recall that schema matching operates solely on schema-level information (attributes and metadata), without having access to the raw data. This adds to the complex- ity, as matching must be performed without the benefit of analyzing the actual data values.
The mapping function f can then be defined as follows:
arg max AsyECi r((A,,dai), (Ars, dej)); f(A,i) if max r((Asi, dsi), (Atz, 2 T: 20 otherwise
where is relevance threshold and f assigns the query attribute Asi to the candidate attribute Ar with the highest relevance score. Conversely, we may assign 25, indicating no match - accounting for the fact that not all source attributes may have a possible match in the target schema. Further details can be found in Appendix A.4
4. Matchmaker
Definition 1 (Schema Matching). The goal of schema
matching is to find mapping function f A. + A, U {Ø) that correctly assigns each attribute of the source schema Sy to a corresponding attribute in the target schema St or to the empty set Ø, indicating no possible match.
We propose Matchmaker, a self-improving compositional language model (LM) program for schema matching (see Fig. 3), defined as a three-step LM program. For further details, see Appendix A.9.
1. Multi-vector documents (Sec. 4.1): Creation of multi- vector documents from the target schema facilitate re- trieval of candidate target attribute matches.
3.2. Schema matching as information retrieval.
As outlined in Sec. 2. schema matching is often formulated as a supervised binary classification problem (match/no match) over the entire Cartesian product of source and tar- get schema attributes. Beyond the computational side, this formulation has several drawbacks: Labeling Cost: It requires manual annotation of attribute pairs by domain experts, which is time-conee consuming and costly. Class Im- balance: The prevalence of non-matching attribute pairs significantly outnumbers matching pairs, resulting in severe class imbalance. Lack of Ranking: does not yield a ranked list of candidate matches, which is critical for human review if multiple possible matches exist.
2. Candidate generation (Sec. 4.2): Employing two types of candidate generation: semantic retrieval and reasoning- based - later refined into a smaller candidate set to evaluate. 3. Confidence scoring (Sec. 4.3): match confidence of a candidate target attribute to query attribute.
define the unoptimized Matchmaker program Finally, a key aspect of Matchmaker is our zero-shot op- timization via synthetic in-context examples to improve performance (Sect. 4.4).
& Steps 1-3
Why LLMs for schema matching? LLMs form the foun- dation of Matchmaker, within compositional program comprised of multiple language model calls. Specifically, LLMs exhibit several appealing properties and capabili- ties for schema matching: Contextual understanding: LLMs have been pretrained on vast corpora of information, equipping them with extensive prior knowledge spanning different contexts and settings (Chowdhery et al., 2022; Singhal et al., 2023). This contextual understanding en- ables LLMs to effectively reason about schema hierarchies and identify potential matches. > Hypothesis proposers: LLMs have been shown to be "phenomenal hypothesis pro- posers" (Qiu et al., 2023; Rauba et al., 2024), making them
1. Candidate generation: For each source query at- tribute Agi E A, from the source schema S., we generate a set of potential matches from the target schema S.. Let C S Ad be the set of candidate target matches for query attribute As:- The candidate generation process defined as function g : A, Ar + P(Ar), where ((Ar) denotes the power set of A,, such that C= = A(ss,,)))
2. Ranking: We rank the candidates based on their relevance to the query attribute. We define a ranking func- tion r (A, x D.) x (A, x D2) + R, where D, and D, represent the contextual information associated with at- tributes in A, and At, respectively. For each source attribute

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
(A)
(B)
(C)
Target Attribute
(Quary)
Target Description
Prompt Asl (rars)) (seed: As See and Tal inlract Noarllo
LLM (Prozery' Finetuned)
Retrieve
Prompt Kwich soura
Matchee "Makkh
Retated (1/0)
candidates
Target
atribun with
LLM [Frozon)
Demiiiida langei aitrbstts
Atribute
Dense Vector
"Machh
Source Desorption
CB (Target)
Step
Slep 2: Candidate Generation
Slep 3:
Multi-vactor Documents
Confidence Scoring
Targela (AID
Source Attribute
LLM Candidare Ressoner (Frazen)
Reasoning based candidates
MCQ
Matches
(Quary)
(A) porson-id id
LLM Confidence soorer
Confidence
[A] 100
Semantic candidates
(Frozon)
age-id
(Pronan)
(0)
(C) 30
Refined candidarte
Mulli-Vector (Targel)
(Leelhoood)
documents
Torget
Compositional Language Model (LM) Program (Steps 1-3)
Language Model Program optimization with synthetic in-context examples. Selection of input-output traces using an LLM Evaluator
(D) Matchmaker (Ours)
LLM
Evaluator
Figure 3: Conceptual comparison of different schema matching approaches. (A) Supervised Matching (Zhang et al., 2021) employs a trained neural network (e.g., a transformer) to predict binary match/no-match labels across all attribute pairs, scaling as (n)) and requiring labeled data, thus unsuitable for zero-shot. (B) LLM LLoo--poppiing (Narayan et al., 2022; Zhang et al., 2023a) uses a frozen language model (e.g., GPT-4) for the same task, with similar scalability. Alternatively, (Zhang al., 2023b) fine-tunes the LLM, which requires labeled data. (C) RAG-Based (Sheetrit al., 2024) improves scalability by retrieving candidates from vector database and using frozen LLM select matches, but its effectiveness is limited to semantically similar options. (D) Matchmaker (Ours) performs schema matching via self-improving, compositional language model program that enables enhanced reasoning. The program includes both retrieval and reasoning- based candidate generation with refinement and confidence scoring, allowing for more accurate ranking. The program is optimized using synthetic in-context examples in the LLM prompts.
particularly useful for candidate generation tasks. Capa- ble rankers: LLMs have been shown to be highly capable at relevance ranking; assessing the suitability of candidates given a query and a set of options (Zhuang et al., 2023; Hou et al., 2024), especially "when ranking candidates retrieved by multiple candidate generators" (Hou al., 2024).
ument for each table consisting of the attribute names and the attribute's 's description and data type. The metadata of each document includes the description of the table itself.
Unlike conventional approaches that encode each document as single high-dimensional vector, Matchmaker utilizes multi-vector representations. Specifically, we use ColBERT- v2 (Santhanam et al., 2022) to encode document chunks, producing embedding per token (i.e. token-level dense vector). capturing token-level interactions shown to im- prove expressivity (Thakur et al., 2021; Lee et al., 2024) and out- oft-of-domain performance (Santhanam et al., 2022). Next, we explain how semantically similar candidates are retrieved using this multi- representation.
Defining compositional LM program. A composi- tional language model program, denoted L, is multi- stage pipeline consisting of multiple LLM calls, i.e., L (1,,2,,..,, ...)). where 1, : (s, k.) + y represents a specific LLM call taking as input a prompt string s and in-context examples k, (which could be Ø). Secs. 4.144.3) defines the different components of L specific to Matchmaker. Finally,
Sec. 4.4 describes optimization process.
4.2. Diverse candidate generation (Step 2)
4.1. Multi-vector documents (Step 1)
To narrow down the search space, Matchmaker identifies a subset of candidate attributes from the target schema that are likely matches for query attribute & E A, from the source schema. We draw inspiration from (Hou et al., 2024), which demonstrates that LLM ranking performance im- proves "when ranking candidates are retrieved by multiple
To efficiently retrieve semantically similar candidates from the target schema, we build vector database that encodes target schema attributes. We begin by representing the target schema as collection of structured documents. Specifically, for each table T' in the target schema S,, we create a doc-
5

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
candidate generators." " Hence, while semantic candidates commonly used, Matchmaker goes beyond and employs two distinct types of candidate generation: (i) Semantic retrieval candidates retrieved from the vector database, and (ii) Reasoning- based candidates using a language model. This is then followed by a candidate refinement step. We outline each type of candidate generation applicable to a given query attribute q A..
date scoring task as an MCQ format, labeling each candidate in C* for query qi as options (A), (B), (C), etc. Additionally, to account for none of the target candidates being good match or there might be no possible match in the target schema, Matchmaker includes an abstain option by adding "NONE of the above" as a choice. This ensures that the LLM is not forced to select a candidate when there is no suitable match (Ren et al., 2023; Ding et al.. 2023).
(i) Semantic retrieval candidates. Given query 4; we code it using CoIBERT-V2, producing a multi-vector query embedding. Matchmaker then uses this query embedding to retrieve the top-k matching target schema attributes in the vector database. The top-k semantically similar can- didates are denoted as Cg. Similarity is computed using a inte-interaction approach (Khattab & Zaharia, 2020), though a Maxsim operator which identifies the highest similarity scores for the query tokens, and these scores are aggregated to generate relevance score for that document. The top-k documents, which contain the most semantically similar attributes to the query, are retrieved as matches.
Matchmaker finally performs candidate ranking, where it is common to evaluate each candidate individually (Hu et al., 2024; Wang et al., 2023a: Zheng et al., 2023). Confidence scores are obtained by prompting the LLM to reason about the relevance of each candidate e; E C to the given query q, Furthermore, prior work has shown that L.LMs can provide good uncertainty token-level (Kadavath et al., 2022) like in our MCQ, which is achievable via prompting (Tian et al., 2023). Consequently, Matchmaker elicits a confidence score by prompting the LLM to provide value between 0 and 100, indicating the relevance of a match. The confidence scores are used to rerank candidates or, if "None of the above" receives the highest score, return an empty list (i.e. no suitable matches for the query).
(ii) Reasoning-based candidates. To complement se- mantic matches, Matchmaker generates reasoning-based candidates using a candidate reasoner LLM denoted as lo: A,) + CR. where q: is the i-th query, A, is the set of all target attributes and CR is reasoning-base candidate set. Matchmaker employs Chain of Thought (CoT) prompt- ing (Wei et al., 2022) to reason about the target attributes Ar given the context of the schema hierarchy, descriptions and data types - generating the most likely and relevant target schema candidate matches for each query g-- This metadata allows the LLM to reason about the schema structure and relationships beyond just attribute names.
4.4. Self-improvemmnt:: Zero-shot optimization w/ synthetic in-context examples
Matchmaker optimizes the language model program C by leveraging the few-shot learning capabilities of LLMs (Brown et al., 2020; Agarwal et al., 2024; Dong et al., 2022). This is achieved by selecting input-output demonstrations (i.e. in-context examples). In Sec. 5, we contrast this with an alternative self- iel-oimproveeent method via reffecefl However, selecting in-context examples is non-trivial for schema matching for two reasons.
Refinement. At this stage, the set of candidates is C = CR UC,. Matchmaker then refines this set by selecting the most relevant candidates for each query attribute, resulting in smaller, prioritized candidate set C* to score and rank. Candidate refinement is achieved with a refiner LLM using CoT, denoted as 4, + where 8 = (C,
(i) Lack of labeled demonstrations: We do not have ac- cess to labeled input-output demonstrations from which to select in-context examples. To overcome this challenge, we use the unlabeled schemas to create a "evaluation" " set D Deval = {e1, ,,, ). made up of different types of source queries. Specifically, we identify "easy queries" where the top-n (n=5) target schema semantic matches have similarity score > 0.95, and "challenging queries" with the lowest semantic matches.
4.3. Confidence scoring (Step 3)
The refined set of candidates, C* remains unordered. Hence, this step aims to obtain confidence scores to rank the candi- dates but also gauge the certainty of each match, recognizing that sometimes no suitable source-to-targee ttrrrett attribute match exists, which requires the system to abstain. While language models may not be well-calibrated at the sequence level, re- cent research has shown that they exhibit better calibration at the token level (Ren et al., 2023), feature notably bene- ficial in multiple-choice question (MCQ) tasks (Kadavath et al., 2022).
(ii) Lack of an evaluator: To assess Matchmaker's 's capa- bilities on the evaluation set and guide the optimization process, we need a validation metric. Since no validator is readily available, we propose to use an evaluator LLM, È (e,, C(e:)) + E. that employs chain of thought (Wei et al., 2022) to score the relevance (from 0-5) of matches obtained from L when evaluated on examples from Devat-
Leveraging this insight, Matchmaker structures the candi-
6

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Algorithm 1 Optimize LM program C
beled" when they are optimized? The synthetic examples generated remain "unlabeled" in a traditional supervised sense, as we never explicitly verify or label them via hu- man annotations. Instead, verification is implicitly done via an LLM evaluator, which assesses the quality of the matches through a scoring system (scale of 0-5). i.c., if the synthetic examples are good, they would naturally lead to good downstream performance, which will be rated highly, Thus, synthetic examples are optimized based on evaluator scoring rather than explicit human labeling. This approach deliberately removes the requirement for manual annotation and supports fully autonomous zero-shot self-improveemnn.
0: Input: Set of evaluation
queries
D
Denat
e1. en,...,en
0: Output: Set of top n demonstrations Ddemo
0: for each input e E Denaa do
0: Do. trace, + [(e,) (Teacher & predicts, storing
outputs and intermediate traces)
0:
8; xo--66 E(es. Ú.) { Evaluation score)
0:
Ddemo Deemo (ee, brace;, Di, 81)
0: end for
0: Sort Ddemo by score
0: return Ddemo (0: n) (Select top n) =0
5. Experiments
Zero-shot optimization w/ synthetic in-context examples. To optimize our multi-stage language model program, we aim to select in-context examples for each component in C. However, in-context demonstrations for the intermediate stages are typically unavailable.
We now empirically investigate multiple aspects of Match- maker ? For qualitative examples that illustrate Match- maker's application, refer to Appendix C.
Setup. We conduct experiments on the MIMIC-OMOP and Synthea-OMOP datasets, which are the standard benchmark datasets used in prior schema matching works (Sheetrit et al., 2024; Zhang et al., 2023b; Narayan et al., 2022: Zhang et al., 2023a; 2021). These datasets are real-world healthcare schema matching datasets and have been widely adopted due to their complexity and their reflection of real- wolld schema matching challenges. Additionally, complex, real-worke schema matching datasets are rare and difficult to obtain, as annotating them requires extensive domain expertise (e.g.. 500 hours for OMOP), making them invaluable test beds for schema matching. Further experimental details and overview of the datasets is provided in Appendix B.
To address this, we simulate fraces by running L on the evaluation examples e, E Deval. A trace captures the inter- mediate input-output pairs of each component in C during the execution of L on a given example. The evaluator& & then scores the final output, assessing Matchmaker's (C) overall performance on each example. We then adopt a bootstrap- ping process (Khattab et al., 2023) that selects the interme- diate input-output pairs from the traces that produced the highest evaluation scores as synthetic in-context examples for each component of L. In other words, we use the input- output pairs generated by Matchmaker itself (which resulted in good evaluation performance) as synthetic in in-context examples to guide the LLM reasoning. This allows us to improve the program in a zero-shot manner, without relying on actual labeled data. Algorithm 2 provides an overview of the process. We refer to L with the systematically selected in-context examples as Matchmaker (Optimized).
experime
Metrics. We evaluate schema matching performance using accuracy k used in (Sheetrit et al., 2024) and is commonly used in information retrieval. Besides, ReMatch, the other baselines treat schema matching as a binary classification using F1 Fi-score as the metric. our setting of m: matching (i.e. one match for each query), accuracy @ is equivalent to Fi-score, precision and recall, if the label is assigned via argmaz. For details see Appendix A.8. Hence, we report accuracy @ for all other baselines for comparison to retrieval- based approaches. Unless otherwise stated, metrics are averaged over five seeds (with standard deviation).
Answering FAQs about self-improvemmen. Before diving into experimental validation, we answer two FAQs about our self improveett mechanism.
Q1: How does Matchmaker's 's initial round work? In the initial round, Matchmaker operates without in-context ex- amples. We first run the unoptimized Matchmaker (with- out any in-context examples) on evaluation examples. We then capture execution traces (intermediate inputs/outputs), which are scored by the LLM evaluator. Finally, the highest- scoring traces (and their input-outputs) are used bootstrap synthetic in-context examples. i.e. Matchmaker "starts cold" with a zero-shot bootstrapping process (using its own suc- cessful traces). This allows Matchmaker to self-improve without requiring labeled data, addressing a key challenge in schema matching.
5.1. Schema Matching performance: Does it work?
We compare Matchmaker's performance to diverse schema- matching baselines (refer to See. 2). These include (i) LLM- based methods such as ReMatch and L.MM-DP, (ii) the state- of-the-art non-LLM supervised model. SMAT, and (ii) Jelly- fish, LLM specifically fine-tuned for data preprocessing, including schema matching. While Jellyfish is fine-tuned
Q2: Are the synthetic examples still "unlabeled" or "la-
7

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Table I: Comparison of schema matching performance of different baselines.
Matchmaker
ReMatch
JellyFish- 13b
Jellyfish-Th
L.MM-DP
SMAT (20-80)
SMAT (50-50)
MIMC
acc 6f
62.20 + 2,40
42.50
15.36 + 5.00
14.25
+ 3.00
29.59 + 2,00
6.05 + 5.00
10.85 + 6.00
@
68.80 + 2.00
63.80
N.A.
N.A.
N.A.
N.A.
N.A.
acc 5
71.10 + 2.00
72,90
N.A.
N.A.
N.A.
N.A.
N.A.
Synthee
acc
70.20 1.70
50.50
35.17 3,90
31.52 + 1.70
41.4 + 5.40
36.23 3.30
44.88
2.60
acc 3
78.60 2.50
58.10
N.A.
N.A.
N.A.
N.A.
N.A.
6
80,90 + 1.10
74.30
N.A.
N.A.
N.A.
N.A.
N.A.
using the same MIMIC and Synthea datasets, giving it an advantage, we include it as a baseline to highlight Match- maker's zero-shot performance using a general-purpose LLM. This comparison spans general-purpon LLMs, tradi- tional supervised approaches, and task-specific fine-tuned models. All L.M baselines use GPT-4 (0613) (OpenAT, 2023) as the backbone for fair comparison to the original works and to isolate the gains of the system not tied to the L.M. Other LLM backbone results are found in Appendix D, showing Matchmaker's gain isn't due the LLM alone.
dom selection of in-context examples rather than our op- timized/systematic selection of in-context examples and (3) Matchmaker Self-RRffeection) which employs self- reflection self-refinement mechanism (Pan et al., 2023; Madaan et al., 2024) an alternative self-improvement approach. i.e., the LLM iteratively self-corrects through feedback and has been used for various LLM tasks to im- prove performance.
The results in Table illustrate the following: Match- maker (Optimized) achieves significant performance gains compared to Matchmaker (No-IC), particularly at low k values (+-5% improvement for @cc@ 1). This highlights the value of the synthetic in-context examples and the poten- tial for zero-shot self-improvement, even in the absence of labeled data or well-defined evaluation metrics. Match- maker (Optimized) outperforms Matchmaker (Random), confirming that our systematic selection of in-context sam- ples is the key driver of performance gains, rather than the mere inclusion of any in-context examples. Match- maker (Optimized), which uses an LLM evaluator to score demonstrations directly, provides better performance gains compared to the LLM self-reflcction approach. This under- scores the importance of input-output demonstrations for Matchmaker, especially considering the multi-stage nature of the program, where the outputs of earlier components affect later components.
Matchmaker has the best overall performance. Match- maker consistently outperforms baselines across all settings, see Table 1. Importantly, we find the largest performance gains (+-20%) for accuracy @1 1. This is a desirable property, as it suggests a better ranking of matches. i.c. higher accu- racy at low Io values enables the use of smaller prediction sets, reducing the human effort required to select the final best target attribute match for given source attribute query.
Formulation as information retrieval outperforms bi- nary classification. A key insight from our experiments is that information retrieval-based approaches (Match- maker and ReMatch) perform substantially better for ac- curacy@1 @1 compared to the other binary classification-basse approaches, which evaluate the full Cartesian product of attributes. This performance gap be attributed to the smaller search space of the information retrieval formula- tion. Notably, Matchmaker and ReMatch are evaluated on all mappings, including matches and nulls ("No possible match"), "), whereas binary classification methods consider a simpler problem by only evaluating true matches.
Table 2: Comparison of different Matchmaker seff-improvement mechanisms. We see the value of our systematic selection of in- context samples alternative mechanisms.
Matchmaker (Optimized Full)
Matchmuker (Random)
Matchmaker (No-IC)
Matchmuker
Remark on source of gain. We ablate the different candidate generation mechanisms in Appendix D.I. The results high- light the value of diverse candidate generation mechanisms to enhance Matchmaker's 's overall performance.
(Self-reteccion)
Symthed MIMIC
62.20 2.40
55.36
57.90
1.20
57.10 0.60
uuc 3
68.80 2.00
62.74 1 4,50
66.40 1 0.60
66.60
1.00
ace 5
71.10 : 2.00
65.00 : 6.42
70.20 + 0.70
70.60 = 0.50
70.20 1.70
67.76 + 1.38
65.40 +: 0.90
67.80
1.40
acc 3
78.60 2.50
76.19 1: 5.28
78.20 - 0.60
75.90
0.70
aocd8 5
80.90 1.10
77.66 + 83.20 + 1.10
81.10 1.90
5.2. Matchmaker self-improvent analysis
Matchmaker self-improves its language model program in a zero-shot manner (no labeled examples) via an optimiza- tion process using synthetic in-context examples (Sec. 4.4). We evaluate the performance of Matchmaker (Optimized) against three alternatives to disentangle the value of our in-context example selection mechanism: (1) Matchmaker (No-IC), which is the vanilla language model program with- out in-context examples, (2) Matchmaker (Random): ran-
5.3. Matchmaker in practice: Human-in-the-lopp deferral and remedial action.
How might we use Matchmaker in practice for schema matching? Let us examine two cases.
RReMatch code is not available, hence we report the best accu- racy @k values with the retrieval step in (Sheetrit et al., 2024). Appendix D shows results for a re-implementation ReMatch.
8

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
(I) Matchmaker with human-in-the-loop deferral: We evaluate the effectiveness of integrating Matchmaker with human-in-the-loop approach by deferring uncertain matches to human experts (i.e., an oracle) for correction. High- uncertainty cases are identified using the entropy of Match- maker's confidence scores, with the most challenging matches (those with the highest entropy) deferred to the oracle. We evaluate different deferral percentages p E {0, 10, 20, 30, 40, 50) and observe that entropy- defer- ral consistently yields greater performance gains compared to random deferral, as shown in Fig. 4(a). This finding highlights the practical value of Matchmaker in real-world settings, where based on entropy, one could strategically seek human oversight for challenging matches and improve overall schema matching performance. The appropriate deferral percentage, however, depends on context-specific factors such as human bandwidth and expert availability.
matching task, we also conduct further evaluation assess Matchmaker's s generalizability to other domains. Specifi- cally, we evaluate Matchmaker on Magellan (e-commerce) and WikiData (general knowledge base). The results in Ta- ble 3 show that Matchmaker achieves superior performance, confirming its generalizability across domains.
Table 3: Performance comparison between Matchmaker and ReMatch across datasets.
Dataset
Matchmaker
ReMatch
Wikidata
0.95 + 0.04
0.84 + 0.03
Magallen
1.00 + 0.00
1.00 + 0.00
However, these datasets represent significantly less challeng- ing matching scenarios compared to our healthcare schemas. This is evidenced by the relatively high performance across all methods. The reason is that these tasks typically involve single-table table schemas with fewer columns and focus on direct Peature-to-feature matching. In contrast, the healthcare tasks require multi-step reasoning, first over dozens of source tables and then hundreds of attributes. i.e. first identify- ing the relevant target table, then performing column- matching. These findings reinforce our focus on health- care schemas, which present more challenging real-world matching scenarios that better differentiate the capabilities of advanced matching techniques.
(2) Evaluating ease of remedial action based on the simi- larity between incorrect predictions and true target at- tributes: Not all errors in source-target matching are equal: some might be easier to rectify than others. We hypothe- size that errors involving semantically similar attributes are easier to correct compared to those involving completely dis- similar attributes. We analyze the cosine similarity between incorrectly predicted attributes and their true target attributes using Pubmed- Rabbed-Bert&oe embeddings. To simulate post-hoc re- medial action, we assess the performance gains achieved by correcting erroneous predictions that exceed different similarity thresholds. Figure 4(b) shows substantial im- provements in accuracy @1 when "fixing" errors, with high semantic similarity between the erroncous prediction and true attribute (e.g., cosine similarity 2 0.8). These results suggest that Matchmaker's 's incorrect predictions are often semantically close to the true attributes (i.e. our errors are not far off), making them more amenable to post-hoc re- medial actions. This demonstrates the viability of post-hoc remedial actions to improve schema matching performance. Further error analysis can be found in Appendix D.
6. Discussion
Matchmaker introduces a novel approach to schema match- ing, using a self-improving compositional LLM program. Its superior performance compared to existing MI.-based approaches underlines its potential accelerate data inte- gration for ML-ready data. Matchmaker's zero-shot self- improvement mechanism, using synthetic in-context exam- ples, showcases the potential of using LLMs to handle com- plex reasoning tasks without relying on labeled data.
Limitations and opportunities. (1) Matchmaker, while effective in schema matching, represents just one compo- nent of the broader data harmonization process and needs to be integrated with other tasks to generate ML-ready data, (2) Despite its advantages over alternative ML-based ap- proaches, Matchmaker is not panacea and does not achieve perfect automation. It is best used with a human-in-the-loop (Sec. 5.3) to ensure reliability in real-world settings.
Eateope adenaal
Randoon dekanal
0.10
830
0.00
(8)
45
0%
Seasande similaitty jesdieteditne lddel
HEU SE Sia MEE 91. 98a I7 WORLD
Figure 4: Examples of Matchmaker in practice. (a) Deferring uncertain samples to humans via entropy deferral improves schema matching performance. (b) Performance gains are obtained when correcting errors semantically similar to the true attribute.
Acknowledgements
The authors are grateful to Fergus Imrie, Nicolás Astorga, Julianna Piskorz, Andrew Rashbass, and the anonymous ICML reviewers for their useful comments and feedback. NS is supported by the Cystic Fibrosis Trust. This work was supported by Microsoft's 's Accelerate Foundation Models Academic Research initiative.
5.4. Additional datasets beyond healthcare
While we primarily evaluate Matchmaker's capabilities on healthcare datasets due to the complexity of the schema

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Impact Statement
Advances in neural information processing systems, 33: 1877-1901, 2020.
Schema matching is essential for data integration. By au- tomating and accelerating the harmonization of heteroge- neous datasets, it enables the creation of larger, more com- prehensive training datasets that can lead to more robust and generalizable ML models. Matchmaker accelerates this process through a self-improving compositional language model, reducing manual effort and improving scalability. Its adoption can enhance decision- decision-making across domains such as healthcare, finance, and e-commerce by providing more representative datasets. However, reliance on automated schema matching comes with risks, including potential er- rors and data inconsistencies. To mitigate these, Match- framework
Chen, W. Large language models are few (1)-shot table rea- soners. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1120- 1130, 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111, 2022.
Colubri, A., Hartley, M.-A., Siakor, M., Wolfman, V., Felix, A., Sesay, T., Shaffer, J. G., Garry, R. F,, Grant, D. S., Levine, A. C.,, et al. Machine-learning prognostic models from the 2014-16 ebola outbreak: data nata-harmmnization challenges, validation strategies, and mhealth applica- tions. EClinicalMedicine, 11:54 64, 2019.
maker should be used in human-in-the-loe with rigorous validation and quality control. By striking this balance, Matchmaker has the potential to drive more reliable AI applications and accelerate data interoperability and integration in different fields.
Ding, W., Feng, S., Liu, Y., Tan, Z., Balachandran, V., He, T., and Tsvetkov, Y. Knowledge crosswords: Geometric reasoning over structured knowledge with large language models. arXiv preprint arXiv:2310.01290, 2023.
Beyond addressing the interoperability challenge, Match- maker also introduces fundamental advancements that ar broadly applicable to compositional LLM programs. In particular, the self-improvement mechanism is broadly ap- plicable beyond the schema matching setting and could be used more generally improve LLM system capabilities.
Do, H. H. and Rahm, E. Coma system for flexible com- bination of schema matching approaches. In Very Large Data Bases Conference, 2002. URL https: /ppi. s semant i wecccccchaaarr...
References
Dong, Q.. Li, L.., Dai. D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui. Z. A survey on in-context learning. arXiv preprint 2XXiv:331.000244 2022.
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S., Anand, A., Abbas. Z., Nova, A., Co-Reyes, J. D., Chu, E., al. Many-shot in-context learning. arXiv preprint MxXiv:2404.11018 2024.
Haddadi, O., Chevalier, M., Dousset, B., El Allaoui, A., El Haddadi, A., and Teste, O. Overview on data ingestion and schema matching. Data and Meladata, 3:219-219, 2024.
Avati, A., Seneviratne, M., Xue, Y., Xu, Z., Lakshmi- Behavior
narayanan, B., and Dai, A. M. of ehr-models under distributional shift-a benchmark. In NeurlPS 2021 Workshop on Distribution Shifts: Connect- ing Methods and Applications, 2021.
Gal, A. Uncertain schema matching: the power of not knowing. In International Conference on Information and Knowledge Management, 2011. URL https: //api
Balagopalan, A., Baldini, I., Celi, I.. A., Gichoya, J., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., and Wagstaff, K. L. Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact. PLOS Digital Health, 344):e0000474, 2024.
semanticscholes wttp/oorppss..::333422...
Giglou, H. B., D'Souza, J., and Auer, S. Llms4om: Match- ing ontologies with large language models. arXiv preprint arXiv:2404. 10317, 2024.
Gilbert, S., Kather, J. N., and Hogan, A. Augmented non- hallucinating large language models as medical informa- tion curators. NPJ Digital Medicine, 7(1):100, 2024.
Balch, J. A., Ruppert, M. M., Loftus, T. J., Guan, Z., Ren, Y., Upchurch, G. R., Ozrazgat- Baslanni, T., Rashidi, P., and Bihorac, A. Machine learning -nabled clinical in- formation systems using fast healthcare interoperability resources data standards: scoping review. JMIR Medical Informatics, 1 11:44227,, 2023.
Goetz, I.., Seedat, N., Vandersluis, R., and van der Schaar, M. Generalization--a key challenge for responsible ai in fatient-facing clinical applications. npj Digital Medicine, 7(1): 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.
Gupta, N., Patel, H., Afzal, S., Panwar, N., Mittal, R. S., Guttula, S., Jain, A., Nagalapatti, I.., Mehta, S., Hans, S.,
10

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
et al. Data quality toolkit: Automatic assessment of data quality and remediation for machine learning datasets. arXiv preprint arXiv:2108.05953 2021.
Kong, K., Zhang, J., Shen, Z.., Srinivasan, B., Lei, C.,, Falout- sos, C., Rangwala, H., and Karypis, G. Opentab: Advanc- ing large language models as open-domain table reason- ers. In The Twelfth International Conference on Learning Representations, 2023.
Hertling, S. and Paulheim, H. Olala: Ontology match- ing with large language models. In Proceedings of the 12th Knowledge Capture Conference 2023, pp. 131-139, 2023.
Lee, J., Dai, Z.. Duddu, S. M. K... Lei, T., Naim, I., Chang, M.-W., and Zhao, V. Rethinking the role of token retrieval in multi-vector retrieval. Advances in Neural Information Processing Systems, 36, 2024.
Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., and Zhao, W. X. Large language models are zero-shot rankers for recommender systems. In European Conference Information Retrieval, pp. 364-381. Springer, 2024.
Lehne, M., Sass, J., Essenwanger, A., Schepers, J., and Thun, S. Why digital medicine depends on interoperabil- ity. NPJ Digital Medicine, 2:79-79, 2019.
Hu, C., Ge, Y., Ma. X., Cao, H., Li, Q. Yang, Y., Xiao, T., and Zhu, J. Rankprompt: Step-by-step comparisons make language models better reasoners. arXiv preprint arXiv:2403. 12373, 2024.
Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W. C. Deep entity matching with pre-trained language mod- els. Proceedings of the VLDB Endowment, 14:50
60, 2020. URL https: / api. semanticscholar. org hes//CorpussD. : 214743379.
Jain, A., Patel, H., Nagalapatti, L.. Gupta, N., Mchta, S., Guttula, S., Mujumdar, S., Afzal, S., Sharma Mittal, R., and Munigala, V. Overview and importance of data qual- ity for machine learning tasks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3561-3562, 2020.
Liu, Y., Pena, E., Santos, A., Wu, E., and Freire, J. Magneto: Combining small and large language models for schema matching. arXiv preprint arXiv:2412.08194 2024.
Lu, W., Zhang, J., Zhang. J., and Chen, Y. Large language model for table processing: A survey. arXiv preprint arXiv:2402.05122 2024.
Jiang, K., Liang, W., Zou, J. Y., and Kwon, Y. Opendataval: unified benchmark for data valuation. Advances in Neural Information Processing Systems, 36, 2023.
Madaan. A., Tandon, N., Gupta, P., Hallinan, S., Gao, I.., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Johnson, A. E., Pollard, T. J., Shen, L.,, Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., An- thony Celi, L., and Mark, R. G. Mimic-iii, a freely ac- cessible critical care database. Scientific data, 3(1):1-9, 2016.
Mirchandani, S., Xia, F., Florence, P. R., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. ArXiv, abs/2307.04721, 2023. URL https:/ www.dessmmattsschho.o.o.o org/ (ooppssI::2550011100
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., seifield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know. arXiv preprint arXivv2207.05221, 2022.
Mudgal, S., Li. H., Rekatsinas, T., Doan, A., Park, Y., Kr- ishnan, G., Deep, R., Arcaute, E., and Raghavendra, V. Deep learning for entity matching: A design space explo- ration. Proceedings of the 2018 International Conference on Management of Data, 2018. URL https: /aapi. semanticscholie o CorpusID:4406333
Khattab, O. and Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development Information Reirieval, pp. 39-48, 2020.
Nahid, M. M. H. and Rafeii, D. Tabsqlify: Enhancing rea- soning capabilities of llms through table decomposition. arXiv preprint arXiv:2404. 10/50, 2024.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang. Z., San- thanam, K., Haq. S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., et al. Dspy: Compiling declarative lan- guage model calls into state-of of-te-of-the-ar pipelines. In The Twelfth International Conference on Learning Represen- tations, 2023.
Narayan, A., Chami, I., Orr, L. J., and R'e, C. Can foundation models wrangle your data? Proc. VLDB Endow., 16:738 7438,746,4 2022. URL https:
//api. seeei.ssmatticccchoee wwt./oor////oooo 248965029.

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Seedat, N., Aharonson, V., and Hamzany, Y. Automated and interpretable m-health discrimination of vocal cord pathology enabled by machine learning. In 2020 IEEE Sisia-accffic Conference on Computer Science and Data Engineering (CSDE), pp. 1 1-6. IEEE, 2020.
Pan, L., Saxon, M., Xu, W. Nathani, D., Wang, X., and Wang, W. Y. Automatically correcting large lan- guage models: Surveying the landscape of diverse self- correction strategies. arXiv preprint arXiv:2308.03.1 1888 2023.
Seedat, N.. Crabbé, J., Bica, I., and van der Schaar, M. Data-iq: Characterizing subgroups with heterogeneous outcomes in tabular data. Advances in Neural Information Processing Systems, 35:23660 -3674,, 2022.
mation and evaluation of the mimic database in the omop common data model: Development and usability study. JMIR Medical Informatics, 9, 2021. URL http::///aap.smma.ccssss... .ee.ntticccooll................. Paris, N., Lamer, A., and Parrot, A. Transfor-
Seedat, N., Crabbé, J., Qian, Z., and van der Schaar, M. Triage: Characterizing and auditing training data for im- proved regression. Advances in Neural Information Pro- cessing Systems, 36:74995-75008, 2023a.
Qiu, L., Jiang, I.., Lu, X., Sclar, M., Pyatkin, V., Bhaga- vatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., et al. Phenomenal yet puzzling: Testing inductive reasoning ca- pabilities of language models with hypothesis refinement. arXiv preprint vrXiv:2310.08550 2023.
Seedat, N., Imrie, F., and van der Schaar, M. Dissecting sam- ple hardness: Fine-grained analysis of hardness character- ization methods. In The Twelfth International Conference on Learning Representations, 2023b.
Seedat, N., Imrie, F., and van der Schaar, M. Navigat- ing data-centric artificial intelligence with de-check: Ad- vances, challenges, and opportunities. IEEE Transactions on Artificial Intelligence, 2023c.
Rahm, E. and Bernstein, P. A. A survey of approaches to automatic schema matching. the VLDB Journal, 10: 334-350.22 2001.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj. N., Hardt, M., Liu, P. J., Liu, X., Marcus. J., Sun, M., et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine, 1(1):1-10, 2018.
Sheetrit, E., Brief, M., Mishaeli, M., and Elisha, O. Re- match: Retrieval enhanced schema matching with Ilms. arXiv preprint arXiv:2403.0 2024.
Shraga, R., Gal, A., and Roitman, H. Adnev: Cross-domain schema matching using deep similarity matrix adjustment and evaluation. Proc. VLDB Endow. 13:1401-1415, 2020. URL https: / org/ /ts//CoopusDD:215554444
Rauba, P., Seedat, N., Ruiz Luyten, M., and van der Schaar, M. Context-aware testing: A new paradigm for model testing with large language models. Advances in Neu- ral Information Processing Systems, 37:1 37:112205-112553, 2024.
/tttps://apia /api..ppp..// semanticscholar.
Si, Y., Du, J., Li, Z., Jiang. X., Miller, T., Wang. F., Zheng, W. J., and Roberts, K. Deep representation learning of patient data from electronic health records (ehr): A sys- tematic review. Journal of biomedical informatics, 115: 103671, 2021
Ren, J., Zhao, Y., Vu, T., Liu, P. J., and Lakshminarayanan, B. Self-evaluation improves selective generation in large language models. arXiv preprint arXiv:2312.09330, 2023.
Renggli, C., Rimanic, L., Gürel, N. M., Karlas, B., Wu, W., and Zhang, C. A data qqualty--drvven view of mlops. IEEE Data Engineering Bulletin, 2021.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Lewes H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, pp. 1-9, 2023.
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Par- itosh, P., and Aroyo, L. M. "everyone wants do the model work, not the data work"; Data cascades in high- stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2021.
Thakur, N., Reimers, N., Rücklé A., Srivastava, A., and Gurevych, I. Beir: A heterogeneous benchmark for zero- shot evaluation of information retrieval models. In Thirty- fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 2), 2021.
Santhanam, K., Khattab, O.,, Maldd-Faccon, J., Potts, C., and Zaharia, M. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pp. 3715-3734, 2022.
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. Just ask for calibration: Strategies for eliciting calibrated confi- dence scores from language models fine-tume tenettundd with hu- man feedback. In Proceedings of the 2023 Conference
12

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Empirical Methods in Natural Language Processing, pp. 5433-53 5442, 2023.
Zhang, J., Shin, B., Choi, J. D., and Ho, J. Smat: An attention-based deep learning solution the tomation of schema matching. Advances in databases and information systems. ADBIS, 12843:260-25 274, 2021. URL httpp://app....... semantt..............mam...mm... cschol lar. htts/CoopuaD::23720100
Tiwari, P., Colborn, K. L.., Smith, D. E., Xing, F., Ghosh, D., and Rosenberg, M. A. Assessment of a machine learning model applied to harmonized electronic health record data for the prediction incident atrial fibrillation. 2020.
JAMA network open, ( 19131699996661119999
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36, 2023.
Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., and McI.achlan, S. Synthea: An approach, method, and soft- ware mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association, 25(3): 230-238, 2018.
Zhuang, H., Qin, Z., Hui, K., Wu, J., Yan, L., Wang, X., and Berdersky, M. Beyond yes and no: Improving zero- shot Ilm rankers via scoring fine-grained relevance labels. arXiv preprint 2.. 2023.
Wang. P., Li, L., Chen, L., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators, arXiv preprint arXiv:2305.17923 2023a.
Wang, 7., Zhang, H., Li, C.-I.., Eisenschlos, J. M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang. J., Lee, C.-Y., et al. Chain-of-table: Evolving tables in the rea- soning chain for table understanding, In The Twelfth International Conference on Learning Representations, 2023b.
Wei, J., Wang, X., Schuurmans, D.,, Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of of-thouught prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Whang, S. E., Roh, Y., Song, H., and Lee, J.-G. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal, 32(4): 791-813, 2023.
Williams, R. D., Reps, M., Kors, J. A., Ryan, B., Steyer- berg, E., Verhamme, K. M., and Rijnbeek, P. R. Using iterative pairwise external validation to contextualize pre- diction model performance: use case predicting 1-year heart failure risk in patients with diabetes across five data sources. Drug Safety, 45(5):563-55) 2022.
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., and Hu, X. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303. 10158, 2023.
Zhang, H., Dong, Y., Xiao, C., and Oyamada, M. Large language models data preprocessors. ArXiv, abs/2308. 16361, 2023a. URL https: h semanticscholar. ora.aaruurrrC/oooos 261397017.
Zhang, H., Dong, Y., Xiao, C., and Oyamada, M. Jellyfish: A large language model for data preprocessing. arXiv preprint arXiv:23/2.01677 2023b.
13

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Appendix Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Table of Contents
A Matchmaker additional details
15
A.1 Matchmaker compared with ReMatch
15
A.2 Schema matching challenges.
15
A.3 Complexity of the MIMIC-OMOP task
A.4 Further details schema matching formalism
17
A.5 Detailed explanation of self- improvemet
18
A.6 Extended related work
18
A.7 Matchmaker within the context of LLM table reasoning.
20
A.8 Metrics: accuracy, precision, recall, FI F1-Sccre
20
A.9 Matchmaker algorithm....
22
B Experimental details: Benchmarks & datasets
23
B.I Benchmarks
B.1.1
Matchmaker
23
B.1.2
ReMatch
23
B.1.3
Jellyfish
23
B.1.4
LLM-DP
23
B.1.5
SMAT
24
B.2 Datasets
24
C Examples using Matchmaker (with prompts)
25
Matchmaker prompt examples..
25
C.1.1 Example 1.
25
C.1.2 Example
28
C.2 LLM Evaluator
31
Additional experiments
D.I Source of gain ablation: Why does it work?
33
Number LLM calls
................................
D.3 Matchmaker with other LLMs
34
D.4 Further performance results: ReMatch reimplementation
D.5 Improving performance: Use of Existing Mappings to remedy emrors...
35
Comparison Matchmaker on ontology matching tasks
36
D.7 Detailed emror analysis
36
Ranking ablation
14

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
A. Matchmaker additional details
A.I. Matchmaker compared with ReMatch
Table 4: Difference between Matchmaker & the closest work ReMatch along multiple dimensions.
Feature
KeMatch Single-step process A: x A - (0.1)
Matchmaker (Ours)
Mulli-sipp compositional
System//Apreach
LM program L (2..2......
4)
Candidate
Semantic
only
Semantic Reasoning- hased
C. = gf Ar)
C Ce C2 (See 4.2)
Chain-of-Thooue
Reasoning Mechanisu
Limited to ranking stage
prompting throughout
based confidence scoring
Single L.M call for hinary decision R(A, Agy E (0.
Ranking
with MCQ format: 7 (A, x D.) x [Ar D2) - +
allowing for uncertainty
Zero-shot with synthetic
examples: [ei, M(ez]] (Sec
LLM Prompts
Static
Dynamic
A.2. Schema matching challenges.
Database Heterogeneity: The number of tables in each schema may differ, i.c., [Ta] * [Tel, making challenging to establish correspondences between attributes across schemas.
Structural Heterogeneity: Schemas may have different architectures, hierarchies, and representational granularity. If we define hierarchy function h(T;) that describes the level of nesting within tables, differences in (T(aa)) and h(Taa) for any j, & can lead to significant challenges in aligning attributes Asj and Ark-
Semantic Heterogeneity: Attributes in different schemas may have the same name but different meanings, different names but the same meaning. Let N₂ = {mayl A; be the set of attribute names for schema S, Semantic heterogencity occurs when 3Aej E A,, E As f(Aa)) = Aek A naj + Tisk or when _Aej E Asx E A f(A.g) # Auk A naj = nakk-
Data Type Heterogeneity: Attributes in different schemas may have different data types, even if they refer to the same concept. Let dij be the data type of attribute Aij. Data type heterogeneity occurs when 3Aaj As, Ath E A f(A.j) = Au A dej # dek.
Some attributes in schema may lack corresponding match in the other schema. This necessitates reasoning about "no possible match" cases, which is important as reasoning about possible matches.
Information Mismatch:
Unsupervised Nature: Schema matching is unsupervised, where no labeled data pairs (Asj, Asx) are available to train or validate the mappings. This necessitates reliance on the intrinsic structure and semantic information encoded in A, making the development of effective mapping function f challenging without external supervision.
A.3. Complexity of the MIMIC-OMOP task
MIMIC-OMOP is real-world healthcare schema matching task, which is reflective of complex structures, interlinking and hierarchies that can be expected in real-world schema matching tasks. Hence, Matchmakers ability to empirically outperform baselines on these tasks highlights its ability to handle complex schemas.
To illustrate the complexity of the schemas that Matchmaker can handle, Figure illustrates the complex schema structure and multiple tables.
15

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Target Schema (OMOP)
PPR
Source Schema (MIMIC)
Langlen
Figure 5: Illsttaation of the MIMIC-OMOP schema matching task showing the complexity and schema hierarchies.
16

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
A.4. Further details on schema matching formalism
In this appendix, we provide further details on the formulation of schema matching. We look at properties that a schema matching algorithm or function should possess, well as, detailing how Matchmaker satisfies these properties.
Properties necessary. In practice, correctness in schema matching is evaluated against expert-validated ground truth mappings between the datasets (e.g. MIMIC to OMOP and Synthea to OMOP). However, this begs the question what properties would be useful ti improve emprical performance.
These along the following dimensions:
Semantic Equivalencc//onniistenyy f(As) = A, implies A and A, represent the same real-world concept (i.e. the mapped attributes serve equivalent purposes)
Type Compatibility: Mapped attributes must have compatible data types
Structural Consistency: Mappings must respect schema hierarchies
Coverage: f identifies all valid matches while avoiding incorrect mappings through abstention. i.c. coverage is maximized by improved aecuracy@k
We can then practically assess if a function f (such Matchmaker) satisfies these criteria based its performance against expert-validated ground truth mappings in real-world benchmark datasets as has been done in the paper.
How does Matchmaker satisfy these properties?
While we have empirically shown Matchmaker satisfies the properties needed of schema matching function f. based on its strong performance on real-world schema matching tasks where it significantly outperforms existing approaches on standard benchmarks. In particular, the strong empirical performance outperforming the baselines implies that Matchmaker better satisfies the properties compared the baseline schema matching algorithms.
However, let analyze how Matchmaker also has specific design aspects within its compositional LLM structure that promotes addressing the properties.
Semantic equivalencee/coniitteny::: Matchmaker employs multiple mechanisms: multi-vector document representation captures semantic nuances beyond simple name matching, while dual candidate generation combines both semantic retrieval and LLM reasoning to identify conceptually equivalent attributes.
Type compatibility: enforced through inclusion of data type information in our multi-vector documents (Section 4.1) and LLM reasoning during candidate generation and refinement (Section 4.2), with examples in Appendix C showing explicit consideration of type compatibility (e.g., string->varchar, integer->bigin).
Structural consistency maintained by incorporating table metadata and hierarchical information in document creation (Section 4. I),, using reasoning-bart candidate generation that considers schema structure (Section 4.2), and including table context confidence scoring.
Coverage is optimized through our MCQ format with "None of the above" option enabling abstention when no good match exists, while confidence scoring helps identify and rank high-quality matches. Our empirical results validate that these properties translate to superior performance practice.
17

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
A.5. Detailed explanation of seff-improvement
The self-improvement mechanism of Matchmaker is pivotal component. We provide the Algorithm below.
Algorithm 2 Optimize LM program C
0: Input: Set of evaluation queries Deval = e2,.....
0: Oufput: Set of top n demonstrations Ddemo
0: for each input 6; Deeaal do
O: Do, traacee C(es) (Teacher & predicts, storing outputs and intermediate traces)
0: 8; + E((e, ū.) { Evaluation score }
0: Ddemo t Demo U (e,, trace;, 81)
0: end for
0: Sort Ddemo by score
0: return Ddemo 00 n] (Select top n) =0
In particular, we clarify that the self-improvement approach aims to address the issue of in-context learning for multi-stage LLM programs like Matchmaker. However, in doing so we need address fundamental challenges in our setting (C1 and C2):
(C1) Lack of labeled demonstrations: We do not have access to labeled input-output demonstrations from which to select in-context examples.
(C2) Lack of an evaluator for selection: To assess Matchmaker's capabilities and guide selection of examples, we need an evaluator.
We address each follows:
Addressing (C1): The process begins by creating an evaluation dataset Devel from unlabeled schemas with two properties: "easy queries" where top-n semantic matches have similarity scores > 0.95, and "challenging queries" with the lowest semantic match scores. This ensures diverse coverage of different matching scenarios. The complete Matchmaker compositional program then run on each evaluation example & E Devat- We capture full execution traces including intermediate reasoning steps, candidate generation and refinement decisions, and final confidence scores and matches. The synthetic in-context examples refer to the intermediate input-output pairs generated by the LLM for the intermediate steps of the compositional LLM program. This deals with the challenge of a lack of labeled examples (i.e. zero-shot).
To handle the lack of an evaluator (validation metric), we use an evaluator L.MM E (i.e. an LLM- as-a-judge) to assess match quality through tf-thughh reasoning, producing scores from 0-5 based on match relevance. Finally, the top-n traces are selected based on these evaluation scores. This systematic approach, detailed in Algorithm enables principled selection of in-context examples based on traces that lead good performance. We then these as in-context examples for the different parts of the LLM program (as they led to good performance) in order to guide the reasoning. As shown in the main paper our novel approach to self-improve outperforms random selection of in-context examples and self-reflection confirming that our systematic selection of in-context samples is the key driver of performance gains, rather than the mere inclusion of any in-context examples.
Addressing (C2):
A.6. Extended related work
Alternative retrieve-and-ramt methods.
At high level, there are other methods which share the retrieve-then-rerank architecture with the ReMatch baseline, especially in its zero-shot configuration. Specifically, the paradigm where the method retrieves candidate matches using embeddings and subsequently reranks candidates with an LLM. While our closest method, ReMatch, does this, other methods like Magneto follow a comparable approach.
A primary difference is the benchmark datasets on which methods like Magneto are evaluated. For instance, in Magneto (Liu et al., 2024), most of the benchmark tasks involve schema matching of a single source to a single target table. This
18

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
contrasts with our healthcare setups, where there are multiple source tables and multiple target tables. Hence, our healthcare tasks are more complex, needing reasoning over the table match first and then the column match.
Additionally, besides datasets, we contrast our Matchmaker framework and highlight how it fundamentally differs from Magneto (and ReMatch) in three important ways:
1. Compositional LLM Program: While Magneto uses a two-stage pipeline (retrieval and reranking), Matchmaker introduces a multi-stage compositional LLM program with candidate generation, refinement and confidence scoring. This structured approach allows more nuanced reasoning about schema relationships.
2. Diverse Candidate Generation: Matchmaker combines both semantic retrieval and reasoning- -ased candidate generation, whereas Magneto relies on semantic retrieval only
3.
Self-Improvement Mechanism: Matchmaker introduces a novel zero-shot self-improvement mechanism using synthetic in-context examples, which doesn't 't exist in other methods.
Classical Schema Matching approaches. Classical approaches to schema matching, as thoroughly reviewed by (Rahm & Bernstein, 2001), use a range of strategies, including heuristic-driven linguistic matching, constraint- methods, and structural analysis. These methods have historically focused on simple relational schemas, matching elements between individual tables or flat structures. In particular, the primary focus is matching between individual tables or simple schemas (such as purchase orders).
Key Weaknesses of Classical Approaches and How Matchmaker Addresses Them:
Single-Table and Flat Structure Focus: Classical methods typically perform schema matching at the element level, treating tables as isolated entities and matching attributes based on direct comparisons of names, data types, or simple structural cues. In particular, often focus was simple relational schemas, where the goal to map elements between single tables. However, this approach fails to handle the complexity of modern data systems, where schemas are often tubti, hierarchical, or require cross-table reasoning. Contrast: Matchmaker, in contrast, uses LLM-based reasoning to connect attributes across multi-table and hierarchical schemas, understanding how data relationships span multiple tables. This makes our approach significantly more capable of handling complex and interrelated schema structures.
Dependency on Heuristics and Limited Semantic Understanding: Classical methods rely on heuristic-driven matching based on linguistic similarities (e.g., name matching using synonyms, hypernyms, or edit distance) and structural constraints like key relationships. While these heuristics work well-defined contexts, they are insufficient for domains where semantic meaning is implicit, such as in healthcare and as we show in Fig 1 only semantic matching is in fact insufficient. Contrast: Matchmaker employs chain-of-thougee prompting and advanced LLMs perform reasoning, allowing it capture relationships that are not explicitly defined in the schema structure names. This enables Matchmaker to handle complex mappings that classical methods cannot infer.
Manual Effort and Lack of Adaptability: Classical techniques require significant manual effort for tuning and adaptation, making them less suitable for rapidly evolving or heterogeneous environments. Constraint-based approaches, in particular, are difficult to scale across different domains without manual intervention. Alternatively, they might also rely on labeled data for effective matching. This makes these classical approaches impractical in real-world environments. Contrast: Matchmaker's ' zero-shot and self-optimization capabilities mean it can adapt autonomously to new schemas using synthetic in-context examples, significantly reducing the need for manual tuning and making it more practical for dynamic, real-world data integration tasks.
Key Weaknesses of SMAT and how Matchmaker improves: We also compared Matchmaker to state-of-the-art (SOTA) methods like SMAT Zhang et al. (2021), which applies attention mechanisms for schema matching. While SMAT represents an important advancement over classical methods, it has several limitations that Matchmaker overcomes:
High Dependency on Labeled Data: SMAT requires extensive labeled data (over 50% labeled matches) for training, which is often impractical in real-world schema matching. Contrast: Matchmaker's 's zero-shot matching capability allows it to perform well without any labeled training data, using L.LMs to generate and refine matches autonomously.
19

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Binary formulation: SMAT formulates the problem binary classification task over the full Cartesian product of source and target schema attributes. e.g. for each pair of source-target attributes. This leads to large amount of comparisons. Contrast: Matchmakers formulation as information retrieval reduces the number of comparisons and leads to greater efficiency - in addition to the better performance.
A.7. Matchmaker within the context of LLM table reasoning.
There has recently been works on LLMs for table reasoning. We contrast them Matchmaker along variety of dimensions below.
The table reasoning papers tackle variety of tasks centered around understanding and interacting with tabular data. Some examples include: TabSQLify (Nahid & Rafiei, 2024) and OPENTAB (Kong et al., 2023) focus on table question answering and fact verification, aiming to extract relevant information from tables to answer questions or verify statements. Chain-of-Table (Wang et al., 2023b) and "Large Language Models are Few-Shot Table Reasoners" (Chen, 2023) explore LLMs' capabilities in reasoning over tables for question answering and fact verification tasks. The survey paper "Large Language Model for Table Processing" (Lu et al., 2024) covers a broader range of tasks, including table manipulation, table augmentation, and text-to-SQI. conversion, showcasing LLMs potential in interpreting and manipulating tabular data. In contrast, Matchmaker addresses the task of schema matching, which aims to find correspondences between attributes across different schemas or tables. The goal is to enable data integration by mapping attributes from a source schema to a target schema, considering the structural and semantic differences between them. This task is crucial for creating ML-ready datasets by harmonizing data from diverse sources.
Task/Goal:
Table reasoning approaches span prompting LLMs for direct answers (Chen, 2023), program synthesis generate SQL/code (Nahid & Rafiei, 2024; Kong et al., 2023), iterative table transformation (Wang et al., 2023b), instruction tuning (Lu et al., 2024), and agent-based methods (Lu et al., 2024). Matchmaker proposes novel self-improving compositional language model program. It leverages LLM reasoning via pipeline with multiple LLM calls for candidate generation, refinement and confidence scoring. It also self-improves without labeled data via synthetic in-context examples.
Approach:
Inputs: The table reasoning papers mostly focus on single tables as input along with a question/query. Matchmaker takes as input two tables/schemas (source and target) that need be matched. operates solely on schema-level information (attribute names, metadata) without access raw data in the tables. This is also key difference compared to the table reasoning papers, which often rely on the actual data values for answering questions or verifying facts.
Table reasoning papers aim to output answers to questions, binary fact verification labels, updated tables after manipulation, generated /Ql/c etc. In contrast, Matchmaker outputs a mapping between the source and target schema attributes, or indicates no match is possible for certain attributes. The set of attribute pairs representing the schema matching results, can be used to guide data integration processes.
Outputs:
Use of the LLM: Table reasoning employs LLMs for direct answer generation (Chen, 2023), program synthesis (Nahid & Rafiei, 2024; Kong al., 2023), iterative prompting (Wang et al., 2023b), as part of an agent system (Lu et al., 2024). Matchmaker uses LLMs for reasoning within compositional program, generating candidates, refining them, and scoring confidence.
Table reasoning works explore fine-tuning (Nahid & Rafiei, 2024), instruction tuning (Lu et al., 2024), and in-context few-shot learning (Chen, 2023). Matchmaker introduces a novel optimization process to select synthetic in-context examples for self-improvement without labeled data or fine-tuning.
Optimization/Trainne
Key differences: In summary, while the table reasoning papers focus tasks like question answering, fact verification, and table manipulation single tables, Matchmaker addresses the distinct task of schema matching across table pairs. Its novel approach of a self-improving compositional language model program operating on schema-level information contrasts with general table reasoning which mostly use LLMs for direct table QA or program synthesis.
A.8. Metries: accuracy, precision, recall, F1-Score
In our 1: schema matching setup, accuracy 11 1, precision, recall, and F1 are equivalent due to the specific constraints of the task and the prediction mechanism employed. Below, we provide a detailed explanation of this equivalence:
2. Task Constraints: The schema matching task is constrained such that each source attribute can match to at most one target attribute (m:1 constraint). This ensures that the number of predictions equals the number of source attributes.
20

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Equivalence of Metrics Given the above setup, the following equivalences hold:
Precision:
True Positives (TP)
Precision
True Positives (TP) + False Positives (FP)
In our setup, every prediction corresponds to exactly one target attribute, and there are no extraneous unassigned predictions. Therefore:
Correct Matches
Precision
Total Predictions Accuracy@1 1.
Recall:
True Positives (TP)
Recall
True Positives (TP) + False Negatives (FN)
Since every source attribute must be matched to target attribute, there are no unassigned predictions in our setup. However, incorrect matches can occur, leading to both false positives (FP) and false negatives (FN). In our m: schema matching setup, prediction is either correct (a true positive, TP) or incorrect. An incorrect match to the wrong target attribute results in false positive (FP) for the predicted target and a corresponding false negative (FN) for the true target. Consequently, the number of and FN are always equal, as they reflect the same prediction errors. In this setup, precision, recall, and accuracy @1 equivalent because they all measure the proportion of correct matches (TPs) relative to the total predictions, with incorrect matches impacting all metrics identically. This equivalence holds when correctness is measured against the ground truth annotations from the benchmark datasets. Thus:
Recall Correct Matches
Accuracy@
Total Predictions
F1-Score:
Precision Recall
Fl-Score Precision + Recall
As both precision and recall are equal to accuracy @1 1 in this setup, the Fi-score simplifies to:
FI-Score Accuracy@11 1.
In summary, due to the constraints of our m:l schema matching task and the a argma x prediction mechanism, accuracy @ 1, precision, recall, and F1 F1-score mathematically equivalent. We report accuracy @ 1 in the main results, but the correspond- ing precision, recall, and F1-scores are identical and can be directly interpreted from the accuracy @ I values. We note this equivalence does not hold for one-to-many mappings
21

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
A.9. Matchmaker algorithm
Below we provide high-level overview algorithm of Matchmakers compositional language model program for schema matching.
Algorithm 3 Matchmaker: Schema Matching with Self-Improving Compositional Language Model Programs
Require: Source schema S., Target schema S
Ensure: Schema matches M
0: Stage Multi- Vector Document Creation
0: for each table T E St do
O: Create document Dr with attribute names and descriptions
O: Append table metadata Dr
O: Encode Dr using ColBERT-v2 to obtain multi-vector representation Vr
0: Add Vr to vector database V
0: end for
0: Stage Candidate Generation
0: for each source attribute 9; E S, do
O: Encode 9i using ColBERT-v2 to obtain query embedding Egg.
0: Retrieve top-k semantic candidates C, from V using Egg
0: Generate reasoning-basee candidates CR using LLM lo(gs, St)
0: Refine candidate set C* + (,C,, Ca. 4:)
0: end for
0: Stage 3: Confidence Scoring
0: for each source attribute qi E S,
O: Format candidate set as multiple-choice question Q:
0: for each candidate Cj E C do
0: Compute confidence score 8, +- (g(
0: end for
0: m, eCE [Select match with highest confidence)
0: Add (g:, m,) to schema matches M
0: end for
0: Self-emmrrveeent Optimization (Over all steps)
O: Generate evaluation set Deval from unlabeled schemas
O: for each example e; Deval do
O: (2i, trace;) +- Matchmaker((s) (Run Matchmaker to get output and traces)
0: 9; + --Ee gi) (Compute evaluation score using LLM E
0: Add (ei, tracei, ÿi, 9:) to Ddemo
0: end for
0: Sort Daemo by score 8;
0: Select top-n examples from Dáemo as synthetic in-context examples
0: Update Matchmaker components with selected in-context examples
0: return Final output: Schema matches M =0
22

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
B. Experimental details: Benchmarks & datasets
All experiments are run on a single Nvidia A4000 GPU with 20 GB of vram. We invoke GPT-4 via the Azure OpenAl API.
B.1. Benchmarks
B.I.1. MATCHMAKER
Matchmaker is a compositional language model program for schema matching made up of multiple component modules formulated the context of information retrieval.
GPT-4 Hyper-parameters. The model version used as the LLM was GPT-4-1106, with the following settings: ('tempera- ture': 0.5, imax_tokens, 1024, 'top_p': : 1, "frequency_penattt" 0, presence_penalty" 0, 'n': I,
Embedding model and documents. We use Colbert- V2 (Santhanam et al., 2022) the embedding model and follow the document creation process as outlined in Sec. 4.1. We use the implementation of Colbert-v2 from RAGatouille http://gwwbb.ooo//llll/v///aaaaa
Candidates. For both semantic and reasoning- candidates, we set k=5.
Optimization. As described the main paper, we generate synthetic in-context samples address the unique challenges of lack of labeled data and demonstrations. As described, to achieve this we follow boostrapping process like in DSPy (Khattab al., 2023). For our experiments we select at maximum 4 synthetic in-context examples
Prompts: We show examples with the prompts for each component of Matchmaker in Appendix C.
B.1.2. REMATCH
In the main text we report the numbers directly from the ReMatch paper, as there is no open-source implementation.
How we selected the numbers report: The ReMatch paper does an exploration of the number of documents retrieved. Hence, use the following two criteria. (i) At least document must be retrieved. i.c. the retrieval step cannot be skipped.
(ii) We then select the result that satisfies (i), with the highest accuracy @5
Our implementation ReMatch follows the original paper (Sheetrit et al., 2024). We use OpenAl Ada embeddings for the embedding model and GPT-4 as the LLM.
We following the document creation procedure and use the prompt template as provided.
GPT-4 Hyper-parameters. The model version used for generation GPT-4-1106, with the following settings from the ReMatch paper: (seed=42, temperature=0.5, max_ max_tokens=4096. top_p=0.9, frequencyy_eenaltya)) )
B.1. JELLYFISH
Jellyfish (Zhang et al., 2023b) is a fine-tuned language model tailored for data preprocessing tasks including schema matching. The 7B and 13B models are fine tuned upon the Platypus2 model.
Implementation (7b): http:/hingginnecc.ooooooooooE///
Implementation (13b): http::/wnggiiiiicccoo/hooooooear
B.1.4. LLM-DP
LLM-DP (Narayan et al., 2022; Zhang et al., 2023a) refer to works which have used pre-trained LLMs like GPT-3.5 or GPT-4 for data processing tasks like schema matching via prompting. Since the papers in the few-shot case use labeled examples we do not use those - given its unrealistic in practice. Hence, for these baselines they operate in a zero shot manner.
Implementation: http:://gthuu..cm//uuyyyesssssse data tasks
23

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
B.1.5. SMAT
SMAT is a supervised learning approach which performs schema matching via an attention mechanism. Of course, the model needs labeled data train on. In our experiments, assess two variants given that labeled training data for schema matching is hard to access: (i) 20-80: 20% train and 80% test and (ii) 50-50: 50% train and 50% test.
We use the default hyper-parameters: (Learning Rate: 0.8, Batch Size: 64, Epochs: 30}
Implementation: https//ggttubbooom////2228881111
B.2. Datasets
We outline the two real-world schema matching benchmarks used in this paper - MIMIC and Synthea. These datasets mapping different clinica//healthcure schemas were chosen as they are the standard datasets used in schema matching literature and consequently, used by prior works providing fair assessment. They are also considered the most reflective of real-world schema matching complexity and challenges. We note that the scareity of complex and challenging real- sol-world datasets, underscores the challenges in collecting and annotating real-world schema matching data. For instance, as noted in Sec 1, annotating MIMIC-OMOP alone required 500 hours from two medical experts.
Table 5 provides a summary of the table properties.
Note there is no specific train-test sets used as in supervised learning. As we perform the schema matching task in a zero-shot manner.
Table 5: Summary of the table properties of our schema matching datasets.
Dataset
Source Tables
Target Tables
MIMIC-OMOP
26
14
SYNTHEA-OMON
12
21
MIMIC Dataset: The dataset contains schema mapping between the MIMIC-III electronic health record (Source schema) (Johnson et al., 2016) and The Observational Medical Outcomes Partnership Common Data Model (OMOP schema) (Target schema).
This dataset is currently the largest publicly available schema matching dataset (Sheetrit et al., 2024) and is the cloest to real-world schema matching use case, wherein proprictary database created for specific purpose (a source schema) is mapped to given industry standard (a target schema) for further uses. In this case the proprietary database schema is MIMIC and the industry standard is the OMOP common data model.
Open-source data: https///ggtubb.commmmmnnnaaatt
2_OMOP
Synthea Dataset: The Synthea dataset is part of the OMAP benchmark (Zhang et al., 2021) and is a partial mapping of the Synthea (Walonoski et al., 2018) (Source Schema) which is synthetic healthcare dataset of a Massachusetts health records and attempts map it to subset of the OMOP CDM (Target Schema). The dataset has widely been used in previous schema matching papers (Sheetrit et al., 2024; Narayan et al., 2022; Zhang et al., 2021) as realistic and challenging real-world schema matching benchmark.
Open-source data: http://ihhttooomccc/oSSSS////ttt
24

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
C. Examples using Matchmaker (with prompts)
C.1. Matchmaker prompt examples
We show two end-to-end schema matching examples with Matchmaker, where other methods fail. (1) Example I: case with No possible target schema match for the source schema query, (2) Example 2: challenging reasoning case, where there is match possible between source and target schema.
In each component, can show the "Optimized" In-context examples.
C.I.I. EXAMPLE 1.
Source schema query: admissions cimissions-mmrrtal status(string): Table admissions details-the admissions table gives information regarding patient's s admission to the hospital., Attribute marital _tattusssttuusss details -describe patient demographics.
Target scheme match: None possible.
Matchmaker: None the above.
Figure 6: EXAMPLE 1: Candidate generation.
Candidate generation
You are an OMOP Schema expert. Your goal include only most likely matches to the input query.
OMOP schema and based input, refine schema
Follow the following format.
Input Schema: Input OMOP schema values Input Query: input query Refined Schema: Five most likely matches input query. Include most likely matches to the input query. Respond with a single JSON object. JSON Schema: ("defs": ("Ex- tractor": (" ("properties": ("related": ("description": "related matches", "title": "Related", "type": "string")), "required": ["related"], "title": "Extractor", "type": "object")], }}. "properties": ("value": ("items": ("ref": "/defs/Exttractor" "title": "Value", "type": "array") ",, "required": ["value"], "title": "Output", "type": "object")
Input Schema: ['visit eccureecc-peersom jetprrrrrsooiiiiiiiiirr id(bigint)', 'visit_ occurrence--vsi.. wwtp:r/iiiiiioccureeeeeeee occurrence iiliiiiitt(iii "procedure cerrrr_ccccccccurerereeeee proviierrrroieeereeeeeeee provider__d(biiinnt) visii_ wttpectailliisiitddtaail_oooo (ecrdddtall__ee visit_detallssuureeelluveeeeeee valuu((rrccvcccc555))
Input Query: secoddurrevents_mmiieeeee
Refined Schema: "value": ("related": "procedure_occeee http:rroocccccccceerrroooiddippe "related"; prooeddue__occureenc.
procedure sourceessssceeeeeee wwtpsssooccccececeeeeeee valuu((aarrcaa(((05"" "related": prapr/eeeeppreeeduuudddddddddeed "related": aonocedure__cccurrnece
gat__occurenne_iddeeee jid(iiitttt"",,. "related": "procedure_ "procedure_eccurreneee-
Input Schema: ['visit_ wetiiooccrreecceperoonddedee iidiigiitt
'visit hwsigppccurreeeeviii.cccrreeeeep
pmocedure_eccurrree--pppe occurrence-provider "itroiiiderddbbgiinnn, 'visit disit_detail-viiit detail Socaaalsssuree____ee (oource_valu(vaaccha(((0))
Input Query: noteevents-text
Refined Schema: "value": !"related": "note-note_texx((acceee (exx(vaacccaaamaa))... "related":
"related": inote-note_source__ hatee-oooe_suurcc_alleevvveeeeee value((aarcaar(50)), "related": doteene_ddcc((tte)"" "related":
"note-
note cooe_ddaetimo((datttmee"
Input Schema:
occurrence-person
wwwpurrcocceviisttcccrreeeeeeeoe wwsittdettii__seeeeeeeeeee saraillsourre_vvee value((arccar((50))
cecertuuu_occuurcccceeeee (rovider__idbigintt)
Input Query: admissions cdmiisiions-marital.. - =statussstting): Table admissions details-the admissions table gives information regarding
patient's admission the hospital., Attribute marital_staie status details -descrihe patient demographies.
Refined Schema:
LLM Response
( ("value": [[ (("related": pperonn-peronn pttpspeeedeppgiiiiiinniii id(bigint)"), ). ("related": "person-gender wetnnngeeeeeeeeeeee_eeerre value((arrhar(550)) ("related":
"person-birth, dateedditn(dattetmmee) ("related": "person-death docatt__datttneeddttttnee""" ("related"; "visit_occurreen" -ccuureece- admitted from_ sourc_surce_rt sowure_evalee (attucccvvcca((00)"11
25

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Figure 7: EXAMPLE 1: Candidate refinement.
Candidute refiner
You are expert OMOP matching ranker. Your task is to take the OMOP candidates and based the input, refine the candidates select the 5 most likely matches input query. Return ONLY the keys.
Follow the following format.
Input Schema: List of key: value pairs
Input Query: input query
Reasoning: Let's think step by step in order to pproduce the refined_ carii_sttinn__iieee list]. We...
Refined String List: Five most likely matches input query. Include maximum of the 5 most likely matches the input
query. Return ONLY the keys
Input Schema: I' perrsnn iidiggginn::::::: Tahle person details-this table serves as the central identity management for all persons in the database. it contains records that uniquely identify each person or patient, and demographic information., Attribute person details -it is assumed that every person with a different unique identifier is in fact a different person and should be treated independently... measurement, concept id field is recommended for primary in analyses, and must be used for network studies']
wwwtrremment-mmsuremeeeee cwt.eptoeeppppeeeeeeeeeeeeeeeeee id(integer): the
Input Query: procedureevents_ matttts_v-iimiidd
Reasoning: Let's think step by step in order to produce the refined string list. We are looking for a match to procedureevents siss_mmv-iiiii""" which seems to relate to procedure event and its item identifier: The procceureevents __mv' part suggests we're looking procedure occurrence, and the 'itemid' part suggests we're looking for identifier or id field. Given that, we should look for keys the 'procedure occurrence' table that represent identifiers.
Refined String List: 'procedure occurrence-eeroon id(bigint)", 'prucedure occurrence-viiit, occurrence id(bigint)", prroceuure pecurrence-proxeeuuee concept id(ineeger).. procedure__ccurrrcee--peee occurreenc-prrvvidr. id (bigint)"
'procedure occurrence-visit detail id(bigint)",
Input Schema: perrson-peroon penson_id(bigint) Table person details- this table serves as the central identity management for all persons in the database. it contains records that uniquely identify each person patient, and some demographic information., Attribute person_ic id details -it is assumed that every person with a different unique identifier is fact a different person and should be treated independently'. measurement_ ceatrremeet_ccccpptttee id field is recommended for primary in analyses, and must be used for network studies']
mueasuurementtmmeaasueeeee cettrrremeeeeeecppttteeeee
the
Input Query: noteevents-text
Reasoning: Let's think step by step in order to produce the refined string list. We are looking for matches to "notcevents- text" in the input schema. The term "noteevents" matches with "note" in the input schema, and "text" matches with "note_re textt."t" So, we should consider those keys that contain both "note" and "text". "note-note_ tilee(varchar((55)))
Refined String List:
('note-note_eex((varchhee (mx)))))ccaammax)).
'note-
note sowucouuueeeeeeeeeeeeeeeeeeeeeee value((archar(50)) 'note-note (ettoooe_eee((aaccrr(aaaa))
Input Schema: ('person-person betppeeooooooiiiiiiiioroo Table person details-this table serves as the central identity management for all persons the database. it contains records that uniquely identify each person or patient, and some demographic information., Attribute person_id jn details -it is assumed that every person with a different unique identitier is in fact a different person and should be treated independently", measurement_ cecpanttt_nn___iiie id field recommended for primary use in analyses, and must be used network studies'] Input Query: admissions-mant marital status(string): Table admissions details-the admissions table gives information regarding a patient's 's admission to the hospital., Auribute marital status details -describe patient demographics. Reasoning: Let's think step step in order to
meessurementtmessurreeeee _ecpenneeeeee____eeee cooccppt_idiiteeeeeee id(iteeger): the
LLM Response
"produce the refined string list. are looking for a match to 'admissions maatisaaaatttttttaee status', which seems to relate to hospital admission and the marital status of the patient. The 'admissions'' part suggests we're looking at a visit occurrence or visit detail, and the 'marital_statur part suggests we're looking for demographic information, which is usually found in the 'person' table. Given that, we should look for keys in the 'visit occurrence'. 'visit detail', and 'person' tables.String List: 'visit occurrence-person id(bigint)', 'visit, occurrence-visit. occurrence id(bigitt)', 'visit -perinn id(bigint)'. "visit detail-visit, accurrence id(bigint".
26

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Figure 8: EXAMPLE I: MCQ Formatter.
MCQ Formatter
are an expert MCQ formatter. Your task to take list schema values and convert them into a multiple choice question format with (letter)Schema value, where the schema values should key(description).
Follow the following format.
Input: input list of schema values Moy MCQ fornat of schema values (A)Schema value, (B (B)Scherna value. Do not include additional options, only the schema values options. where the schema values should be kcy(description). Add No Match option.
Input:
1" procedure
pereepperonnnoee idbbiinn"
procedure occurrence--isit occurrence_i((biiinnt""
"procedure_ procedure_occurrenee www.rreecceeeeeoo procencccceeeeeeee.e... ccoceep__ddeeeeeecce id(integer)', (bigint)""] Moq:
'procedure, occurrence-visit _detail idd(biginn)) (A) "AJ'prceurue pocurrence-persee id(bigint)'.
procedure occurrence-provee peeenccceppoverr....
(B) procedure occurrence-vist occurrence, sccurence_id(biginnt). (C) proceduue escurrenee-proeedumg concept id(integer)'.
(D) procedure_oog -etprduuee_ocuuueeeeeeeee -istieeeettiiiiiiillll id(bigint)" (E) prrl'pruueeeeoo pwwpouure_ocuureeeeeppvvleeee jd (bigint)', (F)No Match
Input: ["['note-note ( text(varchar(max)): "note-note (iddee(vaaccaa((250)) "note-note_ sarooee_ssuueeeeeeee waaruce_aalcevaacarrrr 'note-note_ ____eexx (ax(varcchr(mmax) [' Meq: (A)note-nole aappoteenoieeeeexttaaraaarmrmm tex((varchar(maxx), "hhoolemmie_jilleeaaaaaaaC333333 (C)note-
source_ value((archar(50), (Djnotc-note. text(varchar(max)), (E)No Match
Input:
["'visil_oce occurrence pecurrence-person ppenson_i(bbgiit)) 'visit "visit http://eeiisstocuuureeeeeeeebbee
detail-
person_ caaoo_id(iigint',, 'visit detail-visit, occurrence, pecurrenceii((biginn]] Mcq:
LLM Response
"(A)'visit pecurrence-person peniiiiieee person (ibbgnitdbiii) (D) visit detail-visit occurrence id(bigint)' (E)No Match".
(B) Diviviiitccuuuie ceii_nncureene-viie necurrenee-viiinuueerneneee ancurrencejid(biginnn
(C)vviii_detail-
Figure 9: EXAMPLE 1: Confidence scoring,
Confidence scoring
You are schema matching expert. Your task is given the input and the MCQ format of the schema, predict the likelihood relation score from 0-100 of the input query being related to each option. Your scores will be calibrated. If there is no good match score No Match 100
Follow the following format. Input Meq: Input MCQ format of schema values
Input Query: input query
Relation: Relation score of input query being related to the option as value. Assess each independently including No Match, returming a score from 0|-000 for each. Return with key as MCQ letter e.g (A) and score=value as JSON
Input Mcq: (A) procedure occurrence-person id(bigint)". (B) p3yppcccuure occurrence-visit occurrence id(bigint)', pO'prucedure_
proppedur_ooccurrencee occurrence prucedure senveddur__coccppt... sannceptidd(nttgerf
(D)
(B) peccurrennc--rrovvvee..ee (bigint)'. (F)No Match
Input Query: precedureevents. mv-itemid
Relation: ("(A)": 0, "(By": 0, "(C)": "(D)": 0, "(E)": 0, "(Fy": 100)
Input Mcq: (A)moto-note, cxttvvacharrmax), (B)mote-note titlc((archar((550)). (C)note-note_souuu cooote-oottcoucccceylllvvaaarrrr (Dinote-nole Lext(varchar(maxx), (E)No Match
Input Query: noteevents-text
Relation: ("A": 90, "B" 0, "C": 0, "D": 90, "E": 10)
Input Mcq: visit. occurrenco-persot id(bigint)'. (B) visit. occurrence-visit occurrence occurence_idfiggintt"""" (C)'visit_deette detail-
person_idee jibbinidiiigi))", (D)'visit_de De''vvi_eetailvviiii dettail-viilLocccrreeee occurrencee oecurrenceiillbiiint"" (E)No Match Inpul Query: admissions-
Murital_sttatussttie Table admissions details-the admissions table gives infommation regarding patient's admission to the hospital., Attribute marital_status status details -deccribe patient demographics.
Relation:
L.M Response
"(Axx o, (By (cy o, (DY (By 1000"
27

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
C.1.2. EXAMPLE
Source schema query: admissions-marita status(string): ttatss Table admissions details-the admissions table gives information regarding patient's admission the hospital., Attribute marital status details -describe patient demographics.
Target scheme match: srnccddure_occurreneee- quantity
Matchmaker:
pprocedure-_cccurrenee quantity
Figure 10: Candidate generation.
EXAMPLE 2: Candidate generation.
You are an OMOP Schema expert. Your goal is to take the OMOP schema and based on the input, refine the schema to include only 5 most likely matches the input query.
Follow the following format.
Input Schema: Input OMOP schema values Input Query: input query Refined Schema: Five most likely matches to input query. Include maximum of the 10 most likely matches the input query. Respond with single JSON object. ISON Schema: ("defs": | ("Extractor": ("properties:: ("related": ("description". "related matches", "title": "Related", "type": "string"]., ). "required": ["related"], "title": "Extractor", "type": "object" "properties": ("value": ("items": ("ref": "/defs/Extractor"), "title": "Value", "type": "array").. "required": ["value"], "title": "Output", "type": "object")
Input Schema: I'visit_ secit_occureeeeeeeeeeee person_id(bigint). visit_occcurrnce--e visit wec.urtoccccceeeeeeeeeee "itiiiennneddddiinnt)
'visit_ded detaall-iiii_deeeeeee citiiliviiitdettii___e sesiiiddetai_ssuuceeeo cacill__oure___auueeeee war__valuuvvvrraaee (al_aalac((crccatt55))]] Input Query: procedurcevents_mvvvee miviittmdddm Refined
Schema: ("value": [["related": "procedure_oceee wer_oooccccueeeerseeeee person_id(biiint]]] { ("related": "procedure_ocee -ociuurr_cccrreecee-eee aet_occrrecccccciiiiinnn..aa
("related":
eecurrence-roocedure
("related": "procedure occurrence-
"procedure securrence-pooeddiee source value(varchar(50))) ("related":
procedure concept id(integer)))]]
Input Schema: ('visit occurrence-person id(bigint)'. "visit occurrence-vise visit, occurrence id(bigint)".... "visit detail- detail source, (alue(varchar(500)] Input Query: noteevents-text Refined Schema: I"value": [I"r--
lated": "note-note, mhote-not__eex((yrrchaeee (ext(varchar(max))) ("related": "note-note phoee-not__iilevaacchr222333 tidlevvarcarr(550)) ("related": "note- note_source gone_soure__vluuevvrchhae value((archar((5))) ("related"; netenott_ddtt(tte]] ("related": note-note_ddan pate_dddtttimeeattttieeeell
Input Schema: ['visit_ cesuureccccecceeeeeooo secuurence-pprsonn. person perroo_id(bigint)) "visit_occurriiii www.rreece--viieeee.eee visit_occurrenee wet.urrrecccidddiiiiii""'""" "visit detail- visit detail_ detail_source___ source_vaaueevaae value(varchar(50)) Input Query: procedures beccdurrssicc-seqeeeee nuneeqtg_mmnnter) Table procedures ied details- contains icd procedures for patients, most notably ied-9 procedures, Attribute seq_num details -prvviees the order in which the procedures were performed Refined Schema:
procedure occurrence- visit occurrence id(bigint)), [telated: procedure occurrence-rooceduue date(date)). [ielated: procedure occurrence- procedure concept id(integer)). [felated: procedure occurrence (uaurreneeguantiiy(nnnee]]]]]]]
"[Value" [[telated:
procedure occurrence-person id(bigint)).
[felated:
28

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Figure 11: EXAMPLE 2: Candidate Refinement.
Candidate refinement
You are an expert OMOP matching ranker. Your task is to take the OMOP candidates and based on the input, refine the candidates to select the 5 most likely matches to the input query. Retum ONLY the keys.
Follow the following format.
Input Schema: List of key: value pairs
Input Query: input query
Reasoning: Let's think step by step order (produce refined_ string_listl We
Refined String List: Five most likely matches to input query. Include maximum of the 5 most likely matches to the input
query. Return ONLY the keys
Input Schema: I'procedure cectlluuueecuuuueee occurrence- peeurreeneeeeeonnnneeo person_id(bigin)) Table procedure occurrence details this table contains records of activities or processes ordered by, or carried out by, healthcare provider on the patient with a diagnostic or therapeutic purpose., Attribute person_id details -the person_id of the person for whom the procedure recorded. this may system generated code." "visit_detall ceatttdetalllcrre... care_site_______eeeeeeeeeeeee sibg_id( this field provides information about the care site where the visit detail took place"]
Input Query: procedureeevnts ienetrrroveettsmmm-iemddd myvitmmi
Reasoning: Let's think step by step in order to produce the refined string list. We are looking for match to procedurcevents_og _mviittmdi_-iiii which seems to relate to a procedure event and its item identifier. The 'procedureevents, __teeeetttttt part suggests we're looking procedure occurrence, and the 'itemid' part suggests we're looking for identifier id field. Given that, we should look for keys in the "procedure occurrence' table that represent identifiers. Refined String List: 'procedure occurrence- perennceepprronn.oo id(bigint)' 'procedure occurrence vesurrnceeviiit..... occurrence (ibggiin))) proceeure
grocedure_occcurenee occurrence occurrence-prooeduue caciure__ccccepe sancept_i((integr))
'procedure occurrence visit detail dettaleliebiapppe idbbigitt))
procedune__ccurreeeee
pexcedure_occurrence provoddervviiiiio id (bigint)'
Input Schema: l'procedure occurrence person_ (dd(diiidddi):: Table procedure_ roeedurr_occcurreee details-this table contains records of activitics or processes ordered by, or carried out by, healthcare provider on the patient with a diagnostic or therapeutic purpose., Attribute person id details -the person id of the person for whom the procedure is recorded. this may be system generated code.' "visit detail-care site idd(iiin)))) this field provides information about the care site where visit detail took place"]
Input Query: noteevents-text
Reasoning: Let's think step by step in order to produce the refined string list. We are looking for matches to "noteevents- text" input schema. The term "noteevents" matches with "note" input schema, and "text" matches "note_ "ite__teet"___ So, we should consider those keys that contain both "note" and "text". 'note-note titlee((archar(2500))
('note-note text(varchar(max)). Refined String List: 'note-note_ hete-nooteeeetvaehhaeeee (ex(((((aaccar((ma)) www.uure_valeevvvraaaaaaaa
nnote-
Input Schema:
l'procedure occurrence-pense idd(bigin:):::: Table procedure occurrence details-this table contains records of activities or processes ordered by, or carried by, healthcare provider the patient with diagnostic therapeutic purpose., Attribute person details -the person_is id of the person for whom the procedure is recorded. this may be system generated code.' ... 'visit detail-care, site id(bigint): this field provides information about the site where the visit detail took place"]
Input Query: procedures ied-seq num(integer): Table procedures ied details-contains ied procedures for patients, most notably icd-9 procedures, Attribute seq num details -provides the order in which the procedures were performed Reasoning: Let's think step by step in order
"produce the refined string list. are looking for a match to pprocedures, senddures_icd-sear jcd-seq_nnnt num', which seems to relate to procedure occurrence and its sequence number. The "procedures_ics jidd' part suggests we're looking at procedure accurrence, and the 'seq_num' part suggests we're looking for a sequential number order field. Given that, we should look for keys in the pprocedure_occune occurrence' table that represent order sequence.
Refined String List: 'procedure prooeduue__ccuureeceeepee occurrence-person, sidnindddbii))'' 'procedure occurrence-visit occurrence_ jid(biiidt))). procedure_occureee owcurrence- pecurrece-proceduure conpeetteeccppppeeeee (aacepp_id(integer)
"procedure pccurrenc-proceedue, dateddatttt "procedure cocerddue_ooccuueeeeeequee occurrene-quaaiitiineee"""""""
29

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Figure 12: EXAMPLE 2: MCQ Formatter.
MCQ Formatter
You an expert MCQ formatter. Your task is to take list of schema values and convert them into multiple choice question format with (letter)Schema value, where the schema values should be key(description).
Follow the following format.
Input: input list of schema values Meq: MCQ format of schema values e-g (A) (A)Schema value, (B)Schema value. Do not include additional options, only schema values options. where the schema values should key(description). Add Match option.
Input: ["" proceddeeee sccurrence-perseee id(biii)))'.
'procedure occurrence-visi occurrence id(bigint)', 'procedure occurrence-visil detail id(bigint)) (A) procedure occurrence- peerrennne-eeeonnn..m id(bigint)'.
procedure occurrence proceedueee suncept_id(nteege procedure occurrence-provin prrvviieer id
(bigint)""] Mcq:
(B) procedure occurrence-visit occurrence id(bigint)'. (C)' procedure httprrdduueeocccureeeeeeeee occurrence-roocedure concept id(integer)',
(D)
idiiiidi proceeure_occeueeeee porrreeeeeeeeeeeeoooieee_____ (bigint)", (F)No Match
Input: poooee-nofeeeextttaachaaaaaaaaa iote-nooe_reet peoee_eeettaarcca((maa))) Meq: peteenoot__iilleevarcaaa(33333
hat.eerrrceeallecaaaarrarro/ 33lnooe-noee_iinhe(aaacaaaaaa (C)note- note source value(varchar(55)) (D)eote-note text(varchar(mmu)), (E)No Match
Input:
p""proceuure occurrence-person id(bigint)'.
'procedure occurrence-visit occurrence id(bigint)', prarredure_ooccurenneepeeeeeeeee
procedure occurrence-proceduur: date(date)".
concept_id(integer
procedure accurenc--uaaniiiinneeee
Meq;
"(A)'procede proceduue_occcureeee occurrence pereiceeeppooo__ie "rison_d(biggit",,
(B) www.eooreeceeeviiieeeeeee seccurence-visioocccrrreee www.or/cccceeeeeee (itireence_i(bbggntt)"
(C) procedure_ocaee occurrence petpceeerocceeeeeeeeeee datti
(D) pmocedure_occurreen pecptuuurnncc-opceuueeeeeeeeeeee pnocedure_conceepiiiiieeeee
(E) procedure_occunee cecuurreneeqnantttinnneeee (F)No Match".
Figure 13: EXAMPLE 2: Confidence scoring.
Confidence scoring
You are schema matching expert. Your task is given the input and the MCQ format of the schema, predict the likelihood relation from 0-100 of the input query being related to each option. Your scores will be calibrated. there is good match score Match as
Follow the following format.
Input Moq: Input MCQ format of schema values Input Query: input query Relation: Relation score input query being related to the option as value. Assess each independently including No Match, returning a score from 0-100 for each.
Return with key as MCQ letter e.g (A) and score=value as JSON
Input Meq: (A)' procedure occurrence- perrren j id(higint)", (B)' proceeddure pprocedre__cccureeee occurrence-visit, occurrence nccurrenee_d((hiinnt) (C) procedure procedure_occurrencceee occurrence-proceedeee (B)' procedure pecurrence-providerr id (bigint)', . (F)No Match Input Query: procedureevents, m_v-itemivdddd Relation: ("(A): "(B)": 0, '(C)": 0. "(D)": "(E)": "(F)": 100)
concept jiddiiiegint''
(D) procedure occurrence-visit detail id(higint)',
Input Meq: (A)note-note ptt_tottteeeetteeeeeeeeeeeeeeeet text(yarchar(max)) Shhoot-note_ttleeevacchaea (i5.aee(aarccar(255)) (C)note-note, @Snote-note_souree howee_surree_____toooooo laape__ourc___eeeeeeeeee value(yarchar(55)) (D)mote-nod note (ax.(vacccarmmaxx).. (E)No Match Input Query: noteevents-tex. Relation: ("A": 90, "B": "C": 0. "D": 90. "E": 10)
Input Meq: (A) procedure nccurrence-person, iidbiniitdiiidi""...........ii". (B) pwwpedureeoceeeeeeeee occurrence-visit secureenevviss_ecccureeeee occurrence, sxcurrence_dd(hiiintt) (C) procedur__ccuurrecceeeee accurrence-procedure "areedduureddtt(aatt...... date(date)", (D) tatttrreocccuueeceppppeeeeee cetnocduueeccocppp__eeee concept_id(inteerr) (E) proccedee ppocedure_occureeeee cawpureeooccuueeeeguuue sesuurence-quunittiitteeee
(F)No Match Input Query: procedures
peredurrs_icd-seq icd--seq_____eeee [cd-seq_nun(integer) Table
procedures ied details-contains ied procedures for patients, most notably icd-9 procedures, Auribute seq details -provides the order in which the procedures performed Relation:
"((AY (BY (CY (D) (Ey 90, (Fy: 10)".
30

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
C.2. LLM Evaluator
We provide examples of the LLM evaluator, showing demonstrations achieving high and low scores.
LLM Evaluator Example: High Score
You are a schema matching expert, your task is to rate any of the suggested matches are potential good matches for the query. Be lenient and rate match as good (4 5) if is relevant to the query. Rate the matches from 1-5. If none of the matches are good, rate O.
Follow following format.
Query: The query.
Answers: possible matches
Reasoning: Let's think step by step in order to (produce the rating). We
Rating: Rate any of suggested matches are good for the query from 1-5, Only output rating and nothing else.
Query: noteevents-charrdtte
Answers: [" "ooee_date(date)", "note-note_daiee pooe__dactimme(dactimee) 'note-visit_occee
cecittoccurrece_idoeeee jidiiggiiddii "note-prowe pttppporrooodiieeeeeee
Reasoning: Let's think step by step in order to
LLM Response
produce the rating. We can see that the query is looking for a match related to the date of a note event a chart. The first suggested matches, nntee-ooe date(date)) and notte-note (atetiie(datetime)). are both related to dates and notes, making them potentially good matches for the query. The other three suggestions are not directly related to dates or note events, they not good matches. Rating: 4
Figure 14: LLM evaluator example, rated with high score.
31

---

Bootstrapping Sepf-mmprovement of Language Model Programs for Zero-Shot Schema Matching
LLM Evaluator Example: Low Score
You are a schema matching expert, your task is to rate any of the suggested matches are potential good matches for the query. Be lenient and rate match as good (4 or S) if is relevant to the query. Rate the matches from 1-5. If none of the matches are good, rate
Follow following format.
Query: The query.
Answers: possible matches
Reasoning: Let's think step by step in order to (produce the rating). We
Rating: Rate any of suggested matches are good for query from 1-5, Only output rating and nothing else,
Query: cptevents-uubseccionneadrr
Answers: ['visit_ooe ocw.urencce-viiit......... sisperceeeeisittteee__e__ee ses___ty______eeeeeeeeee sarr__cncept____ concepttid(int',,
"visit detail-visit, detail concept sincept_id(int',, 'procedure occurrence-procedure wttpee-rooocceeeooeeeeeeee serereddrreccncep....e conceptt id(int),, 'procedure_ visit detail id(int)", 'condition occurrence-vis visit occurrence ]d(int)']
proceduue__ccuurrenceee occurrence-
Reasoning: Let's think step by step in order
LLM Response
"produce the rating. We that the query is looking for match related to spseeenns--ubseetioolle However, none of the suggested matches seem to directly relate to this query. They refer to different concepts such as catitnnccrrenee,, 'visit_de detail', pnocedure_occuurenee/ occurrence', and inndition_occurrenee Therefore, none of the matches seem to be good fit query. Rating: 0
Figure 15: LLM evaluator example, rated with low score.
32

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
D. Additional experiments
D.I. Source of gain ablation: Why does it work?
Matchmaker's performance relies on the generated candidate matches. Given its strong performance compared to baselines, we investigate which candidate generation approach contributes most to Matchmaker's success. To disentangle the role of each candidate generation method, we assess Matchmaker with (I) reasoning-bon (reasoning _aliinngoollylggg_________ and (2) semantic candidates via retrieval only poamantic_on_ly
candidates from the LLM only
The results in Table 6 show that reasoning- based candidates outperform semantic retrieval-based candidates. This finding suggests that LLM reasoning over the database hierarchy and data types produces better candidates than semantic matches that do not consider hierarchical relationships. In some cases (e.g.. Synthea @cc 1). the inclusion of retrieval based candidates harms performance. However, the overall results indicate that Matchmaker benefits from both candidate generation approaches, with reasoning massenin--based candidates providing greater value. These results highlights the value of diverse candidate generation mechanisms to enhance Matchmaker's overall performance.
Table Understanding the impact different candidate generation approaches on Matchmaker.
Matchmaker
reasoning only
semant ic_only
Synther MIMIC
@
62.20 + 2,50
61.60 1.50
60.20 = 2.20
acc
68.80 # 2.00
68.70 = 1.60
64.50 2.80
71.10 = 2.00
70:40 1.00
67.10 = 3.10
acc
73.00 1,90
acc
78.60 2.50
78.50 1.50
77.40 0.90
80.90 1.10
79.40
0.30
80.20 0.40
D.2. Number of LLM calls
Goal. To compare the number of LLM calls required by Matchmaker and other baseline methods for schema matching on the MIMIC-OMOP and SYNTHEA-OMOP datasets.
Experiment. We count the number of LLM calls made by each method during the schema matching process on both the MIMIC-OMOP and SYNTHEA-OMOP datasets. For methods that do not rely on LLMs (e.g., SMAT), we consider the number forward passes through the neural network as equivalent to an LLM call for comparison purposes.
Results. Table 7 presents the number of LLM calls required by each method on the two datasets.
Table 7: Number of LLM calls
Method
MIMIC-OMOP SYNTHEA-OMOP
Matchmaker
1340
890
ReMatch
268
178
Jellyfish-13b
24771
29637
Jellyfish-7b
24771
29637
LLM-DP
24771
29637
SMAT
24771
29637
Discussion. The results in Table highlight the efficiency of Matchmaker and ReMatch in terms of the number of LLM calls required for schema matching.
Both Matchmaker and ReMatch formulate schema matching an information retrieval problem, which significantly reduces the search space compared to the binary classification formulation used by Jellyfish- 13b, Jellyfish-7b, LLM-DP, and SMAT.
The high number of LLM calls required by Jellyfish-13h Jellyfish-7b, LLM-DP, and SMAT can be attributed to their formulation of schema matching as binary classification problem over the Cartesian product of source and target attributes. In this formulation, the LLM is prompted to provide label of Yes/No for each pair of source- suuree-target attributes, resulting in large number of LLM calls that scales (O(n²)). Consequently, these methods are computationally expensive and less scalable compared to Matchmaker and ReMatch, which employ a more efficient approach.
The fewer number of LLM calls used by Matchmaker and ReMatch has practical implications terms of computational cost and runtime efficiency. By reducing the number LLM calls, these methods can perform schema matching more quickly and with lower computational overhead compared to methods that rely on large number of calls. This particularly
33

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
important when dealing with large-scale schemas or when schema matching needs be performed frequently in real-world applications.
D.3. Matchmaker with other L.MSs
Goal. To understand the performance of Matchmaker when using a less powerful LLM backbone compared to GPT-4, and contrast it with the ReMatch baseline using GPT-4.
Experiment. We evaluate the performance of Matchmaker using GPT-3.5 as the backbone LLM for all components, instead of GPT-4 which was used in the main experiments. We compare this to the performance of Matchmaker with GPT-4 and ReMatch with GPT-4. All other aspects of the setup remain the same as in the main text.
Results. Table shows the schema matching accuracy @k for the different methods. We observe that Matchmaker with GPT- 3.5 performs worse than Matchmaker with GPT-4, which is expected given GPT-3.5 a less powerful LLM. Interestingly, Matchmaker with GPT-3.55 achieves comparable performance ReMatch with GPT-4, despite GPT-3.5 being much weaker LLM than GPT-4. On MIMIC, Matchmaker with GPT-3.5 slightly outperforms ReMatch with GPT-4 for accuracy @ and is competitive at higher k. On Synthea, performance is similar for accuracy @ but Matchmaker with GPT-3.5 outperforms ReMatch with GPT-4 for accuracy@* @3 and accuracy @5.
Table 8: Comparison of schema matching performance of different baselines.
Matchmaker (GPT-4
Matchmaker (GPT-3.5)
ReMatch (GPT-4)
Synthca MIMIC
ace 1
62.20 2.40 1
48.30 2,80
42.50
@cc@@3
68.80 2.00
62.00 +
63.80
acc
71.10 2.00
70.00 +:
72.90
acc
70.20 + 1,70
47.80 +: 3.20
50.50
3
78.60 + 2.50
63.30 = 4.30 f
58. 10
5
80,90 + 1.10
77.10 = 0.70
74.30
Discussion. These results demonstrate that the Matchmaker approach of using a compositional LLM program is quite robust and can provide good schema matching performance even with weaker L.LM backbones. The fact that Matchmaker with GPT-3.5 is competitive with ReMatch using GPT-4 highlights the strength of the multi-stage Matchmaker approach over ReMatch's 's single-stage stage LLM usage. However, using a more powerful LLM like GPT-4 still provides significant gains, underlining the importance of using an LLM with powerful reasoning capabilities for this complex task.
D.4. Further performance results: ReMatch reimplementation
Goal. To compare the performance Matchmaker against the ReMatch baseline, using both the original reported results from the ReMatch paper and the re-implementation of ReMatch.
Experiment. In the main paper, report the performance of the ReMatch baseline using the results directly from the paper, as code is not available for ReMatch. However, for completeness, we also re- re-implement the ReMatch approach based on the details provided in the ReMatch paper.
Our re-im --implementtaiin uses the OpenAl Ada-002 text embeddings for the retrieval step, following the same procedure as ReMatch for chunking and creating documents. We then use the same prompts as described the ReMatch paper for the schema matching task. We compare the performance of our re-implemented ReMatch with the original reported results and Matchmaker.
Results. Table 9 presents the schema matching accuracy @k for Matchmaker, the original ReMatch results, and our re-implemented ReMatch. We observe that Matchmaker consistently outperforms both the original ReMatch results and our re-implementation across all metrics and datasets. We also find the re-implemented ReMatch achieves lower performance compared to the original reported results.
Table 9: Comparison of schema matching performance different baselines.
Matchmaker
ReMatch (Original)
ReMatch (Reimplemented)
Synthes MIMIC
@cc@@
62.20 2.40
42.50
41.99 0.61
68.80 2.00
63.80
46.63
71.10 :: 2.00
72.90
46.63 1.99
acc@!
70.20 +: 1.70
50.50
29.10 1 0.80
acc60
78.60 + 2.50
58.10
32.71 0.35
80.90 +: 1.10
74.30
33.46 0.63
34

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
Discussion. These results further confirm the superiority of Matchmaker over the ReMatch baseline, even when considering re-implementation of the method. The lower performance of the re-implemented ReMatch compared the original reported results could be due to differences in implementation details, such the choice of text embeddings variations not accounted for. However, it is important to note that even with these differences, Matchmaker consistently outperforms ReMatch (original) by significant margin. The fact that Matchmaker achieves strong performance gains over both the original ReMatch and our re-implementation underscores the value of the novel techniques introduced in Matchmaker, such as the multi-stage language model program, the use of diverse candidate generators and the self-improvement mechanism through synthetic in-context examples.
D.5. Improving performance: Use of Existing Mappings to remedy errors
Goal. To investigate the potential performance improvement Matchmaker when leveraging readily available mappings to rectify errors between directly mapped attributes.
Experiment. In schema matching, certain attributes like pource_valuee and concept id have direct mapping (e.g. in OMOP). If Matchmaker incorrectly maps the source attribute to the wrong target attribute (e.g., mapping to source_value instead of concept id or vice versa), these errors can be easily rectified by leveraging the existing relationship.
To simulate this error correction, we implement post-processe post-processing step where we adjust Matchmaker's predictions the predicted target attribute has a direct mapping to the true target attribute. apply this correction for all values of k and measure the resulting performance improvement.
Results. Figure 16 shows the accuracy gains across different values of k when applying the mapping correction. We observe consistent performance improvements across all k values. These results indicate that leveraging knowledge can indeed help rectify some of the errors made by Matchmaker.
0.72
Matchmaker
w/ Schemn domain knowledge
0.70
Accuracy
0.68
0.66
0.64
0.62
0.60
2
5
K
Figure 16: Performance improvement in Matchmaker when leveraging readily available mappings to correct errors between directly mapped attributes like source_ valueeeaaeeeeeeeeeeeeeeeeeeee and concept id.
Discussion. While the results demonstrate the potential benefit of using existing mappings for error correction, it is important to note that the performance gains are relatively modest compared to other strategies like human-in-the-loop deferral based on Matchmaker's confidence scores (as shown in the main text).
Moreover, the mapping correction relies on the availability of direct mappings between attributes, which may not always exist in practice. Therefore, while this approach can serve as useful prst-prrcessing step, it should be seen as a complementary technique to used alongside other strategies like numan-in-the-Ioee for improving schema matching performance.
35

---

Bootstrapping Self-Improvement of Language Model Programs for Zero-Shot Schema Matching
D.6. Comparison of Matchmaker ontology matching tasks
While Schema matching and ontology matching are seemingly related, in reality, they are completely different tasks. Specifically, schema and ontology matching fundamentally differ in their task and available information. Ontology matching leverages richer contextual info, including properties, axioms, rules, concept hierarchies and additional annotations. In contrast, schemas are sparser, with only attribute names, data types, descriptions and links.
Despite the difference for completeness, we evaluate recent LLM ontology match methods using GPT-4 backbones to mirror Matchmaker namely: OLaLa (Hertling & Paulheim, 2023) and LLMs40M (Giglou et al., 2024).
As shown in Table 10, Matchmaker outperforms these methods on both datasets.
Table 10: Accuracy @ I: Matchmaker vs two LLM-based Ontology matching methods.
Method
MIMIC
Synthea
Olala
33.58 = 0.47
31.53 = 3.37
LLMs40M
44.78 + 0.41
64.50 + 2.02
Matchmaker (Ours)
62.20 2.40
70.20 #1.70
D.7. Detailed error analysis
Goal, We wish to understand different dimensions of Matchmaker's errors.
Discussion, We analyze the errors made by Matchmaker and find two categories of errors.
17% of Matchmaker's errors occur when attempting find matches for source attributes that have corresponding target attribute.
The remaining 83% involve selecting incorrect semantically related attributes. For these incorrect matches, we find mean semantic similarity of 0.862 between the erroneously predicted attribute and the true target attribute. This confirms that Matchmaker typically selects attributes semantically close to the correct match rather than completely unrelated attributes.
These results further provide an understanding of Matchmaker's errors, as well as showing how they can be addressed both via uncertainty deferral and remediation, being easy to identify and correct.
D.8. Ranking ablation
Goal. Assess the importance of ranking to Matchmakers performance.
Results. Below we ablate the ranking step. The results shown highlight the importance of the re-ranking step towards achieving better accuracy@11
Table 11: Comparison of Matchmaker models with and without ranking on MIMIC and Synthea datasets.
Matchmaker (with ranking)
Matchmaker (No ranking)
MIMIC
Acc@1 1
62.20
57.00
3
68.80
66.90
Aco@5
71.10
71.10
Synthea
@cc@1
70.20
62.40
Acc@3
78.60
77.20
Acc@5
80.90
80.90
36


Reviews:
[{"review_id": "FBYefFsPh7", "reviewer": "Reviewer_9YfQ", "summary": "This paper presents Matchmaker for schema matching problem, the task of finding matches between attributes across disparate data sources with different tables and hierarchies. Matchmaker has 3 main stages: candidate generation, refinement and confidence scoring. The authors also propose a synthetic data-based in-context demonstration selection strategy to further improve the approach. Empirical results on 2 medical schema matching benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-9YfQ.``\n\nThank you for your thoughtful and insightful comments. We provide answers to each of the following in turn.\n\n---\n\n### **(A) Clarifying and motivating paper area as healthcare**\n\nWe agree with the reviewer that Matchmaker is generally applicable outside of healthcare. However, we selected the \"Health/Medicine\" track for three key reasons:\n\n- **Significant impact on healthcare/medicine:** Our main motivation is real-world healthcare integration, where schema matching remains largely manual and time consuming (e.g., mapping MIMIC to OMOP took 500 hours [Paris et al., 2021], Sec. 1). Schema matching is critical and has the potential for significant impact in healthcare due to fragmented datasets across institutions, with inconsistent schemas and terminologies. Effective matching enables data interoperability, allowing the creation of larger, integrated datasets which is essential for clinical data integration for downstream models, as well as, for external validation of models (see Sec. 1 and Impact Statement). Hence, advances in schema matching like Matchmaker have a potential for significant impact on healthcare.\n\n\n- **Well-understood problem in healthcare & complexity of healthcare:** We use two real-world healthcare benchmarks—MIMIC-OMOP and Synthea-OMOP—commonly used in prior work (Sheetrit et al., 2024; Zhang et al., 2023; Narayan et al., 2022). Additionally. these healthcare benchmarks also reflect the complexity of healthcare data schema matching.\n\n\n- **Health-specific design:** Matchmaker supports privacy-preserving schema matching, which is essential in healthcare where access to raw patient data is limited. It operates solely on schema-level metadata (Sec. 3.1), a realistic constraint in health contexts. In other words. the constraints of healthcare are relevant to the design.\n\n\nWe hope this clarifies. That said, we thank the reviewer for the suggestion and have added experimental validation on other datasets from other domains as suggested (e-commerce and general knowledge bases) to demonstrate Matchmaker’s generalizability — see *response (B)*\n\n---\n\n### **(B) Additional datasets beyond healthcare**\n\nWe thank the reviewer for the suggestion to assess Matchmaker on other domains, such as e-commerce. In response, we conduct **new experiments** on non-healthcare datasets: Magellan (e-commerce as suggested) and WikiData (general knowledge base). The results are shown below and confirm Matchmaker’s strong performance in domains beyond healthcare. \n\n| Dataset   | Matchmaker | ReMatch     |\n|-----------|------------|-------------|\n| Wikidata  | 0.95 ± 0.04 | 0.84 ± 0.03 |\n| Magallen  | 1.00 ± 0.00 | 1.00 ± 0.00 |\n\nHowever, we note that these datasets pose simpler challenges compared to healthcare. They typically involve single-table schemas with fewer columns and focus on direct feature-to-feature matching. In contrast, our healthcare tasks require reasoning over dozens of source tables and hundreds of attributes—first identifying the relevant target table, then performing column-level matching.\nThese findings reinforce our focus on healthcare schemas as they better showcase the advantages of advanced schema matching techniques.\n\n**Action taken:** We will include these new results and discussion in the camera-ready version. Thank you again for the suggestion!\n\n---\n\n### **(C) Clarifying Novelty**\n\nWe respectfully disagree that Matchmaker is merely an application of existing techniques. We clarify that our novel contributions are fourfold:\n\n- **Novel compositional LLM program:** Unlike prior work using single LLM calls (Sec. 2, Table 3), Matchmaker employs a multi-stage compositional approach that enables more complex reasoning and superior performance (Appendix A.1).\n\n\n- **Novel optimization mechanism for zero-shot self-improvement**: We introduce a novel optimization method via synthetic in-context examples to self-improve without labeled data (Sec. 4.4). This significantly outperforms other self-improvement methods (Table 2). Moreover, the optimization mechanism is applicable to other compositional LLM programs.\n\n\n- **Novel formulation:** As detailed in Sec. 3.2, we reformulate schema matching as an information retrieval task rather than binary classification, resulting in better efficiency (Appendix D.2).\n\n- **Human-in-the-loop deferral:** Unlike existing methods, Matchmaker permits deferral to humans (Sec. 5.3), an essential feature for real-world deployment, especially in healthcare.\n\n---\n\n*We thank the reviewer for helping us improve our work. We hope these answer your points, please let us know if there are any remaining concerns!*", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "No, benchmark datasets are limited.", "theoretical_claims": "N/A, no theoretical claims are made in the paper.", "experimental_designs_or_analyses": "Yes. The authors only verify the method on healthcare schema matching benchmarks, which is quite limited.", "relation_to_broader_scientific_literature": "The methodology seems to aim to solve the general Schema Matching problem (a established area for database research). However, the authors choose to verify the method only on healthcare domain, and select the paper's primary area as \"Applications->Health / Medicine\".\n\nThat said, I would expect the methodology should have health / medical-specific design or related insights to be considered of the selected primary area.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow, and the problem of Schema Matching is a practical problem. \n\nThe methodology itself is not that novel, more like an application-specific adaptation of existing techniques (CoT reasoning, in-context demo generation and selection, etc.)", "other_comments_or_suggestions": "See questions.", "questions_for_authors": "Again, I am confused about the primary area of this paper being Health / Medicine. The selection seems to justify why the two benchmarks in this paper are both health-related, however, it really contradicts with the design and presentation of the seemingly general design of the main approach.\n\nIf there are really no other established benchmarks besides these two, I would say creating a benchmark for domains like finance and e-commerce (like the authors mentioned in the abstract) is a more significant contribution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VfsLzOt5CU", "reviewer": "Reviewer_mppD", "summary": "This paper introduces Matchmaker, a self-improving compositional language model (LLM) program designed for schema matching, a critical task in data integration and interoperability. Schema matching involves finding correspondences between attributes across disparate data sources with different schemas and hierarchies, which is particularly challenging due to structural, semantic, and database heterogeneity. The authors propose a multi-stage LLM program that includes candidate generation, refinement, and confidence scoring. Matchmaker also self-improves in a zero-shot manner by constructing synthetic in-context demonstrations to guide the LLM's reasoning process. The paper demonstrates that Matchmaker outperforms existing ML-based approaches on real-world medical schema matching benchmarks, highlighting its potential to accelerate data integration and interoperability for machine learning-ready data.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-mppD``,\n\nThank you for your insightful comments.\n\nIn *Part 1* we address points **already** addressed in our paper (responses A-E), then in *Part 2* we respond to additional points (F-J).\n\n---\n\n## **PART 1 - Points **already** addressed in our paper (A-E)**\n\n---\n\n### **(A) Scalability and Computational Analysis**\nWe clarify that we provided a detailed analysis of LLM call complexity compared to baselines in **Appendix D.2 (Table 6)** and referenced/flagged on **L120**. Matchmaker significantly reduces LLM calls via our information retrieval formulation (**Sec. 3.2**), thus improving scalability, unlike exhaustive O(n²) evaluations used by LLM-DP and SMAT.\n\n---\n\n### **(B) Human-in-the-Loop Deferral Analysis**\n\nWe clarify that **Sec. 5.3 (Matchmaker in practice: Human-in-the-loop deferral and …)** *already* evaluates human-in-the-loop deferral. We show entropy-based deferral outperforms random deferral - with just 10–20% deferral significantly boosting acc@1 - Fig. 4(a).\n\n---\n### **(C) Missing comparisons with baselines from related work**\nWe clarify that we *already* compare Matchmaker to schema matching methods from our related work & Fig. 3. **Table 1’s** results include:\n\n- Supervised: SMAT (Zhang et al., 2021)\n\n\n- Pre-trained LLM: LLM-DP (Narayan et al., 2022; Zhang et al., 2023a)\n\n\n- Fine-tuned LLM: Jellyfish (Zhang et al., 2023b)\n\n\n- RAG: ReMatch (Sheetrit et al., 2024)\n\n\nTraditional methods are omitted, as prior work shows they underperform on these benchmarks.\n\n---\n### **(D) LLM  reliance**\n We clarify that we use GPT-4 to match LLM baselines. As per **Sec. 5.1 (L339–343)**, all systems use GPT-4 to ensure fair comparison and isolate system-level gains not tied to the LLM itself. While backbone quality matters, Matchmaker is LLM-agnostic.\n\n---\n### **(E) Performance Gains Attribution**\n\nWe agree that attribution is crucial and have analysed it in two ways:\n\n (i) **Sec. 5.2 & Table 2**: The ablation shows our synthetic in-context examples outperform self-reflection & that systematic example selection outperforms random or no examples, confirming it is the main driver of gains\n\n (ii) **Appendix D.1**: The ablation shows Diverse candidate generation (semantic + reasoning-based) outperforms single-type generation.\n\n---\n\n\n## **Action taken**: We realize our paper is dense, and hence, it is easy to overlook these points. To improve clarity and better help the reader navigate the paper, we will add a summary table in Appendix A showing where different issues are addressed.\n\n---\n## **PART 2 - Additional points (F-J)**\n\n---\n### **(F) Novelty**\n\n The reviewer suggests our work lacks novelty and is prompt engineering. We respectfully disagree and clarify our four key novelties:\n\n- **Novel Compositional LLM program:** Unlike prior single-call methods (Sec. 2, Table 3), our multi-stage structure enables complex reasoning. Appendix A.1 compares this with ReMatch\n\n\n**Novel optimization for zero-shot self-improvement:** We introduce a novel optimization method using synthetic in-context examples (Sec. 4.4), which outperforms other methods (Table 2). The process is applicable to other compositional LLM programs\n\n\n- **Novel task formulation:** Schema matching as information retrieval (Sec. 3.2)\n\n\n- **Human deferral support:** Matchmaker enables deferral to humans (Sec. 5.3), vital for real-world use\n\n---\n### **(G) Generalization to Non-medical Domains**\n\nWhile focused on healthcare due to its complexity and real-world importance (Sec. 1), we agree it's useful to test other domains. We conduct **new experiments** on Magellan (e-commerce) & WikiData (general knowledge) datasets, which include Amazon product datasets as suggested. These results confirm Matchmaker’s cross-domain performance, but also highlight the complexity of our healthcare datasets — reinfocing their selection\n|Dataset|Matchmaker|ReMatch|\n|--|--|--|\n|Wikidata|0.95 ± 0.04|0.84 ± 0.03|\n|Magallen|1.00 ± 0.00|1.00 ± 0.00|\nWe will add these new results to the camera-ready. Thanks for the suggestion!\n\n---\n### **(H) Confidence Scoring Validity and Consistency**\nOur MCQ-based confidence scores align with token-level calibration literature (Kadavath et al; Ren et al; Tian et al—Sec. 4.3). Entropy-based deferral confirms scores accurately reflect prediction uncertainty, significantly improving accuracy (Sec. 5.3).\n\n---\n### **(I) Theoretical Contributions**\nWhile empirical, our theoretical contribution is reformulating schema matching as information retrieval rather than binary classification, significantly reducing computational complexity (Sec. 3.2)\n\n---\n### **(J) Dataset Size**\nThe benchmark datasets are complex, real-world medical datasets (not small), e.g. MIMIC-OMOP (26 source, 14 target tables). Highlighting its complexity, it required 500 hrs of expert annotation. The datasets are also standard benchmarks (Sheetrit et al; Zhang et al; Narayan et al)\n\n---\n\n*We hope these answer your points, please let us know if there are any remaining concerns!*", "claims_and_evidence": "1. This paper claims that Matchmaker is more scalable than previous methods, but it does not provide a detailed analysis of computational complexity or runtime performance compared to other methods. This is particularly important given the large number of LLM calls required by some baselines.\n\n2. While the results on medical datasets are impressive, the paper does not provide evidence of Matchmaker's performance on non-medical datasets. Schema matching is a problem that spans multiple domains (e.g., finance, e-commerce), and it would be valuable to see how well Matchmaker generalizes to these domains.\n\n3. The paper discusses the potential for human-in-the-loop deferral based on confidence scores, but it does not provide a detailed analysis of how this would work in practice or how much human intervention would be required to achieve significant performance gains.\n\n4. The confidence scoring mechanism relies on prompting the LLM to provide a score between 0 and 100, which is problematic. Since the LLM is a black box, the validity and consistency of these scores are questionable. Moreover, the scores are generated independently for each candidate, making it difficult to compare them across different queries.", "methods_and_evaluation_criteria": "1. Lack of Methodological Innovation: The paper primarily relies on prompting engineering and does not introduce significant methodological innovations. Each step of the process depends heavily on the reasoning capabilities of the underlying LLM (e.g., GPT-4, GPT-3.5), which raises questions about the originality of the approach. The framework is more of a clever combination of existing techniques rather than a novel contribution to the field.\n\n2. Theoretical Contribution: The paper lacks theoretical innovation. It does not provide new theoretical insights or frameworks that could inspire other researchers. The reliance on LLMs for reasoning and scoring means that the paper does not contribute to the broader theoretical understanding of schema matching or LLM-based reasoning.\n\n3. Dataset Size and Baselines: The experiments are conducted on relatively small datasets (e.g., MIMIC-OMOP and Synthea-OMOP), with only 20-30 tables. This limits the ability to validate the effectiveness of Matchmaker on larger, more complex schemas. Additionally, the paper compares Matchmaker to only a few baselines despite mentioning several related methods in the related work section. A more comprehensive comparison, including classical schema matching approaches and other LLM-based methods, would strengthen the evaluation.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "There are several areas where the experimental design could be improved:\n\n1. The paper compares Matchmaker to several baselines, but it does not provide a detailed analysis of why Matchmaker outperforms these baselines. For example, it would be useful to know if the performance gains are due to the multi-stage approach, the self-improvement mechanism, or a combination of both.\n\n2. The paper does not discuss the sensitivity of Matchmaker's performance to different hyperparameters (e.g., the number of candidates generated and the threshold for confidence scoring). This information would be useful for practitioners who want to apply Matchmaker to their own datasets.\n\n3. The experiments are conducted on relatively small datasets, which limits the ability to validate the effectiveness of Matchmaker on larger, more complex schemas. The authors should consider testing Matchmaker on larger datasets or more diverse domains to demonstrate its scalability and generalizability.\n\n4. There is no efficiency or API cost analysis.", "relation_to_broader_scientific_literature": "This paper's reliance on advanced LLMs (e.g., GPT-4, GPT-3.5) for reasoning and scoring raises questions about the generalizability and reproducibility of the results. The paper feels more like a technical report on the application of GPT-4 to schema matching rather than a research paper that contributes novel insights or methodologies to the field.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Weaknesses:\n\n1. Lack of Methodological Innovation: The paper primarily relies on prompting engineering and does not introduce significant methodological innovations. Each step of the process depends heavily on the reasoning capabilities of the underlying LLM (e.g., GPT-4, GPT-3.5), which raises questions about the originality of the approach.\n\n2. Theoretical Contribution: The paper lacks theoretical innovation. It does not provide new theoretical insights or frameworks that could inspire other researchers.\n\n3. Dataset Size and Baselines: The experiments are conducted on relatively small datasets, and the paper compares Matchmaker to only a few baselines. A more comprehensive comparison, including classical schema matching approaches and other LLM-based methods, would strengthen the evaluation.\n\n4. Confidence Scoring: The confidence scoring mechanism relies on prompting the LLM to provide a score between 0 and 100, which is problematic. Since the LLM is a black box, the validity and consistency of these scores are questionable. Moreover, the scores are generated independently for each candidate, making it difficult to compare them across different queries.\n\n5. Reliance on Advanced LLMs: The paper's reliance on advanced LLMs (e.g., GPT-4, GPT-3.5) for reasoning and scoring raises questions about the generalizability and reproducibility of the results. The paper feels more like a technical report on the application of GPT-4 to schema matching rather than a research paper that contributes novel insights or methodologies to the field.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmheVByYbB", "reviewer": "Reviewer_15kv", "summary": "The authors introduce Matchmaker, a self-improving compositional LLM program, where multi-stage LLM calls are involved for candidate generation, refinement, and confidence scoring for the task of schema matching, which the authors formulate in the context of information retrieval. Its self-improving aspect comes from their optimization process that generates synthetic in-context examples used for the various LLM calls in the program. Their mechanism is tested against other frameworks, such as Jellyfish, LLM–DP, and SMAT using the MIMIC-OMOP and Synthea-OMOP datasets, which are evaluated via the accuracy@k metric. In addition, they tested various versions of Matchmaker that incorporate randomized, zero, and self-reflected in-context examples. Majority of the experiment results show their optimized Matchmaker achieves better performance amongst the other methods and versions.\n\n## update after rebuttal\nI have accepted this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-15kv.``\n\n\nThank you for your thoughtful and insightful comments!  We provide answers to each of the following in turn.\n\n---\n\n### **(A) Clarifications and questions on the dynamic nature of the algorithm**\n\n---\n\n### *Q1: Clarifying the dynamic nature of the algorithm - is it only generating synthetic examples?*\n\nIndeed, the synthetic in-context examples generation is the main optimization step. However, the dynamic nature of Matchmaker is broader in two ways than just synthetic in-context examples:\n\n(1) **Self-improvement mechanism**: As outlined in Section 4.4 and Algorithm 1 (Appendix A.3), Matchmaker evaluates its own execution traces to select high-performing intermediate outputs across the program’s stages. These selected traces are dynamically reused as few-shot demonstrations in subsequent executions (described next).\n\n(2) **Dynamic program behavior**: The result of these different bootstraped traces is not just the generation of synthetic in-context examples, but this also then serves to update the multi-stage LLM program's behavior.\n\nOverall, this results in Matchmaker’s self-improvement without labeled examples — which other schema matching methods can’t do.\n\n---\n\n### *Q2: Clarifying round 0 of the algorithm — how does it work before optimization?*\nTo clarify, in the initial round, Matchmaker operates without in-context examples. As detailed in Section 4.4 and Algorithm 1. \n\n- We first run the unoptimized Matchmaker (without any in-context examples) on evaluation examples from $D_{eval}$\n- We capture execution traces (intermediate inputs/outputs)\n- The LLM evaluator $E$ then scores these executions.\n- The highest-scoring traces (and their input-outputs) are used to bootstrap synthetic in-context examples.\n\nHence, Matchmaker “starts cold” with a zero-shot bootstrapping process (using its own successful traces) — allowing Matchmaker to self-improve without requiring labeled data, addressing a key challenge in schema matching.\n\n**ACTION TAKEN:** We will update Sec 4.4. to clarify this.\n\n---\n\n### *Q3: How exactly is the LLM $E_i$ trained?*\n\nWe clarify that the LLM itself is not specifically trained (or fine-tuned) for schema matching. Rather, Matchmaker leverages a general-purpose frozen LLM (e.g., GPT-4) within its compositional program (candidate generation, refinement, and scoring). Matchmaker's key innovation is not fine-tuning the LLM weights but dynamically optimizing the end-to-end compositional system behavior via synthetic in-context examples. So in \"training\" is in the sense of this optimization of the compositional LLM program.\n\n---\n\n### *Q4: Clarifying synthetic examples: Are the synthetic examples still “unlabeled” or “labeled” when they are optimized? Is there human-in-the-loop intervention?*\n\nTo clarify, the synthetic examples generated remain \"unlabeled\" in a traditional supervised sense, as we never explicitly verify or label them via human annotations. Instead, verification is implicitly done via an LLM evaluator, which assesses the quality of the matches through a scoring system (scale of 0-5). Thus, synthetic examples are optimized based on evaluator scoring rather than explicit human labeling. This approach deliberately removes the requirement for manual annotation and supports a fully autonomous zero-shot self-improvement system. \n\nHence, Matchmaker can operate without human intervention for the optimization step — rather, the system itself generates the quality labels. However, at deployment time, we show in Sec. 5.3 that a human-in-the-loop can further enhance performance, e.g., by deferring high-entropy predictions.  \n\n**ACTION TAKEN:** Update Sec 4.4 to clarify this.\n\n---\n\n### **(B) Typos**\n\nThank you for flagging the typos; we will correct them in the camera-ready version.\n\n---\n\n*We thank the reviewer for helping us improve our work. We hope these answers your points. Please let us know if there are any remaining concerns!*", "claims_and_evidence": "Yes, they are supported via the experiments against the existing methods and ablations of their own method.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria makes sense for the schema matching task.", "theoretical_claims": "There aren’t any theoretical claims.", "experimental_designs_or_analyses": "I did check, including the additional details and analyses provided in Appendix B and D respectively.", "relation_to_broader_scientific_literature": "The main contribution is the multi-call dynamic prompting mechanism for the task of schema matching. This is useful in cases where the dataset cannot be fully accessed for privacy reasons. The most recent previous work, ReMatch, has an LLM-based solution too but it remains static in nature, lacking in-context instances.", "essential_references_not_discussed": "I believe the related works section is sufficient in understanding the paper.", "other_strengths_and_weaknesses": "Strengths:\n* Application-driven ML Based: It is an important application of using LLMs for the task of schema matching.\n* Non-trivial way of finding synthetic examples – significant for cases where private data cannot and should not be accessed.\n* Generally comprehensive paper with satisfactory experiment and ablation setups.\n\nWeakness:\n* Need more clarity about the dynamic-nature of the algorithm (expressed these in the “Question for Authors” section).", "other_comments_or_suggestions": "I noticed the following typos:\n* Line 254: space between attribute’s and description\n* Line 387: improve instead of impfrove\n* Line 967: $D_{eval}$ instead of $D_eval$.", "questions_for_authors": "* Q1: From what I understand, it seems that this aspect is to do with generating synthetic examples. Just to confirm, is that the only aspect that is of dynamic nature?\n* Q2: How exactly does Round 0 work when it is not optimized yet? In Algorithm 2 in Appendix A, I see that the self-improvement optimization happens in all stages, but by first calling the entire algorithm (referring to Matchmaker($e_i$)). So what exactly are the instances used in that first call, i.e., Matchmaker($e_i$)?\n* Q3: How exactly is LLM $E_l$ trained?\n* Q4: Are the synthetic examples still “unlabeled” or “labeled” when they are optimized? How is that verified when there is no human-in-the-loop intervention? Or does this system need to have human-in-the-loop?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "jEjqfCDjL0", "reviewer": "Reviewer_pQV8", "summary": "The paper describes a technique for matching dataset schemas. They use a compositional language model for this.  They benchmark their solution against multiple competing works and usually achieve superior performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear ``R-pQV8.``\n\n\nThank you for your thoughtful and insightful comments! We provide answers to each of the following in turn.\n\n---\n\n### **(A) Related work - Magneto**\n\nThank you for pointing out this work (Magneto), we will incorporate a discussion of it in the related work of the camera-ready.  At a high level: Magneto shares a similar retrieve-then-rerank architecture to the ReMatch baseline, especially in its zero-shot configuration. Specifically, both Magneto and ReMatch retrieve candidate matches using embeddings and subsequently rerank candidates with an LLM, hence making their underlying approaches comparable.\n\nOn the other hand, we highlight that in Magneto, most of the benchmark tasks involve schema matching of a single source to single target table. This contrasts our healthcare setups, where there are multiple source tables and multiple target tables. Hence, our healthcare tasks being more complex, needing reasoning over the table match first and then the column match.\n\nAdditionally, besides datasets our Matchmaker fundamentally differs from Magneto (and ReMatch) in 3 important ways:\n1. **Compositional LLM Program:** While Magneto uses a two-stage pipeline (retrieval and reranking), Matchmaker introduces a multi-stage compositional LLM program with candidate generation, refinement and confidence scoring. This structured approach allows more nuanced reasoning about schema relationships.\n2. **Diverse Candidate Generation:** Matchmaker combines both semantic retrieval and reasoning-based candidate generation, whereas Magneto relies on semantic retrieval only\n3. **Self-Improvement Mechanism:** Matchmaker introduces a novel zero-shot self-improvement mechanism using synthetic in-context examples, which doesn’t exist in other methods.\n\nFinally, as per the reviewer's suggestion, we have evaluated Matchmaker on datasets from the suggested paper to illustrate applicability beyond healthcare. *See response (B)*\n\n**ACTION TAKEN:** We will include a discussion on Magneto in the camera ready\n\n---\n\n\n### **(B) Additional datasets beyond healthcare**\n\nWe thank the reviewer for the suggestion to improve the paper. We first clarify that our primary focus was on healthcare schema matching due to its real-world importance (and value to advance ML in healthcare settings), coupled with the structural complexity (see Section 1). Moreover, the healthcare schema matching datasets are widely recognized and extensively used in the schema matching literature  (Sheetrit et al., 2024; Zhang et al., 2023; Narayan et al., 2022), due to their complexity and realism.\n\nThat said, we agree that evaluating beyond this healthcare domain is valuable to evaluate generalizability. Consequently, we have conducted **new experiments** on datasets from the suggested work, *not* from the biomedical domain. Specifically, we evaluated our approach on (i) Magellan (e-commerce product data) and (ii) WikiData (general knowledge base data). \n\nThe results can be found below. They highlight Matchmaker's strong capability and generalizability compared to ReMatch (which performs similarly to Magneto on the same datasets). \n\nOur experiments show that Matchmaker achieves superior performance, confirming its generalizability across domains. However, these datasets represent significantly less challenging matching scenarios compared to our healthcare schemas. This is evidenced by the relatively high performance across all methods.  \n\n| Dataset   | Matchmaker (Ours) | ReMatch     |\n|-----------|------------|-------------|\n| Wikidata (General knowledge)  | 0.95 ± 0.04 | 0.84 ± 0.03 |\n| Magallen (e-commerece)  | 1.00 ± 0.00 | 1.00 ± 0.00 |\n\nMoreover, these datasets typically involve single-table schemas with a small number of columns. In contrast, the healthcare schema matching tasks (from our paper) are significantly more challenging. These involve dozens of source tables and hundreds of attributes and require the model to first reason over the entire schema to determine the relevant target table before attempting column-level matching. We believe these results also reinforce our decision to focus on healthcare schemas, which present more challenging real-world matching scenarios that better differentiate the capabilities of advanced matching techniques.\n\n**ACTION TAKEN:** We will include these new results and discussion in the camera-ready version. Thank you for the suggestion!\n\n---\n\n### **(C) Framework availability**\nWe appreciate the reviewer’s enthusiasm to use Matchmaker. To confirm, we will release the full implementation upon acceptance, along with detailed documentation and tutorials for usage/extension beyond our evaluated setups. \n\nThat said, we include a base version at the following anonymized repo: https://anonymous.4open.science/r/Matchmaker-base-2641\n\n---\n\n*We thank the reviewer for helping us improve our work. We hope these answer your points, please let us know if there are any remaining concerns!*", "claims_and_evidence": "I have not found unsupported claims. However, I think the impact of the solution lives or dies by how easy it is to use.", "methods_and_evaluation_criteria": "The authors benchmark against several existing solutions of which one (rematch) is the most similar to their solution. The approach seems sound, but it is hard to determine if there are other solutions which should be included.", "theoretical_claims": "There are little to no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experimental designs seem to be sound.", "relation_to_broader_scientific_literature": "After looking for similar solutions, it seems there are an increasing amount of works who claim superior performance. Providing easy-to-implement benchmark tasks and benchmark code is crucial to determine which solution works the best.", "essential_references_not_discussed": "In this area, which is rapidly developing, it is hard to see essential references. Moreover, this is an interdisciplinary field which overlaps with the database community. I could find https://arxiv.org/abs/2412.08194 as example that was not discussed in the paper.", "other_strengths_and_weaknesses": "The paper is very well formatted, contains examples and a few examples\nIt is not clear if the technique will be available and useable for a broader audience which is crucial for actual use and further benchmarking. Adding (anonymous) code that works well and is extensible and maintainable is important.", "other_comments_or_suggestions": "I think this solution could prove useful outside of health datasets and machine learning. Perhaps try benchmarks provided by the database community to see if your solution does indeed translate. A work that does this, for example: https://arxiv.org/abs/2412.08194 or https://arxiv.org/abs/2408.14507", "questions_for_authors": "I would quite like to try out your solution. Is there any manner you can provide me to test it for it's intended use?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Matchmaker, a compositional LLM program for schema matching featuring a novel zero-shot self-improvement mechanism. The reviews were mixed, with positive (4, 4) and negative (1, 2). The authors provided detailed rebuttals addressing concerns, including adding new experiments for generalizability. The positive reviewers were satisfied (yet one of them reported low familiarity with the field). While the negative reviewers acknowledged the rebuttal, they did not update their scores or engage further, leaving some concerns formally unresolved from their perspective, though substantively addressed by the authors. Overall, the strengths and successful rebuttal of key criticisms lean towards a weak accept.

**Consolidated Strengths:**

* **Addresses Important Problem:** Tackles the challenge of schema matching with real-world application, particularly demonstrated in healthcare domains (R-9YfQ, R-15kv, R-pQV8).
* **Well-Presented:** The paper is well-written, clear, and includes useful examples and extensive supplementary material (R-9YfQ, R-15kv, R-pQV8).
* **Practical Features:** Incorporates valuable aspects like human-in-the-loop deferral support (validated in Sec 5.3) and operates on schema-level metadata, respecting privacy constraints (Authors highlighted; R-15kv satisfied; R-mppD concern addressed by authors pointing to existing analysis).

**Consolidated Concerns:**

* **Scalability/Cost Analysis:** Request for more detailed runtime/cost analysis beyond LLM call complexity (R-mppD). Authors pointed to existing call complexity analysis (Appendix D.2/Table 6) showing improvement over baselines, but R-mppD did not acknowledge this specific point was resolved.
* **Confidence Scoring Validity:** Skepticism about LLM-generated confidence scores (R-mppD). Authors defended via literature and practical HITL results (Sec 5.3); R-mppD did not acknowledge this as resolved.

**Mixed impressions:**

* **Novelty/Contribution Debate:** Proposes a compositional LLM program with a zero-shot self-improvement mechanism using synthetic examples, differentiating it from prior work (Authors' claim, accepted by R-15kv, R-pQV8). However, there were concerns raised about methodological innovation (R-mppD, R-9YfQ). Authors rebutted with specific contributions. While R-15kv/R-pQV8 seemed convinced, R-mppD/R-9YfQ did not update their stance post-rebuttal, leaving this point contested by them.
* **Evaluation Scope:** Concern about initial focus only on healthcare datasets (R-mppD, R-9YfQ, R-pQV8). Addressed by authors during rebuttal with new experiments on Magellan/Wikidata showing generalizability; reviewers acknowledged rebuttal but did not update scores based on this. Additionally, the method outperforms existing methods on challenging real-world healthcare benchmarks as reported in the original submitted paper.
Decision: Accept (poster)