Matchmaker: Schema Matching with self-improving compositional LLM programs

Authors: Nabeel Seedat, Mihaela van der Schaar
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: schema matching, data-centric AI, Large Language Models, healthcare

Abstract:
Schema matching -- the task of finding matches between attributes across disparate data sources with different tables and hierarchies -- is critical for creating interoperable machine learning (ML)-ready data. Addressing this fundamental data-centric problem has wide implications, especially in domains like healthcare, finance and e-commerce --- but also has the potential to benefit ML models more generally, by increasing the data available for ML model training. However, schema matching is a challenging ML task due to structural/hierarchical and semantic heterogeneity between different schemas. Previous ML approaches to automate schema matching have either required significant labeled data for model training, which is often unrealistic, or suffer from poor zero-shot performance. To this end, we propose Matchmaker -  a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker also self-improves in a zero-shot manner without the need for labeled demonstrations via a novel optimization approach, which constructs synthetic in-context demonstrations to guide the language model's reasoning process.  Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches, highlighting its potential to accelerate data integration and interoperability of ML-ready data.

Decision: reject
Reviewer scores: 5, 5, 3, 8

Reviews and discussion:
<thread>
creation date:
1730278326167

thread_type:
official_review

<thread_object>
creator: reviewer_jxrd

<content>
summary:
This paper introduces Matchmaker, a novel approach to schema matching using a self-improving compositional language model program. Schema matching is crucial for creating interoperable ML-ready data by finding correspondences between attributes across different databases. Matchmaker operates through three main components: multi-vector document creation, candidate generation (using both semantic retrieval and LLM-based reasoning), and confidence scoring. A key innovation is its ability to self-improve without labeled data through synthetic in-context examples. The method significantly outperforms existing approaches on real-world healthcare schema matching benchmarks (MIMIC-OMOP and Synthea-OMOP).

strengths:
1. Addresses a critical practical problem in data integration that has significant implications for ML development
2. Novel technical approach combining retrieval and LLM reasoning in a compositional program
3. Zero-shot learning capability through synthetic in-context examples, eliminating the need for labeled training data
4. Comprehensive empirical evaluation against multiple baselines
5. Practical considerations for deployment, including human-in-the-loop integration and uncertainty handling
6. Strong quantitative results showing 15-20% improvement over baselines
7. Well-documented implementation details and ablation studies

weaknesses:
1. Limited evaluation to two healthcare domain datasets, though the approach is claimed to be general
2. The synthetic in-context example generation process could be explained more clearly
3. The paper shows strong performance metrics but doesn't provide detailed analysis of where and why Matchmaker fails

questions:
1. How does the performance of Matchmaker scale with very large schemas (>1000 attributes)?
2. Could the approach be extended to handle many-to-many mappings between schemas?
3. How sensitive is the performance to the quality of attribute descriptions in the schemas?
4. What strategies could be employed to reduce the number of LLM calls while maintaining performance?
5. How might the system handle schemas in languages other than English?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628344461

thread_type:
official_review

<thread_object>
creator: reviewer_n216

<content>
summary:
In this paper, the authors considered the schema matching problem for
tabular structured data. In particular, a schema is defined as a set
of tables {T_1, ..., T_m}, where each table T_i has a set of
attributes {A_{i,1}, ..., A_{i,n_i}}. Additionally, it is assumed that
each table T_i is associated with some metadata describing its purpose
and content, and each attribute A_{i,j} is associated with some
metadata describing its type and relational context. Then, given a
source schema S including a set of attributes A_S and a target schema
T including a set of attributes A_T, the goal of schema matching is to
find a partial function f : A_S -> A_T such that if f(A) = B for
attributes A in A_S and B in A_T, then B is the corresponding target
attribute to the source attribute A.

The algorithm proposed in the paper for schema matching works as
follows. Given a source attribute A, the algorithm uses embeddings of
A and the target attributes to retrieve the top-k matching target
attributes. The generated set of target attributes is called the
semantic retrieval candidates for A. Then the algorithm generates a
set of reasoning-based candidates using a reasoning LLM. The union of
semantic retrieval candidates with reasoning-based candidates
constitutes the set of candidates for A. Then the algorithm uses a
refiner LLM to reduce the number of candidates, and it finally ranks
the resulting candidates and filters out the non-suitable ones. If the
resulting set of target attributes is not empty, then the top-scored
attribute can be considered as the corresponding target attribute for
A.

strengths:
S1) The schema matching problem is fundamental to generating large,
    integrated, and interoperable datasets. In this sense, this paper
    addresses a relevant and interesting problem.

S2) The experimental evaluation shows that the proposed approach
    outperforms other schema matching approaches in terms of accuracy.

weaknesses:
W1) The approach proposed in the paper essentially disregards the
    large body of work on schema matching that has been developed for
    decades in the database field.

W2) The schema matching problem is not properly formalized.

questions:
General comments

The schema matching problem is fundamental to generating large,
integrated, and interoperable datasets. In this sense, this paper
makes a contribution by proposing a new method for this problem that
takes advantage of certain capabilities of LLMs. Besides, the
experimental evaluation provides evidence that the proposed method
outperforms other schema matching approaches in terms of
accuracy. However, I have two serious concerns about the contribution
of this paper:

- The schema matching problem is not properly formalized. The authors
  provide a formal definition of this problem where they indicate that
  mapping function f "correctly" assigns each attribute of the source
  schema to an attribute of the target schema. How is the
  "correctness" of f defined? What are the properties that a correct
  mapping function f should satisfy? Does the algorithm proposed in
  this paper compute a function f that satisfies such properties? None
  of these questions is answer in the paper.

- The authors disregard the large body of work on schema matching that
  has been developed in the database area. I am not going to mention
  here the relevant literature on schema matching, which is an
  established area within databases, but I would like to note that one
  can already find surveys on this topic from more than 20 years ago:

  Erhard Rahm, Philip A. Bernstein: A survey of approaches to
  automatic schema matching. VLDB J. 10(4): 334-350 (2001)

  Notice that the problem considered in this paper is schema matching
  for relational databases, which is exactly the scenario discussed in
  this survey.

  A first obvious step in considering the work on schema matching done
  in databases is to compare the method proposed in this paper with
  methods from the database field. But this is just the tip of the
  iceberg and probably not the most fruitful way. The method proposed
  in this paper can be improved by considering the more classical work
  in schema matching. For example, the authors could leverage this
  work in the generation of candidates, and they can address issues
  with the formalization of the schema matching problem by reusing its
  formalization from the database field.


Specific questions

All these questions refer to the definition of the schema matching
problem:

- How is the notion of correctness of a mapping function f defined?

- What are the properties that a correct mapping function f should
  satisfy?

- Does the algorithm proposed in this paper compute a function f that
  satisfies such properties?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670242415

thread_type:
official_review

<thread_object>
creator: reviewer_tbj4

<content>
summary:
This paper presents a zero-shot schema matching approach by leveraging the multi-stage call of LLMs to generate, refine, and score the matchings.  Specifically, it introduces synthetic examples to guide the reasoning of LLMs for improving and optimizing the results of schema matching. The experiments on medical schema matching benchmarks demonstrate that the proposed approach outperforms the selected baseline methods on accuracy. 

The paper does a great job of demonstrating the problem that they are solving and the methodology they presented. 
The main contribution is decomposing the schema matching task into multi-stage sub-tasks that are completed by multiple calling of LLMs, with retrieval from contextual reasoning and prompt optimization based on in-context examples. However, the contribution of this work is limited, as they only introduce the muti-stage schema matching by extending the calling of LLMs from single to multiple.

strengths:
S1: The idea of leveraging the multi-stage LLMs for schema matching is novel.

S2: The authors do a great job of demonstrating the challenges of schema matching in real-world scenarios that they are trying to address and the methodology they presented. 

S3: Several experiments on MIMIC-OMOP and Synthea-OMOP datasets are conducted to empirically investigate and demonstrate the performance of the presented method.

weaknesses:
W1: The contribution of this work is limited, as they only introduce the muti-stage schema matching by extending the calling of LLMs from single to multiple.  

W2: The accuracy@k is the only metrics that is reported in experimental results, the results on precision are missing. 

W3: The prompts are provided in the appendix, but the source code is not provided for reproducibility.

W4: The GPT-4 (0613) is the only backbone model, the results of using llama as backbone are not reported.

questions:
Q1: How does your approach work in terms of precision when compared with the baseline method?

Q2: How confident when ranking the LLM-generated candidates with LLM-based scores? How much does this ranking contribute to the results?

Q3: Could you provide the details of how the vector retrieval works in Sec 4.1?
If I understand well, you retrieve the top-k matching from the target schema attribute based on MaxSim between query embeddings and target schema embeddings. However, the granularity of the query embeddings and target schema embedding is different, query embedding is an attribute-level embedding while target schema embedding is a table-level embedding.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726720149

thread_type:
official_review

<thread_object>
creator: reviewer_fxfk

<content>
summary:
The paper deals with schema matching, and old but very important problem in databases. The idea is that, given one starting (relational) schema and one target schema, to be able to match which attributes in the  starting schema correspond to attributes in target schema. The proposal in this paper is Matchmaker. This system uses a mix of retrieval using multi-vector representation and LLM-based reasoning to produce candidates for the matching, and then applies a final LLM-driven step to refine these candidates. Notably, the program also is built so that it can optimize the last step by providing examples from the databases. All together, the system shows quite an advantage over previous proposals.

strengths:
- Very well written 
- Addresses an importan tproblem that has ben recently identified as a target for the ML community
- Provides a thorough experimental section, including a (very nice) ablation study to understand the impact of different strategies for candidate generation.

weaknesses:
- While the paper is well writtend and scientifically sound, the algorithm itself (matchmaker) is not groundbreaking. Matchmaker resolves basically on building appropriate chain of thought prompts, as well as applying semantic similarity techniques. As such, I see this mostly as a paper describing a particular, LLM-based proposal, to address this problem. 
- There seem to be a lack of LLM-dirven alternatives to compare with, which is both good for the paper (because authors are the first to apply them in this context), but it also raises the question on whether any other similar approach would produce similar results. 
- The problem itself ( schema matching, or more generally data harmonization/interoperability) is not a core problematic of ICLR. I would imagine this paper would be more suitted to a database conference.

questions:
Why haven't you submitted this to a database conference? It seems to me that the reception and impact there would be much higher.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731962389558

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response Overview

comment:
We thank the Reviewers for their insightful and positive feedback, and their time during the review process!

We are encouraged that they found the schema matching problem to be "important" (``R-fXFk``), "fundamental" (``R-N216``), ‘relevant and interesting’(``R-N216``) , that "addresses a critical practical problem"(``R-JXrD``) with "significant implications for ML development" (``R-JXrD``). They highlighted our "novel technical approach" (``R-JXrD``) noting particularly that "LLMs for schema matching is novel" (``R-TBJ4``) and highlighting our "zero-shot learning capability through synthetic in-context examples" (``R-JXrD``), along with its "advantage over previous proposals" (``R-fXFk``). We are pleased they noted our "comprehensive empirical evaluation"(``R-JXrD``) with "strong quantitative results" (``R-JXrD``), including our "(very nice) ablation study to understand the impact of different strategies" (``R-fXFk``) and that the proposed approach "outperforms" (``R-N216``) in various settings.

We address specific questions and concerns below and highlight updates based on reviewer suggestions. 

**We have uploaded the updated manuscript with the changes highlighted in blue.** We thank the reviewers for helping us improve the paper!

On the basis of our clarifications and updates, we hope we have addressed the Reviewers' concerns.

Thank you for your kind consideration!

Paper 7911 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Matchmaker, a schema matching technique using a self-improving compositional language model program. The reviewers agree the paper addresses an important problem, is well written, and provides thorough experiments. During the rebuttal, the authors made significant efforts to address the reviewer concerns. However, there are still remaining concerns on the contributions/scope and mapping correctness despite extensive discussions. There was also an additional discussion among the reviewers, and the consensus is that while the paper is interesting, the contributions are not novel and that there are still unresolved concerns. Summing things up, I think the paper can be improved and should go through another round of reviewing.