Labeling without Seeing? Blind Annotation for Privacy-Preserving Entity Resolution
Authors: Yixiang Yao
Weizhao Jin
Srivatsan Ravi
Venue: Accepted by TMLR ()

Abstract: The entity resolution problem requires finding pairs across datasets that belong to different owners but refer to the same entity in the real world. To train and evaluate solutions (either rule-based or machine-learning-based) to the entity resolution problem, generating a ground truth dataset with entity pairs or clusters is needed. However, such a data annotation process involves humans as domain oracles to review the plaintext data for all candidate record pairs from different parties, which inevitably infringes the privacy of data owners, especially in privacy-sensitive cases like medical records. To the best of our knowledge, there is no prior work on privacy-preserving ground truth labeling in the context of entity resolution. We propose a novel blind annotation protocol based on homomorphic encryption that allows domain oracles to collaboratively label ground truth without sharing data in plaintext with other parties. In addition, we design a domain-specific, user-friendly language that conceals the complex underlying homomorphic encryption circuits, making it more accessible and easier for users to adopt this technique. The empirical experiments indicate the feasibility of our privacy-preserving protocol (f-measure on average achieves more than 90\% compared with the real ground truth).

Published in Transactions on Machine Learning Research (05/2025)
Labeling without Seeing?
Blind Annotation for Privacy-Preserving Entity Resolution
Yixiang Yao
yiriongy&usc.eduv
Deportment of Computer Science University of Southern California
Weizhao Jin
weizhaoj edu
Department of Computer Science University of Southern California
Srivatsan Ravi
srivetsr edu
Department of Computer Science University of Souteenn California
Reviewed on http:://wpparreeeeee/////metdded
Abstract
The entity resolution problem requires finding pairs across datasets that belong to different owners but refer to the same entity in the real world. To train and evaluate solutions (either rulo- based or machine-learning learniig--neeee]) to the entity resolution problem, generating a ground truth dataset with entity pairs or clusters is needed. However, such a data annotation process involves humans as domain oracles review the plaintext data for all candidate record pairs from different parties, which inevitably infringes the privacy of data owners, especially in privacy-sensine sensitivee cases like medical records. To the best of our knowledge, there is no prior work on privacy-preserving ground truth labeling in the context of entity resolution. We propose novel blind annotation protocol based on homomorphic encryption that allows domain oracles to collaboratively label ground truth without sharing data in plaintext with other parties. In addition, we design a domain-specific, user-friendly language that conceals the complex underlying homomorphic encryption circuits, making more accessible and easier for users to adopl this technique. The empirical experiments indicate the feasibility of our privacy-preservic protocol (f-measure average achieves more than 90% compared with the real ground truth).
1 Introduction
Entity resolution is the task of linking entity pairs across datasets that refer to the same entity in the real world. It is an essential problem in massive data integration and quality improvement, which has been widely employed in various domains including academia, industries, and goverument agencies (Christen 2011 Winkler 2014b. Traditionally, requires the participation of multiple owners of the data sources, and it is unavoidable that the data shared among parties. As the example shows in Figure [ datasets DA and Dn, which belong two different parties, consist of the IDs and names of the digital camera lens products but in different representations. Each lens is entity and the goal is to link the same entities between these two datascts. To achieve that, the data owners of DA and DB share the data with each other and jointly determine that A1 from DA and B1 from Dn could be a match since they are all "Canon 24-70mm f2.8" camera lenses though A1 explicitly notes the lens is with "USM" motor and B2 the 'II' generation. On the contrary, and are not a match since they have different focal ranges ("24-105" vs. "24-70") and aperture sizes ("4" vs. "2.8"), as well as A3 and B2 since they belong different brands ("Canon" vs. ") As world entity data usually contains sensitive personal information or is related to intellectual

---

Published in Transactions on Machine Learning Research (05/2025)
property, prisacy-preserving entity resolution (PPER) Gooulalas-Diive et al. 2021 Yao al. 2021 Ghai
et al. 2023 Yao et al. 2024), as solution, is to link entities with the protection for data privacy. The
linking process does not reveal any unnecessary information and the parties will not learn any entities that are not common among all parties. Back to our example of lens products, DA's 's owner only knows A1 has a potential match B1 in Dg but does not learn anything about B2. Likewise, D3 's owner also learns no other lens products from D except the match with A1.
D2
ID
Name
ID
Name
A1
Canon 24-70mm 8L USM
Camon 24-700 24-70mm 12 228LL II
Canon 24-105mm 1941, USM
Sony 16-35mm 12.8 GM
A3
Canon EF 15-35mm 1/2.BL USM
IDg
We
Sampled mooord pairs
Painvise annotation
Figure 1: The entity resolution and annotation example. DA and Dr are two lens product datasets owned by two parties separately. Making pairwise annotation requires the visibility of both parties' sampled records, revealing private data.
In the lifecycle of entity resolution and PPER, a ground truth dataset with entity pairs or clusters needs to be prepared to evaluate the quality of linking or train meaming-based entity resolution models. Constructing such a ground truth dataset, known as annotation, requires the involvement of domain oracles (data owners or trusted delegators) H Normally, the first step is to sample some representative entities from all data sources and convert them into pair-wised forms. The oracles then manually determine if such candidate "pair" is considered to be legitimate in the real world. For example, in Figure assume that all records are sampled, oracles could recognize (A1,B1) is match but (A1,B2) is not. Annotation tools (Ipeirotis 2010 Neves & Leser 2014) including Amazon Mechanical Turk (Buhrmester et al. 2016 Kang et al. 2014 Lease et al. 2013) are for constructing and curating ground truth datasets.
However, this is a human-in human-in-the-loww tn-phe-oopp process and the oracles need to see all the raw data of entities side-by- side as pairs plain. It worth pointing out that even though most of the privacy-aware entity resolution algorithms themselves are considered to be somewhat privacy-preserving the ground truth labels they used, matter what strategies are conducted for minimizing the size of the required labels, are still annotated in plaintext (Kurniawan & Mambo 2022) and can not be identified as privacy- preserving annotation. For example, healthcare facilities aim to perform entity resolution on their confidential medical records, yet the process of creating accurate reference annotations still relies on conventional annolation tools that lack robust privacy guarantees.
Recently, several privacy-preserving annotation methods have boen proposed. While some methods (Kaiino et al. 2014 Zhang et al., 2021 employ perturbation for crowd-sourced annotations, others leverage exist- ing Al-driven ground truth generation in specific tasks (Kavasidis et al. 2012 Wangt et al.) 2001), they are limited to certain domains (D'Orazio et al. 2009 Jing et al., 2017 and almost implausible to have such techniques universally generalized for tial privacy-based methods (Feyisetan et al. (019) protect the Personal Identifiable Information (PIT) of individual records, but the content remains in plaintext. Contrary to the relatively well-researched privacy- preserving entity resolution solutions for the application phase (Schnell et al., 2009 Scannapieco et al. 2007 Inan et al. 2008 2010), there still exists a privacy protection void in the annotation phase for PPER. Making annotations for entity resolution privacy-preservve the data across parties side-by-side in order to determine if two records are the same. Moreover, the raw data can certainly be manipulated accordingly or the annotation process can be semi-auto for the purpose of privacy protection, but the partial content of candidate pairs is still inevitably revealed for oracles at least need to see some of the plaintext to understand the meaning and then make decisions.
entity resolution. Additionally, the differen-
is challenging because the domain oracles must compare
In practice domain oracles are individuals who possess a deep understanding datasets, enabling them to make high-quality annotations.
2

---

It is important to clarify that this paper does NOT aim to address PPER itself. Instead, we propose a novel Prioucy-preseevving method for pairwise ground truhh annolation, which is applicable to any PPER pipeline. annotation method is illustrated in Published in Transactions on Machine Learning Research (05/2025)
Different from the aforementioned approaches that not suitable for pairwise privacy annotation merely hide the individual identifier but not the content, we propose novel "blind annotation" approach of ground truth labeling, based on homomorphic encryption (HE) (Fontaine & Galand 2007 Acar et al., 2018). for privacy-preserving entity resolution where oracles from each party can only inspect the data of their own but are still able work collaboratively. The concept ual model is as shown in Figure 2 where data owners "blindly" annotate the ground truth set, which can later be used the model training and evaluation phases of privacy-preserving entity resolution. No data in the plain view of each party is shared with any other parties the privacy of the data is guaranteed without loss of data utility.
The relationship between PPER and the proposed
Figure We summarize our main contributions as follows:
We propose novel lonoomoppii--eccyypion-naee privacy-preserving protocol that allows multiple parties to collaboratively produce pairwise entity annotations without sharing their data with any other parties in plaintext. To the best of our knowledge, this is the first solution to discover the potential of privacy-preservin ground truth annotation for pairwise entity annotations in entity resolution.
The proposed protocol allows the domain oracles focus solely on the annotation logic, and the protocol handles the optimizations for the underlying homomorphic circuit. Each party involved in the protocol can perform the computation efficiently and independently. The privacy properties of the protocol have been rigorously proven.
We implement the protocol and conduct empirical studies on heterogencous real-world datascts to prove the feasibility of our blind annolation ion protocol.
PPER Phoae
PPER Model
[n]
Entiy Pairs ((LD)
Traning
Evsluation
1: ((P)))
Homomorphic Encryption (P)
whis K(ED
Blind Annotation G
Ground truth G-((L1))
Samplig (X)
V X(E()
Repual until seisfaction reached
The relation between srivacy-preserve entity resolution (upper white dashed box) and blind annotation (lower grey solid box). this paper, we focus on the annotation phase and utilize "blind anno- lation" to label ground truth for PPER with zero-knowledge shared with other parties. The homonorphically encrypted records from each party are first sampled. Then, the data owners or their delegated oracles an- notate sampled records without looking at the other party's data but interacting with the blind annotation protocol. Finally, the ground truth set of triplets G ((). (i and j are record ids from two partics and I is a label indicating if and rj is a match) is formed and can be used for training and evaluation in any PPER task. Note that no record content r₁ or in plaintext is revealed throughout the annotation process. The solid line denotes the data stream in plaintext and the dashed line dcnotes the data strcam in ciphertext. denotes dataset, denotes sampling, E denotes homomorphic encryption, and [r] denotes record ciphertext.
Figure 2:
3

---

Published in Transactions on Machine Learning Research (05/2025)
2 Problem Definition
The privacy-preserving entity resolution (PPER) problem can be defined as triple T (D, M, E) where D = {Di ..,,..D,) is a set of n different datasets consisting of records r, from n different data owners }. respectively. E denotes encoding or encryption algorithm that keeps the from each D in encoded or ciphertext form, that is, [r] E E(D), where [r] explicitly denotes r is in ciphertext. M is the match set containing record pairs between any two dalasets amongst the n parties, so M = {[Ir:], [r;]) |r; = rai Ir: E E(Dx), Iry] E(Dm)), where r; ry indicates that ra and ry refer to the same entity in the real world, despite having different representations.
Finding the complete and precise M depends on the domain-specific algorithn/mooke which relies on the high-quality ground truth data for training, fine-tuning or evaluation. The ground truth set G is a set of triples (i,j.() where i and j are record ids of a record pair rę and ry, and is a Boolean label which indicates if such record pair is the same entity. The process of constructing G is called annotation. Formally, G = () r; E X(D)). r E (Dm)), where X sampling algorithm. I is determined by domain oracles with the content of r; and TJ.
Usually, G is not constructed individually from each party: because the domain oracles have to see r, and Tj in clear in order to determine whether two records are the same real-world entity based on the features the records have. Therefore, the owners of the records need to share sampled records in the plain with other parties which makes the raw content of r; and ry revealed.
To prevent the potential privacy leakage during this process, this paper focuses on creating ground truth data with [ra] and [ry] straight so that no plaintext from any partics reveals any other partics. We name this process blind annotation, that is, G [rd] X(E(())) [rsl E ((E().)))
3 Preliminaries
3.1 Related Works
Privacy-preserving annotation As stated this privacy-preserving record linkage survey pa- per Divaaniss et al. 2021), almost no open method available for making ground truth annotation for PPER due to privacy concerns regardless of its a time-consuming and erroneous process.
The only similar scenario we found that explored the attempts for privacy-preserving annotation is about active learning using differential privacy (Feyisetan et al.] 2019). The general active learning method aims to learn a distribution of the data by selecting less training data that carries the highest information with the active Icarner. The sampled data from the Icarner needs to be annotated by domain oracles or crowd workers. The problem is that crowdsourcing the labels is usually from an open call and transmitting non- public data to crowd workers has an inevitable privacy leakage risk, especially for the identities. For the insufficient privacy guarantee of k-anonymity, the authors adopted differential privacy algorithu (Li et al., 2012), which prevents user from sustaining additional damage by including their data in dataset, binary classification tasks, this method achieves similar accuracy scores as the non-privacy counterpart with a small performance hit but strong privacy guarantee. Though Personal Identifiable Information (PII) is protected from revealing, the content of the sampled data as plain text is still known by the crowd workers for them to understand and annotate. If the content of the non-public data itself requires to be kept private, such a method not yet qualified.
Another method to prepare data for machine learning tasks is to learn the distribution of the original data and generate synthetic data based that (El Emam 2020). Even though this method might work for training statistical models and conducting analytical works, it is not feasible to be applied to entity resolution tasks. Because the representation of the token/ token/word word is an important signal to determine the similarities between records, a tiny modification or substitution of the original representation could cause huge judgment deviations for the annotators.
Yu et al. (2020) provided method of not considering ground truth but instead using unsupervised heuristic measures based greedy matching approach to evaluate and optimize the hyper-parameters for entity
4

---

Published in Transactions on Machine Learning Research (05/2025)
Key Generation: (pk; sk) &- HE. KeyGen (A)
Encryption:
- HE. (m,
Decryption:
m + Dec. (c, sk)
Evaluation:
cf HE. Eval. (J, (01, 02. ,,...,, pk) s.t. HE. DeE.Dec sk) [m1,, ma; mp)
Figure 3: Operations of a typical public-koy homomorphic encryption scheme
resolution models. This method is based on the assumption that a match of record pairs gets the highest similarity score among a set of houristic measures against other candidate pairs. Using houristics to estimate linkage quality is doable some certain scenarios, however, evaluating heuristics by heuristics, is somewhat "the chicken or the egg" problem. Additionally, this method only works in some general cases where the same entity looks similar: in extreme conditions some hard record linking problems, the representations of the record are different even when they referring to the same entity.
general,
In short, labeling ground truth in a grivacy-preservin manner is tough and no direct work exists. Our approach achieves the goal without compromising privacy protection by employing homomorphic encryption.
Comparison PSI and PIR Private Set Intersection (PST) MMorales et al. 2023) or weighted PST are general-purpose solutions for identifying common elements between scts. However, they have limited extensibility, particularly when comes to supporting complex annotation logic. Specifically, PSI does not natively support conditional expressions complex predicates, and incorporating such logic typically requires substantial additional effort. In contrast, the blind annotation protocol can express more sophisticated logical operations more naturally.
Moreover, PSI and weighted PSI can still be used within blind annotation protocol for computing set in- tersections. Although they are not currently available in our DSL, they can be incorporated as functions, provided that the PSI operation can be represented by HE circuits (Chen et al. 2017).
the other hand, is designed for retrieving data without disclosing which item is being accessed. While state-of-the-an PIR extensions support more expres- sive queries, such as keyword-based retrieval (Hao et al. 2025), they share the same limitation as PSI in that they do not easily accommodate complex logical operations. In comparison, the blind annotation protocol provides a more flexible foundation for such logic.
Private Information Retrieval (PIR) (Chor et al.
1998
3.2 Homomorphic Encryption
Homomorphic encryption (HE) allows the computation to be performed over encrypted data while preserving the input/outpin relationship of the function between the plaintext and ciphertext data (Fontaine & Galand,
2018). In general, an encryption scheme is said to be homomorphic if for some operator 2007 Acar et al.
@ over plaintext (OM) and ciphertext (Oc), the encryption function É satisfies: Vmn, m2 E M, E(mi ΦM m2) + ((m, )c 0c E(m2), where M denotes the message in plaintext and C denotes the message in ciphertext. declares the computation is direct without any decryption the middle of the process. Therefore, we let @ be +, computing mi + m2 can be done by a computation unit that receives the original message from the data owner in the encrypted form, that is, E(m) and E(ma), and is able compute the addition of two encrypted messages without decrypting. Only the data owner with the key can decrypt the message from ciphertext back to plaintext.
Figure BigureBdemonsle Idemonstrates a lypical public key homomorphic encryption scheme primarily characterized by four operations. Key generation takes in a security parameter À and outputs public key pair sk). Encryption encrypts plaintext message m with pk and returns ciphertext Decryption, as an inverse, decrypts ciphertext with sk into plaintext m. Evaluation evaluates a function (represented as a circuit) f over ciphertext tuple (c1, co...,,, with pk and returns Cj, which is equivalent to f(m,, [(mm,,...... m2...., after decrypting.
5

---

Published in Transactions on Machine Learning Research (05/2025)
The party that executes the homomorphic functions does not know anything about the data and the party that provides the encrypted data is unaware of what functions have been executed. This property, which is informally named blind evaluation, is the basis of our protocol.
4 Blind Annotation Protocol
We propose a privacy-preserving annotation protocol that allows the ground truth to be annotated among multiple parties "blindly" without inspecting other parties' records.
Intuition: The intuition behind this is that without putting candidate record pair side-by-side for domain oracles to determine if they are the same entity or not, the oracles on each side could extract the core features by just looking at the record content in plaintext of their own and summarize these features into a series of Boolean questions. If any record from the other parties satisfies all conditions defined by these questions, this record is highly likely to be a match of the record that the questions are derived from. Suppose the record content is encrypted, and these Boolean functions can be blindly evaluated over the ciphertexi of the record. In that case, no plaintext of the record content is revealed other parties.
For example, from Pa's 's perspective, determining if Py's record B1 is similar to its record A1 can be based A1's core leatures. If B1 also has keywords "Canon", "24-70" and "2.8" as A1, it is highly-likely that it is a potential match to A1. The other keywords like "mm", "USM" are not necessary for identifying the lens based on the oracle's domain knowledge. Figuring out these features only needs PA access to its own record A1. Moreover, if PA encapsulates the feature detection functions for A1 to be homomorphic functions and PB encrypts B1, PA can label record pair (B) using homomorphic evaluation without knowing the content of B1.
The protocol is elaborated in Figure 4 Succinetly, assume the ground truth construction is between party PA and PB, and party Pg is coordinator for key management and result collection. In the initialization phase, party PA and Pg first send their dataset size to Pc, and Pe randomly samples the record ids. Moreover, Pe also prepares the public key pair (pk; sk) for homomorphic eneryption and sends the public key pk to PA and PB, and keeps the secret sk for decryption.
Protocol:
The next step is called feature questions, which is for data owners or domain oracles to annotate their records using homomorphically executable functions. Concretely; PA and Pa sample records r; and rý respectively according to the sampled record ids from Pc, and prepare a set of Boolean logic-style questions Q₂ and Qj accordingly to the features of the record content for each record. Each question set Q is combined into a form that returns one Boolean result with first-order logic and converted to a homomorphically executable function.
Following that is record encryption, where the clear record data is homomorphically encrypted with pk in PA and PB respectively, and the ciphertext [r:] and [ry] are shared with another party.
The core step is blind evaluation, where the enerypted records from the other party evaluated with prepared feature questions. Concretely, Once PA receives [r;]] from Pn, it evaluates Qi over it homomor- phically, that is HE. AHE.Eval; [r)), and gets [A PB does similar operation and gets [4 Both [A and [ are in ciphertext and sent to Pe. Pc then tests if [A equals [ and decrypts the result with the secret key sk.
The last step is to determine the end conditions. If PA and PD agree on the result, the result will be stored. Otherwise, this pair will be picked out for the next round. In the next round, PA and PB require to refine their questions and evaluate with the encrypted record data from another party again. After t rounds, the pairs with no agreement will be discarded.
Privacy: From PA'S perspective, throughout the protocol, PA has no access to PB'S record content in plaintext but evaluates questions over ciphertext [ry]- The question functions Qi are evaluated on PA's side so Pg does not know anything about the features tested in Q: by PA. The same situation applies PB. Pc, which has the secret key sk, decrypts the final results which contain no record content in plaintext
6

---

Published in Transactions on Machine Learning Research (05/2025)
from either PA or Pg. None of the parties is able apply ciphertext collision attack because homomorphic encryption is semantically secure (Section 4.3
In the following subsections, we walk through the protocol details and dissect each essential component. We also provide the protocol security analysis in Appendix B.2 Note that parties operate independently throughout the process as long as the interactions follow the timeline the Figure + Although Secure Multi-Party Computation (MPC) is potential option for implementing the blind annotation protocol, HE is preferred in practice due to its ability to support offline evaluation. This is especially important because each party may perform annotation at different times, with varying time costs, and HE does not require all parties to remain online concurrently.
Pa
Pg
P
10,1
Sample record its (P.)). from I) and
Generate crysto keys (pkc dk Krytieng and send pir
Foll Doo
Design gn questions Wi X(D))
Design questions
Vrye X(Px)
In| HE. HE.,,,, pio), Vove X(D,)
Disagreement exists
h + HE En pk), X(Op)
and of rounds
[+ - Arotel (oj.ob. vfol
[+] - HE Evalt I.. vO, VI6l
HE. ( 14) v[A
Figure 4: The timing diagram for the blind annotation protocol. The protocol, as well as five main compo- nents (initialization, feature questions, record encryption, blind evaluation, and end conditions) are dissected in Section
4.1 Initialization
Prior to getting involved the annotation, few initialization steps should be conducted first. The dataset owners (PA and PB) need to first consent on the annotation criteria including how similar can two records be identified as the match and what kind of difference is tolerable. For example, consider the records "Canon 24-70mm f/2.8T, USM" and "Canon 24-70mm f2.8T. USM II" (A1 and B1 in Figure 1 Should these two lenses, which differ in generation, be treated as the same? In some contexts, lenses with generation 1 and II should be distinguished, while in others they may be categorized as the same item. The data used for deciding such criteria in this step can be synthetic or be derived from the record content so it does not leak any (sensitive) information about the original records.
A data preprocessing step is necessary for some of the datasets. If the dataset owners believe the format of their data has noticeable distinction, they could apply series of data cleaning and standardization operations, for example, removing the dataset-specific characters, lemmatization or stemming for natural language content, on their data individually.
Pe is responsible for sampling records from PA and PB's dataset. After the dataset size |D and Døl are sent to Po, Po randomly samples some amount of record from DA and D3 and sends back the selected record ids as a list back to the original data owners respectively. Pe then generates the key pairs (pk, sk) HE. KeyGen, (A) with selected homomorphic encryption scheme and parameters X, then
7

---

Published in Transactions on Machine Learning Research (05/2025)
distributes the public key pk to both PA and P3 for encrypting their data and questions. Note that Pa is the only party that can decrypt any ciphertext back to plaintext with access to the secret key sk.
Note that all steps, except for HE key generation, are also required in standard non-private annotation setup and are essential for the final annotation quality. Our proposed method preserves these steps unchanged. Only after the annotation criteria have been defined and consistent data preprocessing has been applied can appropriate feature questions be formulated for the records.
4.2 Feature Questions
Feature questions are designed lest. whether essential features of a record are satisfied and are annotated by domain oracles. This is based on the assumption that if all the core features of two records are the same, then they are highly likely to be the same entity. Specifically, the features of the records could be the sub-strings that are necessary for representing the record. For example, in lens name "Canon 24-70nm f/2.8T. USM IT", one feature can be the focal length "24-70" which can be used to construct question that tests if "24-70" is in the compared lens. Therefore, using this question to evaluate "15-85" lens returns false.
Under this assumption, the data owner can design a set of Boolean questions for all the features of a certain record. Utilizing first-order logic chains turns it into single Boolean value the final result. For example, assume the question set is (Q1, Q2... (2....Qnl Qn), and it could be Q = Q1 V -Q2... A Qn after conversion with first-order logic. Therefore, Q(r) = (i(r) V -Q2(r)... A (r(r) returns Boolcan value that implics whether the record r has all the desired features.
of questions together and the evaluation of record
In our protocol, the feature question set Q is as f in homomorphic encryption that uses to evaluate the encrypted record [r] with HE Eval, that is, HE Eval.(Q, [r].pk), or simply Q(Ir]). Constructing Q that is homomorphically executable non-trivial due to the operator limitation and the hardness of the efficient encryption circuit construction (see details in Section 4.2.2).
Hence, on high level, we design a simple annotation language for the case of use in Section 4.2.1 and pre-defined some primitive functions that encapsulate the details of underlying homomorphic circuits in Sec- tion 4.2.2 With this layered design, the domain oracles focus solely on the annotation logic, and the program handles the tricks and optimizations for the sophisticated underlying homomorphic circuit construction.
4.2.1 Domain-specific Language
The annotation is designed to be written in a domain-specific language (DSL). This language provides general methods for data manipulation and logic computation, which are extensible for defining functions. The syntax of the DST. in Appendix [B..T]
With this DSL, it is efficient and sufficient to "ask questions'. Take the lens example, the record content from PA is "Canon 24-70mm f/2.8L USM II", the annotation from the oracle could be:
Sr lower ($r)
Sc1 is_in ("canon". $r)
condition condition condition
Sc2 is_in ("24-70*, $r) is is_in("2470) $r)
Se3 !is_in (*24 24""0"" " $r)
ret
$c1
$c2 & $c3
where larget record $r is first being lower-cased, and then requires the brand to be "Canon", focal range to "24-70" or simply "2470" as the common abbreviation but can not be "24-105". Therefore, both "Canon 24-70 f2.8" and "Canon 2470" are considered matches, but "Canon 24-105mm USM" isn't. Notice not all features but the important ones, that are consensus the annotation criteria, are tested, e.g., the motor type "USM" the version "IT" not being used for making the decision.
8

---

Published in Transactions on Machine Learning Research (05/2025)
4.2.2 Homomorphic Functions
Writing homomorphic functions from scratch is not straightforward writing some normally "obvious" functions because of all the cumbersomeness and restrictions that homomorphic encryption schemes bring: (1) The homomorphic encryption scheme only supports a collection of limited operators. More complicated functions need to be built using these basic building blocks. (2) Simply porting the logic of a normal function to a homomorphic encryption function would sulfer from privacy and/or efficiency issues. The function needs to revamp or refactor for it to work properly. (3) To achieve desired privacy protection (Appendix B.2), the encryption is end-to-end, so no decryption is allowed in the middle of the evaluation, which confines the use of control flow in programming.
To address these challenges and for the ease of writing effective and efficient functions along with the proper privacy protection, we pre-define several functions that are essential components for constructing Boolean questions Q and can be evaluated homomorphically without exposing the details of the underlying circuits. For the encryption schemes that only support logical operators, the arithmetic operators could be enriched by building upon the lower-level gate circuits, for example, constructing an 8-bit adder with AND/ AND/OR/NOT/XOR gates (Mano) 2017). On the other hand, the arithmetic schemes such as BGV can also be extended to support logical comparisons (lliashenko & Zucca 2021 (e.g., "<" and "=") and that also retrains the benefit of efficient SIMD operations (Smart & Vercauteren 2014) naturally come with these schemes.
It is worth mentioning that, among all the predefined functions, one function is is is_iinnnni [n] if the encrypted string [a] is a sub-string of the encrypted string [6]. This the core to measure if certain feature [a] exists in a record [b]. In terms of extensibility, as long as a function needed can be encapsulated in way that is able to compute homomorphically, it is safe to be added to the framework.
[b]
that tests
4.3 Record Encryption
To make the blind evaluation functions work, the records from both data owners require to be encoded with the same data encoding mothod. For the current case, since we only work on English characters, we simply apply ASCII encoding over the original records. Each character in a record is then represented as an 8-bit integer. All the sampled records r E (D) are encrypted with the public key pk that Pc generated in the initialization step.
The output ciphertext [r]] <- HE. Enc, (r, pk) is different even if the content of r is exactly the same due to the probabilistic encryption property, also known semantic security, that homomorphic encryption carries YYi et al. 2014). For instance, encrypling "Canon 24-70mmn n times results in different ciphertexts. Therefore, no party is even able to test the exact match of the records by comparing the encrypted record ciphertext from other parties.
Finally, the encrypted [r] e X(E(D)) is sent to another data owner for blind evaluation.
4.4 Blind Evaluation
From PA's perspective (the perspective for PB is interchangeable), once it receives the encrypted (EE)D)) from PB, [r>] E X(ED))) is fed into each prepared function Q; for blind evaluation. Computing Q; with Iry] is a typical homomorphic evaluation process that PA does not know anything about what the content is in [r,] but can still evaluate it as the plaintext version ry with all the questions defined in Qi- Because PA does not know which Ir potentially matches its own record ri E (D)),, this process neods to test all [r,]s against all Qis. Therefore, the total number of such process executes |Da] [Da] times for [Qi]= - |DD
The output [AX|= HE. HE.Eval. (Qi, Iry1, pk) for each [r>]] is also in ciphertext and since the secret key sk is only accessible by Pc, PA is not possible to get any information from X X(E(DB)). Note [A associates with the record id pairs (i, so that Pg is able to identify the provenance of the result..
9

---

Published in Transactions on Machine Learning Research (05/2025)
4.5 End Conditions
The encrypted results from PA and PB are collected and evaluated by Pc. On Pe's side, it aligns [A'] and [ according to and j, and tests [A']] [A meanwhile decrypts the result with secret key sk. Pc then has a mapping F with record id pairs (i,j) as keys and Boolean values indicating if two parties have made an agreement on the same record pairs as values. Formally, {(i.5) - Dec [[ = (Bllsk) ri E ((DA).r E X(DB)).
For each record pair (i,j) in F, if the value is true, the agreement has been established between the two data owners, so no additional process needed. Otherwise, the record pair needs further investigation, and Pe extracts all and js from disagreed pairs and returns them to the data owner respectively. The data owners have to conduct another round of annotation only for these records. The annotations for the later rounds tend to be not strict as the former rounds so it increases the possibility of making [4X]] and [ the same. Simultaneously, Pe maintains another list G₁ that stores the annotation result from one of the parties (assuming PA here), that is, G = (i). h E [1, + HE Dec: ([A'): sk)}, where is the label, h is the round number and is a paramcter denoting the maximum number of rounds that should be decided between the data owners the initialization step. When rounds have finished, the protocol ends: the pairs whose value true (consensus archived) in F are added the final ground truth, and others are discarded, Therefore, the ground truth set G' is constructed G (( (i,j, E Gr F(i. truc). Note that increasing tends to improve performance, but it also raises the labeling cost. An empirical analysis of the effect of is presented in Section 5.2
Obviously, when all the values in F are true in the h-th round, it is by no means to recurse into the (h+ 1)-th round. The protocol meets the early termination condition and exits immediately.
5 Experiments
In this section, we conduct experiments to empirically evaluate the feasibility using blind annotation to annotate datasets and the incurred overhead of homomorphic encryption. In general, domain oracles are asked to amnolate datasets using blind annotation. The quality of the final annotation results is assessed by comparing them to the ground truth.
Original
Selected
Dataset
Domain
Attributes
#E
#M
#E
#M
Abt-Buy
E-commerce
name, description, manufacturer, price name, description, manufacturer, price
1081+1092
1097
50+50
50
Goozen
E-commerce
1363+3326
1300
49+-50
50
DBLP-Scholar
Bibliographic
title, authors, yenue, title, authors, yenue, year
2616+66426
5347
49++50
50
DBLP-ACM
Bibliographic
2614+2294
2224
50++50
50
given name, surname, sex, age, title, date of birth, address 1, address 2, phone, soe see id, culture, family role
Febri
Biomedical
1500+33500
1074
50+-50
50
Table 1: Dataset specifications and basic statistios. The underlined attributes are the attributes selected in the experiments. #E represents the number of entities, and #M denotes the number of matches.
5.1 Settings, Datasets and Metrics
5.1.1 Settings.
To empirically study the feasibility of our blind annotation protocol in practical sense, we implement a uscr-friendly GUI and its corresponding HE program (see details in Appendix C.1). Specifically, the web GUT is implemented in Python, in which the DSL syntax is written in Extended Backus-Nant Form
10

---

Published in Transactions on Machine Learning Research (05/2025)
(EBNF) and parsed by Lark library with Look-Ahead Left-to-Right (LALR) parser. The syntax definition can easily be extended for new syntax, operators, and functions. The workflow is follows: each domain oracle g annotates the owned dataset individually, and the annotation program merges and calculates the results. If the annotations do not satisfy the cxit condition, the unqualified records are returned to the corresponding oracles for the next annotation.
The underlying HE program is implemented in OpenF'HE AAl Badawi et al. 2022) an open-source project that efficiently and extensibly implements the post-quantum Fully Homomorphic Encryption schemes. We use the BinFHE module, concrete implementation of SHEW/TFHE (Ducas & Micciancio 2015 Chillotti et al. 2020). Specifically, we set it to work in public-key encryption mode with the crypto context to be STD128, which guarantees more than 128 bits of security for classical computer attacks. Additionally, we implement the program in two versions: serial and parallel. The latter employs OpenMP [ a auulti-platfon sharod-memory parallel programming library, for accelerating independent gate operations be executed in parallel. All the experiments are conducted on a Linux machine with an 8-core CPU @ 3.60 GHz and 32 GB RAM.
5.1.2 Datasets.
We use the real-world entity resolution benchmark (Köpcke et al. 2010 which includes tasks and lies in both e-commerce and bibliographic domains. Each task consists of two datasets and a ground-truth file, which contains all the true matches. To mimic the labeling process, we extract a subset of records from each dataset, annotate them, and then compare the annotated pairs to the ground truth. Specifically, we first randomly sample 50 labeled matches from each provided ground truth, and this covers at most 5% (50 records) of each dataset because one record could link to multiple records. Note that 50 records from each dataset are around 2.5-5% of the original dataset except for Scholar. Even though the proportion of sampled records as ground truth seems small, is sufficient for ER in most practical scenarios (Kasai et al., 2019 The specifications and the basic statistics of the dataset are listed in Table [ We do not make any specific data cleaning and normalization except that all the Unicode characters are mapped into the ASCII range.
Generally, PPER are evaluated using ER's benchmark datasets. However, important application area is in healthcare, particularly with patient data. Unfortunately, these datasets often require permission for use or are restricted to specific purposes, making them less ideal for evaluating the effectiveness of the privacy-preserving methods. As an alternative, synthotic datascts are commonly used. One such synthetic dataset is Febrl (Freely Extensible Biomedical Record Linkage) (Christen 2008), which widely employed for generating census records containing fields such as name, sex, age, and address. These fields align with the structure of many patient datasets. Furthermore, a synthetic dataset, it allows for customizable levels of noise. For experiments, generated the experiment dataset using the settings from Yao al. (2021) The record content contains typos, ORC errors and phonetic festures, and summarize the statistics of missing value in Table 2
given name
surname
SEX
age
title
address 1
phone
1.8
1.3
21.2
19.6
57.5
5.7
5.6
Table 2: The percentage (%) of missing value for each attributes in the Febrl dataset
5.1.3 Evaluation Metrics.
We use precision, recall and F-measure as the standard evaluation metries to measure the accuracy of blind annotation against the scores computed from ground truth labels. We additionally analyze other observed phenomenons, including the relationship between the annotations rounds and the number of labeled pairs that come agreement. As for the homomorphic implementation, the communication and computation costs are evaluated.
Phttps: http://gtttbb/om/////
3 Given that these tasks require only general knowledge, the oracles are individuals who understand the datasets. 4 hhttps:/ http///www..o..rgg.//////
11

---

Published in Transactions on Machine Learning Research (05/2025)
Abt-Buy
Gmazon
DBLP-Scholar
DBLP-ACM
Febri
0.75
0.50
Precision
Precision
Precision
Precision
Precision
0.25
Recall
Recall
Recall
Recall
Recall
F-measure
F-measure
F-meeasure
F-measure
0.00
2 Round
2
Round
Round
Round
Figure 5: Performance evaluation regarding precision, recall and f-masure.
Abt-Buy
Amazon-Google
DBLF-Scholar
DBLP-ACM
Febri
2600
noe antolaation
agreemment
2200
100
2000
3
2
Round
Round
Round
Round
Round
Figure 6: #round to meet agreement. The gray dashed line is the total number of candidate pairs, and the green solid line is the actual number of pains that meet agreement. These two use the left y-axis as the scale. The purple bars at the bottom are the summation of the number records to annotate from two parties. It uses the right y-axis as the scale.
5.2 Feasibility Verification
We try to validate two hypotheses in the following experiments to verify the fcasibility of using blind anno- tation to construct the ground truth dataset for PPER:
(H1) The feature extraction from only one side of the dataset without inspecting the other side and comparing candidate pair bideeb is a feasible approach.
(H2) More annotation rounds are effective for improving the quality of the PPER ground truth labeling.
(H3) It efficient for domain oracles to learn the DSL and perform annolation through the GUI.
H1 From Figure we observe the F-measure for Abt-Buy, DBLP-Scholar, and DBLP-ACM are all above 0.9, but Amazon- Amazon-Google not as good as others. The Amazon-Google dataset used in our experiment can be considered as hard problems for the record representations are full of abbreviations (e.g., software as sw), missing brands models for products, and a limited amount of information (very short names). Tf better data pre-processing, especially normalization, is applied beforehand, a visible performance boost should be achieved. With more rounds of records annotated, precision and f-measure increase sharply for Abt-Buy and maaazon-Google, but not that noticeable for DBLP-Scholar and DBLP-ACM since both of them have already achieved high scores at the initial stage. Surprisingly, the recall values for DBLP-Scholar and DBLP-ACM drop slightly with more rounds executed, this is mainly due to the side-effect of the annotation strategy: when annotations from two parties do not come to an agreement, the oracles tend to make the annotations to be more generalized, that is, relaxing similarities, in the next round. Normally, having slack criterion will diminish precision but rise recall for more negatives become positives; however, another variable, the number of agreements between parties, is also increasing in blind annotation, which leads to the increment of both true positives and false negatives and the recall somewhat drops as a consoquence. For Febrl, performance has a boost. It finally achieves promising results, largely due to the presence of highly identifiable signals, such as names, phone numbers, and keywords in addresses, in the census data.
the strictness of the matching criterion for easier capturing
12

---

Published in Transactions on Machine Learning Research (05/2025)
Record length distribution vs Communication cost
1e6
Computation cost
40-
35
8000
1.6
Serial
50-
Lefggratis:
Abt
1.4
1.28
25-
Amazon Google
6000
Time codt (GAC5)
1.2
20
DSLPACM
1.0
4000
15
Right axts:
0.8
Main text
10
Clpher text
2000
0.54
0.48
0.38
1315
16:30
45.45
45.00
01.75
70.00
01:109
100-220
121.135
150
151.700
221223
236-270
972-289
200-300
0.14
22t
0.0
Amazon- Google (6x46)
DBL P-Scholar (6x118)
DRLPAACM (6x149)
lentth
(6x53)
(a) Communication cost in the context of the record leugth range. The left y-axis indicates the number of such records, and the right y-axis indicates the size of the corresponding plaintext and ciphertext.
(b) Computation cost in serial and parallel mode. The x-axis represents the length of the two records as
{the average token length x length} in each dataset.
the average record
H2 We further explore other effects that occur with the iteration of annotation rounds. As is shown in Figure G the green line, which indicates the number of agreements, approaches the gray line, which the total number of candidate pairs, as the annotation round increases. On the other hand, the number of records that requires
annotate, shown the bars, decreases. Specifically, for Abt-Buy and Amazon-Google the annotation demands drop to 1/4 and 1/2 of the original amount respectively; for DBLP-Scholle and DBLP- ACM, in the second and the third round, this number is even less than 10,, because of the more explicit features the datasets have. For Febrl, the agreement increases notably across rounds, primarily due to the progressive relaxation of the criteria related to missing values.
H3 The domain oracles became familiar with the DSL and GUT in around minutes. For labeling, the average time per record was under 1 minute using the web-based GUI (implementation details are provided in Appendix C.1 The annotation process was fully offline for each party and executed in parallel.
In conclusion, the first hypothesis holds because all evaluation results lic in the acceptable range. With better cleaned and normalized datasets, the performance in terms of precision, recall and f-measure should be promising. The second hepothesis holds because precision, recall, and f-measure surge at the beginning with the increment of annotation rounds, and tend to be steady after a series of such rounds. The number of disagrecments between the partics drops over the rounds before it hits the plateau. The third hypothesis holds as well, as both the learning and annotation costs remain within an acceptable range.
5.3 HE Overheads Analyses
Implementing blind annotation based on HE encryption scheme introduces various overheads. We hereby analyze them from the following two perspectives.
Communication cost We compare the size difference between plaintext and ciphertext with different lengths of input tokens among four datasets. In Figure 7a the x-axis indicates the distribution of record length with a step of 15, the left y-axis indicates the number of records, and the right y-axis indicates the average size of serialized record data in step in KB. clear to find out from the left y-axis that the distribution of the record length is in the range of 300, and the most common lengths are from 16 60 for Abt-Buy and Amazon-Google and 91-160 for DBLP-Scholar and DBLP-ACM This leads to the average record length for each dataset is 53, 46, 118, and 148. From the right y-axis, the increment of plaintext size, as bascline, is nuance since cach ASCII character only takes 1 byte (0.001KB) and 300 is only 0.3KB. As for the ciphertext, the trend of ciphertext size expands linearly, with an average consumption of 32KB per ASCII character after encryption and serialization. Although the size increment after encryption is significant, the total size of the data is still practical, and such communication onc-time operation. For example, the annotation for 50 records, the average length 100, then the total size of the encrypted data is 156.25MB.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Computation cost We evaluate the computation cost 2500 comparisons (50 records from each party) with the average length of the token size (1st argument, see Appendix D.1] and the average record length (2nd argument) used in is_in in function for each dataset: 6x53, 6x46, 6x118 and 6x148. We also compare the cost with ciphertext in serial and parallel. As is shown in Figure 7b the incurred computation cost for ciphertext about 5.4e5 seconds for 6x53, 4.6e5 seconds for 6x46, 1.28e6 for 6x118 and 1.6e6 seconds for 6x148 in scrial mode. The parallel mode exccutes byto-level equal comparison with independent gate operations in parallel so that the time cost is significantly improved more than 3 times.
Two optimizations we consider doing in the future are: 1) The records from both parties are encrypted due to the limitation of BinFHE. For some other HE schemes (e.g., BGV/BFY that allow the HE operations between a plaintext and a ciphertext, only the record from the other party needs to be encrypted so that more computation and communication costs can be further saved. 2) The parallelization is only optimized for the byte-level comparison. Since the 2500 pairwise record comparisons are mutually independent and can also be run simultancously, the fully paralleled program should cost notably less time.
6 Conclusion and Future Work
In this work, we propose blind annotation protocol based homomorphic encryption for labeling ground truth that is tailored for privacy-preserving entity resolution. Unlike revamping the traditional annotation methods with de-identified records, this protocol explores the possibility of annotating without revealing any record in plaintext to other parties. The domain-specifie language lowers the bar of the implementation in the real world and the comprehensive experiment results show the rationality of blind annotation. It can be anticipated that scaling efficiently to more complex datasets and real-world scenarios will become increasingly computationally fcasible as HE implementations become more efficient.
In the future, because of the extensibility of DSL. more qualified homomorphie evaluation functions could be invented and integrated. Additionally, supporting the multi-attribute dataset is practical enhancement that increases information utilization and improves linkage accuracy.
References
Abbas Acar, Hidayet Aksu, A Selcuk Uluagac, and Mauro Conti. A survey on homomorphic encryption schemes: Theory and implementation. ACM Computing Surveys (Csur), 51(4):1-35, 2018.
Ahmad Al Badawi, Jack Bates, Flavio Bergamaschi, David Bruce Cousins, Saroja Erabelli, Nicholas Genise, Shai Halevi, Hamish Hunt, Andrey Kim, Yongwoo Lee, et al. Openfhe: Open-source fully homomorphic encryption library. In proceedings of the 10th workshop on encrypted computing & applied homomorphic cryptography, pp. 53-63, 2022.
Dan Boneh, Eu-Jin Goh, and Kobbi Nissim. Evaluating 2-dnf formulas on ciphertexts. In TCC, volume 3378, pp. 325-341. Springer, 2005.
Zvika Brakerski. Fully homomorphic encryption without modulus switching from classical gapsvp. In Advances in CryptologyCRYYPTO 2012: 32nd Annual Cryptology Conference, Santa Barbara, CA, USA, August 19-23, 2012. Proceedings, pp. 868 886. Springer, 2012.
Zvika Brakerski, Craig Gentry, and Vinod Vaikuntanathan. Fully homomorphic encryption without boot- strapping. Cryptology ePrint Archive, Report 2011/277, 2011. httpss//eptint.ierrro///211111 2011//1//202
Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. Amazon's mechanical turk: A new source of inexpensive, yet high-quality data? 2016.
Hao Chen, Kim Laine, and Peter Rindal. Fast private set intersoction from homomorphic encryption. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1243- 1255, 2017.
Iaria Chillotti, Nicolas Gama, Mariya Georgieva, and Malika Izabachène. Tfhe: fast fully homomorphic encryption over the torus. Journal of Cryptology, 33(1):34 91, 2020.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Benny Chor, Eyal Kushilevitz, Oded Goldreich, and Madhu Sudan. Private information retrieval. Journal of the ACM (JACM), 45(6))965-995) 981, 1998.
Peter Christen. Febri- an open source data cleaning, deduplication and record linkage system with graphical user interface. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1065-1066. 2008.
Peter Christen. A survey of indexing techniques for scalable record linkage and deduplication. IEEE trans- actions on knowledge and data engineering, 24(9):1537-1555, 2011.
Scott D Constable and Steve Chapin. liboblivious: c++ library for oblivious data structures and algo- rithms. 2018.
Tiziana D'Orazio, Marco Leo, Nicola Mosca, Paolo Spagnolo, and Pier Luigi Mazzeo. A semi- automatic system for ground truth gencration of soccer video sequences. In 2009 Sirth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 559-564. IEEE, 2009.
Léo Ducas and Daniele Micciancio. Fhew: Bootstrapping homomorphic encryption less than a second. In Elisabeth Oswald and Mare Fischlin (eds.), Advonces in Cryptology EUROCRYPT 2015, pp. 617-640, Berlin, Heidelberg, 2015. Springer Berlin Heidelberg. ISBN 972-3-662-46800-5.
Khaled El Emam. Seven ways to evaluate the utility of synthctic data. IEEE Security & Privacy, 18(4): 56 59, 2020.
Junfeng Fan and Frederik Vercanteren. Somewhat practical fully homomorphic encryption. Cryptology ePrint Archive, 2012.
Oluwaseyi Feyisetan, Thomas Drake, Borja Balle, and Tom Diethe. Privacy-preserving active learning on sensitive data for user intent classification. In PAL 2019, 2019. URL https:/ wwww waanaazzo.science.
-eaaaaaaaa-----einnooooooo www.iiinn-oo-snnitiieeeeeeeeeeee www-ssssnsitieeaeeeeee data- for ues-uuser-ine wwwee-eiteettccilie classettttssiiiiaatiin
Caroline Fontaine and Fabien Galand. A survey of homomorphic encryption for nonspecialists. EURASIP
Journal on Information Securily, 2007:1-10, 2007.
Craig Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.
Tanmay Ghai, Yixiang Yao, and Srivatsan Ravi. Lessons learned: Building a
entity resolu- tion adaptation of ppjoin using end-to-end homomorphic encryption. In 2023 JEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 117 124. IEEE, 2023.
Aris Soaulalla-Divanii Dinusha Vatsalan, Dimitrios Karapiperis, and Murat Kantarcioglu. Modern privacy- preserving record linkage techniques: an overview. IEEE Transactions on Information Forensics and Security, 16:4966-4999 49666,997772 2021.
Meng Hao, Weiran Liu, Liqiang Peng, Cong Zhang, Pengfei Wu, Lei Zhang, Hongwei Li, and Robert H. Deng. Practical keyword private information retrieval from key-to-index mappings. Cryptology ePrint Archive, Paper 2025/210, 2025. URL https://pprinn.accrro//22233
Ilaa Iliashenko and Vincent Zucca. Faster homomorphic comparison operations for bgv and blv. Proceedings on Privacy Enhancing Technologies, 2021(3):246 264, 2021.
Ali Inan, Murat Kantarcioglu, Elisa Bertino, and Monica Scannapieco. hybrid approach to private record linkage. In 2008 IEEE 24th International Conference Data Engincering, pp. 496-505. IEEE, 2008.
Ali Inan, Murat Kantarcioglu, Gabriel Ghinita, and Elisa Bertino. Private record matching using differential privacy. In Proceedings of the 13th International Conference on Extending Database Technology, pp. 123- 134, 2010.
Panagiotis G Ipeirotis. Demographics of mechanical turk. 2010.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. arXiv preprinl Xiv: 171: 11100999,, 2017.
Yongsoo Song Jung Hee Cheon, Andrey Kim & Miran Kim. Homomorphic encryption for arithmetic of approximate numbers. 2017. URL http://wwww.apooooooooo//////oaa
Hiroshi Kajino, Yukino Baba, and Hisashi Kashima. Instance-privacy preserving crowdsourcing. In Pro- ceedings of the AAAI Conference on Human Computation and Crowdsoureing, volume 2, pp. 96-103, 2014.
Ruogu Kang, Stephanie Brown, Laura Dabbish, and Sara Kiesler. Privacy attitudes of mechanical turk workers and the us public. In Symposium on Usable Privacy and Security (SOUPS), volume 4, pp. 1, 2014.
Jungo Kasai, Kun Qian, Sairam Gurajada, Yunyao Li, and Lucian Popa. Low-resource deep entity resolution with transfer and active learning. arXiv preprint arXiv:1906. 106:0096600 2019.
Isaak Kavasidis, Simone Palazzo, Roberto Di Salvo, Daniela Giordano, and Concetto Spampinato. A semi- automatic tool for detection and tracking ground truth generation in videos. In Proceedings of the 1st international workshop visual interfaces for ground truth collection in computer vision applications, pp. 1-5, 2012.
Hanna Köpcke, Andreas Thor, and Erhard Rahm. Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment, 3(1-2):484-493 2010.
Hendra Kurniawan and Masahiro Mambo. Homomorphic encryption-based federated privacy preservation for deep active learning. Entropy, 24(11):1545, 2022.
Matthew Lease, Jessica Hullman, Jeffrey Bigham, Michael Bernstein, Juho Kim, Walter Lasecki, Sacideh Bakhshi, Tanushree Mitra, and Robert Miller. Mechanical turk is not anonymous. Available at SSRN 2228728, 2013.
Ninghui Li, Wahbeh Qardaji, and Dong Su. On sampling, anonymization, and differential privacy or, k- anonymization meets differential privacy. In Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security, pp. 32 33, 2012.
M Morris Mano. Digital logic and computer design. Pearson Education India, 2017.
Daniel Morales, Isaac Agudo, and Javier Lopez. Private set intersection: A systematic literature review. Computer Science Review, 49:100567, 2023.
Michael Nachrig, Kristin Lanter, and Vinod Vaikuntanathan. Can homomorphic encryption be practical? In Proceedings of the 3rd ACM workshop on Cloud computing security workshop, pp. 113 124, 2011.
Mariana Neves and Ulf Leser. A survey on annotation tools for the biomedical literature. Briefings in bioinformaties, 15(2):327-33) 340, 2014.
Olga Ohrimenko, Felix Schuster, Cédric Fournet, Aastha Mehta, Sebastian Nowozin, Kapil Vaswani, and Manuel Costa. Oblivious multi-party machine learning on trusted processors. In 25th (USENIX) Security Symposium (1 USENIX) Security 16), pp. 619 636, 2016.
Pascal Paillier. Public-key cryptosystems based on composite degree residuosity classes. In Advances in Cryplology EUROCRYPT'99: International Conference on the Theory and Application of Cryplographic Techniques Prague, Czech Republic, May 2-6, 1999 Proceedings 18, pp.
Springer, 1999.
Monica Scannapicco, Tyya Figotin, Elisa Bertino, and Ahmed K Elmagarmid. Privacy preserving schema and data matching. Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pp. 653 664, 2007.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Rainer Schnell, Tobias Bachteler, and Jörg Reiher. BMC medical informalics and decision making, 9(1):1-11, 2009.
record linkage using bloom filters.
Nigel P Smart and Frederik Vercanteren. Fully homomorphic simd operations. Designs, codes and cryptog- raphy, 71:57-81, 2014.
Yi-Fan Tseng, Chun-I Fan, Ting- Chuan Kung, Jheng-Jia Huang, and Hsin-Nan Kuo. Homomorphic encryp- tion supporting logical operations. In Proceedings of the 2017 International Conference on Telecommuni- cations and Communication Engineering, pp. 66-69, 2017.
Yalin Wangt, Ihsin T Phillipst, and Robert Haralick. Automatic table ground truth generation and a backgrouud-analysit based table structure extraction method. In Proceedings of Sirth International Con- ference on Document Analysis and Recognition, pp. 528 532. IEFF, 2001.
William E Winkler. Matching and record linkage. Wiley intendisciplinary reviews: Computational statistics, 6(5):313-38) 325, 2014.
Yixiang Yao, Tanmay Ghai, Srivatsan Ravi, and Pedro Szekely. Amppere: A universal abstract machine for privacy-preserving entity resolution evaluation. In Procesdings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2394-2403, 2021.
Yixiang Yao, Joseph Cecil, Praveen Angyan, Neil Bahroos, and Srivatsan Ravi. Feasibility of privacy- preserving entity resolution confidential healtheare datasets using homomorphic encryption, 2024. URL
hutp::/aaxivv.o/ abs/ 2843.18430
Xun Yi, Russell Paulet, Flisa Bertino, Xun Yi, Russell Paulet, and Flisa Bertino. Homomorphic encryption. Springer, 2014.
Joyce Yu, Jakub Nabaglo, Dinusha Vatsalan, Wilko Henecka, and Brian Thorne Hyper-parameter optimiza- tion for crvacc-prreerving record linkage. In ECML PKDD 2020 Workshops: Workshops of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2020): SoGood 2020, PDFL 2020, MLCS 2020, NEMCP 2020, DINA 2020, EDML 2020, XKDD 2020 and INRA 2020, Ghent, Belgium, September 14-18, 2020, Proccedings, pp. 281-296. Springer, 2020.
Xiaoyu Zhang, Xiaofeng Chen, Hongyang Yan, and Yang Xiang. crowdsourcing with worker updates. Information Sciences, 548:212-232, 2021.
and verifiable online
A Preliminaries
A.1 Homomorphic Encryption
Homomorphic encryptions are widely used for privacy-preserving computation tasks. For example, users outsourcing their data to cloud computing platform would have privacy concerns; with homomorphic encryption, the platform is able to perform necessary computations on the encrypted data directly (Nachrig et al. 2011).
Homomorphic encryption schemne are normally categorized into three types according to the survey (Acar et al. 2018): (1) Partially Homomorphic Encryption (PHE) allows only one type of operation with an un- limited number of times (i.e., no bound on the number of usages), e.g., Pailier (Paillier 1999 (2) Somewhat Homomorphic Encryption (SWHE) allows some types of operations limited number of times, e.g., Boneh- Goh-Nissim (BGN) (Boneh et al. 2005). (3) Fully Homomorphic Encryption (FHE) allows unlimited number of operations for an unlimited number of times. Since Gentry realized the first FHE scheme (Gen- Iry 2009) based on the lattice, a lot of follow-up FHE schemes have been proposed to address the issue it has and optimized to be practical in real- world applications. Some schemes including Brakerski- Gentry- Vaikuntanathan (BGV) (Brakerski et al. (011))
(BFV) (Fan & Vercauteren 2012 Brakerski, 2012) and Cheon-Kim- Kim-Kim-Seee (CKKS) (Jung Hee Cheon (017) support arithmetic op- erations while some others (Tseng et al. 2017 including FHEW (Ducas & Micciancio 2015) is capable for
17

---

Published in Transactions on Machine Learning Research (05/2025)
logical operations. Recently, works are extending the pure arithmetic schemes to run logical comparators without losing advantages such as Single Instruction Multiple Data (SIMD) (lliashenko & Zucca 2021
B Blind Annotation Protocol
B.1
Feature Functions
B.1.1
Domain-specific Language
Some of the features the DSL equips:
The primitive data types are string which is wrapped in double quotes and number.
Three logic operators are supported. Or operator: a returns true if either a or is true. Otherwise, false. And operator: & b returns true if both a and are true. Otherwise, false. Not operator: returns true if a is false, or returns false if is true.
Variables can be defined by Sv=(exp) where $ is the variable indicator, v is the variable name and {exp} is a valid expression.
A preset variable $r is act the target record for comparison and should be used in the annotation the argument of the question Q.
At least one return statement ret is required. The argument the return should be in Boolean. If multiple is provided, the code terminates when the first return statement is found.
Round bracket pair O is for prioritizing the exccution of specified expressions.
Comment starts with #, c.g., this comment.
Whitespaces and empty lines are ignored.
B.1.2 Homomorphic Functions
Besides the challenges mentioned in Section 4.2.2 for constructing pro-defined functions, still need to tackle one more issue that these functions rely As (Yao et al., 2021 pointed out, the control flow including choice (e.g. if) and loop (e.g. for, while) based flows require conditional expressions, which are in Boolean, for execution. These encrypted Boolean values cannot be used in conditional operations unless they are decrypted. However, the decryption of the sonsitive encrypted value causes the exposing of the execution path which not allowed our privacy-preserving settings. The solution is to import ternary operation that converts logic operation to be arithmetic operation, hence bypassing the decryption of the encrypted Boolean condition. The ternary operator is defined as cond:
b where cond is the encrypted Boolean condition and a b encrypted integer values. To return when cond is true to return b when cond is false without decrypting cond, we exploit oblivious-style ternary operators (Ohrimenko et al. 2016 Constable & Chapin 2018) and implement choose (orose([conn) [[cond], [a], [6]) with the arithmetic operators as [b]] + [b] where [cond], [a]] and [6]] are all ciphertexts, and the return of choose, which is either [a] or [b]. also a ciphertext.
ccodd| [[a]
With the ternary operator, we can define more sophisticated functions. Algorithm [ lists some commonly- used functions and the implementation details. All these functions are evaluated and returned in ciphertexts. lower ([s]]) (line 1-6) and upper ([s]) (line 7-12) convert the ASCII character to all lower-case upper-case for standardization. These are not by default applicd because, in some circumstances, letter case is an important signal for identifying record similarity. Both methods take in the encrypted string s and loop the characters in one by one. For each character, it determines table, and uses that as the condition choose whether to modify the value or not. If the homomorphic scheme supports SIMD operation, calculating cond and choose can be run as a batch in one operation so the for-loop is saved. We also provide is in([[], [T] function (line 13-22) which detects if [a] is sub-string
falls in a certain range of the ASCII
18

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 1: Commonly-used Functions
Input return: The imput [s]. [a]] [6] are homornophicllyy encrypted. The return of functions is also
in ciphertext. len returns the encrypled string length in plaintext.
Function lower [ string):
for i <-- to [en([s]] do
3
[d < [=[[]]:
[cond] [[c] 0x40) [[c] < 0x5b);
5
[s[i]] f- choose [cond], [c] + 0x20, [[c]];
return [[}]
Function upper [ string):
for < to [en([=]) do
[c] 6 [+[i]]
10
[cond] (1d 0x60) ([cl < 0x7b);
11
[s[]]] + choose [cond], [c] 0x20, [[c]];
return [[s]] 12
Function is is_in([a]]: string, [b]: string):
14
la 4 [e]]]]
([b];
16
[res] < [False]:
17
for j to
do
18
[r]] + [Tree]:
19
for
l. do
20
L [r] + [r] A ([ali]] = [+] =]]);
21
[res] < [res] V [r]
22
return [[re]]:
of [b]] by scanning the existence of [a] from [[]'s left to right. Using this O(n?) naive mothod without the early exit, even if sub-string match exists, not ideal but unavoidable because of preventing the disclosure of the exccution path for privacy, as stated above.
B.2 Security Analysis
B.2.1 Security Definitions
We model A as probabilistic polynomial-time machine and the parties as interactive Turing machines.
Definition 1 (Adversary). An honest-but centtbbutcurrius adversary A is capable of corrupting a subset of parties in the system, both not the server and one of the data owmers at the same time. A compromised party will divert A all the received messages and act as A requests.
Ideal World. Our ideal-world functionality F interacts with annotation parties as follows:
Each data owner sends its plaintext data r E D and plaintext feature questions Q. F processes and checks Boolean result w as Q+(ry)
VQvr.
or is true, then add the records from both sides to the ground truth.
3. w is false, repeat Step (1) until an agreement reached or exccods a set trial threshold.
Real World. In the real world, F replaced and realized by our protocol described in the previous parts of this section.
Definition (Sccurity). blind annotation protocol is simulation secure if for every adversary A in the real world, there erists security simulator S in the ideal world that also corrupts the same set of parties and produces output identically distributed to A's outgut in the real world.
19

---

Published in Transactions on Machine Learning Research (05/2025)
B.2.2 Security Simulation
We describe a security simulator S that simulates the view of the A in the real-world execution of our protocol. Our security Definition P and ensure both confidentialitty and correctness. receives from F, 2), where C is computing circuits. S sends F(C) to S and obtains fake Homomoprhie Encryption circuits HE fake- ShE generates a random string fake of the same length output. S sends Eakee ffake) to A. As HE circuits distribution is independent, fake is compulationall, indistinguishable from the real НE circuits HE' in the real execution. The random output fake in ideal execution is indistinguishable from in the real execution. In the ideal world, S creates fake circuits H fake and does not use æ for computing. Otherwise, A could use x to evaluate the circuit, which would allow A to distinguish between real and ideal executions.
C Implementation
C.1 User Interface
Figure shows the main page of the web-based GUT which contains the current progress the top of the page and the records that need to be annotated in the table. For each row of the table, it has an indicator of if the record annotated or not, a brief of the record content, dataset name, and the current round of annotation for the record.
Blind Annotation Tool
Tasks
open onnorations seen
Record
Dataset
Phase
elueAnn Bluetooth Voice Control Heecceer
V1
Abt
White hafriger L.BBMMS1I
Netgear 10/100 Switch-
studio
carel movibraccooy
Arazzn
sonic collection plus
Amazon
F7
transfer duds 2 ipod edition (win/macc
Amazon
acobe after effects caB
Amazon
3
now up-to-date & contact vG.0 (mac)
Amazon
final out ixpress 2.0 upgrade
Amazon
piotoo pro upg
vegis sludfoducie
Platinum HDTV Millennim
TV24734
Ab1
P
Gamiis 010 1000-0823-440 200230000000 Slack Muxi 680 Weiide Suction Cup Mount..
Ab1
Z
Samsung DLP Stand In Black TR72BX
Ab1
LG sel LLB LU--8822000 US1 Fully integrati atadd luuit in Stainiese Steel U...
4b1
Panason o VIERA Posma Fat Ranel 1080p HDTV Black
Ab1
2
Mining Quantiariie Assuciation Large Relational Te...
DBLP1
Obttiiing Complete
incomclete Databases A Lex..
lie Reclacement] Franidinn
DBLP1
Efficient lanty search marker basket data
DBLP1
Figure 8: The main page of the web-based GUI
When clicking on the record, an annotation window pops up (Figure 9 It shows the record content from the dataset that. belongs to domain oracles' side, record placeholder $ which represents the record from the
20

---

Published in Transactions on Machine Learning Research (05/2025)
other data owner side, annotation editor with DSL code highlighting, and three buttons to quickly fill the editor with an auto-generation heuristic, discard the annotation, and save the annotation respectively. The "save" operation also runs the syntax check through the DSL parser to make sure the input is syntactically valid.
Dataset
Voice Control Headset V1
te Bottom-Freezer R
bt
Annotation Code (Phase 1)
X
Port 10/100 Rackm
98/me 2000/xp ma
Original: BlueAnt Bluetooth Voice Control Headset - V1
mazon
Target:
ctory 6.0
Sr lower(Sr)
ction plus
Sc is_in("blueant" "'bluennn", $r)
mazon
"e_iin"bbuettotii $r)
pod edition (win/mac
Sc Sc & is_in("headset headset", 5r)
ret Sq
cs3
mazon
contact v5.0 (mac)
2.0 upgrade
mazon
5.x below
iodvd+8 plat
mazon
Millennium TV Ant
bt
Black 660
Quick fill
Discard
Save
Stand Black - TR2
bt
Fully integrated Built Stainless
46' Plasma Flat Panel 1080p HDTV In Black
Abt
Figure 9: The first round record annotation
When annotating records after the first round, the annotation popup window also shows the previous amno- tation and allows domain oracles quickly fill the editor with the previous content (Figure 10
C.2 Details of the HE implementation
Each character in a record is ASCII encoded (1 byte), and the Unicode character is converted to the cor- responding ASCII character. Therefore, a string is represented by a two-dimensional matrix, where each element in the first. dimension represents a character, and each dimension in the second dimension represents bit. For example, a record "Canon" is represented by 5x8 matrix where the first row is 01000011 as the binary representation for 'C'.
Naturally, the encryption is the bit level since the BinFHE module in OpenFHE defines the operations for bits. Hence each item in the matrix is homomorphically encrypted. The logical operators we employed for implernentations from BinFHE are AND (A), OR (V), and XOR (Ø). The detail of the implementation is demonstrated in Algorithm 2 Specifically, byte_equal is the helper function for character-level comparison, which compares cach corresponding bit from two encrypted characters [a] and [b]] is_in the serial version of the function for comparing the encrypted string [a]] a sub-string of the encrypted string [b]. The parallel version utilizes OPenMP and executes character comparison using byte_equal in parallel.
21

---

Published in Transactions on Machine Learning Research (05/2025)
and 198 has been
Annotation Code (Phase 3)
Original: transfer dvds 2 ipod edition (win/mac)
ataset
Target: $r
Voice Control Head
bt
Sr lower(Sr)
Bottom-Freezer
2 $c is_in("i transfer", 5r) I is_in("itransfer" 5r)
bt
3 $c Sc is_in("dvd", $r)
Port 10/100 Rackm
ret Sc
bt
98/me 2000/xp ma
mazon
tory 6.0
on plus
mazon
ed edition (win/mac
mazon
cs3
mazon
ontact v5.0 (mac)
Previous annotation (read only):
1 $r lower($r)
upgrade
2 $c is_in("i transfer", Sr) 1 is=in("itransfer", $r)
mazon
3 Sc Sc "s_inn"dvdds". Sr)
5.x or below
ret Sc
mazon
dvd+8 plat
mazon
Millennium TV Ante
.00 Black Nuvi 660
bt
tand In Black TR7
bt
Fully Integrated B
bt
Plasma Flat
Association Rules
BLP1
with previous
Quick
Discard
Save
Answers from Inc
BLP1
iing and Replacement S Dar, Franklin, B...
DBLP1
Figure 10: The follow-up round record annotation
D Experiments
As the core function for feature extractions in annotation, it is essential to understand the first argument, that is, the input token, in the is in function.
D.1 Feature Length
The length distribution of the tokens demonstrated Figure 11 The length ranges from 16, and most of the tokens have lengths of The average length of the token around (6.65, 6.28, 5.60, 6.61 for Abt-Buy, A Amazon-Google, DBLP-Schola and DBLP-ACM, respectively).
22

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 2: Functions using BinFHE scheme
Function byte_equale
equal [[a]: bit[8), [6]: bu(8):
[res] [True]:
3
for f- to 8 do
[res] [[re]] [[ali]] @ [b]]])
return [[res]:
Function is is_iissnn_ii [_ [b([8], J, [6]: [bi(8], -. :)
la ([a]]]
lb < [bb];
[res] < Falsel]
10
for 1 to l
la 1 do [r]] +- ->rreee]
11
for i to la do
13
L |r] + [r] A byte_equall ([a|i]], [6l + :I];
14
[res] < [res] v [r]:
return [res]:
is in first token length distribution
Abt Buy
Amazon Google
DBLP-Scholar
DBLP-ACM
31
Figure 11: Token length distribution
Abt Buy
Amazon Google
DBLP Scholar
0BLP-ACM
50
In toten
Is taken
Is silen
Ir boom
25
15
10
20
10-
Tobarn
tobin
Talmn
5oom
Figure 12: #tokens appears in the dataset and #is_ in uses extract the corresponding tokens from all annotators. The modified tokens that not in the original token list discarded. Additionally, token is removed from Abt-Buy and Amozln for better visualization.
D.2 Feature Importance
We are curious about if the important features are more likely to be extracted by the annotators. We tokenize the record based on white space without any additional processing steps. Mcanwhile, we count the number of times that function is being utilized to extract the corresponding tokens. Note that annotators could modify the tokens accordingly, e.g., "photoshop" to be "ps", and "international" to be "il8n", we only
23

---

Published in Transactions on Machine Learning Research (05/2025)
use the original token to construct the x-axis, thus none of these non-original tokens are counted if they do not appear in the original token list.
The distribution of tokens and is in usage is summarized in Figure 12 The tokens are present in the long-tail form, whereas the dense invocations of is in function concentrate on its "tail". This observation shows the common tokens contain relatively less information than the rare ones so that the feature extraction depends more on the latter. Some common tokens also attract fair amount of tokens, these are usually the common but special tokens for records, for instance, the brand name of product. Some is is_in functions are called significant amount of times more than the token itself, they derive from the modification of the original token.
E
Case Study
We selected two cases from two different domains. For both of them, two parties hold different opinions at first but establish an agreement finally.
The first case Figure 13 is product records from Amazon-Google dataset. On the Amazon dataset side, the given record content short. The two attempts of the annotation are identical since not too much useful information can be used. On the Google dataset side, besides the brand and product name, it also has the category "music production software". The first annotation has relatively strict rules, whereas the second annotation only tests the brand and the product name.
Dataset: Amazon
Dataset: Google
Record: cakewalk sonar 6 studio Annotation 1:
Record: cakewalk sonar 6 studio edition software music production software
Annotation 1:
Sr lover ($r)
lower ($r)
8c is_in ("cakeaalk". 8r)
is_in ("cakewalk",
8c $c is_in "a_in("s"nar"" $r)
is_in ("sonar", $r)
ret $c
$c is_in $r)
$c $c is_in $r)
$c $c is_in $r)
Annotation 2:
ret Se
Annotation 2:
Sr lover ($r)
Sc is_in(" (akkevakkk', $r)
$r lover (Sr)
8c $c is_in("sonar" $r)
2 $c is_in cakewalkk", $r)
ret $
3 $c $c is_in ("sonna"r''" 8r)
ret Se
Figure 13: A case in Amazon-Google
The second is from DBLP-ACM dataset and shown in Figure 14 The record content from ACM mentioned "database", but in DBLP, it is represented as "DBMS". One possible solution is that annotators could expand "DBMS" to be "database" with their domain knowledge. However, the record content is about the paper title and it usually remains consistent across different websites, so the annotators do not modify it on the token level. Wisely, the annotation from the DBLP's side splits the paper title into two parts, the first part before 4." as the first attempt, and the rest is as the second attempt.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Dataset: DBLP
Dataset: ACM
"Honey, I Shrunk the DBMS": Foot- print, Mobility, and Beyond (Panel) Praveen Se- shadri SIGMOD Conference 1999
Record:
Record: Honey, I shrunk the database: foot- print, mobility, and beyond Praveen Seshadri International Conference on Management of Data 1999
Annotation 1:
Annotation 1:
8r lower (8r)
$ lower (6r)
is_in Sc is in ("honey", $r)
2 $c is_in "p_in("honer"" $r)
$c $c is_in("i", $r)
3
$c
$c is_in (s_in("shrunk" $r)
$c $c is_in ""sin("hrunk'' $r)
4
$c
$c is_in("the" $r)
$c $c a ("thee, 8r)
5
$c
$c is_in ""fiot""ooprrint.""" $r)
Sc $c a is_in ("dbms", $r)
6
$c
$c is_in (".mmbimilll $r)
ret &
7
$c
$c is_in ("and", $1)
$c
$c
is_in ("beyond", $r)
Annotation 2:
Annotation 2:
fr lower (8r)
1s_in ("honey",
$c is_in ("shruuu"
8r lower ($x)
8c is ia_in (* "foopprit", $r)
$c
is_in("the" $r)
$c = $c & is is_in ("mobillyy" $r)
$c
"s.oo("foot,riit", 8r)
$c = Sc is_in a*and", $r)
$c
$c is_in (".omiimiillty.""" $r)
Sc Sc is_in ("beyond", $r)
7
$c
$c
is_in "'_add""nd",, $1)
$c
$c
is_in ("beyond", $r)
ret $c
$c
$c is_in ('sennass"ar""" $x)
10
ret
$e
Figure 14: case in DBLP-ACM
25


Reviews:
[{"review_id": "x2AaBkZn3l", "reviewer": "Reviewer_weJx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. The problem introduced in this paper is interesting and well formulated, and the proposed method is reasonable. The example of camera data is helpful in understanding the problem and method. A few weak things includes: 1) lack of analysis of parameters used in the method; 2) weak empirical analysis. Please find the requested changes for further details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies privacy-preserving annotation, particularly for generating entity link between multiple datasets. The authors introduce the ground truth labeling in entity resolution problem regarding privacy consideration, and propose privacy preserving method for such problem. The main idea of the proposed method is to use homomorphic encryption and the intuition that paired items in two datasets share similar features, which can be checked by multiple questions. If the items passes some amount of questions related to another item, those items are set to pair item. The authors further provide a problem that enable us to use their method for labeling ground truths in entity resolution problem with privacy guarantee.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In Figure 3, font colors are not so visible (especially the orange and purple colors). \n- There are some parameters ($\\varepsilon$ in encryption, $t$ in # of rounds), but they are not formally introduced and studied. As examples, I have questions: what is $\\varepsilon$ doing? and what is the trade-off between $t$ and the performance?\n- Comparison of $t$ and the labeling performance should be studied. Further, is there any reasonable way to choose the number of rounds? \n- If possible, I think it would be better to discuss possible trade-off between privacy and performance. I think the proposed method provides perfect privacy guarantee, but is there a way to improve the labeling performance by relaxing the privacy?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fBFCYg9rtF", "reviewer": "Reviewer_L6HE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem seems to have real-world importance. Figures 1-4 are useful and clear.\n\nMy main criticisms of the paper concern missing detail.\n\n- The experiments are only on the proposed HE-based solution; is there nothing else in the cryptography literature that could apply to the blind annotation problem? For example, I am surprised there is no discussion of private set intersection. The problems seem related and I would have liked to see comparisons. Oblivious transfer and private information retrieval could also be discussed. Additionally, it would have been nice to have a more detailed explanation of why MPC is less preferable than HE: is it just the cost of interactivity (as implied in page 6 of the submission), or is it something deeper? Answering these questions in Related Work (Sec 3.1) would strengthen the paper.\n\n- It looks like a requirement of the proposed solution is for a human being to create feature questions for each of the items to be blind-annotated (Sec 4.2). How much time did this take for each (pair of) dataset? Do you think this is scalable, beyond 50 samples as done the experiments?\n\n- The authors mention that the two parties \"consent on the annotation criteria including how similar can two records be identified as the match and what kind of difference is tolerable\" (Sec 4.1). How does the criteria established here percolate to the feature questions (Sec 4.2)? Why is data needed here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors consider the scenario where two parties each hold a list of items and want to identify which members have a match in the other list, without exposing plaintext to the other party. The items can be natural-language strings such as names of consumer products, which means that two items can match without being completely identical. The authors intend for this annotation process to be the basis for a privacy-preserving entity resolution scheme: the matrix representing matches and non-matches can serve as either training data or evaluation data for a classifier that performs entity resolution.\n\nThe authors propose the following: each party distills every item in their possession into a series of Boolean criteria (\"does this product name have substrings 'Canon' and '123'?\"). This can then be compiled down into either an arithmetic or Boolean circuit, whichever is appropriate for the homomorphic encryption scheme to be applied on the plaintexts. The parties evaluate their questions on the ciphertexts and then pass the results to a third party who is able to decrypt.\n\nThe authors run experiments on four (pairs of) datasets. As one might expect, the less structured data (product names on Amazon and Google) are more challenging to label accurately than the more structured data (entries in DBLP and Google Scholar). The authors also document the communication and computation costs of their proposed solution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From Weaknesses (critical):\n- Add to Related Work\n- Clarify the effort spent to design feature questions\n- Clarify the first paragraph of Sec 4.1\n\nOther (less critical):\n- Figures 5 and 6a should each be split into two, one for each y-axis.\n- I'm not sure I understand what Figure 5 is trying to convey. Are the y-axes correct? It looks like all datasets have the same scale but that does not seem consistent with Table 1.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kgS3kwbsjq", "reviewer": "Reviewer_atV2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work studies an important, yet often overlooked problem. The results seem sound to me, the diagrams are descriptive enough for the reader not familiar with the methodology to understand the proposed method and the examples in the text are well motivated. I do have a couple of points of concern/discussion.\n\nI thank the authors about being very upfront about what their method is (and more importantly isn't) with respect to the problem it addresses. And while some may argue that only targeting the labelling component limits the scope of the work, I would argue it allows the reader to see a more targeted approach to the problem ( and as far as TMLR is concerned limited scope of the problem is acceptable). However, I do also think that while it is fine to limit the work to a specific subset of the problem, the evaluation of the proposed method is on a weaker side. Even though the choice of datasets may be pretty standard in the field (I am personally from the ML field, so unfamiliar with what comprises a set of 'benchmark' datasets in ER), I do not find it descriptive or challenging enough - e-commerce and bibliography management are hardly the most privacy-sensitive examples to illustrate the real use cases for this type of the problem. I would like to see more realistic examples in domains which can actually benefit from privacy-preserving ER, because claiming that a specific camera model is sensitive does seem like a stretch to me with respect to this work's motivation. Domains like healthcare, finance etc. Otherwise while the method itself can be well-motivated, it would not be a) convincing enough with respect to solving the problem and b) would not be particularly relevant to the TMLR community. \n\nI additionally have some concerns regarding the DSL: while a neat idea, this seems to be incredibly domain- and task-specific, making this somewhat limited to domains where extensive labelling expertise is easy to assume. Should the task be even a little more difficult (e.g. mapping the patient records across different hospital departments), I struggle to see how this would generalise. So similarly to my line of discussion above, I would like the authors to clarify and add concrete examples (along with limitations) on how well this idea would perform in more relevant domains. I currently do not believe that the proposed way of labelling is capable of targeting (even somewhat) complex datasets where data criteria can be less trivial to formalise. \n\nFor both of my points above: I believe these are of high importance because in these domains there is a) more variation of data records to account for and b) a higher price of a mislabelled record to pay (and it can be seen as a spectrum of 'matched when shouldn't have, but harmless' to 'mismatched and caused a severely damaging outcome').\n\nWith respect to the methodology of the evaluation, I can somewhat see why the 2.5-5% threshold may be ok as well as the lack of normalisation, but again - in order for the work to be relevant to the readers of TMLR I would argue more realistic usecases are required. It is hardly an inconceivable thought that collaborators normalise their data before performing ER (or any other data-related methods for that matter). So I would also like to see how different data variations can impact the results (e.g. data being normalised, data having many missing/outlying points etc.), so the results become much more relevant to practitioners. \n\nFinally, from my understanding the sampling of records is random, meaning that one can get very unlucky with sampling malformed (since there is no normalisation) or underrepresented data records for which the DSL would fail to account for. Could the authors comment on this and expand the discussion on the record selection here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a novel method for privacy-preserving entity resolution based on homomorphic encryption. The proposed method does not 'solve' the entire process itself, but rather targets the issue of private labels specifically addressing the issue of having to have high-quality labels in an environment where the geographically distant data is sensitive and cannot be visually inspected.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Experiments on datasets from domains where having privacy-preserving ER is actually necessary.\n\nExperiments and detailed discussions on domains where DSL is highly non-trivial and evaluation of how well the method can still perform when the labellers themselves are not fully skilled in accurately addressing the task.\n\nExperiments with more realistic data regimes with more data, different sampling, missing values, normalisation etc.\n\nThe discussion on the above as well as the limitations of the method based on the results of the additional experiments.", "broader_impact_concerns": "So far I do not think that this work has a particularly broad impact based on the description of the methods as well as its experimental evaluation. Since the scope of evaluation is somewhat limited and assumes fairly trivial tasks, I cannot yet say that this work supports its claims well to be relevant to a broad public. I would like the authors to address the issues I outlined above in order to make this work relevant to a broader community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper meets both of the criteria for acceptance to TMLR.
Decision: Accept (poster)