No Pairs Left Behind: Improving Metric Learning with Regularized Triplet Objective
Authors: A. Ali Heydari
Naghmeh Rezaei
Daniel McDuff
Javier L Prieto
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Metric Learning
Representation Learning
Machine Learning for Healthcare
Triplet Loss

Abstract: We propose a novel formulation of the triplet objective function that improves metric learning without additional sample mining or overhead costs. Our approach aims to explicitly regularize the distance between the positive and negative samples in a triplet with respect to the anchor-negative distance. As an initial validation, we show that our method (called No Pairs Left Behind [NPLB]) improves upon the traditional and current state-of-the-art triplet objective formulations on standard benchmark datasets. To show the effectiveness and potentials of NPLB on real-world complex data, we evaluate our approach on a large-scale healthcare dataset (UK Biobank), demonstrating that the embeddings learned by our model significantly outperform all other current representations on tested downstream tasks. Additionally, we provide a new model-agnostic single-time health risk definition that, when used in tandem with the learned representations, achieves the most accurate prediction of a patient's future health complications. Our results indicate that NPLB is a simple, yet effective framework for improving existing deep metric learning models, showcasing the potential implications of deep metric learning in more complex applications, especially in the biological and healthcare domains.

Under review as a conference paper at ICLR 2023
No PAIRS LEFT BEHIND: IMPROVING METRIC LEARN- ING WITH REGULARIZED TRIPLET OBJECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel formulation of the triplet objective function that improves metric learning without additional sample mining or overhead costs, Our approach aims to explicitly regularize the distance between the positive and negative samples in a triplet with respect to the anchor-negativ distance. As an initial validation, we show that our method (called No Pairs Left Behind [NPLB]) improves upon the traditional and current state-of-the-ar triplet objective formulations stan- dard benchmark datasets. To show the effectiveness and potentials of NPLB on real-world complex data, we evaluate our approach on large-scale healthcare dataset (UK Biobank), demonstrating that the embeddings learned by our model significantly outperform all other current representations tested downstream tasks. Additionally, we provide a new model-agnostic single-time health risk defini- tion that, when used in tandem with the learned representations, achieves the most accurate prediction of subjects' future health complications. Our results indicate that NPLB a simple, yet effective framework for improving existing deep metric learning models, showcasing the potential implications of metric learning in more complex applications, especially in the biological and healthcare domains. Our code package as well as tutorial notebooks is available on our public repository: <revealed after the double blind reviews>.
INTRODUCTION
Metric learning is the task of encoding similarity- embeddings where similar samples are mapped closer in space and dissimilar ones afar (Xing et al., 2002; Wang et al., 2019; Roth et al., 2020). Deep metric learning (DML) has shown success in many domains, including computer vision (Hermans al., 2017; Vinyals et al., 2016; Wang et al., 2018b) and natural language processing (Reimers & Gurevych, 2019; Mueller & Thyagarajan, 2016; Benajiba et al., 2019). Many DML models utilize paired samples to learn useful embeddings based on distance comparisons. The most common architectures among these techniques are the Siamese (Bromley et al., 1993) and triplet networks (Hoffer & Ailon, 2015). The main components of these models are the: (1) Strategies for constructing training tuples and (2) objectives that the model must minimize. Though many studies have focused on improving sampling strategies (Wu et al., 2017; Ge, 2018; Shrivastava et al., 2016; Kalantidis et al., 2020; Zhu al., 2021), I), modifying the objective function has attracted less attention. Given that learning representations with triplets very often yield better results than pairs using the same network (Hoffer & Ailon, 2015; Balntas et al., 2016), our work focuses on improving brselet-based DML through a simple yet effective modification of the traditional objective.
Modifying DML loss functions often requires mining additional samples or identifying new quantities (e.g. identifying class centers iteratively throughout training (He et al., 2018)) or computing quantities with costly overheads (Baintas et al., 2016), which may limit their applications. In this work, we aim to provide an easy and intuitive modification of the traditional triplet loss that is motivated by metric learning on more complex datasets, and the notion of density and uniformity of each class. Our proposed variation of the triplet loss leverages all pairwise distances between existing pairs in traditional triplets (positive, negative, and anchor) to encourage denser clusters and better separability between classes. This allows for improving already existing triplet- DML architectures using implementations in standard deep learning (DL) libraries (e.g. TensorFlow), enabling a wider usage of the methods and improvements presented in this work.

---

Under review as a conference paper at ICLR 2023
Many ML algorithms are developed for and tested on datasets such as MNIST (LeCun, 1998) or ImageNet (Deng et al., 2009), which often lack the intricacies and nuances of data in other fields, such as health-related domains (Lee & Yoon, 2017). Unfortunately, this can have direct consequences when we try to understand how MI. can help improve care for patients (e.g. diagnosis or prognosis). In this work, we demonstrate that DMI. algorithms can be effective in learning embeddings from complex healthcare datasets. We provide a novel DML objective function and show that our model's s learned embeddings improve downstream tasks, such as classifying subjects and predicting future health risk using a single- time point. More specifically, we build upon the DML-leamned embeddings to formulate a new mathematical definition for patient health-risks using a single time point which, to the best of our knowledge, does not currently exist. To show the effectiveness of our model and health risk definition, we evaluate our methodology on a large-scale complex public dataset, the UK Biobank (UKB) (Bycroft et al., 2018), demonstrating the implications of our work for both healtheare and the ML community. In summary, our most important contributions can be described as follows. 1) We present novel triplet objective function that improves model learning without any additional sample mining or overhead computational costs. 2) We demonstrate the effectiveness of our approach on large-scale complex public dataset (UK Biobank) and on conventional benchmarking datasets (MNIST, Fashion MNIST (Xiao et al., 2017) and CIFARIO 10 (Krizhevsky, 2010)). This demonstrates the potential of DML in other domains which traditionally may have been less considered. 3) We provide novel definition of patient health risk from single time point, demonstrating the real-world impact of our approach by predicting current healthy subjects' future risks using only single lab visit, a challenging but crucial task in healthcare.
2 BACKGROUND AND RELATED WORK
Contrastive learning aims to minimize the distance between two samples if they belong to the same class (are similar). As result, contrastive models require two samples to be inputted before calculating the loss and updating their parameters. This can be thought of as passing two samples to two parallel models with tied weights. hence being called Siamese or Twin networks (Bromley et al., 1993). Triplet networks (Hoffer & Ailon, 2015) build upon this idea to rank positive and negative samples based on an anchor value, thus requiring the model to produce mappings for all three before the optimization step (hence being called triplets).
Modification of Triplet Loss: Due to their success and importance, triplet networks have attracted increasing attention in recent years. Though the majority of proposed improvements focus on the sampling and selection of the triplets, some studies (Baintas et al., 2016: Zhao et al., 2019; Kim & Park, 2021; Nguyen et al., 2022) have proposed modifications of the traditional triplet loss proposed in Hoffer & Ailon (2015). Similar our work, Multi-level Distance Regularization (MDR) (Kim & Park, 2021) seeks to regularize the DML loss function. MDR regularizes the pairwise distances between embedding vectors into multiple levels based on their similarity. The goal of MDR is to disturb the optimization of the pairwise distances among examples and to discourage positive pairs from getting too close and the negative pairs from being too distant. A drawback of regularization methods is the choice of hyperparameter that balances the regularization term, though adaptive balancing methods could be used (Chen et al., 2018; Heydari et al., 2019). Most related to our work, Balntas et al. (2016) modified the traditional objective by explicitly accounting for the distance between the positive and negative pairs (which the traditional triplet function does not consider), and applied their model to learn local feature descriptors using shallow convolutional neural networks. They introduce the idea of "in-triplet hard negative", referring to the swap of the anchor and positive sample if the positive sample closer to the negative sample than the anchor, thus improving on the performance of traditional triplet networks (we refer to this approach as Distance Swap). Though this method uses the distance between the positive and negative samples to choose the anchor, does not explicitly enforce the model to regularize the distance between the two, which was the main issue with the original formulation. Our work addresses this pitfall by using the notion of local density and uniformity (defined later in $3) to explicitly enforce the regularization of the distance between the positive and negative pairs using the distance between the anchors and the negatives. As result, our approach ensures better inter-class separability while encouraging denser intra-class embeddings. In addition to MDR and Swap Distance, we benchmark our approach againt three related and widely-used metric learning algorithms, namely LiftedStruct Song et al. (2015), N-Pair Loss Sohn (2016), and InfoNCE Oord et al. (2018a). Due to the space constraints, and given the popularity of the methods, we provide an overview of these algorithms in Appendix E
2

---

Under review as a conference paper at ICLR 2023
Traditional Triplet Objective
Triples with Distance Snop
NPL8 Objective (Prupasell
6+
67
Cos 8_
Eol 01
OPLL
E
d(skei), d(u)) d(0(o.).400))
d(spcc.oom)) diolo))-000)
Figure I: Visual comparisons between a traditional triplet loss (left), Distance Swap triplet loss (middle) and our proposed No Pairs Left Behind objective (right) on a toy example. In this figure o refers to a learned operator, 8+ = d(ó(p;), ((a=)). 5_ = d(p(n:), o(a:)) and eo denotes the margin. For this toy example, the network (b-) trained on the traditional objective (left) is only concerned with satisfying 6_ > 5+ + co. potentially mapping positive close to negative samples n;; mgg- which not desirable. A similar case could happen for the Distance Swap variant as well (middle). Our proposed objective seeks to avoid this by explicitly forming dependence between the distance of the positive and negative pair and ó_.. This regularization results in denser mappings when samples are similar and vice versa when the samples are dissimilar, as shown in $4. We describe our formulation and modification in $3.
Deep Learned Embeddings for Healthcare: Recent years have seen increase in the number of DL models for Electronic Health Records (EHR) with several methods aiming to produce rich embeddings to better represent patients (Rajkomar et al., 2018; Choi et al., 2016b; Tran et al., 2015; Nguyen et al., 2017; Choi et al., 2016;; Pham et al., 2017). Though most studies in this area consider temporal components, DeepPatient (Miotto al., 2016) does not explicitly account for time, making it an appropriate model for comparison with our representation learning approach given our goal of predicting patients' health risks using single snapshot. DeepPatient an unsupervised DL model that seeks to learn general deep representations by employing three stacks of denoising autoencoders that learn hierarchical regularities and dependencies through reconstructing a masked input of EHR features. We hypothesize that learning patient reconstructions alone (even with masking features) does not help to diseriminate against patients based on their similarities. We aim to address this by employing a deep metric learning approach that learns similarity embeddings.
Predicting Patient's 's Future Health Risks: Assessing patients' health risk using EHR remains a crucial, yet challenging task of epidemiology and public health (Li et al., 2015). An example of such challenges are the clinically-silent conditions, where patients fall within "normal" or "borderline" ranges for specific known blood work markers, while being at the risk of developing chronic conditions and co-morbidities that will reduce quality of life and cause mortality later on (Li et al., 2015). Therefore, early and accurate assessment of health risk can tremendously improve the patient care, specially in those who may appear "healthy" and do not show severe symptoms. Current approaches for assessing future health complications tie the definition of health risks multiple time points (Hirooka et al., 2021; Chowdhury & Tomal, 2022; Razavian et al., 2016; Kamal et al., 2020; Cohen et al., 2021; Che et al., 2017). Despite the obvious appeal of such approaches, the use of many visits for modeling and defining risk simply ignores large portion of patients who do not return for subsequent check ups, especially those with lower incomes and those without adequate access to healthcare (Kullgren et al., 2010; Taani et al., 2020; Nishi et al., 2019). Given the importance of addressing these issues, we propose a mathematical definition (that built upon DML) based on a single time point, which can be used to predict patient health risk from a single lab visit.
3 METHODS
Main Idea of No Pairs Left Behind (NPLB): The main idea behind our approach is to ensure that, during optimization, the distance between positive p; and negative samples n, is considered, and regularized with respect to the anchors a, (i.c. explicitly introducing notion of distance between d(p;, 74) which depends on d(a,, nt)). We visualize this idea in Fig. 1. The mathematical intuition behind our approach can be described by considering in-class local density and uniformity, as introduced in Rojas-Thomas Thomas & Santos (2021) for unsupervised clustering evaluation metric.
Given metric learning model o, let local density of sample p: be defined as LD(ps) min (((p))) ((p)))) Vp; E C₁ and # j. and let AD(c;) be the average local density of all point in class Ck. An ideal operator & would produce embeddings that are compact while well
3

---

Under review as a conference paper at ICLR 2023
separated from other classes, that the in-class embeddings are uniform. This notion of uniformity, proportional to the difference between the local and average density of each class, i.e.
LD(p))-A((a)) AD(
if cc
Unif(cx)=
0
Otherwise
for 0 < { < 1. However, computing density and uniformity of classes is only possible post-hoc once all labels are present and not feasible during training if the triplets are mined in a self-supervised manner. To reduce the complexity and allow for general use, we utilize proxies for the mentioned quantities to regularize the triplet objective using the notion of uniformity. We take the distance between positive and negative pairs inversely proportional to the local density of a class. Similarly, the distance between anchors and negative pairs is closely related to the average density, given that triplet model maps positive pairs inside an e.-ball of the anchor (c. being the margin). In this sense, the uniformity of a class is inversely proportional to [d(o(p;), o(n,)) - d(o(a;), ((n;))l-
NPLB Objective: Let 66-) denote an operator and T be the set of triplets of the form (pt. az n2)) (positive, anchor and negative tensors) sampled from mini-batch B with size N. For the ease of notation, we will write p(g;) as 2g. Given margin co hyperparameter), the traditional objective function for triplet network shown in Eq. (1):
Crriplet
Σ
[d(Oa: pp) d(0er on) + eol
with [+ max 0} and d(.)) being the Euclidean distance. Minimizing Eq. (1) only ensures that the negative pairs fall outside of an 6--ball around the a, while bringing the positive sample pt inside However, this objective does not explicitly account for the distance between positive and negative samples, which can impede performance especially when there exists high in-class variability. Motivated by our main idea of having denser and more uniform in-class embeddings, we add a simple regularization term to address the issues described above, shown in Eq. (2)
of this ball (illustrated in Fig. 1), satisfying d(da. on) d(óa; Φp)
N
1
LNPLB =
Σ
[d(aa (p)) d( (n) coo + [d(Op, d(øa, on)),
(2)
N
(P..T.N))TT
where p N and NPLB refers "No Pairs Left Behind." The regularization term in Eq. (2) enforces positive and negative samples to be roughly the same distance away as all other negative pairings, while still minimizing their distance to the anchor values. However, not careful, this approach could result in the model learning to map n, such that d(oa, Øp) > maxfes, d(Øp. ox) which would ignore the triplet term, resulting in a minimization problem with no lower bound". To avert such issues, we restrict p (or generally, p = 0 (mod 2)) as in Eq. (3).
LNPLA
1
Σ
[d(0a, 0p) d(cha Φn) + eool [d(0m, Φn) d(
N
(PER..N))
Note that this formulation does not require mining of any additional samples nor complex computa- tions since it just uses the existing samples in order to regularize the embedded space. Moreover
LNPLE = 0 = - [d((pp dh.)) d(p.. ) = [d(øa: hp) d( 4n) + 6o.
which, considering only the real domain, is possible if and only if d(@), on) d(pla Φn), and d(ø.: Φ.)) 2 d(ø.: k,) + c.. explicitly enforcing separation between negative and positive pairs.
'The mentioned pitfall be realized by taking p i.
N
C(pi, 04 n.)
Σ
[d(p., 0p) d(pa, on)) + eo]_ + (d(0): Φo)) (()a;
(PL.O.RDEET
In this case, the model can learn map and such that d( Φn)) C where max (d( &n ), d( d( Φp) m) resulting in
4

---

Under review as a conference paper at ICLR 2023
Triplet Loss
Dist. Swap Loss
NPLB (Ours)
Triplet Loss
Dist. Swap Loss
NPLB (Ours)
Embeed Din
UMAP
Embed. Dim
Embel. Dim 1
Embed. Dim
UMAP1
UMAP1 Fashion MNIST Training Set
MNIST Training Set
Emneed Dim
UMAP
Embed.. Dim
Embed.. Dim 1 Testing
Emhed. Dim
UMAP1
UMAP1
UMAP1
Fashion MNIST Testing
MNIST: #1 02
Fashion MINIS::
T-Shirt
Tronser Pullover e Dress Coar Sandal
Shirt
Sucoker
Bag
Aakll Boot
[A] MNIST
(B) Fashion MNIST
Figure 2: Visual comparisons between a traditional triplet loss, Distance Swap and the proposed NPLB objective on train (top row) and test (bottom row) sets of (A) MNIST and (B) Fashion MNIST. (A) To evaluate the feasibility of the proposed Triplet loss general datasets, trained the same network (described Appendix Appendix K) under identical conditions on the MNIST dataset, with only difference being the loss function used. (B) UMAP-reduced embeddings of Fashion MNIST trained with the three triplet loss versions. Our results indicate both quantitative and qualitative improvements in the embeddings, as shown above and in Table (See Fig. A6 and A7 for higher resolution versions).
VALIDATION OF NPLB ON STANDARD DATASETS
Prior testing our methodology on healthcare data, we validate our derivations and intuition on common benchmark datasets, namely MNIST, Fashion MNIST and CIFAR10. To assess the improvement gains from the proposed objective, we refrained from using more advanced triplet construction techniques and followed the most common approach of constructing triplets using the labels offline. We utilized the same architecture and training settings for all experiments, with the only difference per dataset being the objective functions (see Appendix K for details on each architecture). After training, we evaluated our approach quantitatively through assessing classification accuracy of embeddings produced by optimizing the traditional triplet, Swap Distance and proposed NPLB objective. The results for each data are presented in Table 1, showing that our approach improves classification. We also assessed the embeddings qualitatively: Given the simplicity of MNIST, designed our model to produce two-dimensional embeddings which we directly visualized. For Fashion MNIST and CIFAR, we generated embeddings in g64 and 12 respectively, and used Uniform Manifold Approximation and Projection (UMAP) (McInnes et al., 2018) to reduce the dimensions for visualizations, as shown in Fig. 2. Our results show that networks trained on NPLB objective produce embeddings that are denser and well separated in space, as desired.
Table 1: Comparison of state-of-the-art (SOTA) triplet losses with our proposed objective function. We
present the classification accuracy (Weighted Fl score) standard datasets using XGBoost for five random train-test splits (we randomly split the data into train and (80-20) five time, and calculated the mean and standard deviation
NPLBBttained model was consistent across different classifiers (as shown for different dataset $5). Results for LiftedStruct and InfoNCE are presented in Table A2.
accuracies).
note that improved performance
Trad. Triplet Loxs
N-Puir
MDR
Distonce Swap
NPLB Objective (Ours)
MNIST
0.9859 0.0009
0.9843 = 0.0003
0.9886 + 0.0009
0.9891 + 0.0003
0.9954
0,0003
Fashion MNIST
0.9394 0.001
0.9586
0.003
0.9557
0.001
0.9536 0.001
0.9664 0.001
CIFARIO
0.8036 + 0.028
0.7963 0.034
0.8152 0.027
0.8285 0.022
0.8475 0.025
5 IMPROVING PATIENT REPRESENTATION LEARNING
In this section, we aim to demonstrate the potential and implications of our approach on a more complex dataset in three steps: First, we show that deep metric learning improves upon current
5

---

Under review as a conference paper at ICLR 2023
state-of-the-an patient embedding models ($5.1). Next, we provide comparison between NPLB, Distance Swap and the traditional triplet loss formulations ($5.2). Lastly, we apply our methodology to predict health risks of currently healthy subjects from single time point ($5.3). We focus on presenting results for the female subjects due to space limitations. We note that results on male subjects are very similar to the female population, as presented in Appendix I.
5.1 DEEP METRIC LEARNING FOR BETTER PATIENT EMBEDDINGS
Healthcare datasets are considerably different than those in other domains. Given the restrictions on sharing health-related data (as stipulated by laws such as those defined under the Health Insurance Portability and Accountability Act HIPAA), most DL.-based models are developed and tested on proprietary in-house datasets, making comparisons and benchmarking a major hurdle (Evans, 2016). This is in contrast to other areas of ML which have established standard datasets (c.g. ImageNet or GLUE (Wang et al., 2018a)). 8a)). To show the feasibility of our approach, we present the effectiveness of our methodology on the United Kingdom Biobank (UKB) (Bycroft et al., 2018): a large-scale (~~ 500K subjects) complex public dataset, showing the potential of UKB as an additional benchmarking that can used for developing and testing future DL models in the healthcare domain. UKB contains deep genetic and phenotypic data from approximately 500,000 individuals aged between 39-69 across the United Kingdom, collected over many years. We considered patients' lab tests and their approximated activity levels (c.g. moderate vigorous activity per week) as predictors (complete list of features used is shown in Appendix M), and their doctor-confirmet conditions and medication history for determining labels. Specifically, we labeled a patient "unhealthy" if they have confirmed conditions or take medication for treating a condition, and otherwise labeled them as "apparently-hette -heatthhh". We provide step-by-step description of our data processing in Appendix F,
A close analysis of the UKB data revealed large in-class variability of test ranges, even among those with no current or prior confirmed conditions (the pooparently-healthy subjects). Moreover, the overall distribution of key metrics are very similar between the unhealthy and appurently-healthe patients (visualized in Appendix G). As result, we hypothesized that there exists a continuum among patients' health states, leading to our idea that a similarity-based learned embedding can represent subjects better than other representations for downstream tasks. This idea, in tandem with our assumption of intricate nonlinear relationships among features, naturally motivated our approach of deep metric learning: our goal is to train a model that learns a metric for separating patients in space, based on their similarities and current confirmed conditions (labels). Due to our assumptions, we used the apparently- -healthy patients initially as anchor points between the two ends of the continuum (the very unhealthy and healthy). However, this formulation necessitates identifying a more "reliable" healthy group. often referred to as the bona fide healthy (BFH) group (Cohen et al., 2021)?.
To find the BFH population, we considered all patients whose key lab tests for common conditions fall within the clinically- normal values. These markers are: Total Cholesterol, HDL Cholesterol, LDL Cholesterol, Triglycerides, Fasting Glucose, HbAlc, C-Reactive Protein.; we refer to this set of metrics as the Po metrics and provide the traditional "normal" clinical ranges in Appendix H. It is important to note that the count of the BFH population is much smaller than the apparently-hear healthy group (~ 6% and ~ 5% of female and male populations, respectively). To address this issue and to keep DMI. as the main focus, we implemented a simple yet intuitive rejection-baser sampling to generate synthetic BFH patients, though more sophisticated methods could be employed in future work. Similar to any other rejection-based sampling and given that lab results often follow Gaussian distribution (Whyte& & Kelly, 2018), we assumed that each feature follows a distribution A (M aw) where μl and о₂ denote the empirical mean and standard deviation of feature x; for all patients. Since BFH patients are selected if their Po biosignals fall within the clinically- normall lab ranges, we used the bounds of the clinically normal range the accept/reject criteria. Our simple eijection-based sampling scheme is presented in Appendix I..
Training Procedure and Model Architecture:
Before training, we split the data 70% 30% for training and testing. In the training partition, we augment the bona fide population 3 folds and then generate 100K triplets of the form (a;, Pi, n.)) randomly in an offline manner. We chose to generate
Athhhggh it is possible to further divide each group (e.g. hased on conditions). we chose to keep the patients in three very general groups show the feasibility of our approach in various health-related domains.

---

Under review as a conference paper at ICLR 2023
only 100K triplets in order to reduce training time and demonstrate the capabilities of our approach for smaller datasets. Note that in an unsupervised setting, these triplets need to be generated online via negative sample mining, but this is out of scope for this work given that we have labels priori.. Our model consists of the three hidden layers with two probabilistic dropout layers in between and Parametric Rectified Linear Units (PReLUU) (He et al., 2015) nonlinear activations. We present visual representation of our architecture and the dimensions in Fig. A5, We optimize the weights for minimizing our proposed NPLB objective, (3), using Adam (Kingma & Ba, 2014) for 1000 epochs with Ir 0.001, and employ an exponential learning rate decay ( = 0.95) to decrease the learning rate after every 50 epochs, and set the triplet margin to 6, For simplicity, we will refer to our model as SPHR "Similarity-basee Patient Risk modeling). In order to test the true capability of our model, all evaluations are performed on the non-uugmented data.
Results: Similar to $4, we evaluated our deep-learned embeddings on its improvements for binary (unhealthy or opparentlyhealthhy and multi-class (unhealthy, apparently- healthh or bona fida healthy) classification tasks. The idea here is that if SPHR has learned to separate patients based on their conditions and similarities, then training classifiers on SPHReproduced embeddings should show improvements compared to data. We trained five classifiers (k-nnarest neighbors (KNN), Linear Discriminate Analysis (LDA), a neural network (NN) for EHR (Chen et al. (2020a)), and XGBoost Chen & Guestrin (2016)) 6)) on raw data (not-trannformed) and other common transformations. These transformations include a linear transformation (Principal Component analysis [PCA]). a non-linear transformation (Diffusion Maps (Coifmman & Lafon (2006) [DiffMap]) and the current of nonlinear transformation, DeepPatient. DeepPatient, PCA, DiffMap, and SPHR are E + R-, where n, E N denote the initial number of features and (reduced) mapping space, respectively, with d = 32 (in order to have the same dimensionality as SPHR), though various choices of d yielded similar results. We present these results in Table 2 comparing classification weighted F1 score for models trained on raw EHR, linear and nonlinear transformations. We also evaluate the separability qualitatively using UMAP, as shown in Fig. AI. In all tested cases, our model significantly outperforms all other transformations, demonstrating the effectiveness of DML in better representing patients from EHR.
Table 2: Comparison of binary and multi-label classification performance (weighted F1 score) with vari- ous representations on the female subjects. We kept the same random seeds across different classifiers and randomly split the data into train and test (80-20) five time, and calculated the mean and standard deviation considered BFH patients healthy patients (hence having
of the accuracies. For binary classification, binary labels). Our results show that SPHR significantly improves the classification of all tested classifiers. demonstrating better separability in space compared to raw data and state-of sfate-of-thh-ars art methods (DeepPatient).
Model Not-TTranffrmed
PCA
DiffMap
DeepPatient
SPHR (Ours)
Binary Classification
KNNs
0.6093 + 0.002
0.6067
0.002
0.6281 = 0.003
0.6397 = 0.003
0.7180 = 0.002
LDA
0.6189 + 0.001
0.6144
0.002
0.6309 + 0.002
0.6404 + 0.005
0.7189 + 0.002
NN for EHR
0.6214 1 0.006
0.6127
0.006
0.6228 = 0.003
0.6315 + 0.001
0.7142 0.003
XGBoost
0.6113 + 0.003
0.6088
0.002
0.6196 + 0.005
0.6311 + 0.003
0.7223 0.002
Multi-Label
Classification
KNNs
0.5346 + 0.002
0.5341
0.002
0.5646 = 0.006
0.5740: 0.004
0.6534
0.002
LDA
0.5501 + 0.002
0.5432
0.001
0.5510 0.007
0.5637 + 0.003
0.6640
= 0.001
NN for EHR
0.5374 + 0.004
0,5473
0.003
0.5541 1 0.004
0.5774 1 0.004
0.6667
1 0.002
XGBoost
0,5190 1 0.003
0,5082
0.003
0,5390 = 0.002
0.5688 + 0.002
0.6642 1 0.002
Table Comparison of SOTA Triplet losses with our proposed objective on UK Biobank data (male and female patients). We quantitatively evaluate embeddings by measuring classification accuracy (weighted Fl scores) of XGBoost on five random train-test splits of the UK Biobank for each gender. As shown below, our model significantly outperforms traditional and distance swap loss in both binary and multi-class classification, demonstrating its potential in real-world applications, such as healthcare applications. Results for LiftedStruct and InfoNCE are presented Table A2.
Trad. Triplet Loss
N-Pair
MDR
Distance Swap
NPLB Objective (Ours)
Fematex: Binary
0.6592 + 0.002
0.6727 + 0.004
0.6612
0.003
0.6041 = 0.002
0.7223 = 0.002
Females: Multi-Labe
0.5874 + 0.001
0.6064 : 0.002
0.6047 = 0.005
0,5416 : 0.004
0.6642 - 0.002
Malex: Binary
0.7174 + 0.001
0.7423
: 0.005
0.7208
0.002
0.6563
= 0.003
0.8160 = 0.003
Males: Multi-Label
0.6861 1 0.003
0.6961 0.002
0.6964
0.002
0.6628 0.002
0.7845 0.003
7

---

Under review as a conference paper at ICLR 2023
5.2 NPLB SIGNIHICANTLY IMPROVES LEARNING ON COMPLEX DATA
One of our main motivations for modifying the triplet object was to improve model performance on more complex datasets with larger in-class variability. To evaluate the improvements provided by our NPLB objective, we perform the same analysis as in $4, but this time on the UKB data. Table 3 demonstrate the significant improvement made by our simple modification to the traditional triplet loss, further validating our approach and formulation experimentally.
5.3 PREDICTING HEALTH RISKS FROM A SINGLE LAB VISIT
Table 4: The percentage of apparently healthy female subjects who later develop conditions within each predicted risk group. For the same number of healthy subjects, we use embeddings from each method to classify patients into three health risk groups using the provided health risk definition [Normal (N). Lower Risk (LR), and Higher Risk (HR)). To validate each model's risk prediction. we use future health assessment to calculate the percentage subjects each risk group who later developed Cancer, Diabetes or Other Serious Conditions. Among all methods (top three shown), SPHR- preediccdd Normal and Higher Risk patients developed the fewest and most conditions, respectively, as desired.
Transfrrasooreed
DeepPatient LR
SPHR (Ours)
Normal LR
HR
Normal
HR
Normal
LR
HR
Predicted Healthy Subjects in Each Risk Group
Percentage
74.71%
5.82%
16.26%
77.19%
7.61%
15.2%
68.35%
6.94%
24.71 %
Validation on Future Diagnosis
Cancer
3.17%
0.92%
2.83%
3.21%
<0.1%
<0.1%
1.67%
2.85%
3.22%
Diabetes
1.26%
1.85%
1.41%
0.51%
<0.1%
<0.1%
0.50%
<0.1%
4.80%
Other Serious Cond
9.3%
5.66%
9.29%
8.77%
4.98%
6.81% %
1.47%
8.33%
11.65%
Predicting patients' future health risks is a challenging task, especially when using only a single lab visit. As described in $2, all current models use multiple assessments for predicting health risks of patient; however, these approaches ignore large portion of the population who do not return for additional check-ups. Motivated by the definition of risk in other fields (e.g. risk of re-identification of anonymized data (Ito & Kikuchi, 2018)), we provide a simple and intuitive distance- bastance-based definition of health risk to address the mentioned issues and that is well suited for DML embeddings. Given the simplicity of our definition and due to space constraints, we describe the definition below and outline the mathematical framework in Appendix B.
Definition of 'SngleeTime Health Risk:
We define the health distance as the euclidean distance between a subject and the reference bona fide healthy (BFH) subject. Many studies have shown the large discrepancy of lab metrics among different age groups and genders (Cohen et al., 2021). To account for these known differences. we identify reference vector which is the median BFH subject from each age group per gender. Moreover, for simplicity and interpretability, we define health risk as discrete groups using the known BFH population: For each stratification g (age and sex), we identify two BFH subjects who at the 2.5 and 97.5 percentiles (giving us the inner 95% of the distribution). and calculate their distance to the corresponding reference vector, This gives us a distance interval 487.5 In group g. any new patient whose distance to the reference vector falls inside the corresponding [tg 5. 187.t) is considered be "Normal". Similarly, we identify the [t fgol intervals (corresponding to the inner 98% of BFH group); any new patient whose health distance is within [t,, (ge) but not in [t]. t97.sl I is considered to be in the "Lower Risk" (LR) group. Lastly, any patient with health distance outside of these intervals is considered to have "Higher Risk" (HR). The UKB data is good candidate for predicting potential health risks, given that it includes subsequent follow-ups where a subset of patients are invited for repeat assessment. The first follow-up was done between 2012-2013 and it included approximately 20,000 individuals (Littlejohns et al., 2019) (25x reduction with many measurements missing). Based on our goal of predicting risk from single visit, only include the patients' first visits for modeling, and use the 2012-2013 follow-ups for evaluating the predictions.
We utilized the single-time health risk definition to predict a patient's 's future potentials of health complications. To demonstrate the versatility of our approach, we predicted general health risks that were used for all health conditions available (namely Cancer, Diabetes and Other Serious Conditions); however, we hypothesize that constraining health risks based on specific conditions will improve risk predictions. We considered five methods for assigning risk group to each
Results:
8

---

Under review as a conference paper at ICLR 2023
patient: (i) Euclidean distance on raw data (preprocessed but not transformed), (ii) Mahalanobis distance on pre- pre-prceesed data, (iii) Euclidean distance on the key metrics (Po) for the available conditions(describeel in section 5.1), therefore hand-crafting features and reducing dimensionality in order to achieve an upper bound performance for most traditional methods (though this will not be possible for all diseases). The last two methods consider are deep representation learning methods: (iv) DeepPatient and (v) SPHR embeddings (proposed model). We use the Euclidean metric for calculating the distance between deep-learned representations of (iv) and (v). For all approaches, we assigned all patients to one of the three risk groups using biosignals from single visit, and calculated the percentage of patients who developed a condition in the immediate next visit. Intuitively, patients who fall under the "Normal" group should have fewer confirmed cases compared to subjects in "Lower Risk" or "Higher Risk" groups. Table shows the results for the top three methods, with our approach consistently matching the intuitive criteria: among the five methods, SPHR- sPHR-predicted patients in the Normal risk group have the fewest instance of developing future conditions, while the ones predicted as High risk have the highest instance of developing future conditions (Table 4).
CONCLUSION AND DISCUSSION
We present simple and intuitive variation of the traditional triplet loss (NPLB) which regularizes the distance between positive and negative samples based on the distance between anchor and negative pairs. To show the general applicability of our methods and as an initial validation step, tested our model on three standard benchmarking datasets (MNIST, Fashion MNIST and CIFAR-10) and found that our NPLB- -PLB-ntaieedd model produced better embeddings. To demonstrate the real world impact of DML, such as our proposed framework called SPHR, we applied our methodology on the UKB to classify patients and predict future health risk for the current healthy patients, using only a single time point. Motivated by risk prediction in other domains, we provide distance-based definition of health risk. Utilizing this definition, we partitioned patients into three health risk groups (Normal, Lower Risk and Higher Risk). Among all methods, SPHR-predicte Higher Risk healthy patients had the highest percentage of actually developing conditions the next visit, while -red-cced Normal patients had the lowest instances, which is desired. Although the main point of our work focused on modifying the objective function, a limitation of our work is the simple triplet sampling that we employed. particularly when applied to healthcare. We anticipate additional improvement gains by employing online triplet sampling or extending our work to be self-supervised (Chen et al., 2020b; Oord et al., 201 8b; Wang et al., 2021).
The implications of our work are threefold: (1) Our proposed objective has the potential of improving existing hiplet-based models without requiring additional sample mining or computationally intensive operations. We anticipate that combining our work with existing triplet sampling can further improve model learning and results. (2) Models for predicting patients' health risks nascent and often require time-series data. Our experiments demonstrated the potential improvements gained by developing DML-based models for learning patient embeddings, which in turn can improve patient care. Our results show that more general representation learning models are valuable in pre-pree pre-processing EHR data and producing deep learned embeddings that can be used (or fine-tuned, for more specific downstream analyses. We believe additional analysis on the learned embedding space can prove to be useful for various tasks. For example, we show that there exists a relationship between distances in the embedded space and the time to develop a condition, which we present in Appendix C. The rapid growth of healthcare data, such as EHR, necessitates the use of large-scale algorithms that can analyze such data in scalable manner (Evans, 2016). Currently, most applications of ML in healthcare are formulated for small-scale studies with proprietary data, or use the aualicly-available MIMIC dataset (Miotto et al., 2016; Johnson et al., 2016), which is not as large-scale and complex as the UKB. Similar to our work, we believe that future DL models can benefit from using the UKB for development and benchmarking, (3) Evaluating health risk based on a single lab visit can enable clinicians to flag high-risk patients early on, potentially reducing the number (and the scope) of costly tests and significantly improving care for the most vulnerable individuals in a population.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
Our code package and tutorial notebooks are all publicly available at on the authors' Github at: <revealed after the double blind reviews>, and we will actively monitor the repository for any issues or improvements suggested by the users. Moreover, we have designed our Appendix to be comprehensive guide for reproducing our results and experiments as well. Our Appendix includes all used features from the UK Biobank (for male and female patients), a complete list of model parameters (for classification models), detailed definition of single-time health risk as well as pseudocode and description architectures we developed for our experiments.
ACKNOWLEDGMENTS
Will be added after double blind reviews.
REFERENCES
Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Edwin R. Hancock Richard C. Wilson and William P. Smith (eds.). Proceedings of the British Machine Vision Conference (BMVC), pp. 119.1-119.11 BMVA Press, September 2016. ISBN 1-901725-59-6, doi: 10.5244/C. 30.119. URL https: // dx. doi. org/
/trgg/111 10. 5244/C.30.119
Y. Benajiba, J. Sun, Y. Zhang, L. Jiang, Z. Weng, and O. Biran. Siamese networks for semantic pattern similarity. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pp. 191-194, Los Alamitos, CA, USA, feb 2019. IEEE Computer Society. doi: 10.1109/ICOSC. 2019.866551.. URL https: / /oi.. ieeeccopptersoocceeeee..oooo org/10..///..oooo..ooo.oo. 1109/ 1109//Cos.. Icocc. 201 9. 8665512.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. J. Cowan, G. Tesauro, and Alspector (eds.), Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmann 1993. URL htwp::tttt///t /ttptttps://rrreeeeeeetet ngs. neur ips- CC wwwpppppp/////ccecc..cceeeeeeeea 993/ wttp////ccc0000222222 7bd3df9 Tbc3dd99bbc9360b9c Paperr pdf.
Clare Bycroft, Colin Freeman, Desislava Petkova, Gavin Band, Lloyd T. Elliott, Kevin Sharp, Allan Motyer, Damjan Vukcevic, Olivier Delaneau, Jared O' O'Connell, Adrian Cortes, Samantha Welsh, Alan Young, Mark Effingham, Gil McVean, Stephen Leslie, Naomi Allen, Peter Donnelly, and Jonathan Marchini. The uk biobank resource with deep phenotyping and genomic data. Nature, URL
562(7726):033-009, Oct 2018. ISSN 1476-4687. https://dolt www.oii.org/00/ 1038 44885-18666- 018-0008-100 0579-2.
doi: 10.
Zhengping Che, Yu Cheng, Shuangfei Zhai, Zhaonan Sun, and Yan Liu. Boosting deep learning risk prediction with generative adversarial networks for electronic health records. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 787-792, 2017. doi: 10. D..100//CDM220117 93.
Peipei Chen, Wei Dong, Jinliang Wang, Xudong Lu, Uzay Kaymak, and Zhengxing Huang. Inter- pretable clinical prediction via attention based neural network. BMC Medical Informatics and Decision Making, 20(3):131, Jul 2020a. ISSN 1472-6947. doi: 020- 1110-7.
URL
Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD 16, pp. 785-794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10. 10.1145/29396722299978. URL https: /tttts:://oi... http://ooo.oo//.//////// 1r4//0.111//// 01672.2939988
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020b. URL https: / //arxiiv.org 2002. 05709.
10

---

Under review as a conference paper at ICLR 2023
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 794-803. PMLR, 10-15 Jul 2018. URL https: ptwpppooeeeedeeee ngs ml p.rssss////ee.... / v80/ chen aaten1.a.tmm...
Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16. pp. 3512-3520, Red Hook, NY, USA, 2016a. Curran Associates Inc. ISBN 9781510838819;
Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier Tejedor-Sojo, and Jimeng Sun. Multi-layer representation learning for medical concepts. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16. pp. 1504, New York, NY, USA, 2016b. Association for Computing Machinery. ISBN 9781450342322. doi: 10.11452939672.299933 URL
ht.tps: http:://doooorrrooo
145/ / 29229999772222 2939923.
Rafiqul Chowdhury and Jabed Tomal. Risk prediction for repeated measures health outcomes: A divide and recombine framework. Informatics in Medicine Unlocked, 28:100847, 2022. ISSN w00p:////..................
2352-9148. doi:
100847. URL https: /www///w......////////// scienced irect. com/ htt.:////cceeeceeerrteeeeeeeeeee /ttp:///paaa.2221111222221022110
Netta Mendelson Cohen, Omer Schwartzman, Ram Jaschek, Aviezer Lifshitz, Michael Hoichman, Ran Balicer, Liran I. Shlush, Gabi Barbash, and Amos Tanay. Personalized lab test models to
quantify disease potentials in healthy individuals. Nature Medicine, 27(9): Sep 2021.
ISSN 1546-1 doi: (038//41191-021-001 s41591-021 0646--666666666 6.
URL https: http://oiiooooo /ttp://oooooo.oo./////////o
Ronald R. Coifiaan and Stéphane Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5-30, Jul 2006. ISSN 1063-5203, URL https: /ttp:://wwaw wwwp:/wwwwwww///.o/////.ww wwwenceeieeeeetc........
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245. leee, 2009.
R. S. Evans. Electronic health records: Then, now, and in the future. Yearbook of medical informatics, Suppl ((Suppl D:548-S61, May 2016. ISSN 2364-0502. URL htwph:ttp////ett /pubmed. ncbi..ii...... ml 27199197/pmidj,
m. m.nih.
Weifeng Ge. Deep metric learning with hierarchical triplet loss. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10. 10.1109//CV.22015.11
Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval, 2018. URL https: / http/axxvvvv.../ org/ abs 110/11003..0 06189.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. CoRR, abs/1 701.07737.777 2017. URL htt.:t:t://ae//..rgg/////// hbtgg//bbs///i aww.aa/s..././////////// 07737.
A. Ali Heydari, Craig A. Thompson, and Asif Mehmood. Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions, 2019. URL htwpt:ttt///aat /tap:s::////iiiaa g/ g/abs/1 / 91 2, 12355.
Nobutaka Hirooka, Takeru Kusano, Shunsuke Kinoshita, Ryutaro Aoyagi, and Nakamoto Hidetomo. Association between healthy lifestyle practices and life purpose among a highly health- literate cohort: cross-sectionss study. BMC Public Health, 21(1):820, Apr 2021. ISSN 1471-2455 doi: 10. 186// 02899001--20 10905-7. -21-1000777 URL https: /oii org /10 1186 1889 - 021
10905-7.
11

---

Under review as a conference paper at ICLR 2023
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Aasa Feragen, Marcello Pelillo, and Marco Loog (eds.), Similarity- Based Pattern Recognition, pp. 84-92, Cham, 2015.
Springer International Publishing. ISBN 978-3-319-24261-33
Satoshi Ito and Hiroaki Kikuchi. Risk of re-identification based on euclidean distance in anonymized data pwscup2015. In Leonard Barolli, Tomoya Enokido, and Makoto Takizawa (eds.), Advances in Network- Baased Information Systems, pp. 901-913, Cham, 2018. Springer International Publishing. ISBN 978-3-319-65521-3
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3(1):160035, May 2016. ISSN 2052-4463.
https: doi: 10. 10.1038/sdata.2011 2016..3.. UBL.tppps//a....rrr....00//////
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 21798- 21809. Curran Associates, Inc., 2020. URL ht httpttts::///cccotttttttt Pa.. er/2020/ fi e
neurips. cc /pap
Sundreen Asad Kamal, Changchang Yin, Buyue Qian, and Ping Zhang. An interpretable risk prediction model for healthcare with pattern attention. BMC Medical Informatics and Decision Making, 20(1 1):377, Dec 2020. ISSN 1472-6947. doi: 1011166611111---0-0013 URL https: doi. .org/oo....... 11001/11111/12211000000003303330
/doii
Yonghyun Kim and Wonpyo Park. Multi-level distance regularization for deep metrie learning. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL 1412. 6980.
https://arxiv.orr
Alex Krizhevsky. Convolutional deep belief networks on cifar- 10. 2010.
Jeffrey T. Kullgren, Alison A. Galbraith, Virginia Hinrichsen, Irina Miroshnik, Robert B. Penfold, Meredith B. Rosenthal, Bruce E. Landon, and Tracy A. Lieu. Health care use and decision making among lower-income families in high- deguctbdee health plans. Archives of internal medicine. 170
(21):1918-1925, Nov 2010. ISSN 1538-3679, doi: 10.
wwwcchinnerrme..211000 https:/ /twp//dddddddd////// nabi mlm. nih. gov/2109835.. 21098352(pmid),
URL
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Choong Ho Lee and Hyung-Jin Yoon. Medical big data: promise and challenges. Kidney research and clinical practice 36(1):3-11, Mar 2017. ISSN 2211 2211-9132. URL penttss:/pubbmen ncb i.nlm. nih. ttw 28392994. 28392994[pmid].
Hui Li, Xiaoyi Li, Murali Ramanathan, and Aidong Zhang. Prediction and informative risk factor selection of bone diseases. IEEF/ACM Transactions Computational Biology and Bioinformatics, 12(1))79-91, 2015. doi: 20119,
Thomas J. Littlejohns, Cathie Sudlow, Naomi E. Allen, and Rory Collins. Uk biobank: opportunities for cardiovascular research. European heart journal, 15814):1158-1166, Apr 2019. ISSN 1522- 9645, doi: 10. (993/eurhearti//eh2554, URL https: httph:::/puubedd..t. /ttp///bbddddd///// ncbi nlm. nih g2 531320. 28531333 p2531dd0[pmid].
158-1166,,
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2018. URL https::///// /ttp/aaxvvv/////////eea/aaaaa..a hrrg/abs/1802
00426............. 03426.
Riccardo Miotto, Li Li, Brian A. Kidd, and Joel T. Dudley. Deep patient: An unsupervised representation to predict the future of patients from the electronic health records. Scien- tific Reports, 6(1):26094, May 2016. ISSN 2045-2322 doi: 10.1008//rep26694, URL
htts:://do..rgooe 1038 1008///rep226994.......
12

---

Under review as a conference paper at ICLR 2023
Jonas Mueller and Aditya Thyagarajan. Siamese recurrent architectures for learning sentence similarity. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Mar. 2016. doi: 10.16609anaiiv30i1....
URL https://ojs aaai. paaaiooze org/ index. php/AAAI/articl
e/view/1033 0350..
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Pytorch metric learning, 2020. URL https: //arxiv. hwwpxii..rgg/ab///222/
091 64.
Khanh Nguyen, Hoang Huy Nguyen, and Aleksei Tiulpin. Adaptive gradient triplet loss with automatic margin learning for forensic medical image matching. In Medical Imaging with Deep Learning, 2022. URL https: / wpwpreveeeeee.ee/e//ee/e. net / forum?id-LgD e jjNN.
Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh. Deepr: convolu- tional net for medical records. IEEE Journal of Biomedical and Health Informatics, 21(1)22 2017. doi: 10. N0.110//BHL.22062233999
Takumi Nishi, Akira Babazono, and Toshiki Maeda. Association between income levels and ir- regular physician visits after health checkup, and its consequent effect on glycemic control among employees: A retrospective propensity score-matched cohort study. Journal of Dia- betes Investigation, 10(5):1372-1381, 2019. doi: https://doi.org// http://ii...r//1.111//o/o/// 1/jj///iii// 13025. URL wiley ccm/ htt////o//bbb/b/e 11111/jdi, 13025.
ht.ps:: http:////nnnnnnnneeeee
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018a. URL https: http//axivvvv///// wwwpeiiivooroooooeee abss////////// 007007777...
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018b. URL https: //tppiis////xvai////aa//sss/as/a wwtp:iiwworg/bb//880... 037488
Trang Pham, Truyen Tran, Dinh Phung. and Svetha Venkatesh. Predicting healthcare trajectories from medical records: A deep learning approach. Journal of Biomedical Informatics, 69:218- 229, 2017. ISSN 1532-0464. //www. wwwp::/sieeeeeeieeeccccooo www.e//ncddreet..coo//c/eeeereee
0000011700400001.. URL ht doi: http://cww....c...eeaaaaaaaaaaaa com
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj. Michaela Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, Patrik Sundberg. Hector Yee, Kun Zhang, Yi Zhang, Gerardo Flores, Gavin E. Duggan, Jamie Irvine, Quoc Le, Kurt Litsch, Alexander Mossin, Justin Tansuwan, De Wang, James Wexler, Jimbo Wilson, Dana Ludwig, Samuel L. Volchenboum, Katherine Chou, Michael Pearson, Srinivasan Madabushi, Nigam H. Shah, Atul J. Butte, Michael D. Howell, Claire Cui, Greg S. Corrado, and Jeffrey Dean. Scalable and accurate deep learning with electronic health records. npj Digital Medicine, 1(1):18, May 2018. ISSN 2398-6352. doi: 10. 10.1008//41146-000 101---018-0022-11 URL https: /oo.. org/10 1038/ / s 746 0114446-68881 C029- 1.
Narges Razavian, Jake Marcus, and David Sontag. Multi-task prediction of disease onsets from longitudinal laboratory tests. In Finale Doshi- Velez, Jim Fackler, David Kale, Byron Wallace, and Jenna Wiens (eds.), Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pp. 73-100, Northeastern University, Boston, MA, USA, 18-19 Aug 2016. PMLR. URL https: /ttp:////ooceeeeee ngs. mlr, .rrss// v5 v56/Razave an 6 html.
Nils Reimers and Iryna Gurevych. Sentence-BERRT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 20.18653/11DD9// 1410. URL http:: : aww.aaaaaoantoooo..oooooo www.oollooy.ooooooooooooo wowg//ii--/11111 1-10
Juan Carlos
and Matilde Santos. New internal clustering validation measure for
contiguous arbitrary-shape clusters. International Journal of Intelligent Systems, 36(10):5506- 5529, 2021. doi: http://do...rg/11..0000////.... com/ doi/ abs / 10. 1010/000nnn..io 22521.
URL https: onloneelll ibrrrr wile
Karsten Roth, Timo Milbich, Samarth Sinha, Prateck Gupta, Björn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning, 2020.
13

---

Under review as a conference paper at ICLR 2023
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 761-769, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:////// / arxiv. org/ abs 1.0.9.15555
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL pttpp:tpa.... /ttp::///rcceeeee ngs. negrine wttpcccpp/pp////pc.e http:///......./o...........aa2. Paper.pde .ddf.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding, 2015. URL htttps://arxi....../// http://aaxxoo..raaa/////a/s /1511. 06452.
Murad H. Taani, Bev Zabler, Michael Fendrich, and Rachel Schiffman. Lessons learned for recruit- ment and retention of low-income african americans. Contemporary clinical trials communications, doi: 10.1016/j c/oo////.oncc...... 2020. 100533, URL https: htttps::/ppbmed..re hwtpppubbpdeee. ncbib nlm. nih. 0009/32211555 3221 3221122221a1 1558[pmid].
17: 17:100533-100533 Jan 2020. ISSN 2451-8654
Truyen Tran, Tu Dinh Nguyen, Dinh Phung, and Svetha Venkatesh. Learning vector representation of medical objects via emr-driven nonnegative restricted boltzmann machines (enrbm). Journal of Biomedical Informatics, 54:96-105, 2015. ISSN 1532-0464. doi: http:///bo.orr//1.0111///
15,01.012. URL https:/ /wwp//wwwwooo///wowt sciencedi com/ wttpr//eweeecceeee/ee/eeeeeeeeee science/article///eee 532046415000143.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukeuoglu, and Daan Wierstra. Matching networks for one shot learning, CoRR, Abs/1600.04080, 2016. URL http:://rxixs org/abs/1600 /106. 04080.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedingss of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NIP, pp. 353-355 Brussels, Belgium, November 201 Sa. Association for Computational Linguistics. doi: 10.18663/v 1/W11//W/330 18-6646..
URL https: /aclanthologyyy org / /W18- - 5446.
Feng Wang, Weiyang Liu, Haijun Liu, and Jian Cheng. Additive margin softmax for face verification. CoRR, abs/1 0oss/8801.0599, 2018b. URL httt:://aaxi...rrr/t//// www.xiii..o/gabba. aataaassss.....1
Guangrun Wang, Keze Wang, Guangcong Wang. Philip H. S. Torr, and Liang Lin. Solving inefficiency of supervupe representation learning, 2021. URL https: http://axwwv.ooaaaa//a/aaaaaaa/a 8760.
X. Wang, X. Han. W. Huang, D. Dong, and M. R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5017-5025, 2019.
Martin Brunel Whyte and Philip Kelly. The normal range: it is not normal and it is not a range. Postgraduate medical journal, 94(1117):11--166 Nov 2018. ISSN 1469-0756. doi: 10. 136/ 2018--21-1111
postgradmedi-2011 8- 135983. URL https: /ttp//ummdddmeeeem ncbi. nlm. nih. gow//3425110. 304251405 pm.id..
C. Wu, Manmatha, A. Smola, and Krahenbuhl. Sampling matters in deep embedding learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2859-2867, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmi mnist; a novel image dataset for benchmarking machine learning algorithms, 2017.
14

---

Under review as a conference paper at ICLR 2023
Eric Xing, Michael Jordan, Stuart Russell, and Andrew Ng. Distance metric learning with application to clustering with side- indommnformtion.. In S. Becker. S. Thrun, and K. Obermayer (eds.), Advances in Neural Information Processing Systems, volume 15, MIT Press, 2002. URL https: / /procee dings. neurips cc httpp/ppppp////////// feee//illieeeeee c3e F2a1 b-Paper.pdft pdff
951ac80
Xiaonan Zhao, Huan Qi, Rui Luo, and Larry Davis. A weakly supervised adaptive triplet loss for deep metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Chuang Zhu, Zheng Hu, Huihui Dong, Gang He, Zekuan Yu, and Shangshang Zhang. Construct informative triplet with sto-stage hard-sample generation, 2021. URL https: //l::///rrae http/axxxvvooog///ee abs/2112. 02259.
15

---

Under review as a conference paper at ICLR 2023
Appendix
APPENDIX A RUNTIME AND SCALING ANALYSIS OF SPHR
To measure the scalability SPHR, we generated a random dataset consisting of 100,000 samples with 64 linearl-independentt (also known as "informative") features and 10 classes, resulting in matrix X E IR. 100000044 From this dataset, we then randomly generated varying number of triplets following the strategy described in the main manuscript and in Hoffer & Ailon (2015), representing computationally intensive case (as opposed to more intricate and faster mining schemes). The average of five mining runs is presented "Avg. Mining Time" column in Table A1. Next, we trained SPHR on the varying number triplets five times using (1) A Google Compute Engines with 48 logical cores (CPU) (2) A Google Virtual Machine equipped with one NVIDIA V100 GPU (referred as GPU). The average training times for CPUs and GPUs are shown in Table A1.
Table A1: Average training time of the proposed deep learning model (SPHR). All experiments are done under the same computational The times shown below are the average runs with identical settings.
Number of Triplets
Avg. CPU Training Time
Avg. GPU Training Time
Avg. Mining Time
1,000
8.28 Mins
3.95 Mins
<0.01 Mins
5,000
15.07 Mins
5.96 Mins
0.01 Mins
10,000
31.02 Mins
7.60 Mins
0.12 Mins
50,000
76.24 Mins
16.79 Mins
0.65 Mins
100,000
133.26 Mins
27.15 Mins
1.27 Mins
500,000
512.75 Mins
74.79 Mins
5.58 Mins
APPENDIX B MATHEMATICAL DEFINITION OF SINGLE- HEALTH RISK
In this section, we aim to provide mathematical definition of health risk that can measure similarity between a new cohort and an existing bona fide healthy population without requiring temporal data. This definition inspired by the formulation of risk other domains, such as defining the risk of re-identification of anonymized data (Ito & Kikuchi, 2018) 8). We first define the notion of "health distance", and use that to formulate thresholds intervals which enable us to define health risk as mapping between continuous values and discrete risk groups
Definition Appendix B.1. Given space X, a bona fide healthy population distribution B and new patient p (all in X), the health score sn of p is:
(p) = ((P)
where (H) denotes the nh percentile of B, and d(-) refers to a metric defined on X (potentially pseudo metric).
Definition Appendix B.1 provides measure of distance between a new patient and an existing reference population, which can be used to define similarity. For example, if d(-) is the Euclidean distance, then the similarity between patient and y is sim(x,, y) = 1+d[z.y) Next, using this notion of distance (and similarity). we define risk thresholds that allow for grouping of patients.
Definition Appendix B.2. A threshold interval Ig = It'p te] is defined as the distance between the vectors providing the inner q percent of a distribution H and the median value. Let n = (100 - 4)/2, then we have Ly
I = 11 (F)P
Pso(H)))
Additionally, for the sake of interpretability and convenience, we can define health risk groups using known I, values which we define below.
Definition Appendix B.3. Let M R + V {n} where d denotes the number of features defining patient, with 7) being discrete group and V denoting pre-defined risk groups based on k E N many

---

Under review as a conference paper at ICLR 2023
Trad. Triplet Loss
UMAP
UNNA
UMAP2
UMAA1
UMAP UMAPS
UMAP
UNMAY
UMAP1
UMAP3
UMAP1 1
UMAP2
Non-Transformed (Raw)
Trained
Network
Subject
UMAP2 2
UMAP1
Healthy
UMAP 1
Apparently Healthh
Unhealthy
NPLB (Proposed)
Figure A1: Qualitative assessment of approach UKB female patients. The NPLBBtrained network better separates subjects, resulting significant improvements in classification of patients, as shown in Table 3. Note the continuum of patients for both metric learning techniques, especially among the apparently-healthy patients (in yellow).
k). Using the same notion of health distance s, and threshold interval Ig as before, we define patient p's health risk group as:
intervals (i.e. |V [g
((p) = V if s,(p) E E
otherwise
We utilize our deep metric learning model and the definitions above in tandem to predict health risks. That is, we first produce embeddings for all patients using our learned nonlinear operator G. and then the distance between the bona fide healthy (BFH) population and new patients assign them risk group. Mathematicallyy given the set of BFH population for all groups, i.e.
A (5), B[a6,50). B5,,55), B136,60], 6561,65), ]
we take the reference value per age group to be
Fage G(Pso (Bage))
where Bage denotes the BFH population for the age group age. Then, using definition Appendix B.2, we define:
N Nage
95 [d(P2.5 Pool Bage ()ggge)),, d( ((o77.55 (agee) Pso (age))
(A.4a)
LRage = 1gg = (d(Pe(Bage) Poo (uge )). d(Pa9 (age)) Pso Bage)) ))]
(A.4b)
Note that Nage C. LRage- Lastly, using the intervals defined in Eq. (A.4), we define the mapping M as shown in Eq. (A.5).
Normal
if s, (G(p)) d(G(p), Nage
Mage (p)
Lower Risk
8, (G(p)) d(G(p), ((l()))FFgg)) Fage) E I.Rage' Nage
(A.5)
Higher Risk otherwise
APPENDIX C PREDICTING PATIENT'S HEALTH RISK IN TIME
Given the performance of our DML model in classifying subjects and health risk assessment, we hypothesized that retrieve relationship between spatial distance (in the embedded space)
2

---

Under review as a conference paper at ICLR 2023
and patient's time to develop condition using only single lab visit. Let assume that subjects start from a "healthy" point and move along a trajectory (among many trajectories) to ultimately become "unhealthy" (similar to the principle of entropy). In this setting, hypothesized that our model maps patients space based on their potential of moving along the trajectory of becoming unhealthy. To test our hypothesis, we designed experiment to health risk, and the NPLB embeddings to further stratify patients based on the immediacy of their health risk (time). More specifically, we investigated the correlation between spatial locations the serrentyyhhealty patients in the embedded space and the time to which they develop a condition.
distance-basee definition of
Similar to the health risk prediction experiment in the main manuscript, we computed the distance between each patient's 's embedding and the corresponding reference value in the ultra healthy pop- ulation (refer to the main manuscript for details on this procedure). We then extracted all healthy patients at the time of first visit who returned in (2012-2013) and 2014 (and after) for reassessment or imaging visits. It is important to again note that there is significant drop in the number of returning patients for subsequent visits. Among the retrieved patients, we calculated the number of individuals who developed Cancer, Diabetes or Other Serious Conditions. We found strong negative correlations between the calculated health score (distances) and time of diagnosis for Other Serious Conditions (r 0..72, = 0.00042) and Diabetes (r = -0.64, p = 0.0071), with Cancer having the least correlation -0.20 and 0.025. Note that the lab tests used predictors diagnosing metabolic health conditions and less associated with diagnosing cancer, which could explain the low correlation between health score and time of developing cancer. These results indicate that the metric learned by our model accounts for the immediacy of health risk, mapping patient's s who are at higher risk of developing health conditions farther from those who are at lower risk (hence the negative correlation).
associated with
APPENDIX D NPLB CONDITION IN MORE DETAIL
In this section, aim to take closer look at the minimizer of our proposed objective. Using the same notations as in the main manuscript, we define following variables for convenience:
5 - d(øa: 0p)
8 d(0a: dn))
p d(pps (n))
With this notation, we can rewrite our proposed No Pairs Left Behind objective as:
LNPLB
1
Σ
64 - 5_ +
+oo+ + + (p -8_)
(A.7)
N
(PL.L.M))
Note that the since [6+ 5_
+ enl 2 0, ENPLE = if and only the summation of each term is identically zero. This yields the following relation:
- ( -p- (p- = (6+ 8- +
(A.8)
which, considering the real solutions, only valid if 8_, and if 5__ 6_ + co. and therefore p > 84 + ég. As result, the regularization term enforces that the distance between the positive and negative to at least as much 8, - eg, leading to denser clusters that are better separated from other classes in space.
NPLB can very easily implemented using existing implementations in standard libraries. As an example, we provide likee pseudo code showing the implementation of our approach:
from typing import Callable, Opticnal 2 import torch
class NPL81oos (torch. nn. Module)
3

---

Under review as a conference paper at ICLR 2023
def init (self,
triplet_criterion nerch.nn. (erch....r..ppletaarghhh..ss metric: Optional [Callable[
Tensor, TercorTensor],
Corch.Tenson torch. nn. functional, www.ooasaaaaaiiieeeeeeee hettwiisddistaees
11
"*mnnitializs the instance with backbone triplet and distance netric
12
super init_() self selt.triplet triplet_critezie self.metric metric
15
def (owwrd anchor: torch. Tensor, positive: torch. Tensor, 16
negative: torch. Tenser) -> forch.Tenson
"" """ooward method NPLB lcss. """
19
20
Traditional triplet as the first component of the loss function. triplct_loss = self.triplct (anchor, positive, negative) This
21
23
melff negative, keepdin-Crue) anchor_to_negattive self self.metric ( negative, keepdim-True) Here we use the reduction to be 'mean', but it can be any kind that the DL library would support.
positive_to_negative
24
25
26
return triplet loss torch.mean
28
torch. (positive to negative anchor to negative), 2)
Listing Pytorch implementation of NPLB.
APPENDIX E LIFTEDSTRUCT, N- LOSS, AND INFONCE
In this section, we provide a brief description of three popular deep metric learning models that related to our work. We also describe the implementations used for these models, and present complete comparison of all methods on all datasets this work. In LiftedStruct (Song et al., 2015), the authors propose to take advantage of the full batch for comparing pairs as opposed to traditional approaches where positive and negative pairs are pre-defined for an anchor. The authors describe their approach as "lifting" the vector of pairwise distances for each batch to the matrix of pairwise distances. N-Pair Loss (Sohn, 2016) a generalization address the "slow" convergence of traditional triplet models through considering N negative examples instead of the one negative pair considered in the traditional approach. InfoNCE (Oord et al., 2018a) is generalization of N- N-pair loss that is also known the normalized temperature -ccaled cross entropy loss (NT-Xent). This loss aims to maximize the agreement between positive samples. Both N-Pair and InfoNCE loss relate our work due to their formulation of the metric learning objective that are closely related to Triplet loss. To compare our approach against these algorithms, leveraged the widely used Pytorch Metric Learning (PML. package (Musgrave al., 2020). The complete results of our comparisons on all tested datasets are shown below in Table A2.
the traditional triplet loss which aims to
Table A2: Comparison state-of-the-ar (SOTA) triplet losses with our proposed objective function (com- plete version of Tables and 3 of the main manuscript). The classifications were done on the embeddings using XGBoost for five different train-test splits, with the average Weighted Fl score reported below. We note that the improved performance of the NPLB-trained model was consistent across different classifiers. The UKB
results are for the multi-class classification.
MNIST
MashionMNIST
CIFARIO
UKB (Females)
UKB (Males)
Trad. Triplet Loss
0,9859
0.0009
0.9394 + 0.001
0.8036 + 0.028
0,5874 = 0.001
0.6861 + 0.003
N-Puir
0.9863
0.0003
0.9586 + 0.003
0.7936 + 0.034
0.6064
0.004
0.6961 + 0.002
LiftedStruct
0.9853
0.0007
0.9495 + 0.002
0.7946 + 0.041
0.5994
0.003
0.6989 + 0.014
MDR
0.9886
0.0003
0.9557 + 0.003
0,8152 + 0.027
0.6047
0.005
0.6964 + 0.002
InfoNCE
0.9858
0.0002
0.9581 + 0.004
0.8039 + 0.026
0.6103
+ 0.002
0.6816 + 0.003
Distance Swap
0.9891
0.0003
0.9536
+ 0.001
0.8285
1 0.022
0.5416
0.004
0.6628
1 0.002
NPLB (Ours)
0.9954
0.0003
0.9664 0.001
0.8475 1 0.025
0.6642 0.002
0.7845 1 0.003
4

---

Under review as a conference paper at ICLR 2023
UKB Database 0-501
Guerying
Appropriate
75% Completeness Filter
190.386
n
Male
Fonale
Biolegical Sex Fitter
Nuit Fiiter
n: 902.451
301327
Confirmed
Apparently Unhealthy Females
ConditonssMeddiaaioon Fiiter
Unhealthy Femeles
Age Group
36-45
46-10
S1-60
36-455
46-60
14-15
Apparently Unhealthy Males
26-00
Atria
46-26
56-40
66-86
65-76
Unhealthy Males
Figure A2: Visualization of the data processing scheme described Scction Appendix
APPENDIX F UK BIOBANK DATA PROCESSING
Given the richness and complexity of the UKB and the scope of this work, we subset the data to include patients' age and gender (demographics), numerous lab metrics (objective features), Metabolic Equivalent Task (MET) scores for vigorous/moderatee activity and self-reported hours of sleep (lifestyle) (complete list of features Appendix M). Additionally, we leverage doctor-confirmed conditions as well as current medication to assess subjects' health (assigning labels and not used as predictors).
After selecting these features, use the following scheme to partition the subjects (illustrated in Fig. A2);
1. Ensure all features are at least 75% complete (i.e. at least 75% of patients have a non-null value for that feature)
2. Exclude subjects with any null values
3. Split resulting data according to biological sex (male or female), and perform quantile normalization (as Cohen et al. (2021))
4. For each sex, partition patients into "unhealthy" (those who have at least one doctor- confirmed health condition or take medication for treating serious condition) and "appar- ently healthy" population (who do not have any serious health conditions and do not take medications for treating such illnesses). This data is used for training our neural network.
5. Split patients into six different age groups: Each age group is constructed that the number patients each group is on the same order, while the bias in the data is preserved (age groups are shown in Fig. A2). These age groups used to determine age-specific references at the time of risk prediction.
APPENDIX G -IMILARITY OF DISTRIBUTIONS FOR KEY METRICS AMONG PATIENTS
Although lab metric ranges seem very different at first glance, look at the age-stratified ranges of tests show similarity among the apparently healthy and unhealthy patients. Additionally, we further stratify the data based lifestyle, the similarities between the two health groups becomes even more evident. The additional filtering is as follows: We identify the median sleep hours per group as well as identifying "active" and "less active" individuals. We define active someone who is moderately active for 150 minutes or vigorously active for 75 minutes per week. We use the additional filtering to further stratify patients in each age group. Below in Fig. A3 and A4 we show examples of these
5

---

Under review as a conference paper at ICLR 2023
similarities for two age groups chosen at random. These results motivated our approach in identifying the bona fide healthy population to be used as reference points.
Median Sleep ior Higher)
Less than Median Sleep
Median Sleep for Higher)
Less than Median Sleep
35
35
PTL PTS EITE miiimii
PTS LIP
40
50-
Cholesterol immol112
LDL (mmolit)
20-
20
10-
Active Subjects
Active Subjects
Active Subjects
Active Subjects
55-
45
35
10-
25-
20
10-
Active Subjects
Less Active Subjects
Less Active Subjects
Less Active Subjects
"Unhealdhy" (at least one confirmed condition)
Traditicnal Fixed Range
Extimated range for "unhealthy" subjects
"[lealthy" (No confirmed conditions or related meds)
Estimated range for "healthy" subjeects
Figure A3: Distribution similarity of key lab metries between apparently-healthy and unhealthy female patients. We present the violin plot for Total Cholestrol (left) and LDL Cholestrol (right) for patients between the ages of 36-45 (chosen at random). This figure aims to illustrate the similarity between these distributions based on lifestyle and age. That is, by stratifying the patients based on their sleep and activity we can see that health status alone can not separate the patients well, given the similarity in the signals.
APPENDIX H NORMAL RANGES FOR KEY METRICS
Below we provide list of the current "normal" lab ranges for key metrics that determined the bona fude healthy population:
Key Biomarker
Gender Specific?
Range for Males
Range for Females
Reference
Total Cholesterol
No 2 5.18 mums
5 5.18 mous
Link to Reference 1, Link to Relerence 2
HDL
Yes
> 1.3
Link Reference Link to Reference
LDL
No
3.3 mqual
3,3
mqua
Link Reference Link to Reference
Triglycerides
No
1.7
sqqal
<. 1.7 squat
Link to Reference Link to Reference
Fasting Glucose
No
6 170, 100) +
6 [20, 100) mF
Link to Reference 3
HbAle
No
<> mung!
42
muse!
Link to Reference
C-Reactive Protein
No
< aja
Link Reference
APPENDIX I PERFORMANCE OF SPHR ON MALE SUBJECTS
In this section. we present the results of the experiments in the main manuscript (which were done for female patients) for the male patients. The classification results are presented in Tables A3, A4, A5, and A6, and the health risk predictions are shown in Table A7.

---

Under review as a conference paper at ICLR 2023
Median Sleep for Higher)
Less then Median Sleep
Medion Sleep for Higher)
Less then Median Sleep
153
4
231-
7.53
2.00
34
Willycerides
1.75
25-
1.50
1:3
Triglycerides (mmolll.
1.
1.0
HDL [mmo.L)
033
80-
Active Subjects
Active Subjects
Active Subjects
Active Subjects
139
157
SAR BAR
200
Tiglyceries
1,5
155
1:50
22-
40
15
149
100
03
059
153
Less Active Subjects
Less Active Subjects
Less Active Subjects
Less Active Subjects
"Unhealthy" (at least one confirmed condition)
Traditional Pixed Range
Estimated range for "uahsaathy" subjects
"Healthy" (No confirmed conditions or related meds)
Estimated range for "bealthy" subjects
Figure A4: Distribution similarity of key lab metrics between opparently-heattee and unhealthy female patients. We present the violin plot for HDL Cholestrol (left) and Triglycerides (right) for patients between the ages of 55-60 (age group chosen at random). This figure aims to illustrate the similarity between these distributions based on lifestyle and age. That is, by stratifying the patients based on their sleep and activity see that health status alone can separate the patients well, given the similarity in the signals.
Table A3: Comparison binary classification performance (weighted Fl score) with various representa- tions on the male patients. In this case, we consider the bona fida healthy patients as healthy patients and train each model to predict binary labels. We keep the same random seeds across different classifiers, and for the supervised methods, we randomly split the data into train and test (80-20) five time, and calculate the mean and standard deviation of the accuracies. Our model significantly improves the classification all tested classifiers,
demonstrating better separability space compared to raw data and state-of-the method (DeepPatient).
Model
Non-Transformed
ICA
PCA
DecpPatient
SPHR (Ours)
KNNs
0.6200) + 0.004
0.6167
0.003
0.6077 + 0.003
0.6224 + 0.001
0.8163 + 0.002
LDA
0.6275 + 0.005
0.6227
0.004
0.6227 + 0.003
0.6385 + 0.002
0.8141 + 0.002
NN for EHR
0.5926 + 0.014
0.6301
0.018
0.6105 + 0.021
0.6148 + 0.032
0,8092 + 0.011
XGBoost
0.5975 + 0.004
0.5804
0.004
0.6157 : 0.004
0.6101 + 0.004
0,8160 + 0.003
APPENDIX EFFECTS OF AUGMENTATION ON DML
In order to evaluate the effect of augmenting the bona fide population and to determine the appropriate increase fold, we trained SPHR with different levels of augmentation, and evaluated the effect of each increase fold through multi-label classification performance. More specifically, created new augmented datasets with augmentation, 1 x, 3x, 5x and 10x and generated the same number of triplets (as described previously) and trained SPHR. We evaluated the multi-label classification using the same approach and classifiers before (described in the main manuscript) and present the results of XGBoost classification in Table A8. Based on our findings and considerations for computational efficiency, we chose 3x augmentation the appropriate fold increase.
7

---

Under review as a conference paper at ICLR 2023
Table A4: Comparison of binary classification performance (micro Fl score) with various representations on male patients. this case, we consider bona fide healthy patients as healthy patients and train each model to predict binary labels. We keep the same random seeds across different classifiers, and for the supervised methods, we randomly split the data into train and test (80-20) five time, and calculate the mean and standard deviation of the accuracies. Our model significantly improves the classification all tested classifiers, demonstrating better separability space compared raw data and state-of-the-ar method (DeepPatient).
Model
Not-Transformed
ICA
PCA
DeepPatient
SPHR (Ours)
KNNs
0.6490 + 0.004
0.6480
0.004
0.6469 + 0.003
0.6639 + 0.001
0.8185
0.002
LDA
0.6529 + 0.004
0.6509
0.002
0.6509 = 0.002
0.6664 + 0.003
0.8138 0.002
NN for EHR
0.6345 + 0.003
0.6419
0.003
0.6380 = 0.003
0.6527 + 0.005
0.8176 + 0.004
XGBoost
0.6573 + 0.002
0.6488 0.003
0.6469 = 0.003
0.6701 + 0.003
0.8180 - 0.003
Table A5: Comparison of multi-label classification accuracy (weighted F1 score) with various representa- tions the male patients. We keep the same random seeds across different classifiers, and for the supervised methods, we randomly split the data into train and (80-20) five time, and calculate the dard deviation the accuracies. Our model significantly improves the classification for tested classifiers, demonstrating better separability in space compared to raw data and state-of-the-ae method (DeepPatient).
and stan-
Model
Not-Transformed
PCA
ICA
DeepPatient
SPHR (Ours)
KNNs
0,5852 + 0.005
0.5820
0.005
0.5734 = 0.003
0.5834 = 0.002
0.7819 = 0.001
LDA
0.6011 + 0.004
0.5953
0.003
0.5952 = 0.002
0.6080 = 0.003
0.7865 = 0.002
NN for EHR
0.5926 + 0.004
0.5925
0.004
0.5838 + 0.003
0.5918 + 0.001
0.7884 0.005
XGBoost
0.5439 + 0.005
0.5583
0.004
0.5587 = 0.005
0.5896 + 0.003
0.7845 0.003
APPENDIX K ADDITIONAL DETAILS ON MODEL ACRCHITECTURES
K.1 SPHR'S NEURAL NETWORK
(Proposed)
Linear Operator
Probabilistic Dropout
Linear
Probabilistic Dropout
Linear Operator
Operator
PRELU
PReLU
PROLU
Di
G(æ,)
R bxn
R2x0012
0.1
2122226
1.1
IR 256xx
Boxd
Figure A5: Architecture of SPHR. Our neural network is composed of three hidden layers, probablistic dropouts (p 0.1) and nonlinear activations (PReLU) in between. In figure above, n denote to the number of patients and features, respectively, with d being the output dimension (in our case = 32).
For readability and reprodueibility purposes, also include Pytorch snippet of the network used for learning representations from the UK Biobank:
import torch. nn as
class SPHR (nn. Module)
def iinit (self, input_dim:ln 64, output_dim: int= 32) self sii_nindii = input_di self self.cut_dim output_dim
super ()__nnn_n init ()
8

---

Under review as a conference paper at ICLR 2023
Table A6: Comparison of multi-label classification accuracy (micro Fl score) with various representations on male patients. keep same random seeds across different classifiers, methods, we randomly split the data into train and test (80-20) five time, and calculate the mean and standard
the supervised
deviation of the accuracies. Our model significantly improves the classification for all tested classifiers,
demonstrating better separability in space compared to raw data and statc-of-the-ar method (DeepPatient).
Model
Not-Transformed
PCA
ICA
DeepPatient
SPHR (Ours)
KNNs
0.6364 + 0.004
0.6355
0.004
0.6358
0.004
0.6358 + 0.001
0.7921
0.001
LDA
0.6405 0.003
0.6393
0.002
0.6392
0.003
0.6383 = 0.003
0.7811
0.002
NN for EHR
0.6345 0.003
0.6342
0.003
0.6342
0.004
0.6438 + 0.001
0.7930
0.003
XGBoost
0.6409 = 0.003
0.6380
0.003
0.6384 0.003
0.6403 - 0.002
0.7940
0.003
Table A7: The percentage of apparently- male patients who develop conditions in the next immedi- ate visit within each predicted risk group. Among all methods (top three shown). SPHR predicted Normal High risk patients developed the fewest and most conditions, respectively, as expected.
PO (Not- Transformed)
DeepPatient
SPHR (Ours)
Future Diagnosis
Normal
LR
HR
Normal
Higher Risk
Normal
HR
Cancer
2.76%
0.62%
2.75%
2.86%
0.33%
<0.1%
1.52%
2.96%
4.14%
Diabetes
1.88%
194%
1.63%
0.85%
0.73%
1.62%
0.73%
0.55%
5.29%
Other Serious Cond
9.57%
6.44%
9.47%
9.33%
4.41%
7.28%
2.73%
8.60%
12.45%
sclf.nonlinear_net Sn.uenuentaal
nn. In.Linear (self. inp__dnn, 512),
10
nn.Drop (p-0.1), nn.PRell (),
11
I.iner 1. (512, 256),
nn.Dropout (p=0.1), nn.PReLU
14
15
nn.Linear (256, self.out dim),
16
nn.PReLU
18
19
def Sewaard_oneSample(eee input_tensor)
20
# useful for the forward method call and for inference (input_tensor)
return
23
def ferward positive, anchor, negative): # forward method for training
24
25
(positive), f. felf..orwwaddooeeaang anchor), self, forwarddoneeaaneeerr (negative)
return self.
Listing 2: SPHR's network architecture
We train SPHR by minimizing our proposed NPLB objective, Eq.(3), using the Adam optimizer for 1000 epochs with (r 0.001, and employ an exponential learning rate decay (7 0.95) to decrease the learning rate after every 50 epochs. We set the margin hyperparameter to be = 1. In all experiments, the triplet selection was done in an offline manner using the most common triplet selection scheme (e.g. see ht https: /www. loss opt
K.2 CIFAR-10 EMBEDDING NETWORK AND EXPERIMENTAL SETUP
To further demonstrate the improvements of NPLB on representation learning, we benchmarked various triplet losses on CIFARI0 as well. For this experiment, we trained a randomly-initializn VGG13 (Simonyan & Zisserman, 2015) model (not pre-trained) on CIFAR10 to produce embeddings m E I228 for 200 epochs, using the Adam optimizer with ir = 0.001 and a decaying schedule (similar to SPHR's optimization setting, as described in the main manuscript). We note that the architecture used this experiment is identical to a traditional classification VGG model, with the difference being in the "classification" layer which is re- re-purposed for producing 128 128-dimensional embeddings. CIFAR- 10 images were normalized using the standard CIFAR- 10 transformation, and agumented during trainnig (i.e. we did not any augmentations in training the model).

---

Under review as a conference paper at ICLR 2023
Table A8: Studying the effects of different augmentation levels classification proxy for all down- stream tasks. We followed same procedure as other classification experiments (including model parameters). The results for 1x augmentation are omitted since the results are very similar to no augmentation.
No Augmentation
3x
5x
10x
Females: Multi-Label
0.5730 0.002
0.6642 + 0.002
0.6619
0.003
0.6584 + 0.003
Males: Multi-Label
0.6247 + 0.005
0.7845 + 0.003
0.7852 + 0.004
0.7685 + 0.002
K.3 MNIST EMBEDDING NETWORK
For ease of readability and reproducibility, we provide the architecture used for MNIST as a Pytorch snippet:
import torch. nn as nn
class MN MNTST_ Nett__e Inn.Modulee..
def _init__ (self, embedding imnnnsionn
( ()__init___ self.conv net nn Sequential
nn. Conv 32, 5), nn.PReIU()
Mn.Maxooll2d stride=2), nn.Dropout (0.3), nn (on2 64, 5), nn.PReLU
10
11
12
MaxxxPool2 (2, stride=2), nn.Dropout (0.3)
14
15
16
self feedForward net Sequential In. 464+4+4, 512),
18
19
nn. nR. (), nn. Lineir (512, beteddiingddiiensee
20
23
def (sele, :
24
conv_output colf.conv_net (input_tensor) conv_output = conv_output. view 64+4+4) return sclf.feedForr feedForward_net (conv_output)
25
Listing 3: Network architecture used for validation on MNIST.
We train the network for 50 epochs using the Adam optimizer with ir 0.001. We set the margin hyperparameter be 6o 1.
K.4 FASHION MNIST EMBEDDING NETWORK
For readability and reproducibility purposes, we provide the architecture used for Fashion MNIST as a Pytorch snippet:
import torch. nn as nn
class FMNIST_Networs Monulee;
def _init_ (self, diiodding
() self.conv_net nn sequential(
(n.Coonvddiin_ccee (i_chammelann out cut_channels=16, kernel_size=3), nn.PReID
(n.MaxPool2dd,, stride=2),
10
nn.Dropout (0.1), nn. ww..Conn2((in.cameelool out_channels=32, kernel_size=5),
11
12
nn. Max stride=1),
Dropout (0.2),
10

---

Under review as a conference paper at ICLR 2023
nn. Conv2d channels=32, out channels=6,, kernel size=5), nn, nn.AvgPool24 (kernel_size-1), nn. (n.PReLU()
16
17
18
self. feedForward net Sm..entnnt (
21
nn, Linei (64x4*4, 512), nn.PReIU(), Ln.Linear (512, beteeddingdiimensseeee
22
23
25
26
def @awar, input_tensen
27
conv_output self.conv self.conv_net (input_tensor) conv_output = conv cottto view 64.4.4) self. feedForward (conv (conv_output)
Listing 4: Network architecture used for validation on Fashion MNIST.
We train the network for 50 epochs using the Adam optimizer with Ir = 0.001. We set the margin hyperparameter be to 1.
K.S CLASSIFICATION MODELS
The parameters for NN for EHR were chosen based on Chen al. (2020a). The "main" parameters for KNN and XGBoost were chosen through randomized grid search while parameters for LDA were unchanged. We specify the parameters that were identified through grid search in the KNN and XGBoost sections.
K.5.1 NN FOR EHR
We follow the work of Chen al. (2020a) and construct a feed forward neural network with the additive attention mechanism in the first layer. As in Chen el al. we choose the learning rate to be = 0.001 with an L2 penalty coefficient = 0.001, and train the model for 100 epochs.
K.5.2 KNNs
We utilized the Scikit-Learn implementation of K-Nearest Neighbors. The optimal number of neighbors were found with grid search from 10 to 100 neighbors (increasing by 10). For the sake of reproducibility, we provide the parameters with seikit-learn terminology. For more information about the meaning of each parameter (and value), refer the reviewers to the online documentation: https: /ttp///ttitittttcetteeeeeeeaaaet org/ wwtpsotaggtsssootooeoooee stable/ staabe/modulee mwt//oodulleeee/eeemm hwwe/eeeerreeeeeeeeeee sklearn. neighbo KNe www.eeihhorsllassiiiiii..... html.
Algorithm: Auto
Leaf Size: 30
Metric: Minkowski
Metric Params: None
n Jobs: -1
n Neighbors: 50
p: 2
Weights': Uniform
K.5.3 LDA
We employed the Seikit-Leam implementation of Linear Discriminant Analysis (LDA), with the default parameters.
11

---

Under review as a conference paper at ICLR 2023
K.5.4 XGBOOST
We utilized the official implementation of XGBoost, located https: / //ggoootx readthed ocs. io so/en///eee st able/. We optimized model performance through grid search for learning rate (0.01 to 0.2, increasing by 0.01), max depth (from to 10, increasing by 1) and number of estimators (from 10 to 200, increasing by 10). For the sake of reproducibility, we provide the parameters the nomenclature used in the online documentation.
Objective: Bogiste
Max Delta Step: 0
Use Label Encoder: False
Max Depth: 4
Base Score: 0.5
Max Leaves:
Booster: gbtree
Minimum Child Weight:
Callbacks: None,
Missing: NaN
co bolssmple_by_leeelll.l
Monotone Constraints: ()*
colsample_by_noder
n Estimators: 50
co col_sample_by__rree::
Jobs: -1
Early Stopping Rounds: None
Number of Parallel Trees:
Enable Categorical: False
Predictor: Auto
Evaluation metric: None
Random State: 0
? (gamma): 0
reg_alpha::
GPU ID:
reg_lambda:
Grow Policy: depthwise"
Sampling Method: Uniform
Importance Type: None
gaale_pos_weightt :1
Interaction Constraints:
Subsample: 1
Learning Rate: 0.05
Tree Method: Exact
Max Bin: 256
Validate Parameters:
Max Categorical Onehot: 4
Verbosity: None
12

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Proposed Augmentation of Electronic Health Records Data. The proposed strategy will ensure that each augmented feature falls between pre-determined ranges for appropriate gender and age group, which crucial in diagnosing conditions.
Require: X A mapping between gender/age condition groups to raw bloodwork and lifestyle
matricies
Require: condust: A list of all present conditions # c.g. bona fide healthy, diabetic, etc. Require: U: A matrix storing upper bounds for feature, given condition;
Require: L: A matrix of the lower bounds for featurej given condition;
Xdiet +- +-Zrros(Xiic
2: for condition; in condiist, do for feature; in Aiit ( do
3:
4:
<- Mean(feature;)
5: <- STD(feature)) Standard deviation
6: c -101 initialize
7:
while = [Lij, Usy]. do
8:
Z-- ~ (,,, # sampled value from the Gaussian distribution
9: end while
10: Xdiet (coddition:]/feaauree +- z augmented feature
11: end for
12: end for
13

---

Under review as a conference paper at ICLR 2023
APPENDIX L DATA AUGMENTATION SCHEME
APPENDIX M COMPLETE LIST OF FEATURES
M.I UKB FID TO NAME MAPPINGS FOR FEMALE PATIENTS
Lab Metrics
30110: Platelet distribution width
30760: HDL Cholesterol
21003: Age
30770: IGF-1
30160: Basophill count
30010: Red blood cell
30780: LDL Direct
30220: Basophill percentage
(erythrocyte) count 30070: Red blood cell (erythrocyte)
30810: Phosphate
30150: Eosinophill count
30830: Sex Hormone-Bindee
30210: Eosinophill percentage
Globulin (SHBG)
30030: Haematocrit percentage
distribution width
30850: Testosterone
30020: Haemoglobin
30250: Reticulocyte count
30840: Total bilirubin
concentration
30240: Reticulocyte percentage
30860: Total protein
30300: High light scatter
30000: White blood cell
30870: Triglycerides
reticulocyte count
(leukocyte) count
30880: Urate
30290: High light scatter
30620: Alanine
30670: Urea
reticulocyte percentage
aminotransferase
30890: Vitamin D
30280: Immature reticulocyte
30600: Albumin
21001: Body Mass Index
fraction
30610: Alkaline phosphatase
30140: Neutrophill count
30120: Lymphocyte count
30630: Apolipoprotein A
30200: Neutrophill percentage
30180: Lymphocyte percentage
30640: Apolipoprotein
30090: Platelet crit
30050: Mean corpuscular
30650: Aspartate
Lifestyle Metries
haemoglobin
aminotransferase
22038: MET Minutes per Week
30060: Mean corpuscular
30710: C-Reactive Protein
for Moderate Activity
haemoglobin concentration
30680: Calcium
22039: MET Minutes per Week
30040: Mean corpuscular
30690: Cholesterol
for Vigorous Activity
volume
30700: Creatinine
22037: MET Minutes per Week
30100: Mean platelet
30720: Cystatin C
for walking
(thrombocyte) volume
30730: Gamma
22040: Summed MET Minutes
30260: Mean reticulocyte
glutamyltransferaee
per week for All Activity
volume
30740: Glucose
22033: Summed Days of Activity
30270: Mean sphered cell
30750: HbAlc
22034: Summed Minutes of Activity
volume
30130: Monocyte count
30080: Platelet count
30190: Monocyte percentage
1160: Sleep Duration
14

---

Under review as a conference paper at ICLR 2023
M.2
UKB FID TO NAME MAPPINGS FOR MALE PATIENTS
Lab Metrics
30110: Platelet distribution
21003: Age
width
30760: HDL Cholesterol
30160: Basophill count
30190: Monocyte percentage
30770: IGF-1
30220: Basophill percentage
30130: Monocyte count
30780: LDL Direct
30120: Lymphocyte count
30750: HbAlc
30810: Phosphate
30180: Lymphocyte percentage
30730: Gamma
30830: SHBG
30050: Mean corpuscular
glutamylraasferase
30850: Testosterone
haemoglobin
30740: Glucose
30840: Total bilirubin
30060: Mcan corpuscular
30010: Red blood cell
30860: Total protein
haemoglobin concentration
(erythrocyte) count
30870: Triglycerides
30040: Mean corpuscular volume
30070: Red blood cell (erythrocyte)
30880: Urate
30670: Urea
30100: Mean platelet
distribution width
30890: Vitamin D
(thrombocyte) volume
30250: Reticulocyte count
30140: Neutrophill count
30260: Mean reticulocyte
30240: Reticulocyte percentage
30200: Neutrophill percentage
volume
30000: White blood cell
30090: Platelet crit
30270: Mean sphered cell
(leukocyte) count
Lifestyle Metries
volume
30620: Alanine
1160: Sleep Duration
30080: Platelet count
aminotransferase
21001: Body Mass Index
30150: Eosinophill count
30600: Albumin
22038: MET Minutes per Week
30210: Eosinophill percentage
30610: Alkaline phosphatase
for Moderate Activity
30030: Haematocrit percentage
30630: Apolipoprotein A
22039: MET Minutes per Week
30020: Haemoglobin
30640: Apolipoprotein
for Vigorous Activity
concentration
30650: Aspartate
22037: MET Minutes per Week for walking
30300: High light scatter
aninotransferase
reticulocyte count
30710: C-Reactive Protein
22040: Summed MET Minutes
30290: High light scatter
30680: Calcium
per week for All Activity
reticulocyte percentage
30690: Cholesterol
22033: Summed Days of Activity
30280: Immature reticulocyte fraction
30700: Creatinine
22034: Summed Minutes of
30720: Cystatin C
Activity
15

---

Under review as a conference paper at ICLR 2023
APPENDIX N
HIGHER RESOLUTION FIGURES
Triplet
Swap Triplet
Objective (Proposed)
MNIST
Ranboodding Dow
Basbookiing Drww
Rastodking
Embeediing
Eabeeding
Imbelling
Training Set
Embbooking Diee
Enbbeding lirer
Embedding Din
Embedding Dian
Exihadding
Testing Set
Figure A6: Visual comparisons between traditional triplet loss (left), Distance Swap (middle) and pro- posed NPLB objective (right) train (top row) and test (bottom row) sets of MNIST. To evaluate the feasibility of proposed loss on general datasets, we trained the same network (described in Appendix Appendix K) under identical conditions on MNIST dataset, with the only difference being the loss function used. As expected by our mathematical intuition described in $3, the model trained using our proposed objective learns embeddings that much denser within classes and farther apart from other class compared to traditional Triplet and Distance Swap variation, leading better classification results (see Table 1).
Traditional Triplet
Swep Triplet
Objective (Proposed)
Fashion MNIST
Items
Trouser
UMAP1
Pallaver
Coal
Sandal
UMAP1
UMAPP
UIMPP1
Seeaker
Troining
Bag
Anble Bocc
UIMAPS St
UMAPT
Testing
Figure A7: UMAP-reduced comparisons between traditional triplet loss (left), Distance Swap triplet loss (middle) and proposed NPLB objective (right) train and test sets of Fashion MNIST (top and bottom rows). Similar results MNIST, see both quantitative and qualitative improvements the emheddings. as shown Table
16


Reviews:
[{"review_id": "T3JSSqJKx2", "reviewer": "Reviewer_LF9L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel formulation of the triplet objective function by explicitly regularizing the distance between the positive and negative samples in a triplet. \nThey evaluate their approach on (Fahsion) MNIST and EHR data. \n\n", "summary_of_the_review": "Major flaws in empirical analysis.", "strength_and_weaknesses": "While the approach is interesting, I have major concerns regarding the empirical analysis. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N2_GPli5HQT", "reviewer": "Reviewer_7q7N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a simple modification to a metric-learning triplet loss by adding penalty term which encourages the distance between the anchor sample and negative  sample to be similar to the distance between the positive sample and the negative sample. \n\n", "summary_of_the_review": "I vote reject as I do not think that this work is ready for publication, given that the comparison to related triplet/contrastive methods on public benchmark datasets is severely limited. Adding more related methods and experiments should not present a significant hurdle. \n\n\n--------\n## Post Rebuttal Update\nI thank the authors for working to improve their submission. I am keeping my score since metric learning work published at top ML conferences should follow a more thorough benchmarking on diverse datasets and against recent competitive metric learning approaches. In addition, in its current form, the submission is still lacking convincing arguments as to why the proposed approach should work better than alternative metric learning approaches that also make use of additional relationships between positive and/or negative pairs. I encourage the authors to take this feedback into account to mature the submission. ", "strength_and_weaknesses": "Strengths\n- The authors provide results for interesting applications of the method on healthcare data. \n\nWeaknesses\n\n- The proposed idea is extremely simple. In essence, the submission uses the fact that a triplet contains two negative pairs: (a,n) and (p,n). In addition to the standard triplet loss, the authors add a loss that encourages the distance between the p and n pair (a negative pair) to be similar to the distance between the a and the n pair (another negative pair). Thus, the paper just adds an extra loss to the typical triplet loss that encourages all negative pairs in the dataset to be roughly equidistant. \n\n- Given the extreme simplicity of this idea, in my opinion the paper is lacking: (A) a thorough analysis how it compares to other recent triplet/contrastive losses, and in what cases equidistant negative pairs are desirable, (B) a thorough benchmarking comparing to more related work on triplet and contrastive losses*, and (C) an evaluation on a much larger set of common (metric learning) benchmark datasets used in related contrastive/triplet methods, rather than just MNIST and FashionMNIST. \n\n      *To just name a few methods for which I think comparisons are lacking: Lifted Structured Loss [1], Multi-Class N-pair loss [2], Noise Contrastive Estimation [3], InfoNCE [4], Circle Loss [5]\n\n      [1] Oh Song, H., Xiang, Y., Jegelka, S., & Savarese, S. (2016). Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4004-4012). \n \n      [2] Sohn, K. (2016). Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29. \n\n      [3] Gutmann, M., & Hyvärinen, A. (2010, March). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 297-304). JMLR Workshop and Conference Proceedings. \n\n      [4] Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.\n\n      [5] Sun, Yifan, et al. \"Circle loss: A unified perspective of pair similarity optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n     \n\n\n- Finally, the motivation via the definition of uniform in-class embeddings is unclear to me, in part because the notation is not well defined at the moment. I encourage the authors to elaborate more on this point. Notation issue: Rojas-Thomas & Santos (2021)  define the local density (LD) over a point x_i in c_k. The authors in this submission define LD on c_k directly, without specifying if p_j is also in c_k or any data point in the dataset. \n\t\n\t", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "24eo_OtPAM", "reviewer": "Reviewer_wL6B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A variation of a triplet loss for deep metric learning, dubbed NPLB, is proposed. It is inspired by the condition that distance between positive and negative examples should be bigger than distance between anchor and positive. It is illustrated that optimizing such objective leads to more compact clusters. Empirical results on benchmark datasets (MNIST and fashionMNIST) and Biobank dataset show increased prediction accuracy (weighted F1 score), when using embeddings from NPLB.", "summary_of_the_review": "Given that modeling novelty seem to be adding distance based regularization to known objective, and novel application of using the learned metrics as health risk is not quite clear to me, I am leaning towards the rejection, although open to turn to the positive side if my concerns are addressed. ", "strength_and_weaknesses": "Strengths\nMotivation for, and derivation of the new triplet objective is clear.\nEmpirical results for cluster density and separability, as well as embeddings utility in downstream prediction tasks looks compelling\n\nWeaknesses\nAt moments it is not easy to follow the flow of the paper. Last paragraph of the Introduction appears to have redundancy in summarizing the main 3 aspects of the contribution: novel variation of triplet objective, embeddings’ utility for classification tasks, and distance based risk indicators. Claim that this approach does not require additional hyperparameters, as it is based on distances, might be misleading. In equation 3, the added “regularization term”, ie the square of distance, could have a hyperparameter multiplicator to tune the impact on overall loss. Just in this case it is set to 1 (similarly as it is set to 0.1 in MDR case).  \n\nMinor suggestions\nEquation 1 has an extra right bracket.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject