Task-Relevant Feature Selection with Prediction Focused Mixture Models
Authors: Abhishek Sharma
Catherine Zeng
Sanjana Narayanan
Sonali Parbhoo
Roy H. Perlis
Finale Doshi-Velez
Venue: Accepted by TMLR ()

Abstract: Probabilistic models, such as mixture models, can encode latent structures that both explain the data and aid specific downstream tasks.
We focus on a constrained setting where we want to learn a model with relatively few components (e.g. for interpretability).
Simultaneously, we ensure that the components are useful for downstream predictions by introducing \emph{prediction-focused} modeling for mixtures, which automatically selects data features relevant to a prediction task.
Our approach identifies task-relevant input features, outperforms models that are not prediction-focused, and is easy to optimize; most importantly, we also characterize \emph{when} prediction-focused modeling can be expected to work.

Published in Transactions on Machine Learning Research (05/2024)
Task-Relevant Feature Selection with Prediction Focused Mixture Models
Abhishek Sharma
abhisheakhharmo@@g.hraaaa eduu
School of Engineering and Applied Sciences, Harvard University
Catherine Zeng
callerriesenn@@oolleeeevvvvrrrre edu
School of Engineering and Applied Seiences, Harvard University
Sanjana Narayanan
Brnyjamna.bookk@@mmai..
School of Engineering and Applied Sciences, Harvard University
Sonali Parbhoo
s.parbhoo Gimperialllmmm.....coo.o
Imperial College London
Roy Perlis
rpertis@@ggt.arrard.comm edu
Massachusetts General Hospital
Finale Doshi-Velez
School of Engineering and Applied Sciences, Harvard Universily
Reviewed on OpenReview: https: openreview. net/ forun? id= JO
Abstract
Probabilistic models, such as mixture models, encode latent structures that both explain the data and aid specific downstream tasks. We focus on constrained setting where we want to learn model with relatively few components (e.g. for interpretability). Simultanc- ously, we ensure that the components are useful for rownstream predictions by introducing prediction-focuser modeling for mixtures, which automatically selects data features rele- vant to a prediction task. Our approach identifies task-relevant input features, putporfss models that not prediction-focusedd and easy to optimize; most importantly, we also characterize when prediction-focused modeling be expected to work.
1 Introduction
The subfield of tosk-fo generative modeling uses tasks identify relevant structure embedded complex data. For example, suppose we want to discover HIV subtypes (clusters) from electronic health records. Health records complex and contain structures irrelevant to our goal (e.g. effects of insurance type). A model small enough for us to inspect for medical insights (that is, one where underspecify the number of clusters), risks finding irrelevant patterns, while a model that is too large may be too complex to inspect or learn efficiently.
Task-focused approaches to this problem introduce a downstream prediction task to separate relevant struc- ture from irrelovant. For example, the clusters can predict whether patient will develop specific mutation, then it likely those clusters are relevant to identifying HIV subtypes. This motivation results in our desiderata: goal is to train mixture model that is (a) small enough for human inspection, and

---

Published in Transactions on Machine Learning Research (05/2024)
(b) predictive for some task. Because inspection is our ultimate goal, simply creating predictive model is not sufficient. Additionally, focusing only on relevant structures also facilitates sample-efficient learning.
Unfortunately, task-focused generative modeling is not easy: identifying clusters relevant to our task is the challenge. As we detail in Section 6, the naive approach of augmenting a mixture model with labels (supervised generative modeling) fails because the signal from the input features overwhelms the signal from the labels. More principled approaches involve challenging optimization (Hughes al., 2018b). Works like Ren et al. (2020) achieve more tractable optimization by identifying graphical model whose maximum likelihood can be used the solution. However, they do not address important theoretical questions, like if and when their objective makes the desired trade-off between predictive and generative performance.
In this work, we introduce and analyze predict ion-focused Gaussian mixture models (pf-GMM) and hidden Markov models (pf-HMM). All of our models identify relevant structure by automatically curating imput features as relevant or irrelevant. This curation enables us find predictive clusters even when the model is misspecified. Unlike the prediction-oonstrrineed approach by Hughes et al. (2018b), we have formulated our problem as graphical model with single hyperparameter (the probability that an imput feature is relevant). Unlike Ren et al. (2020), we provide precise theoretical analysis identifying data regimes in which this approach will work.
Specifically, we make the following contributions:
1. We develop a probabilistic graphical modeling approach (pf-GM,, pf-HMMM that automatically curates and clusters features relevant to downstream task. We outperform standard model-basod clustering approaches like Gaussian Mixture Models (with and without supervision) at predicting the label and can use existing inference algorithms e.g. Expectation Maximization (EM).
2. We analytically characterize representative scenarios where our auto-curation model achieves the desired relevant structure, and support this analysis with empirical evidence using simulations and real datasets. Unlike Ren et al. (2020), we characterize the true likelihood objective, and not its our analysis applicable irrespective of approximation in inference. In doing lower so, we show in which data regimes this relatively simple and robust approach will work.
3. We demonstrate that pf-GMM achieves predictive clusters in even in misspecified-cluster settings on several synthetic and real-world datasets. We show that our approach not only yields computational benefits but is also casier to optimize than the prediction-constrainee approach (Hughes et al., 2018b). pf-GMMs thus provide the best of both worlds they achieve a good predietive--eneraaivee trade-off and are easy tune and optimize.
Outline
The rest of the paper is organized as follows. In Section 2, we review the literature on semi-supervised learning and task-focused generative modeling. In Section 3, we describe the data setting and the assumptions we make about the data. In Sections and 5, we introduce our prediction-focuset models and describe how to perform inference them. Then in Section 6, we analyze the properties of prediction-focused mixture models and characterize the data regimes in which they will work. In Sections 7 and 8, present experiments and results. Finally, in Section 9, we discuss the implications of our work.
2 Related Work
Semi-supervised Learning Semi-supervised Learning augments the generative modeling of inputs X with supervisory target Y. Methods that incorporate supervisory labels under the paradigm of "supervised clustering" fail to provide a density of the observed data (Eick et al., 2004) and are not aligned with our goals. For methods that train the joint likelihood log Y) (Nigam et al., 1998; Kingma et al., 2014; Ghahramani & Jordan, 1994; Fralcy & Raftery, 2002), the generative process typically modeled with latent variables as X Z ++ Y. However, these models fail to recognize the inherent asymmetry between the dimensionalities of X and Y and tend to ignore Y in favor of allocating the model's capacity for the
2

---

Published in Transactions on Machine Learning Research (05/2024)
much more structured X. We address this issue by incorporating specific input-focused latent variables (called 'switches') that treat the inputs and targets differently.
Task-focused Generative Modeling Most work on learning generative models for specific downstream
tasks (Lacoste-Julien et al., 2011; Cobb et al., 2018; Stoyanov et al., 2011; Futoma et al., 2020) train for the discriminative loss only. Closer our goal of managing the generatire trade-off, Hughes et al. (2017) proposed prediction-constrrinned lcarning in the context of mixture and topic models, formalizing the problem as maximizing the likelihood of X constrained by the prediction objective. However, this objective is hard to optimize in practice, and unlike our work, does not correspond to the maximum likelihood of a valid graphical model.
Downstream use cases of Generative Models Semi-supervised generative models that use a discrim- inative task to learn more useful generative representations have found several applications in the literature. The review by (Chen et al., 2021) highlights the importance of generative models for understanding the dif- ferences in survival time of patients with melanoma with different demographics for aiding in better planning of disease management. Halpern et al. (2016) describe the importance of generative models for identifying groups of clinical conditions that can subsequently used to predict disease subtypes and clinical tags. Works such as (Prabhakaran Vogt, 2019) and (Prabhakaran el al., 2012) also show how generalive models can be used to learn clusters of similar chemical compounds or proteins in HIV treatment which can subsequently be used to predict response patterns for patients who receive those treatments. Furthermore, mixture models have been used in several classes of downstream tasks. For example, Sharma et al. (2023) use the density p(X) from a mixture model to plan an elevator scheduling decision-making task, Silva & Deutsch (2018) use the gaussian mixture model to impute missing values in dataset of lateritic Nickel deposits, and Attar et al. (2014) use gaussian mixture models for anomaly detection. Besides these, the interpretation of clusters for exploration and science-oriented lasks a common use case of the mixture models (Kuyuk et al., 2012; Zhuang et al., 1996). Our work supports these downstream efforts by providing a way to build a mixture model by focusing on the relevant features.
Switch variables In the context of topic models for documents, Chemudugunta et al. (2007) and Ren et al. (2020) use switch variables to include or exclude words based on their predictive relevance. Both approaches rely on independence assumptions between switches and topic-word distributions in the approximate pos- terior. Furthermore, the authors only study lower bound to the tikelihood-makee their results are due the choice of model or inference. In contrast, we develop a innerativediiscrminnate model for more general context. More importantly, we show that the prediction-focused properties exist by our model structure alone (independent of the choice of inference), and (unlike previous work) we describe analytically when our approach will work. Finally, some works impose structural constraints on the target variables and covariates of Conditional Random Fields (CRFs) (Wainwright & Jordan, 2008), such as (Jiao et al., 2006; Chen et al., 2015; Zheng et al., 2015). Unlike these works, we focus on the setting where a model does not have enough parameters to fully model the input dimensions.
it unclear whether
Feature Delection/attribution methods Feature selection (FS) methods pick out small subset of features from the original feature set, and run subsequent analysis on this small subset of features (Jovié et al., 2015). This feature selection can be unsupervised or supervised. Among unsupervised FS methods, clustering FS methods remove the irrelevant features without considering the prediction task (Witten & Tibshirani; Dash & Liu, 2000; Jovié et al., 2015). However, their notion of relevance does not consider the alignment of the clusters to a supervisory signal available in the form of labels. This can lead to a clustering of features that are irrelevant to the labels because the irrelevant features can be correlated to each other. In contrast, utilize the provided labels to cluster the relevant features. This leads to clusters that are predictive of the labels.
Supervised feature selection and altribution methods directly identily the features that are useful for a prediction task (e.g., Random Forest classification, LASSO classification, correlation coefficients) by removing irrelevant features or by providing importance scores. These scores are also an interpretability tool since they help the user understand which features were useful for the task. However, these methods do not directly provide generative model (i.e. clustering) of the relevant. features of the data. In addition, these met
3

---

Published in Transactions on Machine Learning Research (05/2024)
do not prioritize finding all features that are relevant to the outcome; result, the structures they find may be incomplete and thus confusing for domain experts to interpret. In contrast, our prediction-focusel modeling provides model-based clustering using the relevant features and the relative importance scores of these features in the joint task of clustering and prediction.
3 Data Setting
In this section, we formally describe the data distribution that assume in the analysis-related sections of our paper. This is done to emphasize that this distribution is much more complex than the model distribution we introduce in Section 4. That said, we emphasize that our prediction-focused model is specifically designed to work well in 'misspecified" settings where the model has fewer components than the true data generating process, regardless of the specific data generating process.
3.1 Preliminaries
We denote random variables by capital letters (c.g. X, Y), (column) vectors by bold-face letters (c.g. u), and random vectors by bold-faced capital letters (e.g. X). We denote an identily matrix of size D x D as ID. We use v] to denote the concatenation vectors u and v along the last dimension. We use Op. and 1p to denote constant vectors of length D with values 0 and 1 respectively. In our discussions, we use the terms 'feature' and 'dimension' interchangeably. The true data-generated densities and parameters are [*, 1",0", whereas the estimated quantities denoted by f. We use Φ6:6) to denote the multivariate normal density with parameters θ (u,,20}.
3.2 Assumptions
Hn where the input (()) RD and the target (n) E (0,1) when ( and (()) are not time-series. In the time-series setting, the input x(n) E RTXD and the target y()) E {0, 17" are sequences of length T.
We assume that we are given the dataset = {(x )
We consider the setting where the data is concatenation of multiple sources with multiple irrelevant dimen- sions each to one source that contains relevant dimensions and each source distributed as an independent Gaussian Mixture Model. The relevant mixture is called because it also generates Y. Specifically, we assume that the input (m) is concatenation of M sources:
(") = (x("),
x
Vn (1,...,,N)
(1)
where X (n) m E Pm
Vm E (I....,, and D DD D.
(2)
We also assume that there are much fewer relevant dimensions than irrelevant dimensions. This assumption is the main source of the challenge since we only care about the relevant dimensions x(") and the target (n) when explaining the input (") and predicting the target (n).
3.3 Data distribution
We describe the data-generating process for single data point Y) in the non-tine-series setting (the description for the time-series setting appears in Supplement Section 3.4). As stated above in the assump- tions, the ground truth data density f" is product of the relevant source fi and each of the irrelevant sources f (for m 2 2). The data density is (see Figure for a graphical model depicting this process):
('X,Y; (1 Y;O;) II
(= Om)
(3)
m=2
where O" (0).... ..., 0%, are the parameters of the mixture components. Mixture has Km components.
4

---

Published in Transactions on Machine Learning Research (05/2024)
X (n)
x(m) (n)
(n) d
X
D1
D2
DM
n
Y())
2 n
Z
Z(n) M
1
N
N
N
Relevant Mixture (m = 1)
Irrelevant Mixtures (m E [2, M))
Figure Data generating process concatenes M mixture models. The first mixture model also generates the target variable Y (and hence relevant), while the remaining M mixture models are irrelevant.
The first density in the above equation (x,1)): is special because correlated with Y:
K1 (( = Σ (x
(4)
k=1
where ai, are the mixture weights, 87 are the component's input data parameters, and mú are the compo- nent's target parameters.
Every subsequent density fin (Xmi 07n) (m 2 2) is independent of in Equation 3:
Km
(Xmi )
Σ fm, (Xmi B,, for 2 2
(5)
where Th and & are similarly defined as above.
For each mixture density f (.), the mixture weights sum to one:
Σ
Ti Vm.
(6)
Note that we consider the case of Gaussian emissions, ie. ( (m,,))))) B%, of dimension Dm, though approach applies to any exponential-family distribution. We assume (5m()) to be the Bernoulli probability mass function, but again approach extends to any exponential-family distribution.
Example: HIV subtyping. As an application of this data-generating process, consider the HIV subtyping example from the introduction (Section 1). The goal of the HIV subtyping problem is to identify data- driven HIV subtypes in the form patient clusters sharing similar attributes. The data for this problem is the patient's 's electronic health records, which include several sources of information such as blood count information, genomics, and hospital visit codes during the patient's lifetime. Among these, the relevant features (X,) would include measurements of immune response and of co-infections expected to be correlated with the mutations (Y). The irrelevant features correspond to groups of diagnosis and medicine codes that are unrelated to the patient's 's HIV condition (Xm, m 2 2). For instance, patient's depression diagnosis, along with related medical codes, is irrelevant to the HIV subtyping problem. The relevant features and each group of irrelevant features can be thought of as being distributed as a mixture with Km, components since is reasonable to expect that each condition (c.g. depression) has multiple subtypes. While challenging to reconstruct all of the structure in the data as a mixture with K₁ Imzz Km components, our primary focus, the discovery of HIV subtypes, does not necessitate perfect modeling of these irrelevant features.
5

---

Published in Transactions on Machine Learning Research (05/2024)
3.4 Data distribution for the time-series case
In this subsection, we describe the case where a single input-target data pair (X, Y) a sequence of length T, i.c. X = [ x (Rxx(D+*<+aal and Y E {0,1)7. 1)". This case analogous to the non-time-scries setting, except. that assumee that the data is generated by M independent Hidden Markov Models (HMMs) instead of M independent Gaussian Mixture Models (GMMs). The data density is:
M II mh 2
Sm (T, )im)
(7)
T II Si. (i, &i (Yssm))
Ai
(8)
ky=1
Amm
Km
Km
T
T
Jm fx(Xmi:
(
π m,
A
A
ki
(X, Sm,ki)
(9)
11=1
ky=1
(10)
where 0* = (oi.... }, are the parameters of the HMMs and HMM m has Km latent states. क T are the initial state probabilities, Amj are the transition probabilities:
F, = Sii,k 171. k) Amj.h = Sm.kk (Zem = Z V [2,
(11)
(12)
where Zem is the latent state of HMM m at time t. These probabilities sum to one: Σn Tm 1.
Ex A A = 1 for all m.j. Finally, the emission distributions are parameterized the same way as above: I are the parameters of the input-emission distribution (+) of HMM m's latent state k, and oi are
the parameters target-emission fi, (.) of HMM I's latent state
3.5 Consequences of our assumptions
The data may have exponentially many components but we only care about a few. In the GMM case, (.) is the probability density function of a Kn, -oomponent mixture distribution, and fx,,, Y) the density of mixture distribution with exponcntially many components (IIm Km in total). The HMM case is the same except that the mixture components are replaced by HMM components. However, modeling all the components needed to reconstruct all of the structure in the true data-genent gatt-geeneaaing distribution may be unnecessary if are interested in only the structure relevant for prediction.
The data-generating process allows for several forms of correlation. The key consequence of our assumption about the data setting is the form of correlations allow in relevant and irrelevant features: GMM
(a) correlation induced by being distributed component, and (c) independence of relevant and irrelevant mixtures. Together, these correlations allow the relevant features to have complex structure, while keeping the irrelevant features separable from the relevant ones.
HMM, (b) correlation within each mixture
3.6 Problem Statement
We want a mixture model (GMM or HMM) that uses user-specified budget of K (< II. Km)) components to: (a) predict targets Y at test time, and (b) maximize the data likelihood f(X) for a given prediction quality of Y. To address these desiderata, we introduce prediction-foeused mixture models in the next section. Then in Section 6, we show why GMM (with or without supervision) does not achieve good predictive performance in this setting, and how our prediction-focused model is preferable.

---

Published in Transactions on Machine Learning Research (05/2024)
Sd
(m)
B,
Sa
X(m)
B,
t
y-(n)
P
p
0
((n)
Y
7
0
7(m)
2
A
N
(a) pf-GMM
(b) pf-HMM
Figure 2: Graphical models for models pf-GMM (a) and pf-HMM (b). Here, Sa denote the per-dimension switch variables, X denotes the data, Y denotes the target variables. For pf-GMM, Z denotes the cluster assignments, and for pf-HMM, Ze denotes the hidden states at time t.
4 Our Models: Prediction-focused Mixtures
In this section, introduce prediction-focusee mixture models which satisfy the desiderata of maximiz- ing the data likelihood while still performing well on the prediction task. This model is specifically designed to work in the underspecified setting, i.c., when the model is provided much fewer components than the data requires.
Our key assumption is that the input data dimensions can be partitioned as being either 'relevant' or 'irrolevant'. Only 'relevant" dimensions are prodictive of the target, Y. This assumption matches the data- generaling process: we preserve the connection between the relevant dimensions and the target and ap- proximate the irrelevant dimensions with background distribution. We start by introducing prediction- focused Gaussian Mixture Models (pf-GMM) and then extend them to the time series setting by introducing freuuce Hidden Markov Models (pf-HMM) (see Figures 2a and 2b for their graphical models).
4.1 Prediction-focused Gaussian Mixture Model (pf-GMM)
Consider the dataset D ((x("), y()) where N is the total number of data points, and each input (*) E R has D dimensions. The data-generathe process under our model pf-GMM assumes the following way of explaining the dataset DD
First, we draw a 'switch' variable Sa per dimension to indicate whether the dimension relevant irrelevant. If Sa = 1, then the dimension is relevant, and if Sa O, then the dimension is irrelevant. We also draw cluster identity 7() per data point to indicate which mixture component generated the data point:
Sa ~ (prn(p) Ad (L.....
z(n) ~ Cat(@) Cat(B) Vn E fl.....
where p is the prior probability that a dimension is relevant, and 0 is the prior probability of each mixture component.
Next, we draw the input data dimensions x(") from a mixture distribution with K components, where S is binary vector which is for relevant dimensions (i.e. S = [Si; :bi:...Sppp] and x(" is the subvector of (m) with only the relovant dimensions. We draw the remaining dimensions (n) from a background distribution (1 - is the binary vector which for irrelevant dimensions):
(m)) =j Bern(n:), (()) =j ~ Ni ).),
(m) x N("", " ΣT) Vn E NJ.
(13)
Note that chose specific distributions for X and Y for the sake of clarity but we could have chosen any exponential family distribution.

---

(n) from hidden Markov model with K Gaussian Published in Transactions on Machine Learning Research (05/2024)
The joint distribution for pf-GMM, FPIGMM (X, Y, Z, is:
FDRMM Y. (Y, Z.
II f(Sa)
(1) (X Il f( n=
(m) (2() (2))) (((())))))))))))))) ((n))
(14)
Switch variables select relevant dimensions by tuning its prior p The switch variables Sa are the key to our model. They allow to model the 'irrelevant' dimensions X using background distribution, and to jointly model the 'relevant' dimensions and target (Xs, Y) using mixture distribution. Furthermore, our models depend on only one hyperparancter-toe switch prior, p. As we will see in Section 6, the switch prior can be luned to trade off generative quality with predictive performance. This trading-off ability allows our model to achieve good downstream performance even when they are (inevitably) misspecified with respect to the gettrrteng process.
4.2 Prediction-focused Hidden Markov Model (pf-HMM)
The generative process for our pf-HMM model as follows. In addition to the switch variables Sa, we draw the state sequence variable 7(") for each data point and time step l:
Sa (ern(p)
Vn (1, (1....,N)) N). z (a)) (n) jj Cat(A,) Vn E (I..., 2,
Yd (1, .., D},
z(m) Cat(@)
where p is the prior probability that dimension is relevant, is the probability distribution of the first state, and Aj is the probability distribution of the next state given the current state j.
For the set of relevant dimensions, we draw the data Sy components. For the set of irrelevant dimensions, draw the data x (n) distribution.
(1-s), from background Gaussian
1 (1) (n) = ~ Bern(1j).
x =j- -j~NNin -j~N(u5 -3),
x ~N("", "0p,,"" E*), Vn,
(15)
The joint distribution for pf-HMM, fTTHM (X, Y, Z) is:
D
N
T
) (firx (1 (((!"
FрниM X, Y,Z,S)
I
((S))
II a=1
f(Z("))
II s(Z) (2-))
(( 8.t
((()))
(16)
The density fAMMMM(X, Y, S) is similar in form to the Pf-GMM case (Equation 14). Both models use a complex structure to explain the relevant dimensions and the target while using a background distribution explain the irrelevant dimensions. Furthermore, both models use a switch variable Sa select relevant dimensions. The main difference for pf-HMM is that a Markov chain of variables {2, (x) ]T= has replaced the single variable (n) to generalize the mixture model to the time-series setting.
5 Inference
Since Df-GMM and pf-HMM are valid graphical models, we can use efficient inference algorithms for these models instcad of resorting generic optimization methods. The likelihood of Pf-GMM is:
L( Σ fNRMMM (X. Y,Z.S,)
(17)
Z.S
This objective intractable to compute for high dimensions because it scales exponentially with the di- mensionality of X. Therefore, in practice, we Variational Inference (VT) to optimize lower bound
8

---

Published in Transactions on Machine Learning Research (05/2024)
(the ELBO) to the intractable log-likelhhood. We choose the following form for the approximate posterior, ((Z, s)),
N II n=
D g q (1) X Y pp b) II (( (pa)
(Z. p)
(18)
where O refers to the model parameters. Note that we have added an index to the switch variable, i.e. s(m) is a function of the datum as well the dimension d. This allows the posterior to factorize over the data. To avoid learning separate switch posterior for each n, we tie their parameters across obtain ((S(r) (pa). Therelore we still get per-dimension relevance.
The distribution ((Z()) ((()) sp ①) is simply the posterior distribution conditioned on the variational parameters and can be computed as:
q k|X,Y
k) (Y)
At test time, the distribution g(Z - x p) is computed without the Px(Y) term but the feature selection porameter sp helps select the right. features in the term (x)(
This results in the following ELBO objective (full derivation in Supplement Section 1):
ELBO(g)=
E [log fXX, Y| 0)] - KL (o (l0, <(Z.S)
(19)
The model is trained using the variational EM algorithm which alternates between learning the parameters θ (M-step) and inferring the approximate posterior qZZ (|O,) (E-step). We also try Gibbs sampling for the E-step since the variables Z and S are conditionally conjugate, We provide the conditional distributions for Gibbs sampling in the Supplement Section B.3.
Similarly, we train pf-HMM by variational EM using the following factorization for gZZ
NV
D Ta (S f. Pa)) f( x()), y(1), ,)
q (Z) = II IIII
d=1t=l
where subscript / indexes time within a sequence, and (a) (1) ( are variables for each sequence n. We use the Backwwd algorithm (Bishop, 2006) to infer f(Z (n) ((n) ( sp, θ) during the E-step (see Supplement Scction B for details).
6 Analysis of Correctness of Prediction-focusee Mixtures
In this section, we show how the switch variable in the pf-GMM is able trade off generative and dis- criminative performance when faced with misspecification in the number of components. We explore the limitations of GMMs in such setting, even with supervision (i.e. for sup-GMM), Importantly, propose analytical expressions to determine how pf-GMM identifics relevant dimensions in various data scenarios, and conclude the section with an empirical evaluation of the expressions.
6.1 What is enabling the generative-discriiinen
trade-off?
The cuneraaiveediscrimnnatt trade-off can either be achieved by the choice of model and objective, or by the choice of inference algorithm. Ren et al. (2020) motivated using switch variables to distinguish relevant and irrelevant dimensions by providing a lower bound to the marginal log-likelihood objective which exhibits an explicit generative-diseriinnnaie trade-off:
log [XX, Y) 2
E log (XX[)) + (1- p) log ((X;)) + E [log ((Y f(2)
(20)
((S)
where p is the switch prior. This suggests that the modeling choice of using switch variables is what enables the generaiivedisccimnnatve trade-off. However, this bound is not optimized during training (it is

---

Published in Transactions on Machine Learning Research (05/2024)
computationally expensive to compute), and is mainly used to justify the use of switch variables. In fact, their trade-off is achieved by the specific inference algorithm they use (variational inference with a specific posterior approximation). Furthermore, favorable properties of a lower bound do not guarantee favorable properties of the model and the objective.
In this section, we demonstrate analytically that the true pf-GMM likelihood can balance generation and prediction in certain data regimes through the switch prior p. This result implies that the pf-GMM model itself is able to trade off generation and prediction, providing theoretical justification for the use of switch variables. Furthermore, the desired trade-off. We note that Hughes et al. (2017) also achieve such a trade-off through their proposed objective, but their objective is hard to optimize. We demonstrate that our objective is easier to optimize than theirs in the experiments section.
precisely characterize the circumstances in which the user can adjust to achieve
A: Define Data Regimes
B: Component Types
Alggned Companents
Number Relevant Dimansions, Dg. Number of irreevant Dimensions, Di. SNR for Relevant Mixiure, An for Inrelevant Mixture, Ag
Fix
+ An
Single Companent
Unaligned Companents
Dn
Vary
An/ An/An.Da
An/An: Dataset Space (2-Dims)
Define Dataset Features
Midoren Ooree 1200 De)
Aligned
Deaeceet (laoe viare (ng)
Lagg inwllrued of componenis
-15 200
-25
Not aligned
Inspect one dataret
10:1
10:1
101
10
Redacioo SAR (Agl
Robativn SMR (ookk)
GMM
GMM
pf-GMM
Ph. A. RC Swith Pia.p
LL for A Alinndd
L1 for Unailgnad
Allgned componerss Oyellow regian) regines
11 for Singl Componerr
C: Dataset Regimes with Aligned Components
D: Effect of switch prior, p for Pf-GMM
Figure (A) We create representative datasets by varying the number (DR, Drr) and the signal-to- ratio (SNR; AR, AIr) for relevant and irrelevant dimensions. We fix the total dimensions (D DR + Die) and the SNR for relevant dimensions; the resulting datasets are evaluated by each model to get the figures in panel C. (B) There are three local optima to the Objective 25 in a two-dimensional feature space: aligned components with label Y (left), unaligned components (middle), and just single component (right). (C) The yellow region in each plot corresponds to the datasets for which the model learns aligned components, and the blue region corresponds to the datasets for which the model learns unaligned components. From left to right, there is an increase in the space of datasets for which aligned components can be selected (i.e. GMM< sup-GMM< pf-GMM).
(D) For any dataset, we see how pf-GMM can achieve aligned components by tuning p. Setting p to value in (PI.G/atPuja) (yellow region) learns aligned components, whereas setting to a value in (0,P/-G//) U (Pu/a: 1) (blue region) learns unaligned components or a single component.
6.2 Defining Pertinent Data Regimes
Consider data formed by concatenating multiple independent mixture distributions, one of which is relevant; the rest are irrelevant. The possible ways in which cach mixture distribution (relevant or irrelevant) can vary are (a) separation between the components, (b) variance of components, and (c) relative counts of relevant and irrelevant dimensions in X. Finally, the number of independent irrelevant mixtures can also vary. We first consider the case where the relevant dimensions come from one mixture, and the irrelevant dimensions come from another mixture (see Figure 3(A)). This case generalizes to the analysis for multiple irrelevant mixtures since it highlights the trade-offs the model must make between relevant and irrelevant mixtures.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Consider imput data X == [ X1r] E Ph with DR relevant and Dr. irrelevant dimensions, and binary output label Y. Both the relevant dimensions Xn and irrelevant dimensions Xtr are distributed equal mixtures of two Gaussians. The data- data-generating process is:
ZR Bern(0.5), 7 Bern((.5),
(21)
XR ZR. NO oR Da) + (11-Ap) ZR) -N(HR Du of Da):
(22)
X1 ~ Zr N(OD or ID.) + (1 Z1) N( 1p, ofe IDn);
(23)
Y ZR
(24)
where Zn are the cluster identities for the relevant mixture and Zi. are the cluster identities for the irrelevant mixture. Note that the output label Y is perfectly correlated with the relevant mixture's cluster identity ZR. Finally, let A == /=/ /o" as the signal-to-noise ratio (SNR) for both relevant and irrelevant mixtures (i.e ÅR, Are.. Note: Our analysis carries over to full-covariance Gaussians; the expressions are simply more involved. We discuss the isotropic case here to convey the core idcas and provide the analogous derivations for the full-covariance case in Supplement Section C.
6.3 Formalizing the Question
What are the datasets (i.e. values of AR, Ar. DR. D1.) for which misspecified model can still select the relevant components?
We answer this question by characterizing the population log-likelihood objective under the model density foodel O) and data density (X(,Y):
m(0mel))) [log fmodel (X, Y:
(25)
for three models: GMM, sup-GMM, and our pf-GMM. The GMM learns a Gaussian mixture model on alone, whereas the sup-GMM is the supervised variant that also uses Y. Upon optimization, the likelihood objective 25 can result in three locally optimal parameters. As illustrated in Figure 3(B), these parameters correspond to:
1. components that model relevant data (XR, Y) well (0 or aligned parameters).
2. components that model irrelevant data (Xt,) well (Q" unaligned parameters).
(for pf-GMM only:) a single component (e (G); this case arises the pf-GMM considers all dimen- sions irrelevant.
Among these, the aligned components have the highest predictive accuracy because they correlated with Y.
6.4 Results for the Analysis
6.4.1 Likelihood Gaps for GMM, sup-GMM, and pf-GMM
The likelihood gaps for GMM and sup-GMM depend on the data features (AR, An. DR, Dı:):
A0O 0%MM) = QR Qi
(26)
O SupCMM- O% (KPGMM) = log 2 + QR Q1e
(27)
where QR is shorthand for the following term that only depends on DR and AR (and similarly for Qt.):
1 DR - A2)
QR 2 DRA2 +1
+ loogog(DDD (DRA3 1)
1
Drr(1 -
Q 2
log D DI.
D +1
11

---

Published in Transactions on Machine Learning Research (05/2024)
For pf-GMM, the gaps also depend on the switch prior p:
A(6 A PrGMM: PRAMA)) (DR D1:) log
P + log 2
QR
Q1
(28)
1 1-p
A P PPGGMM) = DR log
P
+ log + QR
(29)
1-p
When can a misspecified model select relevant components? The models will select aligned compo- nents (over unaligned) when the likelihood gap moodel Qw modul) the gap PoOMMM OL-G PFGMM) to be positive, that is the likelihood associated with the aligned components must be higher than that of collapsing the model to a single Gaussian.
positive. For our pf-GMM, we also need
6.4.2 Tuning PM-GMM prefer relevant components
For pf-GMM, tune the switch prior to prefer relevant components in Equations 28 and 29, We perform this selection of relevant components by identifying values of for which A( PIGGMM + PIGMM) > 0 O1-G
and A(O PTMMM PFGMM) 0. Doing so gives us the following constraints on p:
log + QR Qir
p < Pufa
where
Pufe
if Dn Dir
(30)
DR D1r
log + QR Qii
p > Pufa
where
Pu/o
Dn. Dirl
if DR Drr
(31)
log 2 + QR
p> P1-G/a
where
Pi-G/e =
(32)
DR
where ((x) 1+exp(-z) is the sigmoid function. The feasible range of exists when Pu/a > PI-G/a- which is
when:
D1. log 2 DRQ + DI.Q 0 DR Dr.
6.5 Empirical Illustrations of Analytical Results
We empirically illustrate when these gaps can be positive by setting DR + Dr = 8 and AR = 5 as an example in Figure 3CC. For each model, we vary Ar and Drr to create datasets with different SNR and a mumber of irrelevant dimensions. The yellow region in each plot corresponds to the datasets (i.e. values of (Airs Dıe)) for which the model learns aligned components as per Equations 26-29. The blue region corresponds to the datasets for which the model fails to learn aligned components. We see that the range of datasets for the aligned parameters (i.e. the yellow area) is largest for pf- 2f-MMM followed by sup-GMML and smallest for GMM.
Furthermore, in Figure 3(D), we illustrate how our pf-GMM model selects the aligned components by luning p. By setting p to a value in (P1-G/e: Pufa), we learn aligned components (corresponding to the yellow region in the figure), whereas by setting to a value in (0, P1-G/a) U( 1), we either learn unaligned components or just single component (corresponding to the blue region in the figure). This figure provides us with guidance performing hyperparameter tuning for p: to learn aligned components, decrease p if currently learned unaligned components, and increase if we learned single component.
6.6 Conclusions from the Analysis
The pf-GMM allows us to identify relevant structure by changing To choose the aligned parameters, the gaps in Equations 26- 29 must be positive. Comparing Equation 26 with 27 shows supervision is useful over just using the GMM-the gap for sup-GMM log larger than the gap for GMM. However, the pf- GMM does better: its gaps (Equations 28 and 29) also include a term involving p, allowing us tune p so that the gaps positive.
12

---

Published in Transactions on Machine Learning Research (05/2024)
When the relevant dimensions are outnumbered by irrelevant dimensions, learning the unaligned compo- nents would be preferred by GMM (and sup-GMM after extent) since doing so increases the generative performance of the model. However, maximizing predictive performance would require learning the aligned components. Sctting p to a value in (PI-G/a: Pu/a) allows us to select aligned components, and setting it in (Pu/as] 1) allows us to select unaligned components (Equations 30 and 32). In this manner, we can trade off between predictive and generative performance: the aligned parameters achieve predictive performance while the unaligned parameters achieve generative performance and p lets us decide which ones to choose.
pf-GMM identifics predictive mixtures even with many irrelevant dimensions. Suppose we fix means PR Pir and variances , oir (i.e,, fix År and Ar.) For the interesting case of fewer relevant dimensions DR < Drr lowering p below Pu/a forces the aligned parameters to be preferred over the unaligned parameters. The other models cannot achieve this: the gap A(0 (xna) is negative for the GMM and only increases by log 2 for the sup- sup-GMM (Equations 26, 27). However, there are limits to pf-GMM, too: if we lower p below P1-G/a when DR Dir pf-GMM, prefers the single Gaussian solution (as seen in Figure (())..
Gf-GMM is more robust to high irrelevant SNR. We vary the irrelevant SNR ALr by varying the gap between irrelevant component (Hit), while fixing the means of the relevant components (#R) and all variances (oR, of The case of o >> of is analogous. For all models, the likelihood gap decreases on increasing Ar, (see Equations 26-29). However, pf-GMM selects aligned components for a larger range of Air. For the interesting case of DR DIc, we can see this by setting p < Pa/a in Equation 30 doing so adds positive bias to the gaps of GMM and sup-GMM, making it casier to select aligned components. Nonetheless, if Air is too high, decreasing p no longer works because Pu/e < P--G/a we go directly from the unaligned solution O" to the single Gaussian solution IIGG bypassing the predictive solution O
This case highlights the properties and limilations of all three models: our pf-GMM has the widest. range of datasets for which it will still select the desired aligned parameters, but there will be level of gegal-to-noise ratio at which all models will fail.
Cases of DR > Drr and DR Dir- DR > Drr is the (easy) regime where GMM (and therefore sup-GMM) work as desired and introducing pf-GMM is not necessary. Finally, for DR Die the effect of p disappears irrespective of the parameters. This case is an unlikely technicality, which we rarely expect in real datasets.
Effect of the number of irrelevant mixtures. When multiple irrelevant mixtures are present, we only need to focus on the irrelevant mixture that yiclds model parameters (with model components aligned with mixture components) with the highest population log-likelihood, I(0). Then the same analysis applies.
7 Experimental Details
In this section, we empirically demonstrate that the Pf-GMM successfully identifies relevant input signal, regardless of whether the number of model components misspecified or not. We show that the switch parameters and their prior p can recover the relevant dimensions and maintain predictive performance even under misspecification. Specifically, we address the following questions:
Q1: Can prediction-focused models perform well when we misspecify the number of components?
Q2: Can prediction-focused models perform well as the number of irrelevant dimensions increases?
Q3: Do prediction-focused models select relevant dimensions, even when irrelevant dimensions are not fully independent of the signal?
Q4: Is the optimization stable for these models?
For both GM- and HMM we constrain the models to have diagonal covariance for faster inference; analogous results for the full-covariance case in Supplement. Section 3.2.
13

---

Published in Transactions on Machine Learning Research (05/2024)
A
B
C
Disc-Gen
Landscape as K varies 200
log P(Xnw) as K varies (0nr=20)
Predictive Performance as D varies (K=3)
Disc. Term. log P(Y)))
0.00
-30
1.0
-0.25
lagPiXaal
-40
AUROC
0.8
-0.75
0.6
-1.00
-60
-600
-400
-20
20
30
40
50
Gen. Term, log PIX)
Drel
D
E
Inferred Switches
F
AUROC as Correlation b/w & Irrel.
Runtimes (D=100)
(Dre 20, Km3)
Mixture Varies (Drer=0,, 20, K=
Runtime (s. Log Scale)
10a
Posterior Probabilig
1.0
pf-GMM (This Method)
Pc-GMM
102
Y AUROC
sup- Sup-GMM
0.8
2-Step
10a
Sparse Clustering
0.6
Random Forest
109
Class-specific GMM
2
1 20
100
0.00
0.25
0.50 0.75 1.00
Pf-GMM (Pred. Term Only)
Index
Correlation b/w & Irrel. Mixture
Figure 4: Plots for synthetic data (Section 7). (A) pf-GMM an order of magnitude faster than pc- GMM, with respect to the average runtimes (log scale) per run. (B, E) pf-GMM trades discriminative and generative objectives better- takes small hit in generative quality to achieve discriminative quality competitive with pf-GMM trained only on the predictive term. The benefits are largest when misspecification is larger (fatter markers). (C) Constrained pf- pf-GMM achieves optimal predictive performance by identifying the relevant dimensions (random forest baseline provides the upper bound). f(S, = 1 Y) for the switch variables correctly identify the first 20 dimensions as relevant.
(D) Posterior probabilities
Table 1: Comparison of discriminative and generative metrics for the synthetic (complex) dataset shows that pf-GMM, outperforms other basclines.
Model
log P
log P(X)
log P(Xn)
Y AUROC
Pf-GMM
-0.05
105.665
7.38
0.99
pc-GMM
-0.50
-130.41
-15.51
0.66
sup-GMM
-0.66
-30.81
-15.48
0.58
2-Step
-0.70
5.79
-26.23
0.52
Class-specific GMM
-5.92
14.87
-20.66
0.73
7.1 Baselines
We compare the generative and predictive performance of pf-GMM to (1) a two-step approach that learns a GMM and trains a logistic regression classifier on the posteriors f (2-Step-GMMD), (2) supervised GMM trained jointly on X, Y (sup-GMM), (3) predictlion-constraineed GMM (Hughes et al., 2017) (pc-GMM) and (4) sparse K-Means (Witten & Tibshirani) followed by logistic regression on cluster values (sparse clustering). In Supplement Section F, we also discuss class-specific GMM baseline that learns different GMM for cach class value and shows that it is qualitatively similar to GMM baseline. We also provide random forest baseline a prediction upper bound. We discuss the utility of these baselines in depth in the Supplement Section 4 For pf-HMM, comparable approaches are sup-HMM and 2-MM
7.2 Datasets
Synthetic We generate two synthetic datasets, where the first dataset simple and the second dataset is complex. For the first dataset, we generate the relevant and irrelevant dimensions of X from independent
14

---

Published in Transactions on Machine Learning Research (05/2024)
A
B
Syrthetic (Simple)
Synthetic (Complex)
Swiss Notes
0000
-2.5
0.0
2.5
75
10.0
12.5
15.0
-1.5
-1.0
5
0.
0.5
1.5
1.5
-25
-2.0
-1.
-10
-05
0.0
0.5
3.6
1.5
X001 (tclovad))
K19] IRC evar)
2.5
15.0
0.0
0.0
1.0
Duta Dersit
deniiit
Class spectic dcraity
0.0
6.2
Moan value data hittgraam bit
Figure 5: Densities of relevant (top) and irrelevant (bottom) dimensions of inputs for Synthetic (left and center) and Swiss Bank Notes datasets. The Synthetic (complex) in the middle column compler because all the classes have multimodal densities. By this definition, both Synthetie (simple) and Swiss Bank Notes are considered simple. We also see that the pf-GMM, corresponds to the pf-GMM, model density) in all the dalasets. In contrast, class-specific GMM (olive line) prioritizes modeling irrelevant dimensions' data density (bottom rows) and is only able to capture relevant dimensions' data density (top row) when the datasets are simple.
model the relevant dimensions well (red line
A
GMM Performance on HIV Data
GMM Performance on Swiss Bank Notes Data
1.0
AUROO
YURRO
0.8
0.6-
CMM
0.5
Hxp-CMM
2-5hpp
0.4
0.4
Number of Componants, K
Numbar of Components, K
C
HMM Performance on Data
D Performance on Synthetic Data (Drar
0.9
0.7
YAUROC
0.8
0.6
0.7
0.5
0.6
pRINN (Our Melhod)
Sup-HNM
2-Step
3
6
9
10
Number of Components,
K
Number of Components, K
Figure Target AUROC performance as we vary the component budget K in cach of the domains for GMMs (Top Row) and HMMs (Bottom Row). pf-GMMs and pf-HMMs outperform methods that do not trade off prediction and generation for different choices of K. Synthetic data for the HMM models has 2 relevant dimensions (out of 20 total).
3 3-component mixtures of Gaussians. The target Y is binary, with the assignment only depending on the cluster identities in the relevant GMM. The dataset is simple in the sense that the relevant dimension densities for each class are sufficiently unimodal: there only one mode for the class Y 1 and most of (sce the histograms in the top left panel of Figure
the mass concentrated in one mode for class for visual representation). In the second dataset, we generate the relevant and irrelevant dimensions from independent. 4 comppoeett mixtures, and again the target is binary and depends the relevant
15

---

Published in Transactions on Machine Learning Research (05/2024)
GMM's cluster identities. By the same definition of complexity, the second dataset is complex because the relevant dimension densities are multimodal for each class (see the top center panel of Figure 5). The total dimensionality of each observation is fixed at 100. To generate time-series data, we use independent 4-state HMMs with 20 dimensions. The emission distribution f (x Z = k) given component & is N(: 6k 1. for relevant and irrelevant dimensions. The larget Y is binary and depends on the relevant HMM (details in Supplement Scction 4). We vary the number of model components (K) and the ratio of relevant dimensions for different levels of misspecification. For Q3, we also simulate irrelevant dimensions that are not completely independent of Y; for this experiment, we vary the correlation between Y and the cluster identity.
Swiss Bank Notes The Swiss bank notes dataset (available in the MCLUST library in R (Serueca et al., 2016)) contains 6 features about Swiss 1000-franc banknotes: length of bill, widths of left, right, top, bottom edges, and length of diagonal. The dataset contains 100 counterfeit and 100 genuine notes. We want the clusters to be predictive of whether a banknote is genuine or counterfeit (label Y). However, not all features are relevant to the task of predicting this label. Therefore, prediction-focused modeling is appropriate for this situation. To further demonstrate the robustness of pf-GMM to noisy features, we augment the existing features with 30 irrelevant features generated using a 2- comppeentt mixture of Gaussians with components W(-3, I) and (3,3 I) to make prediction harder, and finally normalize the dataset. For a method to do well, it must. identify clusters that are predictive of the genuineness of the notes even when the cluster count is small (e.g. when only 2 clusters are allowed).
HIV Therapy for HIV involves administering cocktails of antiretroviral treatments to bring the viral load below detection limits (S 40 copies/ml), These antiretroviral treatments belong to five classes: Non- nucleoside Reverse Transcriptase Inhibitors (unRTIs), Nucleoside Reverse Transcriptase Inhibitors (nRTIs), Protease Inhibitors (PIs), Fusion Inhibitors (FIs), and Integrase Inhibitors (IIs). Our task is to predict whether a treatment will bring the viral load below detection limits in the next time-step, or whether it will increase the viral load (Y). Each input observation (X) contains 267 features about the patient, including their CD4+to their genetic mutations, their treatments in terms of drug classes, and their lab results. We study 53, 236 patients with HIV from the EuResist Integrated Database. Each person has a time series of average length of 16 steps where time step is approximately 4 months between consccutive treatments. Though it is common to have many genetic mutations, only few of these may be relevant for inducing drug resistance thus increasing the viral load. Therefore, this problem is perfect for inddiction-fcusse modeling.
We compare both HMM and GMM modcls, ignoring the time dependencics for the latter case. Further details about the experiments are provided in Supplement Section D.
Psychiatric Medical Records We analyzed dataset of electronic health records from two large academic medical centers and their affiliated community hospitals and outpatient clinics. We selected cohort of 16,653 patients with at least one diagnosis of major depressive disorder ICD-10 diagnosis code during psychiatric care between 2017 and 2022. We remapped the ICD-10 codes to 423 CCSR codes (i.e. features). We mapped each feature with a value of 1 if the diagnosis code was observed, and 0 otherwise. An indopendent standard normal observation noise with a standard deviation of 0.1 was added. After excluding codes with high sparsity (2 90% patients with the code) and codes that were highly correlated with another code, we were left with 106 features for a patient. Our prediction task is to ascertain a substance- use disorder outcome for the patient, and the goal of the clustering step is to discover subgroups relevant to the prediction task. This task would benefit from prediction procuee modeling because, without the prediction task, the clusters would correspond to common comorbidities of MDD. However, we wish to discover clusters of substancc--ss--roltte comorbidities.
7.3 Evaluation
We measure the predictive performance of the models by log P(Y[X) and area under the receiver operating characteristic (AUROC) for the classification of Y on heldout test data. We measure generative performance by log P(X). For synthetic datasots, we know which dimensions are relevant, so we can also mcasure the generative performance of the models on only the relevant dimension, log P(XR). Since we do not have this metric in practice, we use Y AUROC for model selection. Using Y AUROC works well in practice because we
16

---

Published in Transactions on Machine Learning Research (05/2024)
Aicahal-related disorders
Prevalence of Codes per Cluster (Median IQR)
Method
AUROC (M ( d) Time (8)
3.5-
3.0-
pf-GMM
0.75 0.01
293
2.5-
GMM
0.60 0.00
Cluster
pe-GMM
0.75 0.09
6828
sup-GMM
0.62 1 0.01
278
1.5-
2
Class-spocific
0.74 1: 0.01
13
GMM
0.5-
Sparse Clustering
0.54 0.02
1218
Code Prevalence
RandomForest
0.85 0.00
Other specified and unspecilied meed disorders
3.5-
Outcome Prevalence Cluster
(Oracle)
PfGMM Top 8 features
2.5-
Cluster
Other specified and unspecified mood disorders Alcohol-related disorders
1.5-
Tobacco-related disorders
1.0-
Other specified and unspecified liver discase
Trauma- and stressor-related disorders
0.00
0,25
0.50
0.75
1.00
Superficial injury; contusion, initial encounter
0.0
Outcome Prevalence
Depressive disorders
Tobacco related disarders Other specifeed and unspecifle... Alcohol-related disorders
Auxicty and fear-related disorders External cause codes: intent of injury, acciden- tal/unintentional
DAGNN Derstyy
Class- speil-pcificcc GNN Dansiy
Dutcome Prevalence
0.2 0.6
0.6
1.0
Per cluster Outcome Prevalence
Data Dansity: Outcome Prevalence
Figure 7: Results for psychiatric medical records dataset. Top Left: pf-GMM learns clusters that are predictive of the substance use disorder outcome, as scen by a high Y AUROC valuc. Other clustering methods either produce clusters unrelated to the outcome (2-StepGMMM, sup-GMM, Sparse Clustering) or are very slow to run and have higher variation (pc-GMM). Bottom Left: The top features that pf- GMM clusters on are indeed related to substance use disorder. Center: The class-specific GMM does not model relevant dimension (alcohol and mood disorders) density despite having a high AUROC, whereas the pf-GMM, correctly captures the density. Right: The pf-GMM clusters show significant variation in the outcome and the relevant feature values, providing evidence that the clusters are both predictive and relevant.
can achieve good predictive performance with the clusters only if the relevant dimensions are well modeled (the class-specific GMM can create false positives with this metric because it has more parameters, as we discuss later).
Model selection is done using a heldout validation set. For Swiss Bank Notes Data, we use 3-fold Stratified Cross-validation because the observations are too few have separate validation set.
8 Results and Discussion
Prediction-focused models balance the discriminative and generative objectives effectively. Figure 4 (A) compares pf-GMM to competing approaches on the aberriinaative-eeeeaatvy landscape. A higher number of irrelevant dimensions or a lower number of components are exactly the settings which the tradc-off of discriminative and generative objectives critical and pf-GMM manages to achieve superior predictive performance in these siluations. In contrast, sup-GMM and 2-Step-GMM are only useful when the number of relevant dimensions (the number of components respectively) is increased predicted by our analysis in Section 6. The class-specific GMM fits independent GMMs on smaller subsets of the data, causing it to overfit its log f(Y[X) density to the training set. Finally, directly optimizing for log (Y (orange) gives up a lot on generative performance.
Prediction-focused models identify paramcters that perform significantly better at the down- stream task. Prediction-foeused models identify components that align with the relevant dimensions (Figure top left). This helps them outperform their non-prediction-focused counterparts predictive
17

---

Published in Transactions on Machine Learning Research (05/2024)
performance on several real and synthetic datasets (Figures 4, 6, and 7). While sup-GMM also maximizes the joint likelihood of inputs and the target, it treats the larget as just another input dimension. In contrast, prediction- focusse learning identifies the asymmetry in the problem, and it emphasizes dimensions that are predictive of the target. The class-specific GMM can achieve good predictive performance, suggesting it also identifies the relevant dimensions. However, we see in Figure that aligns its components with the irrel- evant dimensions (bottom row), and can not align with relevant dimensions when the datasets are complex (top row, center).
Prediction-focuseee models seek out the most correlated dimensions first. As we show in Figure 4(C and F), our constrained pf-GMM achieves optimal predictive performance (random forest bascline provides the upper bound). Ignoring irrelevant dimensions can be easy if the irrelevant dimensions are independent of the relevant dimensions (i.e. correlation to Y is zero). But the pf-GMM seeks out relevant dimensions even when the irrelevant dimensions 'distract' the model by being weakly correlated to Y, as seen Figure 4(F) The rest of the models do a poor job at this: their performance is proportional to how correlated the irrelevant dimensions are to Y. A correlation of one is trivial because the notion of an 'irrelevant' dimension disappears.
Prediction-focusee models are faster to compute and they are casier to tune and optimize. In Figures 4DD and 7(top left), we see that pf-GMM is orders of magnitude faster than the Pc-GMM the only other baseline that can trade off generative and discriminative objectives. Moreover, pc-GMM's performance is unstable and it requires more independent trials to run, as can be seen from the large error bars corresponding to percenlee intervals (Figure (CC) and 6(A)). The switch parameter p the only hyperparameter that must be tuned. In contrast, while pe-GMM also has one hyperparameter x-used for trading off generative with predictive performance the model is still considerably harder to optimize. Training requires a grid search over both learning rates and X, and it takes much longer to converge than the EM-based methods. This makes it prohibitively expensive to apply to HMMs.
Prediction-focused model learns an accurate density of the relevant dimensions even for com- plex data distributions. In Figures 4(B) and 5(top row); we see that pf-GMM, learns a better density for the relevant dimensions even when the model is constrained (i.e., for small values of K), as measured by log P(Xg). In contrast, the class-specifee GMM learns poor density of the data even when it does well at predieting the larget Y. This phenomenon is obscured on simple data that does not have a multimodal data density in X for the values of output class Y. This is because the class-specific GMM can use its components to model the irrelevant dimension density but still do well on the prediction task and model relevant dimension density well if the data set is simple. However, for a complex dataset with multimodal relevant dimension density per class, this is no longer possible and the class-conditional baseline fails. We discuss this further in the Supplement Section F.
Prediction-focused models identify relevant dimensions in both misspecified and high-noise settings. In Figure 6, we see that prediction-focused models maintain high (UROC-indicating that the components discovered are predictive of the downstream task. In particular, we see that pf- GMM maintains good performance even when K is small. The model also correctly identifies that the first 20 dimensions contribute to clustering. In contrast, the baselines do not offer ways of inspecting the models, with the exception of Sparse Clustering. Both small K and feature attribution in the misspecified sctting are critical in our motivating example of disease subtyping.
Conclusions from HIV dataset In HIV data (Figure 6, left column), for patients whose treatment is predicted to fail by pf-GMM and pf-HMM, we observe that several resistance-relevant mutations are identified as 'relevant'. For example, for patients taking nRTIs experiencing virologic failure, prodiction- focused models predominantly identify certain mutations such as K65R, I116V, and M184V as relevant. These are consistent with Wensing et al. (2019). Patients who are more likely to achieve treatment success tend to be younger female patients or those with mutations consistent with subtypes that are known to be casier to treat e.g. subtype B. This is consistent with some bodics of work that have examined the success of mreatment varies across diffeernnt HIV subtypes (e.g. Gatell (2011))
18

---

Published in Transactions on Machine Learning Research (05/2024)
Conclusions from Psychiatric Medical Records dataset From the relevant features table in Figure 7, we see that pf-GMM correctly identifies features relevant to substance use disorder (SUD) outcome. The model correctly identifies that the presence of diagnosis codes related to alcohol and tobacco disorders means that the patient has SUD. It also identifics that a history of trauma, mood disorders, depression, and liver disease can be indicative of SUD prevalence. These discovered features are consistent with the comorbidities of SUD the literature (NIDA, 2020). On comparing the pf-GMM clusters on code prevalence (top right panel in Figure 7), we observe that codes for mood, tobacco, and alcohol-related disorders vary significantly between clusters. This variation explains the difference in outcome prevalence (bottom right panel). Cluster 0 consists of patients with high prevalence of all three codes and high outcome prevalence. Cluster 5 is another pure cluster, consisting of patients with mood, tobacco, and alcohol-related disorders. Cluster 3 only consists of patients with mood disorders but no lobacco or alcohol use, suggesting that these patients used other substances since the outcome prevalence still high. However, mood disorders are not a necessary condition for high SUD: cluster 4 consists of patients with SUD but without diagnosis of mood disorders. pf- GMM allowed to recover these clusters despite other significant (but irrelevant) clusters in the population. The other model-based clustering models did not recover a meaningful clustering along these features and did not identify the features they focused on.
9 Conclusion
We developed prediction--ocusee generalive modeling, a model-based feature selection method for mixture models and hidden Markov models. More importantly, we developed an understanding about when this type of approach to feature selection will work and when it will fail. Prodiction-focused modeling selects the 'right' features to cluster even when the inputs are corrupted with several independent irrelevant features. It achieves this trade-off by maximizing the likelihood of the data directly, instead of resorting to any property of modified objective as proposed by Hughes et al. (2018a) or of the approximate posterior as done by Ren et al. (2020). Our main assumption is in the types of correlations we assume for the relevant and irrelevant variables (i.e. mixture distribution and hidden Markov distribution). In our analysis, we show that centictioo-focusee models envelop supervised mixture models in the data space they can model in the constrained- constraineed-oomponentt setting. Finally, we validate our findings on both synthetic and real-world datasets, including medical dataset with large amount of noise.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. IIS- 1750358, Grant No. IIS-2007076 and by NIH award R01MH123804. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the official views of the National Science Foundation or the National Institutes of Health.
References
Ali El Attar, Rida Khatoun, and Mare Lemercier. Gaussian mixture model for dynamic detection of abnormal behavior smartphone applications. In 2014 Global Information Infrustructure and Networking Symposium (GIIS), pp. 1 1-6, September 2014. doi: 1..109/GGS..014.6666
Christopher M. Bishop. Pattern Recognation and Machine Learning. Springer, 2006.
Chaitanya Chemudugunta, Padhraic Smyth, and Mark Steyvers. Modeling general and specific aspects of documents with probabilistic topic model. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007. URL https://proceedding pdf.
Irene Y. Chen, Shalmali Joshi, Marzych Ghassomi, and Rajesh Ranganath. Probabilistic Machine Learn- ing for Healthcare. Annual Review of Biomedical Data Science, 4(1):393-415, 2021. doi: 10.1146/ amnueex-bocatascc:0099000000000
19

---

Published in Transactions on Machine Learning Research (05/2024)
Tianqi Chen, Sameer Singh, Ben Taskar, and Carlos Guestrin. Efficient second-order gradient boosting for conditional random fields. In Artificial Intelligence and Stalistics, pp. 147-155. PMLR, 2015.
Adam D. Cobb, Stephen Roberts, and Yarin Gal. Loss-calibrated approximate inference in bayesian neural networks. arXiu:1805.00 039001 fes, statj, May 2018. URL http: sttp://arxiv..rr/abb//laat 03901.
arXiv: 1805.03901.
Manoranjan Dash and Huan Liu. Feature selection for clustering. In Takao Terano, Huan Liu, and Arbee L. P. Chen (eds.), Knowleddge Discovery and Dala Mining. Current Issues and New Applications, pp. 110-121, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-45371-4.
C.F. Fick, N. Zeidat, and 7. Zhao. Supervised clustering algorithms and benefits. Tn 16th TEEE International Conference Tools with Artificial Intelligence, pp. 774 776, 2004. doi:
Chris Fralcy and Adrian E Raftery. Model-based clustering, discriminant analysis, and density estimation.
Journal of the American Statistical Association, 97(458):611 631, 2002. doi:
URL https://dee org/10 wttgg//...////11111e 11988006221400770447713
Joseph Futoma, Michael C. Hughes, and Finale Velei-Velez. Popcorn: Partially observed prediction con- strained reinforcement learning. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020. URL attps://art http::///xxxvvvoooee wwtppdf/g//d//////// wttp://pp//.1..0000333333 /tt////.1........221 04032.pdf.
José M Gatell. Antiretroviral therapy for hiv: do subtypes matter? 1153 1155, 2011.
Clinical infectious diseases, 53(11):
Zoubin Ghahramani and Michael Jordan. Supervised learning from incomplete data via an em approach. In Advances in neural information processing systems, pp. 120-127, 1994.
Yoni Halpern, Steven Horng, and David Sontag. Clinical Tagging with Joint Probabilistic Models. In Proceedings of the 1st Machine Learning for Healthcare Conference, pp. 209-225 PMLR, December 2016.
Michael Hughes, Leah Weiner, Gabriel Hope, Thomas H. McCoy Jr., Roy H. Perlis, Erik B. Sudderth, and Finale Noshi-Velcz. Prodiittion-cnnstrained training for semi-supervised mixture and topic models. 1xXx:1707.0734 jes, stalf, Jul 2017. URL sttp:///rxiie http///xxv...g/aaaaaaas/ 1707 07341. arXiv: 1707.07341.
Michael Hughes, Gabriel Hope, Leah Weiner, Thomas McCoy, Roy Perlis, Erik Sudderth, and Finale Doshi- Velez. Semi-supervised prediction-constraineed topic models. In Proceedings of the Twenty-Fiie First Interna- tional Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 1067-1076. PMLR, 09-11 Apr 2018a.
Michael C Hughes, Gabriel Hope, Leah Weiner, Thomas H McCoy Jr, Roy H Perlis, Erik B Sudderth, and Finale Boshi-Velez Semi-supervised prediction-constrained topic models. In AISTATS, pp. 1067-1076, 2018b.
Feng Jiao, Shaojun Wang, Chi-Hoon Lee, Russell Greiner, and Dale Schuurmans. Semi-supervised conditional random fields for improved sequence segmentation ation and labeling. 2006.
A. Jović, K. Brkić, and N. Bogunović. A review of feature selection methods with applications. In 2015 38th International Convention Information and Communicatton Technology, Electromics and Microelectron- ics (MIPRO), pp. 2200-1205 2015. doi: MALL:0//MMRRO.201..7760000
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling, Semi-supervised learning with deep generative models. arXiv preprint 40X6c:1406.52 5298 2014.
H. S. Kuyuk, E. Yildirim, E. Dogan, and G. Horasan. Application of k-means and Gaussian mixture model for classification of seismic activities in Istanbul. Nontinear Processes Geophysics, 19(4):411-44) August 2012. ISSN 1023-5809. doi: 10.5194 11--19--11--2112
20

---

Published in Transactions on Machine Learning Research (05/2024)
Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss- bayesian. In Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, pp. 416 424. JMLR Workshop and Conference Proceedings, Jun 2011. URL http://proceedigg: mlr. html.
NIDA. Common comorbidities with substance use disorders research report, 2020.
Kamal Nigam, Andrew McCallum, Sebastian Thrun, and Tom M. Mitchell. Learning to classify text from labeled and unlabeled documents. In Jack Mostow and Chuck Rich (eds.), AAAT/IAAT, pp. 792-799. AAAI Press / The MIT Press, 1998. ISBN 0-262-51098-7.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Porformance Dcep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett. (eds.); Advances in Neural Informalion Processing Systems 32, pp. 8035. Curran Associates, Inc., 2019. URL 2015-pytorch- seserchhan-imperee wwwernttieesttlleeeeeeeee www.gg--poorroaeeeaeeeeee
ctt.ippp.cpppppr/////c//pp 1irin-ibbrrrrrrrr pdd.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 22222522830 2011.
Sandhya Prabhakaran and Julia E. Vogt. Bayesian Clustering for HIVI Protease Inhibitor Contact Maps. In David Riaño, Szymon Wilk, and Annette ten Teije (eds.), Artificial Intelligence in Medicine, Lecture Notes in Computer Science, pp- 281-285, Cham, 2019. Springer International Publishing. ISBN 978-3- 030-21642-9, doi: 10.1007/978-88 --00-----0---11-----99 35.
Sandhya Prabhakaran, Karin J. Metzner, Alexander Böhm, and Volker Roth. Recovering Networks from Dis- tance Data. In Proceedings of the Asian Conference on Machine Learning, pp. 349-364. PMLR, November 2012.
Jason Ren, Russell Kunes, and Finale Doshi-Velez. Prediction focused topic models via feature selection. International Conference on Artificial Intelligence and Stalistics, pp. 4420 4429. PMLR, 2020.
Luca Scrucca, Michael Fop, T. Brendan Murphy, and Adrian E. Raftery. mclust 5: clustering, classification and density estimation using Gaussian finite mixture models. The R Journal, 8(1):289 317, 2016. URL
https://bbi.org/ 10. 111114//J-216---11
Abhishek Sharma, Jing Zhang, Daniel Nikovski, and Finale Doshi Velez. Travel-time prediction using neural- network-based mixture models. Procedia Computer Science, 220:1033 -038,, January 2023. TSSN 1877- 0509. doi: 10.1016/j 201.:ocs.222.03.144..
Diogo Silva and Clayton V. Deutsch. Multivariate data imputation using Gaussian mixture models. Spatiol Statistics, 27:74 90, October 2018. ISSN 2211-6753. doi: 1...16//.spasta.211.......0
Veselin Stoyanov, Alexander Ropson, and Jason Fisner. Fappirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In Proceedings of the Fourteenth International Conference on Arlificial Intelligence and Slatistics, pp. 725 733. JMLR Workshop and Con- ference Proceedings, Jun 2011. URL https: https://proceeding..t httt:://pppocccnngg...ererssssss http://gg......rreee////////aoaa http:////tsstttooolioooooooooolt 1a html.
tsurumeso. Github pysparcl, 2024. URL http::://gittuu..c////aarrrmmi
Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.
Annemaric M Wensing, Vincent Calvez, Francesca Coccherini-Slbeerttinn, Charlotte Charpentier, Huldrych F Günthard, Roger Paredes, Robert W Shafer, and Douglas D Richman. 2019 update of the drug resistance mutations in hiv-1. Topics in antéviral medicine, 27(3):111, 2019.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Danicla M. Witten and Robert Tibshirani. A framework for feature selection in clustering. 105(490):733-76 ISSN 0162-1459. doi:
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1529 1537, 2015.
Xinhua Zhuang, Yan Huang, K. Palaniappan, and Yunxin Zhao. Gaussian mixture density modeling, de- composition, and applications. IEEE Transactions Image Processing, 5(9):1293 1:00129233,,,,,, September 1996. ISSN 1941-0042. doi:
A Supplement Outline
In Section B, provide the derivation of ELBO and coordinate ascent updates for our prediction-focused models. In Section C, we provide the derivations for the analysis we presented in the main paper. The section is divided into three parts: first, we provide the derivations for the sup-GMM case, then for the GMM case, and finally for the pf-GMM case. For each method, we first derive general expressions, and then expressions for the special case to build intuition for the theory. In Section D and E, we provide additional experiments and further details about the experiments.
B Inference Details
In this section, provide the derivation of ELBO and coordinate ascent updates for our prediction-focusel models. We provide the derivation for the pf-HMM case, where N is the number of sequences and T, signifies the length of sequence n. The results for pf-GMM case follow by setting T, to 1 for all n. The parameters of the model are θ =
B.1 ELBO derivation
The ELBO for sequence X, Y with the posterior distribution (Z.S|O,, is:
log (2,,,
log (X, 2
E ((zs))
(Z,S)0,p)
E (Zzs)
[log [XX, Y/Z, KL (4
= E f Z B, [log fYY|,, ,"'"")
KL (log ((Spp)] log (S(p)
KL (log 9 [ 0) log (Z]0,,
where the individual terms are (define q( 7 X, Y, 0, ((z.))
T D ΣΣΣ d=1
[log S, (X),
E
Sh. log Px( ,B) + (1 - ST.a) log Px (X
7(S
T.
D
Dd E
[log
B)] (1) log Pa (X
22

---

Published in Transactions on Machine Learning Research (05/2024)
N T
E S, ( (,pp))
E
ΣΣ n t=1
log f(Y,") Zt mpte
q
E
[ogp(Y;" n1" y
(Z?)
1( Y)
1
KL (Z) Y, 4, +) log fZ A))
E
log
E
log
2
(29)
22 (X,)
((, 27-1 X,
N
T
D
f(ST,alp)
KL (log log = E
log
((S
q( (pd)
d=1
P
1-
Pa
+ (1 pa)
Pd
1 - PP
B.2 Coordinate Ascent Updates
Rewriting the ELBO again:
L Σ Pd
E a(Z((x
[log (rx(xal (1 101-pp0 paa) log Per (Xua)
E a(Zr(x,V))
log p( Y, V.12z,, [z, 1/ 11
log g(Sal(pa)
log g(Zo) X, Y)
(2,112,,1, X, Y)
E
Σ
E
log
((aukea)
f(Salpa)
2Z X.Y)
s(Zol0)
((2x-1,((xX))
f(Ze Zz
= E log [(X 1 S, B, m]
E log /( Y [2 11 k) + E [log /Z]0, A)] + E [log [S)pp
E. flg
g( S [p) [Z Y, s2, θ)
B.2.1 o0 update
Relevant terms:
Lpp E [log] (X B) + E [log ((S)p) E [log ((() 9
q
N Tn
ΣΣΣ
Pd ((2))
[log (x Z, B)] + (1 (2) log (xPl + pa P. 4Pd
+ (1 (1) log
1-
1 - PPa
The gradient of the loss terms:
V Vig Tap
E [lg Pa Zz B)] log Pa Px (x) loog
p
log
q (2))
1 p
4d
Update:
E
E 1x2
(log Pa Pa(X)) 7,, B - -lo P+ (x id (T)
P
Pd 4 o
log
1
N Th
23

---

Published in Transactions on Machine Learning Research (05/2024)
B.2.2 B update
n
Relevant terms:
N
T
D
1 8
9d
E ((7)))
llog PB (B Dr", B)]
==1
T,
D
-1
E
X
11 .d
B
to
(2,))
2
n=
[=1==
2g .
The gradient the loss terms:
RB
L
4B
Pd
WBB Σ 3.d'
7(66 = k))
10 k.d
-1
(Xis
(R.d)
k=1
N T.
(Xts
u j.d'
Pe =])
B
n=1
k,d
Update:
ΣN ΣTh (Z x)X Pm ΣN Ein 9( =)
Σ
where the qZ, 5) torm computed using the forward-backward algorithm.
o Ba
Relevant terms:
D
Dd
E ((27)
2
1
log
_12
za,d
B²
1
8
1227 d,
B
)2
+
d=1
2gdd
Gradient of the loss terms:
82
LOB2
Σ
Pa
q ZT =
B0 B2
2 20 5,00 )²
1
(X B
j.d
Update (if # 0):
Em
N Tn
Et= ((. = 1) X u )²
B2 j
+
ΣN In (Zt =)
where the q(.b. = j) term is computed using the
algorithm.
B.2.3 π update
1
24

---

Published in Transactions on Machine Learning Research (05/2024)
Relevant terms:
T,
D
bup
Σ
Σ
(1
(1-4a)lo pa) log Pr (
T
D
Qd) -1
(Xia m3) +
20
The gradient of the loss terms:
T
(X
V
Lur
(1 pp))
td
xx
#=1 t=1
d'
Update (if 9v # 1):
Σ N n= 21=1 td
Ia
14
Σ Σ'e
Σ ==1
ELn id
Σm Tp
xx2
Relevant terms:
N T. Σ (1 Pa) (-2 log R2
L
logo"2
1
(Xis - tc)
20
d=
The gradient of the loss terms:
To Σ (p))
1
Ligez
20x d'
+
2 2
(X - m3g
(Xis
n=1==
Update:
T2 d'
-
N
21=1
To
(X p.r)
Σ >== 1==
Tn
(XT, )2
m=
1=1
(107)2
N
Tn
n=
B.2.4 0 update
Relevant terms (including the constraint and the regularizor):
N
K
Lg ((gg +
a(Z = k)log f(ZT (101
Ao (206-1) B
1=1 k=
The gradient of the loss terms:
N
Vo, La = - + (a - 1) On
+ Σ
a(Z = k¹)
O
nml
K
V No L1 =
0k
k=1
25

---

Published in Transactions on Machine Learning Research (05/2024)
Update:
8g a 1 Σ ( =)
Ka K +
1 N
Ka-K
where the lerms q(] = k/) are computed using the forward-backward algorithm.
B.2.5 A update
Relevant terms:
K L ΣΣΣΣ (Z=1 = j, 2 =oooo f(r = =JA)
K
A Σ
Au 1)
The gradient of the loss terms:
1
Ap LA LA= == ->Ap
( k))
Ay.
1=
VAA
LA = Σ
Apsk
k=1
Update:
ΣN I
MN -
Ay +-
n=
Σ nml 2 km
> E 1 =
E [NNy.e]]
Σボ= =1 E [
where the terms q(,,- = = are computed using the cerward-bacwward algorithm.
B.3 Gibbs Sampling Conditional Distributions
While variational EM is quite stable when working with diagonal covariance matricos for in fa and fx. we require Gibbs sampling we are working with full covariance matrices:
f(z(m) m() X. Y,S) & (( (2())))))))))))))))))) (1) (z(n))
((Sa) II N (X (1) S. , S-,)
f(S -,, Y, = Exe(o.1) f(Sa =s) II fx (x
Sa s, S, a) (X (m) ((()), Sa = S
26

---

Published in Transactions on Machine Learning Research (05/2024)
C Derivations for Section 6 (Analysis)
C.1 Preliminaries
We first state some useful notation and results that we will use in the proofs.
1. We use (e, Σ) to denote the density of multivariate Gaussian distribution with mcan and covariance E.
We use faroda(+) denote the density of the data under model Model.
We use an asterisk superscript to denote the true / data distributed quantities (e.g. f*() for data pdf, 8", , M' for true parameters ctc.)
We use XR E PR and X1r E Pt. to denote the relevant and irrelevant components the data X RD respectively. Similarly, we use similar notation for PR, Prr, ER, Er..
Data Density First, we write the density of the data ([R; Y). In our setup, we assume that (XR, Y) and X1r are distributed as independent, equal mixtures with K and I components respectively:
(XX)) Σ 1 Mi
|
E
Σ
L (Xi:)
fin
1
ΣΣ (X
K
KL
Note that any of the components can be indexed mid = (ki, 1) where k and I are the indices of the 'relevant" and 'irrelevant" components respectively:
(X.Y)
fmu (X,Y)
KL
We will use this notation in the proofs. We assume that each of the input data components Xn. and X1, are Multivariate Normal variables and the label Y is Bernoulli variable:
1. (-) = o(: k
2. (1) p0: Mi k)
(-)()) Bern(:: mp)
Lemma C.1. To add in the proofs, we state the following useful results about the expectation of a function g under the data density f
Bbo(X, KL
E Gaati
(x(X
(33)
Sa
k=
E [[((X)]
E S
Xx(
(34)
KL
k=
K
E [l((k))]
E
gg(
(RR)
(35)
K
k=
[o(
7,
1
Σ
E
Xx))
(36)
lin
1=1
K
Blo(Y)] = K
Σ
E
[Y
(37)
27

---

Published in Transactions on Machine Learning Research (05/2024)
Lemma C.2. For multivariate Gaussion distributions Q(x) (X; Σq) and f(x) = pp: Ep), the density 9, the entropy H (Q), cross-entropy H H(Q, P) and KT--divergence KL (QIIP) are:
(Q: Σq) =
1
exp
1 (
Eq' (x mem
(38)
2
h In(2
k
1
H(Q)
+
In اوكا
(39)
2
2
2
1
P) 2
[up (2)) "nq"EEp Ep (up uQ) + In(T)) + |Ep| + Tr Tr)
(40)
KL (QII (Q, - HQQ))
(41)
28

---

Published in Transactions on Machine Learning Research (05/2024)
X(n)
p1, E
((n)
, Σ
0
(nn
(n)
m
0
Z(n)
Y(n)
m
N
N
sup-HMMM
(b) GMM
Figure 8: Graphical models for the GMM and sup-HMM models.
C.2 sup-GMM
We have the same setup as in Section 6 with the exception that we don't restrict the data and model to isotropic Gaussians for now. We first state our assumptions about the data first and then derive the population licellisss for sup-GMM and pf-GMM. We will use the sup-GMM case to derive the GMM
case.
Assumption: Model and Density We assume that the density of the data under the sup-GMM model is an equal mixture of Gaussians with J components (see Figure Sa for the graphical model)
Z Cat(l/J),
X| j ~ NN,,, 2,);
Y = Bern(n;)
The model parameters Q (ui, Egj. 11 1 The density of the data under the model is:
((xY)) Σ - E 1=1
Assumption: Well-separated components We assume that the components of the mixture distribu- tions are well-separated. The main reason we require this to simplify the analysis by allowing the posterior cluster-identity distribution f(Z|X, Y) be cither 0 for all (X, Y). An important consequence of well- separated components is that each data point assigned to exactly model component with probability 1. Therefore, each data component will fall under exactly one model component. We will use this fact in the proof of the population log- likeliidd below.
Re-indexing data components Due to our "well-scparated components" assumption, each data com- ponent will fall under exactly one model component. If we call the set of data components that fall under model component j as Mj, i.e.
Mj [mal data component falls under model component j),
then partition the set of data components {ma) into {M,lj E {1,.....)}.
Lemma C.3. Assume that the data is generated with density f* Y) specified by Equation C.1 and the model is sup- GMM with J components. Then, the population likeliheldoood for the parometers O SuPGMM
29

---

Published in Transactions on Machine Learning Research (05/2024)
ds:
1(0 sMp(MM) ==
[log [x, log j + 1x supprmm) + (QuppMM)))) + [ Y]]
(42)
where
1
1x(O (aapGMM)))
E
llg [xg
KL
Sma
j=1 (k,))
1
(y
E Dlo [(Y:
KT
Fink
j=1
H Z =
E f(2|x,))
[log f(Z
Proof. We derive general expression for the population log-likelihood with parameters SapGGMM- For brevity, omit SappGMM in the density functions.
I(O ((PPGMM)) = E loog f(X,)) Y)
E
E ((2(X.))
[log f(X, Y)
E
E
[og f(X, Y, Z) log f(z|x, yi]
f'
[(((X.))
E
E
[og [(2) + log /XX /) + log [(Y
E
[log ((2(X Y7]
1"
[(Z(x,))
)(
4 E
E ((Z)X())
log j + log f(X 2) + log f(Y f(x12)
+ H|Z X
8"
1
log J
+ +
93 [X, log log
+E Σ 93 (X. Y) log log
+ [
1x (x(ppmaMMM)
by (y(@upasrs)
1
= llogg j + Ix ( (0supGMM)) + ly (OsupGmM) + X Y]]
where (x, == = and EX(ZIX,Y) ==) -H In the above derivation,
step (a) follows from the fact that the expectation of constant doesn't 't change, step (b) follows from the fact that f(X, f(x, Y, / step (c) follows from the factorization of the model and step (d)
follows from the fact that f(Z)= = 1/J.
Expanding the Ix term:
E Ix (OOEppG)))
Σ 9i (x, Y) log fx,
log
KT.
E Dog [og]xx((X;6))
KL 1=1 mag marM
Img
30

---

Published in Transactions on Machine Learning Research (05/2024)
Similarly, we expand the ly (QsuGMM)) term:
ly (y@aupGMM))
= E
Σ log
Σ
Σ marddyy
E
(Y)
log (Y;
KT.
Fmga
Σ Σ
E
(Y;;
KT.
Fmgi
MREM,
C.2.1 Special case: J= 2 = =, 2,
We now consider the special case where the data is generated with hyperparameters K = L = and the sup-GMM model has = 2 components. This means that the data density is a mixture of components, while the model has only 2 components. We further assume that true parameters for Y components are vi = 0, n5 1. Using this case, we determinc the conditions that favor relevant components over irrelevant ones.
Theorem C.4. Assume that the data is generated with density f*X, and parameters O DuppiiM 2. The sup-GMM model 1 رر= We define Oee SuparM the param-
(1), Ej ni specified by Equation C.1, and has hyperparameters K with J = components, learns parameters @supGMM { {D,, E. nj
elers the wup-GMM the model learns when ils components are aligned with dala's 's relevant components, and Ou MMpGMM when unaligned. Then, the population bogeliieldoood for the parameters O SupGMM and Q is:
SMpGMM
2
( ((PGpGMMs)) = log 1/2 +
1
Σ)
H
Holuh, 2x,)) + H (6(Kigg VT,)) + KL (ø(Ming E),))ol(m Eir)
4
k=
1=1
1
I(O SupGMM) llog
1/2 2
H (0(Mir Eira)) + H (o(ph Ent)) KL (ő(Un, loo ER))
k= 1=1
Furthermore, the difference in the population log-likelihoods between the aligned and unaligned cases is:
1 2
I(O (uPGMM)) 1(0" SupGMM) log 2
Σ
[LL (o(Kn Ehol( ER)) KL (0(Kirg M7ra))l Eir))
4
Proof.
Aligned case (0a (Oup)MMN) Aligned parameters O MMPGMM correspond the case when the model compo- nents aligned with the relevant components of the data. Using this information, first. write out the
31

---

Published in Transactions on Machine Learning Research (05/2024)
Description
Values
j
Model component index
==
j ==2 2
(k, E M2
Data component index k=1,7=1 assigned to model component j
h ==1,1=2
== 2, 2 == =1
Aligned Parameters, O SPPGMM
JR. -
Learned model density for XR
fi.1
fū,1
fR
fū
fin. =p(āir: Eir)
Learned model density for X
o(Hi, Eir)
@(pir: Eir)
@iir Eir)
o( Eit)
fY. = -fY
Learned model density
frn
f&.
f8.
f&Z
for Y
Unaligned Parameters, O SMPGMM
fR.a (iR ER)
Learned model density for XR
Ó(ĀR, ER)
@(pR ER)
6(RR, ER)
Ø(ØR, ER)
Jig = fir,
Learned model density
Jig
fize
fiç
fīr,
for X1r
fY. = (055 Learned model density for Y
Bern(0.5)
Bern(0.5)
Bern(0.5)
Bern(0.5)
Table 2: sup-GMM: Summary of the learned model densities for the aligned and unaligned parameter cases.
individual terms in O SMPGMM = {(ng. S2g ng Sig,ng 12 j=1
[ AAe
2
Ag =
for j E {1,2}
where
Pir
Mir.
(43)
L
1=1
ERR 0
0
for j {1,2}
where
Σn.
Mir + Eirs1
T
(44)
Eir
1=1
0
if =1
16
where
7
(45)
1
if j =
32

---

Published in Transactions on Machine Learning Research (05/2024)
We first expand the Ix term:
Ix ( SupGMM)
KL Σ
Σ
E
fx (X:
Finga
j=1 (R.))M;;
KL Σ
Σ
E
log fng (Xui ER + log (Xii Pi E
Sint
j=1 (,
KT. Σ Σ
E
[og [X RR. )R E fie.
[log fix, (Xiri Pir.s )irs))
j=1 (K.))M,
R.
E
E
Ji + E fie.
[log (Xir; Eir)
KL
Jack
j=1 (K,))EM
Σ
HH( 2R,))) H H( Mi,,), o(Tri Mile Enr)
KL
j=1 (R.))EM;
Σ
-H (o(KRA ER.)) - H (6(Mi,,, Ei,,), Min Enr) + H (o(Hi Eir,))
KL j=l (R.))EM,
Σ
-H (H(URA - H (d(Min,, Ei,)) KL (d(Min, E E1s))
KT.
j=1 (K,D)M,
2 2
H (O(Mi, x), - H (o(pir, Eir,)) - KL (o(Mir, Eir (il())l Eit)
where equation (a) uses the learned model densitics expressions from Table 2, (b) uses the entropy and cross-entropy definitions, (c) and (d) add/subtract the entropy term and use the fact that KT-divergence is the difference between cross-entropy and entropy, and (e) uses the fact that the data is generated with hyperparameterss = L = and the model has J = 2 components, and {(k,1) E M,li E [1,2}} = {(1, 1), (1, 2), (2, (2,2)}.
Next, expand the ly (0 MM) term:
(y (OMPGMMM)
Σ
E fmna
[log (Y: (f))
KL
j=1 MarEMy
E
[log (x)
KL
Syuu
j=1 marE
Σ (H))
KT.
1=1 MaEMM
The rest of the terms in Equation 42 are:
log j = log
2
E 1.
[H = 0
33

---

Published in Transactions on Machine Learning Research (05/2024)
The likelihood for the aligned case is therefore:
I(0 SupGMM) log 1
H (Ó(MNR kRt))) H (o(pi, 21,)) + KL (o(Mir, f Ei.))
4 k= 1=1
Unaligned case (0 (MmGMM)) Unaligned parameters O MMMGMM correspond to the when the model components unaligned with the relevant components of the data. Using this information, we first write
out the individual terms 6% SuPGMM
[.. Ey 15 j=1
1 2
AR Minl
for j E {1,2} where
PR
Σ
PR.
(46)
K
En
0
for j E {1,2} where
ER =
1
Σ
HRAMRR + ER,
PRAR
(47)
0
K
N=
1
1j
(48)
2
We can similarly expand the Ix term as we did for the aligned case:
2
Ix 10 (MPGMM)
Σ
Σ
log
/x.j (X: mny
KI.
j=1 (k.))EM;
1
Σ
-H (o((tr Mi..)) H (0(UR, (R - KL (Ó(uRA ,R ( ΣR))
KL
j=1 (A.))M;
2
H (é(vie (ie))) - H (oUURA ER,,) - KL (ő(MR, Ml ER))
1=1
We can also expand the ly ( (upGMM) term we did for the aligned case:
ly ( (()P)M)))
Σ
-H(Ben((0.))
=
KL
==1 maEM,
= log
The rest of the terms in Equation 42 are the same as for the aligned case: log = log and E,- X Y] 0.
The likelihood for the unaligned case is therefore (we separate the log 2 term to highlight that is the ly (0%ppGMdd) term):
e (PMMMM) log log 2
1
Σ
H (o(Mfa, Ein + H (Ø(UR ER, KL (ølMR, E Er)))
k=1 1=1
This gives the difference in the population log- ligliikl between the aligned and unaligned cases:
1
e ((MMMMM))) 1(0% ) log 2
KLL (0(HR E (l En)) KL (6(1 Eir) (1e)
k= 1=1
34

---

Published in Transactions on Machine Learning Research (05/2024)
C.3 GMM General Case
The general case for GMM is similar to sup-GMM except. that Y is not modeled part of the mixture distribution. Therefore, we state the model density and the theorem for GMM case by dropping the log term since is present for both aligned and unaligned paramcters.
Assumption: Model and Density We assume that the density of the data under the GMM model is an equal mixture of Gaussians with J components but Y generated independently (sco Figure 8b for the graphical model):
Z Cat(L/J),
X j NN Ejj; Y ~ Bern(1)
The density of the data under the model is:
((xY))= = fv ー 5)= Σ 1 =
Theorem C.5. Assume that the data is generated with density f (X,Y) and parameters GGMM
(ng, 2g mz 11 j=1 specified by Equation C.1 and the model is GMM with learnt parameters 1GMM {as, Egg f3 j=1
Further assume that the data is generated with hyperparameters K L= and the model has == 2 components. Define G&MM as the parameters that GMM model learns when its compo- nents aligned with the relevant components of the data, and VoaMm as the parameters that GMM model learns when ils components are unaligned with the relevunt components of the dula. Then, the population log- bigliikeldoood for the parameters PoMM and O is:
1
1(0G)M)
H
( R, ER,K)) + H (o(ping (((())))) KL (o(pin; Eiai) Eir)
4
k=1
1=1
1
2
1(0%)M)) - log 2-
H (oliing Eia)) H (o(pR, KL (olRRu ER En))
4
Furthermore, the difference in the population likeliiossso between the aligned and unaligned cases is:
((0))M) - -((9))M)) log 2 +
Σ> Σ KL ER. ER)) KL (0(Mir, Eiri)lo((m Eir))
35

---

Published in Transactions on Machine Learning Research (05/2024)
C.4 pf-GMM General case
Assumption: Model and Density
Sa Bern(p) Vd, 2 ~ Cat(1/J),
YIZ j ~ Bern(1j). Xs = ~ N
X1-3 ~Np" _T) Vn.
Assumption: Well-separated components and point-mass switch posterior
We again assume that the components of the mixture distributions are 'well-separated:
2. We also assume that the posterior for the switches is point-mass at of the values of S, i.c. S(S s|X, Y; PrGMм)) for some {0, 1J For the parameters consider, the following are the values of s:
(a) S(S = s|X, Y; OD P(GMM) = for s = DR Op.]
(b) S(S s) sXX, Y; Oo pFGMM) = for [0pri p
(c) S(S s Y; + I-G
pfGMM) = 1 for [0pRi 0p,)
The density of the data under the model is (see Figure 2a for the graphical model):
D (sa)
((xY)= Σ
Σ
II
fsu
=
sE(0.110 j=1
d==1
デ Σ
II Ld=1
(sa)
fx (X-x11- = Σ 1x, (x = -
DE(0.1)
Lemma C.6. Assume that the data is generated with density f (XXYY specified by Equation C.1 and the model DM-GMM with J components. Then, the population log-lik likelihoodd for the parameters O is:
1(0 = E [logf(XXX)) = log j + +1s(0 + 1x(0 ly (O)GGMN) + E S X X.XY Y]]
(49)
where
1s(0p)GMM) E 1
Σ log fsa (sa)
1x(O) 1x(9pfaMu)) KL
E
log fxi Oysammn] +
Σ
E
llog f fx.z (X.: θ pp(amm)]
k=1
1=1
Imss
KL
j=1
(k.))
Jmms
(x(0ptMM)) KL
Σ
E
[log (Y;
Fr
1=1 (,.))M,;
H[S,Z|X,Y| = - /" x(
Dlog fogf(S.2 ((S.2(x.)))
(X,Y = S(S = x,,Y),
which is 1 for some (0, 1 and 0 otherwise.
36

---

Published in Transactions on Machine Learning Research (05/2024)
Proof.
The derivation is similar to the sup-GMM case:
1(6 p(GMM) E [log (X)
[og f (X, Y,S, 2) - log log yyl
E
E ((
E
log
js, (Sa) Ld=1
II
fz(Z)
2 @prOMM))
E
log f(s, 2| Y)
((
Specifically looking at the first term:
D
log
[[z(2)]
+
Σ -==1
log Ss (Sa)
+ E
[log (V (peccmn))]
[(Z|(.))
S(S|X.Y)
l- [(ss.2x..))
+ E
[oog F PIGGMM) + log fxi- (XI-s: [pIcMmm]]]
b log
+ E
Σ $E(U,
(X.
Σ
log (8d)
d=
+
Σ
Σ
=|==
8E Ø=1
+
Σ
U X, x) X, (log ( PPGGMM) - log fxi-, OptcMN))
SE $0(0.110 j=1
c log j +
Σ
log f (sa)
K
Σ
E Immge
f(z = j|s s, log fr, PFMMM)
KL
k=
j=1
K
E Sinzi
log /x, . (X1-si PFGGMM +
Σ [|==jS-= =,, Y) log fx, (Xx: OpEGMm)
KL
1=
1=1
log
+ E
Σ log (sa)
+ KL j=1 ak
E
log fv, 2 PpGMmM)
la (PPEcINAN)
by (8proMM)))
E mag
[log fx, (X1 OpKaMm))
Σ Σ
E fimss
[log (X,: [pGaMm)]
KL
KL
J=1 (A.1)EM;
Ix (0)cомM)
where equation (a) expands the log terms, (b) expands the expectation over and S, (c) uses the assumption that via (Xx Y) =1 for some {0, P) and (d) swaps the order of summations over j over (k, in similar way to the sup- GMM case.
37

---

Published in Transactions on Machine Learning Research (05/2024)
Description
Values
j
Model component.
j=1
==
index
1k, E M2
Data component index = assigned to model component, j
==1,1=2
k=
=
k
k= =
Aligned Parameters, Og PrGMM
S = [1gi ODul
Switches
fxs.j = -RR
Learned model density
fR11
fR.1
JR
fR.2
for Xs = XR
fx, - ((itee Eir)
Learned model density for X X1r
Mir Eir)
o(Ates Etr)
@ Exr)
oij Enr)
fY. = -Y.
Learned model density Sia1
fi11
fi2
Pi.
for Y
Unaligned Parameters, O PIGMM
= S000n Dul
Switches
fxs.j = -ird
Learned model density for Xs X1r
fir1
fine
fi,
fice
fx - ò(ūR, ER)
Learned model density for X1-s XR
MiRR ER)
Ø(āR. ER)
MiRR En)
o(ūR, ER.
JY. (er5(0.5)
Learned model density
Bern(0.5)
Bern(0.5)
Bern(0.5)
Bern(0.5)
for Y
Single-Gaussian
Parameters, O
S = 0p 0pill
Switches
fxg =1
Learned model density for Xs = 0
fx-_ = o(1,E) Σ)
Learned model density
( Σ)
o(A Σ)
p( Σ)
of Σ)
for X1-s
fY. (05)
Learned model density
Bern(0.5)
(0r5
Bern
Bern(0.5)
for Y
Table 3: pf-GMM: Summary of the learned model densities for the aligned and unaligned cases.
Theorem C.7. Assume that the data generated with density f'XYY) and parameters O OOJMM (1g, Ej, vi J
j=t specified by Equation C.1 and the model is pf-GMM with learnt paramelers PруGМм
and the model has J = components. Define On @yGMM as the parameters that of Gf- model learns when its components are aligned with the relevant components of the data, and @ SHGMM as the parameters that pf- GMM model learns when its components are unaligned with the relevant components of the data. Then, the population log- likelihood for the parameters O @fGMMM and A OpGaMM is:
{Ry, Eg, /3
,p 1.. Further assume that the data is generated with hyperparameters j=1
38

---

Published in Transactions on Machine Learning Research (05/2024)
1 D logg 1o,1og(1 p)
1(0 (rGau))) log
2
2
(o 2m,x)) + H (0(H)r (ir + KL (o(uir,) 2n _ Err
4
k=
1=1
1
1(6 ()GMM) log
Dr, logp + D log p) -
2
2
1
+
H (6((jr) ( H (M(HRN ER,)) - KL (ò(URA: ERR)ll((RR En))
4
k=
1=1
1(0..G P/GMM) log
1
Dlog(1 p) loog
2
2
+
H (o(un,e Em,)) (olHin Eirs)
4
k=
2
1
+
KL (Ø(Un, En Dl En)) - KL (0(pin Ein,)lo(fr Σrr)
4
km
Furthermore, the difference in the population logllikess between the aligned and other cases is:
((GGMM))) - 1(0 (((MM))) (Dn Dir) log
p log
-
KL (0(MR, Enn ER)) KL (o(ping Elr)) Err)
2 2
( (())M)))) - 1(01-G PjGMM) = Dn log
P -
+ log 2+
ΣΣ KL (0(pn E E En))
k=1
Proof. Like in the sup-GMM case, we first write out the individual terms for OIGMM. PpCGMM:
Ope PMGMM
Aligned case (0 (PMMMM) Aligned parameters Oo PIGMM correspond to the case when the model components are aligned with the relevant components of the data. We first write out the individual terms in Op PpGMMM
[RJ
1
A5
for (1,2) where
Σ
Mir,
(50)
Pir
L
1=1
DR
1
2
bgg
for j E (1,2] where
Σ1r
Mirll + Eins
(51)
Eir.
L
0
if j
17 = nī
where
(52)
1
if =
[....]
2
1TT
whore
PA =
Σ
HR.e
(53)
Pir
K
k=1
SITa
0
where
Σn
TT MR. + ER.
-
(54)
0
Σir
K
Note that the mixture parameters for Df-GMM are the same as sup-GMM and GMM parameters, and the only additional parameters are the last two elements (corresponding parameters f S2aa).
39

---

Published in Transactions on Machine Learning Research (05/2024)
Computing the ly term (using (X, 1 for [1 OD.. and 0 otherwise):
Is ( (ffGMM)) E
Σ LEE
log fsa (sa)
log (sa)
dED
Dn log p + Dr, log(1 p)
Computing the Ix term:
K
Ix (frOMM)
E [log /Xxxxxx-xX111 OpEGMNN)
E fime
Dlog Ix, 0x)]
KL
KL
j=1 (k.D)M
2
[ Ales Eir) +
Σ Σ
[log (XR,))
KL
KL
j=1 (R.)EM,
2
(o(uie,) Vi.))) KL (olvi,,: 2i Ioo End)
Σ
-H (o(M,, 2R,))
KL
KL
k=
j=1 (k.1)MM
2
2
Σ
( ER,)) H (ő(Mie, Ei.)) KL (d(Hi,) Ei,))l Enx)
which is the same expression as in sup-GMM case. Here, step (a) follows from the fact that X, = X and X₂ = XR for the aligned case, and the learned model densities for these cases are given in Table 3. Step (b) follows from the definitions of entropy and KI--divegennc..
The by, log J. Ey [H[] Y] terms same in sup-GMM
(p(0GM)))) 0
1 log j log
2
E T
| = 0
The likelihood for the aligned case is therefore:
1
(()MMM)) log 2
+ DR logp D1r log(1 p)
2 2
Σ
(0(M ERai) + H (0(Min. Di.)) + KL (0(Hi Eidilb((r Enr)
4 k=
40

---

Published in Transactions on Machine Learning Research (05/2024)
Unaligned case (6 (PRGMM)
2
PR Mirgj
1
P3
for j E {1.2}
where
PR
MR.k
(55)
K
k=1
ER
1
Σy
for j E {1,2]
where
ΣR K
Σ PR..RR + 2R,
ARRR
(56)
0 1
1) 2
(57)
nT
PR.)
where
PIr L
1
Σ
Pir.
(58)
ER
ST
where
Σ
H Mirdiie + Ziree - MiPA P1:
(59)
0
1
K
Computing the ls term (using (X, Y) for [0 1p,] and 0 otherwise):
Is (0
E f
Σ log (sa)
LdED
D1, log p + Dn log(1 p)
The rest of the terms are same as in the unaligned Sup-GMM case:
(x ((OGGM))
H. (o(vie, Ei.))) (Ó(MRA 2k,)) KL (o(ph, [R.))l ER))
ly (0""FMMM)) = log 2
log j log 1
2
| 0
get:
1
1(" = logg 2 + Dt. log + DR log(1 p) 2
H (o(pi.) Ei.)) (Ø(MRA Ei.)) KL (MlRA DR En))
and the difference in the likelihoods between the aligned and unaligned cases is:
(()MM)))) (()PrMM)) = (DR Drr) log
p + log
1-p
2
2
KL (o(RR, E Elo En)) KL (p(pin Ei,))lo( Eie)
1=1
Single-Gaussian Gaussian case (01-G ( The Gaussina case a special case for the Pf-GMM model, where the model has only component. (i.e. it considers all the data to be generated from single
41

---

Published in Transactions on Machine Learning Research (05/2024)
Gaussian distribution). The parameters for this case
p/.0 =pal-c=
[RR for j E {1,2} where
Pn
1
2
PR,
(60)
Pr
K
k=1
1
and
Prr L
Σ
Mirl
(61)
1=1
Ei- 5m1-G
En 0
for j E [1,2]
where
Σn K
1
T Pi..FR, 2RR
Pntn (62)
1tr
and
Err = K
TT Minkklink + Lir,k
- -i (63)
Computing the ls term (using (X, Y) for
[0 0D. and 0 otherwise):
Is (s(0) = E Σ
log Sse (sa)
LdED
D log(1 p)
Next compute the Ix term:
K
1x (x(M)MMMM)
E log (Xi=:
= KL
mage
2
E
[log ((xn Pun EnT] +
E finl
[log (xxii Exr)]
KLL H ( FR, KL (olKk E {l En))
+ Σ Σ HH (0(mi, EL,) KL (ø(Mi Ex El,)l Eir)
The ly, log J. Ey- | Y] lerms are same as the unaligned pf-GMM case: (y-0G pFGMM)
- log 2, log = log 2 E, [H X, Y|] = 0. Therefore, the likelihood for the
case is:
1 + D log(1 p) 10--ogg 2
1(61. PfGMM) = log 2
H (0(pft Ei,e)) H (o(Mie, Ei.)))
2
Σ
KL (0(PR, ( ER)) KL (ø(Mic, )lo)lo End)
4
2m1
and the difference in the likelihoods between the aligned and single-aaussian
Gaussian cases is:
((GG))MM) - I(0))GMM)) = DR log 1-p p
p + log 2 + KL (DUKR All ER)
42

---

Published in Transactions on Machine Learning Research (05/2024)
Spherical Gaussian case
To develop an intuition for the difference in likelihoods we derived above, we analyze the case where data and model components are isotropic Gaussians. First, we define the data:
Pi,1 MRLRR
Mi, MRd
2i,1 2,, OR 2
DDa
Pi,1 -ir
1i. = lr
Ei.
E1r, of Di
jiR = OR.
ΣR PR 22 1R. + oo LDa
and jrr Ole>
Σn 11 1 + I Di.
(64)
where 1t is Di,-dimensional vector of all ones and Ir. is a Dí. x Drr matrix of all ones (similarly for 1R 1R.R). We also define the signal-to-noise ratio (SNR) for the relevant and irrelevant components as:
PR n
Vi
AR
A
Using the above expressions, we can obtain the model parameters Oº, 0", and O using the parameter definitions for each of the models.
For each of the models, we can compute the difference in log- likelliohsods as follows (we use the notation A(O", O²) = (oode) 1(0 Model) for each of the models):
A (MM)) QR Qr
A(O O% @uPGMM) = log + QR
Qi
A APPPGMM
PIGMM PpGGM))= (DR Dir) log
p + log + QR Qıe
1-p
A PFGMM PPGMM) = DR log P
--p + log 2 + QR
where QR involves terms in A(O@MM: @MmM) that only depend on DR and AR (and similarly for Qı+):
1 (R A
QR 2
DRAD +1 Iog(DDA
Drr(1 A
Qr 2 Dr. +1 + log Dr.47 + 11)
When are relevant components preferred by each model? For GMM, the relevant components are preferred when O MMM has a higher likelihood than G (and similarly for sup-GMM). For pf-GMM, the relevant components are preferred when O" PMGMM has higher likelihood than 6% O"GMMM and e el-G PGGMM
Tuning Pf-GMM to prefer relevant components For GM-MMM, we can tune the parameter to prefer relevant components. We can do this by setting A(ORGGMM 4t OFGGM)) > and A(0 ()G > 0. This gives us the following constraints on p:
log log 20 Qtr DR Did
if DR Drr if DR D
p < Pu/a
lug 2+ 2n
1R Diel
log + QR. DR
p PI-G/a
43

---

Published in Transactions on Machine Learning Research (05/2024)
where axx) 1+exp(-z) is the sigmoid function.
The feasible range of p exists when Puja PI-G/as which is
when:
Dr. log 2 - DRQI, + Dr.Q 0f DR < Dr
Dr. log DRQU, DrQQR < <0 if DR Dr
D Additional Experiments
D.1 pf-HMM Simulated Data Experiments
HMM Performance on Synthetic Data (Dry 2)
HMM Performance on Synthetic Data (Drar
YAUROC
YAUROC
4
HMM Performance on Synthetic Data (Dre)= 4)
HMM Performance on Synthetic Data (Dial
AUROC
AUROC
0.8
0.6
4
HMM Performance Synthetic Data (Drel 8)
HMM Performance on Synthetic Data (Drei = 12)
AUROC
AAURO
0.8
HMM Performance on Synthetic Data (D, 20)
0.9
YAUROC
pANNN Vedud)
0.8
0.7
Figure 9: Df-HMM performance with respect to target AUROC on simulated data when we vary the compo- nent budget for the mixture models. Each panel corresponds to dataset with specified number of relevant dimensions, while keeping the total number of dimensions fixed at 20.
44

---

Published in Transactions on Machine Learning Research (05/2024)
D.2
Pf-GMM Simulated Data Experiments (Full Covariance Parameterization)
Predictive Performance as K varies =Dr
Predictive Performance as K varies (D)
1.0
1.0
0.9
Y AUROC
0.8
Y AUROC
0.8
0.7
0.7
0.6
0.6
0.5 0008
0.5
000λ
2
3
4
2
3
4
K
K
Predictive Performance as varies (3)
Predictive Performance as varies (D) 1.0
1.0
90ah
@
0.9
0.9
0.8
0.8
AUROC
AUROC
0.7
0.7
0.6
0.6
0.5
0.5
0008
3
4
2
3
K
K
Predictive Performance as K varies (Diu=S))
1.0
#9ph
0.9
AUROC
0.8
GMM IThI Mathod)
PC-GMM
Sup- GMM
2.Step
Figure 10: pf-GMM performance (target AUROC) as we vary the component budget (K) for the mixture models on simulated data. We now model the full covariance matrices for all components. Each panel corresponds to a dataset with the number of relevant dimensions fixed, while keeping the total number of dimensions fixed prediction-focus (in pf-GMM and pe- pc-GM) helps when the relevant dimensions are fewer. We also see considerable optimization variation in the models, owing to larger number of parameters needed to learn full-covariance matrix
We see that 2-Stcp-GMM model under-performs when the K small, while
45

---

Published in Transactions on Machine Learning Research (05/2024)
log P(Y
Iog P(X)
Iog P(Xe)
Model
20
p4-GMM
-1.05
-105.65
7.38
pf-GMM (Pred. Term Only)
-0.18
-150.18
-20.33
pe-GM
-0.50
-150.41
-15.51
sup-GMM
-0.06
-30.81
-15.48
log P(T[X)
lng P(X)
log P(Xx)
2-Step
5.79
Model
Class Cond. GMM
-5.92
14.87
-20.6
P&-GMM
-95.90
pF-GMM
-0.05
1105.6
7.38
PM-GMM (Pred. Term Only]
0.19
-143.28
-33.05
DE-GMM (Pred. Term Only)
-0.18
-150.18
-20.93
Po-CMM
00.1
-126.11
-25.99
P6-GMM
sup-GMM
-0.06
-45-79
sup-GMM
-0.06
-30881
15.48
2-Step
-0.09
-10.33
-22.51
2-Stop
-0.70
5.79
26.21
Cond.
12.13
Class Coud. GMM
21.66
40
pf-GMM
-0.04
-86.07
18.19
6 PE-GMM
-0.05
-92.20
7.49
pá GM-M (Pred. Term Ouly)
-0.09
4131.10
10.66
PF-GMM (Pred. Term Only)
-0.10
-151.88
-20.02
Po-GMM
-0.49
-116.71
-30.75
po-GMM
-0.51
-129.12
-5.48
sup-GMM
-0.61
-06.95
18833
sup-GMM
-0.63
1.76
-9.23
2-Step
2-Stop
-3.77
Class Cond. GMM
-3.00
-23.39
16.91
Class Cond. GMM
-2.25
28.47
6.96
50
pf-GMM
-0.09
466.3
23.15
8 pE-GM
-0.05
-57.23
8.37
P4-CMM (Pred. Term Only)
-0.25
-135.90
-56.48
PFGMM (Pred. Only)
Po-GMM
-0.51
-112.77
40.08
pe-GMM
-0.50
-130,48
-15.55
sup-GMM
sup-GMM
-0.56
8.80
0.89
2-Step
-0.68
-66,60
-30.58
2-Step
-10.75
Coud. GMM
-7.56
-14.29
20.76
Class Cond. GMM
-2.07
44.05
8.16
90
pf-GMM
-0.06
10.66
37.69
pá P5-GMM (Pred. Term Only)
-0.18
-125.77
-95.61
po-CMM
-0.49
-94.83
-53.85
sup- sup-GMM
-0.41
-31.33
-3.01
2.Stop
-0.07
Class Cond. GMM
-7.65
19.98
37.27
Figure 11: Discriminative and generative performance of all the methods for different dataset complexities (varying DR) and different model complexities (varying K) on simulated data (complex).
E Experimental and Baseline Details
E.1 Discussion: Baselines
Our central purpose for choosing baselines was to span the space of outcomes with generative and discrimi- native models, and then position our model in that space.
2-Step: This baseline without using any information about Y. This learned model is then used for predicting Y.
fully generative model for X, in the sense that it learns parameters
2. sup-GMM: This baseline uses fully generative model in a semi-supervised setting, iee. it uses information about both X and Y when fitting its parameters.
pf-GMM: This model explicitly trades off generative and discriminative quality as described in paper.
pc-GMM: This bascline also trades off generative and discriminative quality in a different way.
pf-GMM (Pred. Term Only): This baseline has the same model pf-GMM but the objective is purely discriminative, ie. log f(Y]X). This objective hard to optimize, though. Therefore, use this method to demonstrate the predietive-generatin trade-off but not for the rest of the tasks.
6. Random Forest: This is purely discriminative baseline. We expect it to be similar in performance to "pf-GMM (Pred. Term Only)". Random Forests not a method of interest, though. We want a generative model that able to cluster the data, and RF clearly cannot achieve
Sparse Clustering: This baseline is another 2-step method commonly used in the clustering literature to ignore certain features in the data. Tt. has generative model.
46

---

Published in Transactions on Machine Learning Research (05/2024)
E.2 Training Details for Baselines
1. 2-Step: We used the scikit-learn (Pedregosa et al., 2011) implementations of GMM for the gener- ative model and logistic regression for the discriminative model. The GMM is trained using the algorithm.
Random Forest: We used the scikit-learn (Pedrogosa et al., 2011) implementation of Random Forest.
3. Pc-GMM: We used the Adam optimizer (Kingma & Ba, 2017) to optimize the Pe-GMM objective. We used the PyTorch (Paszke et al., 2019) library for automatic differentiation. We search for the best value of and learning rate parameters using a grid search.
SMp-GMM: We used the same implementation as pf-GMM since the special case p = 1 is equivalent to sup-GMM.
Sparse Clustering: We used the implementation from the pysparcl (tsurumeso, 2024) library. We used 25 permutations to search for the best value of the w parameter as proposed in (Witten Tibshirani).
pf-GGM (Pred. Term Only): We used the same implementation as pe-GMM with a very high value of À ignore the generative term.
E.3
Simulated datasets for the GMM experiments
E.3.1 Synthetic (simple)
We follow the following generative process to generate a collection of datasets:
[Po .. -Pko-1]
Kp
X1,1]
0 W(K 1) Bern(p:)]
k=
Kn-1
X₂
Σ
N 6,
X = X
X E
X2 G
where (ern(a) is Bernoulli distribution with parameter a, and 6 is constant vector. This process ensures that the target is correlated with the first DR dimensions of the input X but not the last Dr dimensions. The gap between successive components fixed to 6 so that the clustors are well separated and so that the relovant and irrelevant dimensions have equivalent signal-to- toise ratio a-priori.
In our experiments, we set:
KD 3
DR E {20, 30, 40, 50, 80}
DR + DT. 100
R normalize (0.5 + [0...., K K 1))
g normalize(1 + JO..... K 1))
where normalize(x) = x/ |x]]] makes sure the vectors are valid probability distributions.
This gives us knob tweak: DR (number relevant dimensions). It is common for the input only
have few relevant dimensions. Therefore, we vary DR in 10, 20, 30, 40, 50, 80, 100
47

---

Published in Transactions on Machine Learning Research (05/2024)
E.3.2 Synthetic (complex)
The generative process for the complex dataset simple, except for the following differences:
KD =4 4
= [0.05, 0.95, 0.05, 0.95] DR E {20, 30, 40, 50, 80)
E.4 Simulated datasets for the HMM experiments
[1, ~
Σ
( 6, T), Bern(p)))
k=
K-1
X2
Σ
OK -KKI))
X = X1; X2]:
X1 E EROu
X₂ E Ri.
In our experiments, we set:
K = 4
= [0.05, 0.95, 0.05, 0.95] DR. 2,,,,,,,,,1
DR D1. 20
g = llr 2,
-21
A + (ormalizGrour11
+ Ikk
-2k
-2--
A + normalizerom (.01 +
+ Ilk
-
2, 2, ~ Cat(1/K), 2,, 2/ E R
where normalizeroml (M) applies normalize to each row of matrix M. Also, the numbers .1 and .01 are arbitrary choices to differentiate A and A and the simulation would work with different values too.
E.5
HIV Experiments
Therapy for HIV involves administering cocktails of antiretrovirals from five classes namely, Non-mucleoside Reverse Transcriptase Inhibitors (nnRTIs), Nucleoside Reverse Transcriptase Inhibitors (nRTIs), Protease Inhibitors (PIs), Fusion Inhibitors (FIs), and Integrase Inhibitors (Ils) to bring the viral load below detection limits (S 40 copics/ml). We study 53 236 patients with HIV from the EuRosist Integrated Database. Each person has a time-series of an average length of 16 sleps where a time slep is approximately 4 months between consecutive treatments. Our task to predict whether a treatment will bring the viral load below detection limits in the next time-stop. Each input contains 267 features including CDuncounts genetic mutations, treatments in terms of drug classes, and lab results. Though it is common have many genetic mutations, only few of these may be relevant for inducing drug resistance thus increasing the viral load.
48

---

Published in Transactions on Machine Learning Research (05/2024)
Table Comparison of the baselines with pf-GMM.
Class-specifi- GMM
GMM
pf-GMM
Y AUC
1.0
0.6
1.0
Dataset (Simple)
log P(X)
-10.0
-16.9
-04.66
log P(XR)
-19.6
-27.0
10.2
Y AUC
0.7
0.5
1.0
Dataset 2 (Complex)
log P(X)
10.4
5.6
-106.3
log P(XR)
-21.4
-26.8
10.1
F Discussion: Naive Baseline of Fitting Class-Specific GMMs Fails
A simple alternative to the frediction-focused approach might simply try to learn separate GMM for each class label. In this case, the prediction Y for an input X can be predicted based on the probability of the input under each class-specific GMM. First, we note that this approach which splits up the data by class does not allow us to identify structures that have commonalities between the classes. Second, this approach will often produce poor predictions, as we demonstrate below. In this section, we demonstrate using two datasots that bascline that uses separate GMM for each Y label is insufficient in identifying and modcling the relevant dimensions in the dataset.
F.1 Setup
'Simple' Dataset with unimodal class-specific density along relevant dimensions. This dataset contains 100 dimensions, where the first 20 dimensions relevant and generated using 3-cluster GMM, and the rest of the dimensions are irrelevant and generated using separate 3-cluster GMM. Therefore, the data density along relevant dimensions has three modes, where the middle mode corresponds to the Y = class and the other two modes correspond to the Y = 0 class (see the first row of Figure 12 for the histogram of a relevant dimension of this datasct). The data density along irrelevant dimensions also has three modes but they do not differ the proportion of Y labels (see the second row of Figure 12 for the histogram of an irrelevant dimension). We will show below that this demonstrates the case where the class-specific GMM can give good predictions without learning a good density for the relevant dimensions (i.e. for the wrong reasons).
'Complex' Dataset with multimodal class-specific densities along relevant dimensions. This dataset contains 100 dimensions, where the first 20 dimensions are relevant and generated using a 4-cluster GMM, and the rest of the dimensions are irrelevant and generated using separate 4-cluster GMM. Tn contrast to the previous dataset, the data density along relevant dimensions has four modes, where the first and the third modes correspond to Y = O class and the second and the fourth modes correspond to Y = class (see the third row of Figure 12 for the histogram of a relevant dimension of this dataset). The data density along irrelevant dimensions also has four modes but again they do not differ in the proportion of Y labels (see the fourth row of Figure 12 for the histogram of an irrelevant dimension). We will show below that the class-specific GMM will fail give good predictions and learn poor density for the relevant dimensions in this case.
Evaluation We evaluate the models on Y AUC, log P(X), and log P(XR). AUC measures how well the model clusters can discriminate the label and log P(X) mcasures how accurate the model density is for all the dimensions of X. Finally, log P(XR) measures how accurately the models estimate the density of only the relevant dimensions (known to us because generate the data).
49

---

Published in Transactions on Machine Learning Research (05/2024)
Class-specific GMM
GMM
pfGMM
Top: X5 (Relecart) Dottom: X30 (melevant)
Tep: X5 (Re evant) Bottom: X33 (irelevant)
Top X5 (Relevents Bothom: X30 (melevant)
denaity tey, 1,60)
sneddy
orGAN
Hodely 1
6--
0.
2.5
Dataset 1
PrGMM densily for X30.
3.00
MAXT Moddly 1
-20
-18
-03
0.5
10
15
-2.
-13
Clacsopecite CNN
CMH dens
prGM denally
Top: X5 (Relecert) Botiom: X18 (melevent)
Tha: (Se wvant) Bettom: X30 (Imeeecent)
Top (Reeeconts Bathom: X10 (melevant))
Cins-specili CNM dunilty for
Varill CMM dumalty for
p1o derchty 1.00)
dexcity
proen sansily
30-
0.5
Dataset 2
Clasospaciic GNN denaity for X30
Vanilla CMN density for XID
plotM
Huabiyy wit
3.00
p0oen cantity
Dal -1
6.5
338
3000
Comparison of the class-specifie- GMM baseline (left column) on both datasets reveals that it is similar to a GMM (middle column) in ignoring relevant dimensions. In contrast, pf-GMM (right column) can ignore irrelevant dimensions even when they outnumber the relevant dimensions.
Figure 12:
F.2 Results
Class-specific GMM cannot filter out irrelevant dimensions In the left column of Figure 12, we see that the class-specific GMM baseline clusters on the irrelevant dimensions, and fails to learn good densily for the relevant dimensions, log P(XR), for both datasets. The densities learned by it are similar to densities Icarned by the simple GMM baseline (middle column of Figure 12). In contrast, the pf-GMM model learns a good density for the relevant dimensions, gives good predictions, and ignores the irrelevant dinensions (see right column of Figure 12 and Table 4).
Class-specific GMM can be predictive of Y without clustering the relevant dimensions well In Table 4, we see that the class-specific GMM can give good predictions on the dataset where the relevant dimensions have a unimodal class-specific density (i.c. the first dataset in the Sctup). This is because it can model the P(X = density well and tell when the data comes from it (as seen in the top row, left column of Figure 12). However, this good predictive performance is not indicative of good density model for the relevant dimensions, as discussed in the previous paragraph. Furthermore, when the relevant dimensions have all multimodal class-specific densities (i.e. in the second dataset), the class-specific GMM fails to give good predictions and learns a poor density for the relevant dimensions.
Class-specific GMM is qualitatively similar to simple GMM The behavior of the class-specific GMM bascline is qualitatively similar to that of a simple GMM: when faced with many irrelevant dimensions, both these methods just cluster on the irrelevant dimensions (see Figure 12, middle column). Both these methods achieve a much better overall density log P(X) than the pf-GMM model because they learn a good density for the irrelevant dimensions and the irrelevant dimensions dominate the overall density. Furthermore, both these methods learn poor density for the relevant dimensions, log P(XR). The class-specific GMM still perforrsss the simple GMM quantitatively (Table 4) since it uses twice many parameters. In contrast.
50

---

Published in Transactions on Machine Learning Research (05/2024)
the GMM model is inherently different in its behavior, as it learns to ignore irrelevant dimensions, learns good density for the relevant dimensions by clustering on them, and uses these clusters to make predictions.
51


Reviews:
[{"review_id": "dBZKo5JkPs", "reviewer": "Reviewer_kVkz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1) The paper gave a detailed study of using prediction-based GMM and HMM models on both synthetic datasets and two real datasets. The paper is clearly written, providing insights into when the feature selection will work and when it will fail.\n\nS2) The appendix includes a detailed derivation of the technical details, including the ELBO-based inference procedure.\n\nW1) I wish there are more discussions about the choice of the datasets---they seem reasonable to me but I think more background concerning their motivations and their particular application to this type of methods would be helpful.\n\nW2) I am also wondering how does the proposed method compare with standard EM type methods---also, it seems that the training algorithms for fitting the mixture models are not sufficiently discussed in the current manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors aim to fit a mixture model (such as GMM or HMM) that uses a user-specified number of components to a given data distribution. As is typical in this literature, the authors focus on maximizing the data likelihood for a given prediction quality of the labels.\n\nFor this setting, the authors introduce prediction-focused mixture models, including one case for GMM and another case for HMM. After specifying these models, the authors went on to formalize the questions, such as dataset characteristics for which a misspelled model can still select the relevant components.\n\nThrough a detailed case study, the authors derive a series of conclusions from the analysis, including:\n\n- The prediction-focused GMM can identify relevant structure by changing the switch prior $p$ (this is a variable to distinguish relevant and irrelevant dimensions).\n- The prediction-focused GMM identifies mixtures even when there are many irrelevant dimensions, and it is more robust to high SNR.\n\nLastly, the authors use synthetic data and an HIV patient database and another Swiss ban notes dataset to evaluate their proposed models.\n\nIn summary, the paper developed a model-based feature selection method for mixture models and hidden Markov models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- See weaknesses above. In particular, addressing both points under Weaknesses.\n- Also, there is a section called \"Formalizing the Question\" in Section 6.3---I'm not sure if these are the focused questions of the paper or just follow-up questions after the discussion in 6.2. It would be helpful to clarify.\n- In the introduction, even though you listed a few items as contributions, they don't really talk about the actual empirical analysis you showed. This can be clarified.\n- The appendix is very long; it can benefit from having an organizational paragraph detailing the list of contents to follow. Likewise, in the main text, you can include a preview of the main sections before starting the related work paragraph.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hIpoBnMjoj", "reviewer": "Reviewer_ZjeR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n        - Intuitive model definition.\n\n        - Well written paper.\n\n        - Interesting problem addressed.\n\nWeaknesses:\n\n        It seems that the authors describe a generative model, but do not carry out inference in such a generative. They suggest an alternative model on which they do carry out inference. That is not very well explained. In particular, why it is not possible to use the generating model to carry out inference and why there is a need to describe another model for inference. That is counterintuitive. Is not this inference model another way of writing the generative model assumed first?\n\n        It is not clear VI is used for inference. It seems that you have an expression for the joint probability of the observed variables in the inference models. Thus, one can simply maximize the likelihood without having to approximate it via VI.\n\n        The analysis in Section 6.2 seems to be valid only for data without correlations since the covariance matrices in Eq. (22) and (23) are diagonal. It seems also limited to only two Gaussians.\n\n        No analysis similar to than of Section 6.2 is carried out for temporal data.\n\n        It seems that no real-world experiments are carried out with the HMM model. There are only synthetic experiments in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This papers describes a generative model for observed data consisting on a set of explaining attributes, which can be either relevant or irrelevant for prediction, and a set of observed labels. The labels can be categorical or sequential data. The authors describe alternative models to infer the relevant information from the observed data on which and expectation maximization based algorithm using variational inference is proposed. The method proposed is validated in synthetic data, and on two real-world datasets, showing improved results with respect to alternative methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Revise the writing of the paper. E.g., in the beginning of Section 3.1, in the notation, there is a bold capital letter referred to as a random variable, which is confusing.\n\n- Notation [N] is confusing. I would suggest using {1,...,N}, which is understood better and does not use that much writing space.\n\n- Explain why there is a need to use VI to estimate the model parameters, instead of maximum likelihood.\n\n- Include the synthetic and add real-world experiments for the HMM model in the main part of the manuscript.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0XP7KeU0Mi", "reviewer": "Reviewer_jmg6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe idea of task-focused generative modeling seems interesting, and I'm not aware of any established methods for fitting generative models like GMMs or HMMs that account for a response variables. The proposed approach seems reasonable, and it can be fit efficiently using standard optimization procedures (like EM). The paper is mostly well written and easy to follow. \n\nI have some comments below the experiments, but I appreciated that the authors investigated the run-time of each method, the sensitivity to the $K$ parameter, and used a synthetic dataset to study the effect of a couple aspects of the data-generating process (the results shown in Figure 4). \n\n## Weaknesses\n\nOne main weakness is the motivation for this work: in what scenarios is it important to fit a probabilistic model of the data distribution *and* predict the response variable? The experiments don't discuss any uses of the generative modeling capability. It's doubtful that the GMM-like models used here accurately estimate the data distribution - for example, I wouldn't expect that they could generate realistic samples, and the paper doesn't explore anything of this nature (for this we would probably require some kind of deep generative model). With respect to predicting the response variable, the proposed approach underperforms a simple random forest. Finally, the authors at times lean into the interpretability aspects of this approach, even writing \"because inspection is our ultimate goal, simply creating a predictive model is not sufficient\" (page 1). However, the experiments seem to barely explore the inspectability of these models, and the authors overlook a range of other approaches that create interpretable predictive models (e.g., feature selection or post-hoc feature attribution, both of which can accommodate more expressive nonlinear models).\n\nAnother weakness has to do with the presentation: the assumptions about the data distribution in Section 3 seem disconnected from the modeling approach in Section 4. It takes roughly 3 pages to explain the data setting, but the proposed methods correspond to a far simpler model of the data-generating process. Also, a couple questions and concerns about Section 3: 1) the reference to \"Supplement Section 3.4\" looks like a typo, 2) assuming that all data sources are independent seems unrealistic for most datasets (this perhaps doesn't matter because Section 4 is so different from Section 3), 3) it was unclear whether we would assume existing feature groupings (again, this bit is ignored in Section 4), and 4) I couldn't understand what the authors mean by \"our approach extends to any exponential-family distribution\" on page 4 (again, because Section 3 is not obviously related to the proposed method).\n\nNext, the proposed models are interesting, but I wonder if the authors considered a simpler way to integrate a response variable into a GMM or HMM. For the iid case with a GMM, instead of the pf-GMM (modeling $Y$ jointly with a subset of covariates), or modeling $Y$ as a covariate (the \"supervised GMM\" baseline), or predicting $Y$ based on the mixture components, what about fitting two separate GMMs for data with $Y = 0$ and $Y = 1$? This simple probabilistic model would make it easy to calculate the posterior $P(Y \\mid X)$, and fitting it is just as easy as a standard GMM. If I understand correctly, this baseline was not used in the experiments, and I wonder if the authors could add it.\n\nSome other thoughts on the method:\n\n- In Eq. 13, it seems like one weakness in the model is that features with more complex, non-Gaussian distributions may be pushed towards having $S = 1$ to benefit from the expressiveness of the mixture distribution. Is this intuition correct, and do the experiments address this point? \n- Eq. 14 appears to have some typos, it omits $X_S$ and $Y$'s conditioning on $Z$\n\nAbout the experiments:\n\n- Figure 4B/E show the landscape of discriminative and generative modeling performance for each model, and that pf-GMM offers a reasonable trade-off. However, like I mentioned above it's not clear why the $\\log p(X)$ term is important. \n- Given the emphasis on interpretability and inspection of the model, I expected to see some of this in Section 8. The only bit I see is the very last paragraph, where the authors mention two mutations that are consistent with prior work. Is there any kind of more robust evaluation the authors can offer? For example, would it be more appropriate to share a full list of the features deemed relevant, and perhaps analyze the stability of this set as parameters like $K$ and $p$ are varied?\n- The experiment are run using just two real dataset, and I'm not sure that's enough to draw the conclusions discussed in Section 8. The model seems fast to train, so perhaps the authors could add some additional datasets? And perhaps some more similar to the HIV dataset, because the Swiss Bank Notes one seems very small (6 features, 200 examples).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a class of probabilistic generative models that account for a response variable in addition to the input/predictor covariates. Specifically, it develops a \"prediction-focused Gaussian mixture model\" (pf-GMM) and \"prediction-focused hidden Markov model\" (pf-HMM) for iid and temporal data, respectively. The models can be fit using conventional optimization approaches, and they enable both fitting the input data distribution (maximizing the data log-likelihood) and predicting the response variable, thereby bridging the capabilities of existing generative and discriminative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Several requests for clarification and changes to the experiments are described above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a detailed study of task-relevant feature selection, with two prediction-aware mixture models proposed. These models are analysed both theoretically and empirically, with both synthetic and real-world data. There were some initial concerns on the motivation and comparison to relevant baselines; following the author discussion and revisions, all three reviewers found the paper's execution to be convincing, and were in favour of acceptance. The AE concurs with this view.
Decision: Accept (poster)