Gaussian Copula Embeddings
Authors: Chien Lu
Jaakko Peltonen
Venue: NeurIPS 2022 Accept (2022)
Keywords: representation learning
gaussian copula
heterogeneous data
vectorial embedding
probabilistic models

Abstract: Learning latent vector representations via embedding models has been shown promising in machine learning. However, most of the embedding models are still limited to a single type of observation data. We propose a Gaussian copula embedding model to learn latent vector representations of items in a heterogeneous data setting. The proposed model can effectively incorporate different types of observed data and, at the same time, yield robust embeddings. We demonstrate the proposed model can effectively learn in many different scenarios, outperforming competing models in modeling quality and task performance.

Gaussian Copula Embeddings
Chien Lu
Jaakko Peltonen
Tampere University
Abstract
Learning latent vector representations via embedding models has been shown promising in machine learning. However, most of the embedding models are still limited to single type of observed data. We propose a Gaussian copula embedding model to learn latent vectorial representations of items in heterogencous-dal setting. The proposed model can effectively incorporate different types of observed data and, at the same time, yield robust embeddings. proposed model can effectively learn in many different scenarios, outperforming competing models in modeling quality and task performance.
We demonstrate that the
Introduction
Representation learning is prominent machine learning approach for working with originally non- vectorial data. Embedding models learn latent vectorial representations for data items that appear together with a context, through modeling the interactions between each center item and its context items. The approach was first introduced as a language model [14] which learns word representations through modeling the probability of the appearance of central word given surrounding context words. The word appearance is modeled as an observation from a multinomial word distribution. Building on this notion, exponential family embeddings were proposed [19] which further generalized the original model to a class of models suitable for many observed data types, which have been shown promising in different domains. However, the ability of such models to incorporate heterogeneous data is still limited.
Data in many modern domains is heterogeneous, involving simultaneous observation of different data types such as categorical values, integers and real-valued numbers, and having varied distributions within each data type, hence it is difficult to model them together as observations in vectorial embedding; naive unified solutions that ignore the difference of the data types would not yield good models. In particular, the different data types often distributed over varied scales and with various distributional shapes: naive normalization coupled to modeling with a single distributional assumption would not suffice yield robust embedding models, and would leave them vulnerable to extreme values and distributional shapes not corresponding to the assumed ones. An equally pressing problem is how to flexibly model relationships (dependencies) between the several observed variables with their differing distributions: naive modeling strategies ignoring the variable dependencies would again yield poor embedding models.
We solve the mentioned challenges by introducing a novel Gaussian copula based latent representation learning model. The model learns vectorial embedding representations for items leveraging the center- context item interactions, but unlike previous embedding models the proposed model is able to learn embeddings in setting with multivariate data having heterogeneous data types and distributions.
For computational efficiency, we introduce set of variational auto- ento-encoder based inference algo- rithms. In experiments on five different scenarios, the proposed model is shown to be effective, outperforming competitive methods in fask-based evaluations and yielding insights in a social media analysis task.
Our contributions
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We introduce general-purpose purpose representation learning framework which incorporates multiple, heterogeneous data. Our work allows embedding models include multiple data types and distributional assumptions in well-founded probabilistic joint modeling through the Gaussian copula; other approaches such as exponential family embeddings have generalised to different data types but still treat them individually.
To our knowledge, ours is the first work which brings the advantages of Gaussian copula to learning representation vectors from heterogeneous data. The Gaussian copula is intuitive and has proven effective in machine learning research, thus it is an attractive solution which was neglected in representation learning. We close this gap, and the result shows it handles heterogeneous data well.
We develop an efficient inference procedure based on semiparametric estimation and varia- tional autoencoder. Previous works have used MCMC for inference and in many such works, lack of scalability has limited their application to larger amounts of data.
We demonstrate the effectiveness of our model on five different scenarios, each with real-world data set and corresponding quantitative and qualitative evaluation.
This paper is organized follows. Section introduces the necessary background notions of embedding models and copula models. Section introduces the Gaussian copula model and Section 4 develops the inference algorithms. Five different scenarios of using the proposed model provided 5. each evaluated with an experiment on real-world data set. Section draws the conclusions.
2 Background
2.1 Embedding models
Learning latent representations based the interactions between the observed item and its contexts has been an imperative topic in machine learning. It was first introduced as a language model to model relations between words [14]. The framework has been later generalized model other different co-appearance patterns such as in community embedding (23)
In brief, let the item and its context j be two items (such two words, or two communities). The probability of them appearing (D = 1) in the same context (such as in the same sliding window) can be modeled
P(D P(D)1 (1-E-aTo)
where p; and Pj are the embedding vectors of the items. Note that instead of using the same kinds of vectors for both roles, the context items can have their own context vectors thus the above probability becomes
(1 1
((D=11;)))
Exponential Family Embeddings [19] is an extension which has further generalized the model to data beyond text. Let x be the value of the item at the location n, which has its context c... In exponential family embedding, the value of v., depends its context c, and is generated from exponential family distribution
C, ExpFam (n/ (Ie..) ) (2n))
where 7/, (Zt.)) is the natural parameter, and (In.)) denotes the sufficient statistics. The natural parameter modeled as function of an inner product of the embedding vector p and the context vector a so that
Th (Ie.. =
Σ Ee
a
(4)
C.
As the exponential family can model different observation distributions, the embedding models are no longer limited to modeling co-appearance (binary) observations, It has been applied to different domains such grouped data [18] and graph data [1]

---

Negative sampling or sub-sampling is a common practice when training embedding models. The notion consider only randomly generated subset of the items that do not occur at the location n. That is, if an item i does not occur at location, the probability of that negative occurrence
P(D = 0/i.5) ( ( is integrated into the objective function. In exponential family embeddings, if the item is generated as negative sample, the corresponding pseudo observed value encoded as 2 0.
2.2 Gaussian copula
A J-dimensional copula C is probability distribution on [0, 1 where each of its univariate marginal distributions is uniform distribution on [0, 1]. That is, given set of uniform distributed random variables U1,...,Up copula is the joint cumulative distribution
C(u uJ) = P(U1 S ui,..., UJ S uJ)
(5)
The key idea of copula modeling is to use the copula to model the dependencies between several variables having arbitary types and marginal distributions. Let x be random vector of length and let j E ....J index the elements (random variables) in x. According to Sklars' theorem [21] the cumulative distributions (CDFs) of the variables x can be modeled by a copula
F( x5) = C Fa(zz))
(6)
where F is the joint CDF and (x( P(X, S is the j-th marginal CDF. Since each marginal CDF value is [0, 1). the right-hand side is copula regardless of what distributions the individual marginal CDFs have. If every F, is continuous, then the C is unique. In this way, the copula encodes the structure variable dependencies, while allowing each of the variables 21 IJ, to different type and to have differing kinds of marginal CDFs.
In this paper, we consider a Gaussian copula, which is one of the widely used copula models; we introduce the model to the representation learning task. A Gaussian copula is defined as
C(u (,) = 2 (Φ ( (m1) (w))CC)
where 1 is a J-dimensional Gaussian CDF with a correlation matrix C, and 6-1 is the inverse function of the standard univariate Gaussian CDF. With the Gaussian copula, the joint CDF of observed data can be modeled as
F F xJ) ((Fi F,(z)) = 2. ( (8)
The Gaussian copula can also be expressed in terms of latent Gaussian variable representation. First, a latent vector z is generated from a Gaussian distribution
(0,
wih covariance matrix Ω which corresponds to the correlation matrix C in equation 7 Then for each j. the observed data value I, is obtained from the inverse of the univariate marginal F3 according the generated latent variable 2, so that
2j
(10)
/Wis
where Wjj the j-th diagonal element S2. In principle, equations 9 and (TO) could be used to derive an equation for the likelihood of the data values x, at each observation, in order to use such likelihoods for parameter fitting. However, instead the practical approach in Gaussian copula research is to derive an extended likelihood, and we will do that for the vectorial embedding task.
3 Gaussian Copula Embeddings
We now introduce Gaussian Copula Embeddings (GCE), which perform representation learning for data with heterogencouu-type observations of items in contexts. Let {x]...Xx) be the observations.
3

---

Each observation X,, is of a particular item, occurring in context; moreover, each occurrence of the item is observed with multiple feature values (variables), Let Xn æn denote that the item occurs at location n and is observed carrying J variables Fn = p.......... sn.J ( where the location n comes with a context Cn which contains a collection of context item indices.
The principle of the GCE model is that the multiple heterogeneous observations will be generated based on the relationship of the central item its context, characterized by several embedding vectors, and the dependencies between the observations will be characterized by Gaussian copula. For each central item there is an underlying embedding vector P; R 1xx1, In addition to occurring as central item, each item may also occur as part of a context. The context Cn of the location will contain several items i'. The roles of the items in the context will be characterized by context vectors: unlike a traditional embedding model that generates only one type of observation, here each context item? i' has a set of variable specific context vectors {ar J.).
We develop the GCE model based on the latent representation equations the underlying embedding vector P; E IR Kx1 is generated from multivariate normal distribution
10
For each item i
p;
N(0,
(11)
Following the latent variable representation of Gaussian copula, all J observations of an item at location will be generated based from latent vector: The latent variable vector 2, is generated as
zn
N( Rn = (0 N (R...::
(12)
where the matrix R. RR KXJ is constructed based on the embedding vectors of the items in the context Cn for all observation variables. For each observation variable j. the corresponding column
Tn. of the matrix Rr constructed
Tn.j
Eays at'
(13)
ccal
deEnn
where i' are the items in the context of the location n, Cn. Here, for simplicity, set the prior of a to be a multivariate normal distribution
OF ~ N(0, \T
(14)
with diagonal covariance matrix where the constant Aa is a precision parameter which controls the constraints on Using the exchangeability in equation (12). the above generating process can be also written as
1 (( (6) I), where 1a = 1 Mnnal and Png
Σ
ors
(15)
S'Ec.
The observations then obtained from the latent variables according the Gaussian copula equations. The jth observed value (3) nn is obtained as
1 (1
n.
3
(16)
Σ
j.j.
where 2n.j is the jth element of the latent vector zn and Tnjjk is the kth dimension of the context representation column Tn.j- and F is the inverse CDF of the marginal distribution of variable j. Inference of the embedding parameters based on the observations will be done with an extended likelihood approach introduced in the next section.
The major difference between our approach and other embedding models is that the GCE can take heterogenous, multivariate observed data into account. Another difference is that GCE further specializes the roles of embedding vectors p and context vectors a: in traditional exponential family embeddings their roles can be seen as ambiguous, having the same form and a very similar place the generative equations.
contrast, here the roles are made distinct: the p are used to model a general representation of each item as a central item, and the variable-specific context vectors a govern the role of each item as a context item for the multiple variables, telling how the different variables interact with the central item well as controlling the dependencies between the variables.

---

Inference
The task of the inference is to fit the embedding parameters p and a of all items the observations. We will use a variational inference approach; key aspect of it is to replace direct evaluation of the likelihood by an extended likelihood approach described next.
4.1 Extended Likelihood
As pointed out by [7]. the naive inverse of equation (66) during the inference is only recommended when the observed data continuous and follows an easy-to-invert parametric distribution. Moreover, directly inverting the CDF for discrete variables can lead to pushing the latent variables to extremes and affect the validity of the inference. Therefore, an extended rank likelihood has been proposed for the inference of Gaussian copula.
Consider all observations En,j of variable j at all locations 1....,N, NN, and the corresponding latent variables 2n.j where for brevity we drop the item indices i; each observation may arise from a different item. Denote the latent variables together by vector 2j and observed variables together by vector xj. Since each žn.j is related to the corresponding observed In.j by two monotonic functions (a gaussian CDF and inverse CDF), the rank order of the Zn.j is the same as that of the Un.j- The vector 2, is one of a set (x,y) of vectors having that rank order:
Zj Dxx)) {z, E ERR. Inn < Zn' 2nd < Zn'
(17)
The (x,) is the set of possible z, = (ZI.)...,, Fn.)) which preserve the ordering of the observed data 20 Let D = {Z E RJXN 3j E D(x,) LA S S J} be the set of possible latent variable combinations, such that the rank order satisfied for each observed variable j. Note that ties may happen in the rank orders, for example when discrete variables yield the same value at multiple observations.
In the Gaussian copula model the observed variables are directly obtained as unique transformations of the latent variables. Because the possible latent variables must satisfy the rank orders of the observations, the rank perservation can be inserted into the full likelihood which can then be factorized
FI FJ) P(X, Z e D Fy) =P E D/C) P D,C.F,,...,
(18)
The P(Z DJC) is then taken the alternative likelihood. It has been proved [ that it shares same information bound as using estimator the full data. Ranks also bring an advantage of robustness they unaffected by precise value and thus are less prone outliers. Using an extended likelihood based on the rank of observed data is the current state the art practice for Gaussian copula inference [15 3
4.2
Amortized variational autoencoder
Most of the previous works taking the extended likelihood use Gibbs sampling for inference 15| Despite some recent improvement such [9, the sampling process inevitably must update the latent variables for every location n,, which makes the computation inefficient when the volume of data grows large. To avoid this inefficiency, develop stochastic variational inference procedure exploiting the idea amortized inference [4] and a variational autoencoder [II]
Since the essence of the estimator using P(Z E D(C) is to keep the ranking of zj corresponding to the ranking of æj. we then employ the Plackett-Luce model with ties 24 an alternative likelihood. Let r(In..) denote the rank of Ænjj and ræz;) denote the vector of rankings corresponding to æj. We then have
p(((,))),)) = II Enn.
(19)
9 Envec @Pala
where g in the product goes over the rank positions, Ag = {n r(En.) = q) denotes the items ranked position g (there may be more than one due to ties), and C2 (n r(xn.)) 2 q) are items ranked at q or higher. Note that if there are ties, equation (99) reduces to the standard
5

---

Plackett- Luce distribution. The ties are considered in order to handle discrete data. The likelihood for all variables j is then simply II, log p(g((r())) Moreover, the likelihood can be optimized using a stochastic procedure with a random subset of each æ..
Let us consider again the J latent variables per location, denoted 2n (6) To avoid exhaustively updating every (! follow the framework proposed by M. According to the "reparameterizzttion trick", equations [3 can be rewritten as
+ c
( N(O.)).
(20)
Therefore, during the stochastic inference. the latent variables 2 can be simulated based on the other parameters and the random noise. This will save computation and memory because the ā only need to be simulated at the subset of positions samples in the ongoing mini-batch of optimization. The
noise is first sampled from standard normal distribution and pseudo variable ă is then generated as
7 = p + (1)
(1) ~ I).
(21)
where ja Pi Ccn
Evec, O. and P and & are point estimates of the embedding vectors which are optimizing. The log- ligliiee function becomes C(p, x; Ej log logp(r(z;))z)) and the objective function is computed as
F = F= ax + Σ Logg((pi) + ΣΣ log p(av,)).
(22)
In each iteration, the gradients of the log-likelihood with respect to p and á can be simply obtained with à by chain rule via V vpC E; alog p(a; az; 0z, Up and va Blog p(æ; [1,) 0z, 0% Da, The p(p:) and p(ay)) are set to N(O. I) and N(0, A-1I) according to equations 11 and 14p. The gradients of alogp(a)
the log-priors are Σ: pleg and Σr Σ, 80,', respectively. The gradient update " or à with respect to F is then the sum of the corresponding likeliooddo and log-prior gradients.
Sp,
The complete stochastic inference procedure is given in Algorithm 1 We optimize the embedding vectors iteratively over epochs. In each epoch data is partitioned randomly into mini-batches, and negative samples are generated for each batch in addition its positive samples: a negative sample has the same location and context as a positive sample but a randomly chosen different item i, and its observed variable values are all set zero since the item did not occur at that location. The items for the negative samples are chosen from distribution proportional to a power of the overall item distribution, as is done in word embedding [14. In experiments we use M 1000 bini-batches and 5 negative samples for each positive sample. Due to the stochastic partitions, for each epoch the latent variables only need to be simulated at the positions in the mini-batch. The optimization then updates the embedding vectors in each epoch by gradient steps with step sizes chosen by the Adam optimizer.
5 Empirical Case Studies
In this section we describe 5 different scenarios of using GCE to model the observed data. The precision parameter Aa is set to 0 corresponding very wide prior for We have also tried another setting with a constrained prior, they yield similar results (see supplementary materials).
5.1 Product rating data
Data. The Anime rating data is set of user ratings on anime movies and series collected from myanimelist.net. It contains 17562 different anime rated by 325770 different users. Unlike typical product rating data sets, the data set provides how many episodes (integer) the user had watched when rating the anime (discrete), thus there are multivariate heterogencous observed variables.
Modeling. We evaluate GCE by comparing with two other models on their capability to predict held-out ratings. Compared models are exponential family embeddings (Poisson distribution, p-emb) and Poisson matrix factorization model ((2), Pois-MF). The p-emb is selected as a comparison method

---

Algorithm 1: Inference Algorithm
input :Observations { ... N Context (c1... (c....e)) initial learning rate &
output: : Point estimates of embedding vectors
and context vectors â
foreach epoch do
Divide input data into M random partitions.
Generate negative samples.
for f 1 to M do
Use the m-th batch of the data
Simulate 2n with (15) for every in the mini-batch
Compute gradients V v Ej
logg p(æ; (R;)
02;
0zj
8p
+
Σ:
9 log
Up:
Va, plagpia, a5
+ +2 E; d
d0,
Update p and a with p p {* * F and & is set with Adam 10
a - V
end end
Table 1: Left: Held-out MAE for Anime rating. Right: Held-out MAE for Match Records.
Kills
Deaths
Model
K = 50
K = 100
Model
K 50
K = 100
K = 50
K = 100
Pois-MF
7.4866
7,4872
n-emb 30,0125
31.3224
29.4128
30,6495
p-emb
3.0857
3.0691
p-emb
32.5609
32.5409
32.5620
32.5433
GCE
1.2170
1.2207
GCE
14.0867
14.0766
13.5245
13.5106
since model can been
extension of exponential family embeddings; the Pois-MF is model for the user-rating scenario and it was also a compared method to p-emb in [19]. For all methods, when training the model, we hold out 10% of the data the testing data set, and the trained models are used to predict the ratings in the test data. For each anime rated by a specific user, other anime rated by the same user are context. We train GCE with two variables, anime rating and number watched episodes, whereas p-emb and Pois-MF which can only model one variable are trained model the anime ratings.
To compute the predictive ratings we use 16) with Znd set to the means [n.. computed from optimized embedding vectors and with F computed the inverse of the empirical CDF in the training data. The held-out mean absolute error (MAE) used as the performance metric. The results are shown in the Table (Left): our model strongly outperforms p-emb and Pois-MF.
5.2 Player modeling in online games
Data. The HLTV match record data is collected from HLTV.org and records professional match histories multiplayer first-person shooter game Counter-Strike: Global Offensive [22]. We used a web crawler to gather the histories of 34900 matches of 4751 professional players from the website. For each match, we collect the match ID, player ID, and records of each player in the match including number of kills and deaths, again yielding multivariate heterogencous observations.
Modeling, We train GCE learn representation vectors for each player. The context for the player in each match is the set of other players in the same match. Observed data are the numbers of kills and deaths. We again compare our model to two exponential family embeddings (normal and Poisson) because they are the methodologically closest approaches. We train GCA incorporating the two variables the same time whereas the exponential family embeddings train the model for each variable separately. The predictions for the variables are done in Section 5. We measure MAE for both variables separately. The results are shown in Table T (Right): we strongly outperform the comparison methods.

---

Table 2: Results for Darknet Traffic Classification
Model
Precision
Recall
F1
Accuracy
DeepImage 6
0.86
0.86
0.86
0.86
Random forest (original features)
0.8374
0.7963
0.8117
0,8917
GCE (K 20) + Random forest
0.8955
0.8789
0.8846
0.9347
GCE (K 30) + Random forest
0.8945
0.8803
0.8851
0.9355
GCE (K 40) + Random forest
0.8952
0.8786
0.8844
0.9346
Idio Max
Iote Max
Idle Mean
Src. Port
Src Port
kdle Meen
FlowPackes
Fidd Packets
V19
FlowlAT
FlowIATMax
Pow Bytes 8
Packet
Rwd Mea
Bwd Packets
V15
Flow MATM Mean
V10
FWD
Flow.Duratio
Flow Dursbon
V21
Header, Lnnth
JAT
Flow
Fed MATM
Flow (iT
Packet
Fed Seg. Size Min
V1
2500
5000
7500
2500 5000
7500
Figure I: Importance of top-20 variables in the random forest classifiers, x-axis is the mean decrease of Gini impurity, the higher the more important the variable. Left: Variable importances in the random forest for original input variables. Right: The random forest model trained with additional 30 learned features. Learned features V28, V19, and other features are in the 20 most important variables.
5.3 Internet traffic classification
Data. The CIC Dark-net traffic data set (6 contains 141532 records of darknet traffic. Each record categorized into a traffic category (Audio-Streamm, Browsing, Chat, Email, P2P, Transfer, Video- Stream, and VOIP, categories in total) and contains the source IP, destination IP, and communication observations such as forward and backward bytes, flows, duration, subflow and so on.
Model. We use GCE to learn the latent representation for each source IP. In each traffic record, the destination IP the context the source IP. The observed data used to train the GCE model are IP co-appearance (Boolean), TCP Flag counts (SYN, RST, PSH, and ACK; integer), and three subflow related measurements (continuous value). We first train the embedding model with different vector dimensions. After training GCE, incorporate the learned representation vector of each source IP as additional features to the original input variables and train a random forest classifier to predict the traffic category. We compare our model to DeepImage, which is convolutional neural network based, end-to-end solution proposed in (6. The results in Table 2 show that the learned additional features not only improve performance of the random forest classifier, but also outperform DeepImage, the state-of-the-an art deep learning based classifier. Figure further demonstrates that the learned features play important roles in the classification task.
5.4 Graph embedding with node meta-data
Data. Spanish Twitch gamers is subgraph of the Twitch gamers graph data [17] each node is Twitch gamer; an edge denotes mutual friendship. The data has 5538 nodes and 85893 edges.
Model. We train the embedding vectors incorporating the node-level observations including number of views and life duration. Following the customary graph embedding procedure, we first generate random walks on the graph to simulate node sequence input data: 80 walks per node with length 10 steps. Conventionally, limited by the capability of embedding models, most graph embedding models only take appearance of nodes into account. With GCE, incorporate not only the appearance of nodes but also the views and lifetime of the nodes into the model.
To evaluate our model, we take on link prediction, classical task for graph embedding models. We hold out 50% of edges randomly into a test set while keeping the remaining training graph connected. In both training and test sets, randomly sampled negative edges are added in equal amount to the

---

Table 3: Results for Link Prediction: area under the curve (AUC) of the link classification
Model
Deepwalk
Node2vec
EFGE-bern
EFGE-pois
EFGE-norm
GCE
K = 50
0.7151 0.7063
0.7143
0.5795 0.5887
0.5934 0.6004
0.6063 0.6291
0.7853
K 100
0.6612
0.7832
Table Example GCE model output with the subreddit hyperlink network data. We show the top closest subreddits in terms of the embedding vectors p and three different context vectors. The a corresponds the fraction of the characters, 02 corresponds to the fraction of the digits, and ay corresponds to the semantics.
Top 4 closest subreddits
Embedding
r/environment
p
r/corbbreadiberals r/climate r/conservation
r/basicincome
r/energy r/oil
r/northcarolina
r/green
r/water r/tdes
Q22
r/climate
r/likeus
Or7
r/inaasivespecies
r/lockcarbon
r/metageopolities
rlearthdisaster
r/cryptocurrency
P
r/liccciin
r'noblecoin
r/siacoin r/ripple
r/bicconnseriou
011
r/altcoin
r/blackcoin
r/karmacoin
0g
r/cannabis
r/xdp
r/flappycoin r/ethdev
r/litecoin
07
r/dogenews
r/ethtrader
r/vos
positive edges. logistic regression classifier is trained based on the reduced training graph and the training negative edges, using Hadamard product of embeddings of the edge endpoint nodes as input features; the classifier is used to classify the held-out test-set edges. We compare our model otate-of-the-art, random walk based solutions including Deepwalk [16]. Node2vec [5] and Exponential Family Graph Embeddings (EFGE; 1]. The results in the Table show our model outperforms the other competitive models.
5.5 Social media community interactions
Data.The Reddit Hyperlink Network 12 is a data set 858488 hyperlinks between 55863 subreddits. For each hyperlink, the data set records the source and destination subreddit, and the description of the hypertext including, e.g., number of words, sentiments, fractions of 5 different character types (i.e., alphabetical, digits, uppercase characters, special characters, white space) and on.
Model. We train the model based on the pairs of source and destination subreddits. The source subreddit in each hyperlink is the context for the destination subreddit. The 5 fractions of different character types, the number words, and the sentiment are taken as the observed variables. We train GCE model with K 100.
We demonstrate the closest subreddits based on embedding vectors, and three context vectors. Fach reflects a different aspect: take the r/environment for example, when it comes to fraction of alphabetical characters, closest subreddits are related resources such as r/elimate, r/green, and r/oil, and r/water. However, when it comes to sentiment, the closest subreddits for r/environment become rfinaaiveespecies r/lockcarbon, r/metagcopoliitis, and reearthdisaster.
Discussions and Conclusions
We introduced Gaussian copula embeddings (GCE), representation learning model that can in- corporate observed data of different data types. A stochastic variational inference algorithm based on semi-parametric estimation for efficient computation introduced. The empirical case studies demonstrate that our model is effective in many domains outperforming competitive comparison methods, and can provide analytical insights. Moreover, our model can extend the representation learning task to more complex settings and thus bring more opportunities to the research community.

---

ndtsports
moba
rsportsbook
asyoball
mobaskataajaaaaa abioaa
nyylnicks
indoommeial
omelalmemes
niatannt wars
nihuuddr
inbaspurs
rposthardoore
chopubllootruss
sradichhad
ripoggockertstteh (hoolighlers
iituuusbeats
eoondseeaasquuuees rlwnniors sheat
stedhotehilppppees rigortiia
ridlasscc
iiuuoabss
rthostrokes
steninghcads rlunband riuibandgvitallttteer duliiideeandd
wingtrajjedoors
rilllispalo.zz
rimashupe
(RRPARRS
ricorposer
Figure 2: t-SNE visualization of embedding vectors p. The green contains the subreddits related to basketball and green area contains subreddits related to music. The learned representation from GCE are semantically meaningful.
In this paper we used streightforward parametric construction of context and its combination with embedding vectors; however, the GCE framework can be flexibly adapted to other parameterizations, such as integrating as a layer within deep learning architectures, and integrating context selection mechanisms such as [13]. Our method brings new way of analyzing data through vectorial embed- ding which has the potential to bring greater understanding of several phenomena; as usual such tools must be used responsibly to avoid negative societal impact.
Acknowledgement
This work is supported by the Academy of Finland decisions 312395 and 327352.
References
[1] A. Celikkanat and F. Malliaros. Exponential family graph embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3357-3364, 2020.
[2] D. Cortes. Fast non-bayesian poisson factorization for inplicit-feedback recommendations. arXiv preprint arXiv: 2018
[3] R. Cui, P. Groot, M. Schauer, and Heskes. Learning the causal structure of copula models with latent variables. 2018.
[4] S. Gershman and N. Goodman. Amortized inference in probabilistic reasoning. In Proceedings of the annual meeting of the cognitive science society, volume 36, 2014.
[5] A. Grover and Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference Knowledge discovery and data mining, pages 855-864, 2016.
[6] A. Habibi Lashkari, G. Kaur, and A. Rahali. Didarknet: A contemporary approach to detect and characterize the darknet traffic using deep image learning. In 2020 the 10th International Conference on Communication and Network Security, pages 1-13, 2020.
[7] P. D. Hoff. Extending the rank likelihood for semiparametric copula estimation. The Annals of Applied Statisties, 1(1):265 2007.
[8] D. Hoff, X. Niu, and J. A. Wellner. Information bounds for gaussian copulas. Bernoulli: official journal of the Bernoulli Society for Mathematical Statistics and Probability, 20(2):604, 2014.
[9] A. Kalaitzis and Silva. Flexible sampling of discrete data correlations without the marginal distributions. Advances in Neural Information Processing Systems, 26, 2013.
10

---

[10] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. 2015.
[11] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 13 1312.6114
2013.
[12] S. Kumar, W. L. Hamilton, J. Leskovec, and D. Jurafsky. Community interaction and conflict on the web. Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 333-943. International World Wide Web Conferences Steering Committee, 2018.
[13] L. Liu, F. Ruiz, S. Athey, and D. Blei. Context selection for embedding models. Advances in Neural Information Prucessing Systems, 30, 2017.
[14] Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their conpositionality. In Advances in neural information processing systems, pages 311 11111119, 2013.
[15] J. S. Murray, D. B. Dunson, L. Carin, and J. E. Lucas. Bayesian gaussian copula factor models for mixed data. Journal of the American Statistical Association, 108(502) 2013.
[16] B. Perozzi, R. AI-Rfou, and S. Skiena. Deepwalk: Online learning of social representations. In Proceedings the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pages 701-710, 2014.
[17] B. Rozemberezki, R. Davies, R. Sarkar, and C. Sutton. Gemsec: Graph embedding with self clustering. In Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2019, pages 65-72. ACM, 2019.
[18] M. Rudolph, F. Ruiz, S. Athey, and D. Blei. Structured embedding models for grouped data. Advances in neural information processing systems, 30, 2017.
[19] M. Rudolph, F. Ruiz, S. Mandt, and D. Blei. Exponential family embeddings. In Advances in Neural Information Processing Systems, pages 478-486, 2016.
[20] J. Segers, R. Van den Akker, and B. J. Werker. Semiparametric gaussian copula models: Geometry and efficient rank-based estimation. The Annals of Statistics, 42(5):191- 2014.
[21] M. Sklar. Fonetions de repartition an dimensions leurs marges. Publ, inst. statist. univ. Puris, 8:229-231, 1959.
[22] H. P. E. Valve Corporation. Counter-strike: Global offensive, 2012.
[23] I. Waller and A. Anderson. Quantifying social organization and political polarization in online platforms. Nature, 600(7888):264 268, 2021.
[24] R. C. Weng and C.-J. Lin. A bayesian approximation method for online ranking. Journal of Machine Learning Research, 12(1), 2011.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] Provided in supplementary material.
Did you include the license the code and datasets? [No] proprietary.
The code and the data are
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In supplementary.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rre using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "KF5dmybX7Zo", "reviewer": "Reviewer_jaJh", "summary": "This paper proposes using Gaussian copula to learn the embeddings under heterogeneous data setting. A rank likelihood is proposed to maintain the rank correspondence of the observed variables and the latent variables. The evaluations on heterogeneous data are sufficient to show the efficacy of the proposed methods.", "questions": "1. Why there is no KL term in the training objective of variational auto-encoder? If there is only log-likelihood, probably it should not be called ''variational'', just ''amortized inference''.", "limitations": "There is not negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using Gaussian copula for embedding learning is novel and interesting. As there are heterogeneous data types in real worlds, the Gaussian copula could help to naturally capture the underlining correlations with one model. This embedding learning mathematical tool could be broadly influential to various application domains.\n2. The author also propose an order preserving likelihood and the corresponding variational auto encoder, for efficient and effective inference.\n\nWeaknesses:\n1. It seems there is no ablation study in the evaluations. As the model contains three parts: copula, order likelihood and amortized inference, it is expected to have a thorough ablation to show what improvement each part brings.\n\n------ \nThe author addressed my concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "83kywO1Xpa", "reviewer": "Reviewer_pvbn", "summary": "The authors propose a Gaussian copula embedding model to learn latent vector representations of items in a heterogeneous data setting. The proposed model can effectively incorporate different types of observed data and, at the same time, yield robust embeddings. In the experiments, the authors show the proposed model can effectively learn in many different scenarios, outperforming competing models in modeling quality and task performance. \n", "questions": "What are the advantages of the GCE  in the comparison with deep neural nets-based embedding methods?", "limitations": "\nExperimental comparison with neural network-based methods, e.g. paper[1] and ELMO,  are needed to make the paper stronger. \n\n[1]Vashishth, Shikhar, et al. \"Incorporating syntactic and semantic information in word embeddings using graph convolutional networks.\" arXiv preprint arXiv:1809.04283 (2018).\n\n\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n\n1. The paper is well written with good structures. \n\n2. The flow of the technical sections is clear and easy to follow. \n\n3. Extensive experiments have been conducted to validate the method. \n\n\n\nWeakness:\n\n1. The contribution of the paper is insignificant. The authors apply Gaussian Copula to embedding problems and try to use it to capture interactions in heterogeneous data, and the idea is simple and direct. \n\n2. The authors did not provide a deep explanation of why the Gaussian copula is better than other approaches.  Other embedding methods can also be used for heterogeneous datasets.  I believe by leveraging deep neural nets many existing methods can also learn the interactions in heterogeneous datasets.\n\n=====\n\nThe authors addressed some of the concerns and clarified the benefits of the proposed method.\n\nHowever, the technical and theoretical contributions of the paper are not significant. \n\nI increase the score by 1. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7xNXqtNmlk8", "reviewer": "Reviewer_SEhr", "summary": "The authors present a model for embedding discrete items such that they model the distribution of multiple heterogeneously distributed context dependent observations. This method employs latent variables that are distributed as multivariate Gaussians, and use Gaussian copulas to capture the dependence between dimensions. The paper presents a key insight that these Gaussian copula models can be used as part of an embedding approach. The paper relates this to previous embedding approaches including word2vec and the more general exponential family embeddings. The paper then presents the model and an inference approach based on the extended likelihood. There are a series of experiments on a wide variety of domains and the results show that the proposed method significantly outperforms comparator methods for each of these.", "questions": "For the experimental benchmarks:\nAre these the best existing methods, or ones with readily available implementations, or what? \n\nAlso, Graph embedding with node meta-data is missing key information. How is the data interpreted as item predictions with context?\n\nOn the authors construction of context: are you able to relate this notion of context to other papers/models. Context doesn't just appear in the papers cited but does appear quite widely elsewhere in deep learning and beyond. I wonder if the contribution here is potentially more widely relevant.\n\nI am not sure whether $r_{n,j}$ Equation (13) is as general as it could be? This relates to Equation (4) but I am left wondering what the meaning of making context averaged in this way is, particularly in the context of the authors' own model. For instance, in the HLTV data (Sec. 5.2) the presence of either one of two particularly lethal players in a game may greatly reduce other players scores, but the presence of both players may not have \"double the effect\" (it may be more or less than that). Could the authors maybe comment on the justification and/or limitations of this?", "limitations": "Right at the end of the paper the authors state:\n> [A]s usual such tools must be used responsibly to avoid negative societal impact.\n\nPerhaps they could indicate what the concerns are here a little more explicitly.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "One significant characteristic of the presented model is that context is first represented as a collection of items (discrete entities) each of which carries its own context embedding vector for each observable dimension of the data. There has been a number of deep learning models over recent years that represent the context of observations in some form or another (as is discussed to some degree in the paper) but this seems to me to be a particularly neat and flexible way. Although the authors do relate this context approach to word2vec, node2vec and Exponential Family embeddings, the paper could do more to relate more generally to other (deep learning) models that employ a notion of context. I am not sure precisely how this relates to what has gone before but I would be interested to know. \n\nThe authors run a series of experimental evaluations on different data. The details of these are mostly well described, but some justification for the choice of comparator methods/benchmarks could be given. There does appear to be some missing information (see Questions).\n\nNonetheless, this is a really interesting approach and, to my mind, represents a significant advance in flexiblity for embedding models. The experiments appear to be well designed, test a broad array of domains and the performance improvements with respect to existing methods are convincing. I believe this is a strong paper and will be of interest to a number of communities, not just those directly interested in embedding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept