Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a Gaussian copula embedding model to learn latent vector representations of items in a heterogeneous data setting. The proposed model can effectively incorporate different types of observed data and, at the same time, yield robust embeddings. In the experiments, the authors show the proposed model can effectively learn in many different scenarios, outperforming competing models in modeling quality and task performance. 

strengths_and_weaknesses: Strengths:


1. The paper is well written with good structures. 

2. The flow of the technical sections is clear and easy to follow. 

3. Extensive experiments have been conducted to validate the method. 



Weakness:

1. The contribution of the paper is insignificant. The authors apply Gaussian Copula to embedding problems and try to use it to capture interactions in heterogeneous data, and the idea is simple and direct. 

2. The authors did not provide a deep explanation of why the Gaussian copula is better than other approaches.  Other embedding methods can also be used for heterogeneous datasets.  I believe by leveraging deep neural nets many existing methods can also learn the interactions in heterogeneous datasets.

=====

The authors addressed some of the concerns and clarified the benefits of the proposed method.

However, the technical and theoretical contributions of the paper are not significant. 

I increase the score by 1. 


questions: What are the advantages of the GCE  in the comparison with deep neural nets-based embedding methods?
limitations: 
Experimental comparison with neural network-based methods, e.g. paper[1] and ELMO,  are needed to make the paper stronger. 

[1]Vashishth, Shikhar, et al. "Incorporating syntactic and semantic information in word embeddings using graph convolutional networks." arXiv preprint arXiv:1809.04283 (2018).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes using Gaussian copula to learn the embeddings under heterogeneous data setting. A rank likelihood is proposed to maintain the rank correspondence of the observed variables and the latent variables. The evaluations on heterogeneous data are sufficient to show the efficacy of the proposed methods.
strengths_and_weaknesses: Strengths:
1. The idea of using Gaussian copula for embedding learning is novel and interesting. As there are heterogeneous data types in real worlds, the Gaussian copula could help to naturally capture the underlining correlations with one model. This embedding learning mathematical tool could be broadly influential to various application domains.
2. The author also propose an order preserving likelihood and the corresponding variational auto encoder, for efficient and effective inference.

Weaknesses:
1. It seems there is no ablation study in the evaluations. As the model contains three parts: copula, order likelihood and amortized inference, it is expected to have a thorough ablation to show what improvement each part brings.

------ 
The author addressed my concerns.
questions: 1. Why there is no KL term in the training objective of variational auto-encoder? If there is only log-likelihood, probably it should not be called ''variational'', just ''amortized inference''.
limitations: There is not negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a model for embedding discrete items such that they model the distribution of multiple heterogeneously distributed context dependent observations. This method employs latent variables that are distributed as multivariate Gaussians, and use Gaussian copulas to capture the dependence between dimensions. The paper presents a key insight that these Gaussian copula models can be used as part of an embedding approach. The paper relates this to previous embedding approaches including word2vec and the more general exponential family embeddings. The paper then presents the model and an inference approach based on the extended likelihood. There are a series of experiments on a wide variety of domains and the results show that the proposed method significantly outperforms comparator methods for each of these.
strengths_and_weaknesses: One significant characteristic of the presented model is that context is first represented as a collection of items (discrete entities) each of which carries its own context embedding vector for each observable dimension of the data. There has been a number of deep learning models over recent years that represent the context of observations in some form or another (as is discussed to some degree in the paper) but this seems to me to be a particularly neat and flexible way. Although the authors do relate this context approach to word2vec, node2vec and Exponential Family embeddings, the paper could do more to relate more generally to other (deep learning) models that employ a notion of context. I am not sure precisely how this relates to what has gone before but I would be interested to know. 

The authors run a series of experimental evaluations on different data. The details of these are mostly well described, but some justification for the choice of comparator methods/benchmarks could be given. There does appear to be some missing information (see Questions).

Nonetheless, this is a really interesting approach and, to my mind, represents a significant advance in flexiblity for embedding models. The experiments appear to be well designed, test a broad array of domains and the performance improvements with respect to existing methods are convincing. I believe this is a strong paper and will be of interest to a number of communities, not just those directly interested in embedding.
questions: For the experimental benchmarks:
Are these the best existing methods, or ones with readily available implementations, or what? 

Also, Graph embedding with node meta-data is missing key information. How is the data interpreted as item predictions with context?

On the authors construction of context: are you able to relate this notion of context to other papers/models. Context doesn't just appear in the papers cited but does appear quite widely elsewhere in deep learning and beyond. I wonder if the contribution here is potentially more widely relevant.

I am not sure whether $r_{n,j}$ Equation (13) is as general as it could be? This relates to Equation (4) but I am left wondering what the meaning of making context averaged in this way is, particularly in the context of the authors' own model. For instance, in the HLTV data (Sec. 5.2) the presence of either one of two particularly lethal players in a game may greatly reduce other players scores, but the presence of both players may not have "double the effect" (it may be more or less than that). Could the authors maybe comment on the justification and/or limitations of this?
limitations: Right at the end of the paper the authors state:
> [A]s usual such tools must be used responsibly to avoid negative societal impact.

Perhaps they could indicate what the concerns are here a little more explicitly.
review_title: null

Meta-review: metareview: 
This paper was a difficult case, where numerical review scores might be in disagreement with several statements during the discussion phase, and also with some of my comments below.
Let me explain this in more detail.
All reviewers mentioned several strong points, such as:
- the general novelty of the concept of using copulas for learning embeddings
- the overall soundness and clarity of writing
- interesting conceptual parts, for instance the proposed ordering-preserving likelihood 
- good experimental validation

On the other hand, also several negative points have been raised by the reviewers, and some of them remained valid after the rebuttal. To me, the most important of these negative points is the statement by one of the reviewers that "the technical and theoretical contributions of the paper are not significant".

During the discussion phase, we discussed several points that were listed as "strengths" in the original reviews, such as the statement in one of the reviews that "the Gaussian copula could help to naturally capture the underlining correlations with one model." My counter-argument was that a Gaussian copula model can capture the underlying dependencies if and only if the true underlying "pure" dependency after removing the marginal effects (i.e. the copula) was indeed Gaussian. But this might be rare in practice, and an essential part of the copula literature deals with non-Gaussian dependency models that can explain practically relevant phenomena such as tail dependency etc.. 
In the end, we somewhat agreed that the usefulness of Gaussian copulas for this purpose might be not so clear, and that some ablation studies might be needed in order to better understand the contribution of the Gaussian copula to the overall method.

A second question I raised during the discussions concerned the efficiency of the proposed inference method, which had an important role in the paper, since the authors explicitly motivated their work by efficiency problems in "traditional" Gaussian copula estimation procedure. If that really is a main contribution of this paper, I would argue that there should have been a comparison with the Hamiltonian MCMC method for Gaussian copula inference in (Alfredo Kalaitzis & Ricardo Silva, NIPS 2013), which - to my experience - is indeed quite efficient in practice, and this argument was also supported by one of the reviewers.

In the end, to me as an area chair,  this is one of the classical "borderline" cases, where a paper does certainly contain some interesting aspects, but on the other hand, there are also many potential problems and limitations. Since all reviewers finally saw this paper above the threshold, I also recommend to accept this paper, but I still would like to mention that I am not fully convinced about this recommendation.
confidence: Less certain
Decision: Accept