Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Authors introduce TriMap based on triplet constraints that preserves the global accuracy of the data. A measure of global accuracy is proposed to reflect the global accuracy of the embedding. Experiments on various datasets the better performance than baselines.

Authors define the minimum reconstruction error from the embedding as the global measure in reflecting the global structure of the data similar to PCA. From the definition, this measure has preferences to the linear projection model such as PCA, so the score becomes lower for non-linear projection methods such as t-SNE and UMAP.  The illustrated S-shape example in Figure 1 somehow demonstrate the difference of the proposed method with PCA, t-SNE and UMAP, but the usage of the embedding is not clear since Figure 1(d) looks like a 2-d visualizing of the original 3-d data visualized from certain angle. In addition, the initialization of the proposed method is the PCA method, which prefers the GS measure. It is interesting to see how the GS measure will change if the random initialization is used. 

Authors demonstrate GS and AUS for all the tested data. It might be interesting and more important to see how to get the better embedding with a balanced score for a given data since GS and AUC seems always opposite measures.

The TriMap method defines the loss of triplet based on unnormalized weighting schema and the weights are adjusted by applying a non-linear transformation that emphasizes the small weights. These formulations are quite heuristic and constructive. It is better to have some formal explanation on the proposed method.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Authors suggest a new technique for embedding point to low-dimensional space. The technique is reminiscent of t-SNE, with the difference that it get weighted triplets (i,j,k) as inputs (meaning that j is closer to i thank). Further a loss function is defined which directly follows t-SNE ideology.

A paper is purely experimental. The only way to judge the quality of its results is to compare 2D pictures of TriMap with other pictures. I did not see any evidence that images of TriMap somehow give a new insight into data (in comparison with other methods).
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposed ``TriMap’’---a novel dimensionality reduction technique that learns to preserve relative distances among points in a triplet. The paper has done extensive experiments and presents the results in very nice visualizations. The paper is clearly written. 

Major merits of this paper are: 

1. The proposed method seems effective. 

On some datasets (e.g., S-curve), the learned low-dimensional embeddings indeed look very good. And the proposed method has much less runtime than other baselines as shown in table-1. 

2. The paper is well written. 

However, I am still leaning towards rejecting this submission because the proposed method is lack of necessary justification. 

1. Many important technical decisions on this method seem arbitrary, including the parametrization of functions (e.g., s, \omega, \zeta, etc) and values of hyperparameters (e.g., \gamma and \delta). For function forms, the authors should justify why particular parametrization has been chosen; for the hyperparameters, the authors should clearly explain how they are picked---maybe using domain knowledge or tuned on data? 

2. The argument for ``global score’’ is not clear enough. There are at least two points that need clarification. 

First, ``non-local’’ is not equal to ``global’’. The proposed method indeed considers non-local (or non-near) points while learning embeddings, which helps preserve non-local information. But I am not convinced that the preserved information is actually global. Maybe what helps is to first define ``global’’ in a dimensionality reduction context. 

Second, the definition of ``global score’’ depends on another baseline method (i.e. PCA), which seems odd. A principled evaluation (or a score) should be method-independent. What seems right to me is to compute global score (i.e. how much global information has been preserved) by comparing to some statistics in the (high-dim) x space, not to another method. 

Moreover, the authors had a strong claim that the proposed ``GS is the only DR performance measure that can reflect this property’’---it doesn’t sound right and why one can’t just use another score which is monotonic wrt the proposed score? The authors mentioned that ``PCA has the lowest possible MRE’’---but this is only right up to the use of a linear transformation and F-norm, so this shouldn’t be a justification for my questions above. 

3. What is the reason for the successful runtime? 

The authors didn’t clarify why the proposed method is theoretically faster than the baselines. 

What I noted is: the authors chose a subset \mathcal{T} for the TriMap method and used \mathcal{T} throughout the paper---is it a typo or is a subset always chosen? If the latter holds, then how was it chosen and how large is it compared to the training data used by other methods? In the end, is the proposed method faster because it uses less data? 

Besides the weakness above, I also suggest the authors evaluate their method with some extrinsic evaluations. What’s currently used is only intrinsic---the embeddings are trained to preserve relative distances and are evaluated on a trade-off between local accuracy and the defined global score. It is fine because extensive visualizations are provided and readers can subjectively judge the quality of the learned embeddings. However, the experimental section can be stronger if the authors can show the learned embeddings are better at helping some downstream tasks than other baselines (by preserving non-local information?). 

review_title: Official Blind Review #3

Meta-review: This paper proposes a new dimensionality reduction technique that tries to preserve the global structure of the data as measured by the relative distances between triplets. As Reviewer 1 noted, the construction of the TriMap algorithm is fairly heuristic, making it difficult to determine how TriMap ought to behave “better” than existing dimensionality reduction approaches other than through qualitative assessment. Here, I share Reviewer 2’s concern that the qualitative behavior of TriMap is difficult to distinguish from existing methods in many of the figures. 

Decision: Reject