Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to demystify the power of embedding and embedding space, trying to connect the structure of learned embeddings and knowledge generation process. The authors tested Gaussian model, Preferential Placement (PP) model, and Directional Preferential Placement (DPP) model for node generations. The authors also tried to learn embeddings by graph representation learning algorithms after running BA models with several variances with gravitational movement and centroid node/edge weighting.


(Main Contributions)
1) Propose reasonable processes that can explain the evolution of knowledge embeddings in terms of preferential attachment and attractive-repulsive force.

2) Compare observed and generated embedding spaces via non-parametric statistics, identifying frequency concentration and clustering velocity properties.

3) Evaluate and find the best generative process: incremental insertion process with spatial context-dependent preferential attachment.

main_review: (Strengths)
This paper proposes interesting approaches to review popular static word embeddings. The authors try to reconstruct a number of different embeddings based on different node generation models, edge generating models, and their combinations.

1) It is exciting to read the authors’ endeavor on revisiting the old successful embeddings that brought a wide range of breakthrough in NLP.

2) Their approach of viewing embeddings purely as a result of node-only generations vs a product of graph that has both node and edge generations continued by graph embeddings
 seems to be thorough instrument to investigate the embedding spaces.

3) Their findings of frequency concentration (not only highly frequency terms getting richer) and clustering velocity (but also they tend to be together) is useful to support an important observation: there are significant amount of frequency information in learned embeddings.

4) The authors revisit many interesting concepts and ideas in different fields like geography that covers spatial relationships and corresponding philosophy. 


(Weaknesses)
1) The primary concern is that reading this paper may not add useful intuitions in the era of contextual embeddings and large-scale language models. If the paper suggests any type of treatment that may alleviate too much dependency on the frequency information; or that can improve the performance of state-of-the-art language models by feeding better initial embeddings, the research will become a gold. But the current draft itself is less likely to impact on our community.

2) All the models and experiments are designed and operated rather passively by tuning the hyperparameters and parameters then comparing the results with the popular learned embeddings. It is of course useful to have such models analyses. However it is more natural for Machine Learning community to fit each model into the training portion of data and validate it on the hold-out set before making comparisons. Otherwise, we cannot say whether each suggested model actually has some degree of generalizability of the phenomena of interests.

3) Indeed, we already understood in both high-level and low-level that these static embeddings largely encode frequency information though they were often argued to encapsulate semantic information. Usual definition of the context in computational linguistics is a set of words in the sliding window of each word, which inevitably connects to learning frequential/distributional similarities rather than precisely semantic and syntactic similarities. It was also studied that frequent words are more frequently updated in SGD framework, sharing the large norms. If we use Gaussian like kernel, it will be not surprising to see these are grouped.



summary_of_the_review: (Questions)
1) Why do you believe that embeddings are optimized to preserve the natural clusters of the real-world knowledge in its metric space? At least the three embeddings used in the paper never explicitly optimize any clustering objectives. They want the words closely in the raw texts locate closely in the Euclidean space. 

2) Geometry often means the characteristics of the manifold or topological essence defined by a collection of open sets (that will indeed define suitable metric for metric spaces). Some expressions about geometry or invariance would be too subjectively used.

3) It is interesting to see how clustering velocity is defined as area under the curve in the graph of increasing number of connected components. Different types of graphs (maybe different scale-freeness and degree distributions) naturally differs in this clustering velocity, but it is a bit doubtful that this metric can be drastically different even for two graphs with similar degree distributions by adversarially tweaking couple of local connections. As both models and new metrics are proposed, there must be a natural question about the stability and sensitivity of this metric.

4) More active form of contributions that can impact modern representation learning and transfer-based language modeling will benefit the audiences in the community.


(Minor Comments)
A2) left figure  attach the figure reference as the actual figure is located in the previous page.

A3) figure 5 not in the main draft.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
In this paper,  the authors try to study if we can learn about the human's process of generating new ideas or concepts from embeddings. The authors first define a set of models for generating embeddings. Then the authors observe two properties: (1) frequency concentration, and (2) cluster velocity. The authors finally compare the embeddings from generative models with the embeddings from real-world data driven methods.
main_review: 
Strengths.

In general, I find the question the authors try to address is interesting. The author's methodology of approaching this problem seems fine.

Weaknesses.

- One concern I have is the embedding the authors use. Now NLP community is adapting to embeddings from pre-trained models, e.g., BERT. It seems to me that analyzing these contextual embeddings would be better to understand the question the authors propose.

- Somehow I feel like the evaluation between the embeddings from the generative model and from the real-world is rather weak. I guess another more straightforward way is to show the performance on downstream tasks using the generated embeddings.
summary_of_the_review: 
In sum, I think the authors look at an important problem. But there are some limitations of this study, e.g., not looking at contextual embeddings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to explore the natural processes that generate new knowledge or concepts. Specifically, the authors propose two metrics to characterize embeddings trained on different datasets. Then, they compare some synthetic data generated by models and real-world data according to the aforementioned metrics. Finally, they conclude that the real-world data can be well simulated by a certain generative model.
main_review: Strengths

1. This paper studies data generation processes with the help of embedding spaces. It is an interesting problem, and I think the idea of bridging embedding spaces and data generation processes is worth further exploration.
2. The authors give a detailed illustration of several data generation methods.



Weakness
1. The title is a bit confusing. The authors ask a question "why do embedding spaces look as they do" in the title. However, it seems that they did not answer it throughout this submission.
2. The submission is not self-contained. For example, at the bottom of Page 6, the authors put an important figure in the appendix. However, as noted by the ICLR committee, reviewers are not required to read appendices.
3. The linear correlations in the experiments seem to be weak, especially in Figure 2. The authors may want to calculate the Pearson's correlation between Freq. Concentration and ranking.
4. As described in the conclusion section, experiments in this work are not sufficient to identify a single dominant model for reasoning about the origins and structures of natural embedding spaces. I agree that this work gives a good start for an interesting problem. However, since the authors did not give a clear conclusion, the technical contribution of this submission seems to be weak.

summary_of_the_review: This paper studies an important and interesting problem. However, it does not provide a good enough solution to the problem to meet the bar of ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes two measures of word/graph embedding to character the evolution of word/graph, i.e., frequency concentration and clustering velocity. Many existing graph generation models are surveyed. The proposed measures are calculated on many real-world datasets.
main_review: The strengths:
S1: This paper studies an interesting problem, to analyze structure evolution from the geometry in embedding space.
S2: Two novel measures are proposed.
S3: This paper is well-written.

The weaknesses:
W1: There is no problem formulation. It's hard to know what is the computational problem studied in this paper.
W2: It's not clear how do the proposed measures answer the raised problem.
W3: There is a lack of discussion of the rationale of the proposed measures.
W4: There is a lack of empirical validation for the proposed measure.

I have some concerns about this paper. There is no problem formulation. It's hard to know what is the computational problem studied in this paper. It seems that the goal of this paper is only to reveal and explain preferential attachment. It's not clear how do the proposed measures answer the raised problem. What is the intuition of the proposed measure in terms of analyzing structure evolution?  There is a lack of discussion of the rationale of the proposed measures. There is a lack of empirical validation for the proposed measure.
summary_of_the_review: This paper studies an interesting problem and proposes two novel measures. This paper lacks some key elements, such as problem formulation, rationale discussion, and empirical validation. As its too many weaknesses, I have to recommend a reject.
review_title: null

Meta-review: The authors of this work introduced new metrics for node embedding that can measure the evolution of the embeddings, and compare them with existing graph embedding approaches, and experimented on real datasets.

All reviewers agreed that the work addresses interesting problem and that the proposed measures are nove, but there are too many flaws in the initial version of the paper, and despite the thorough responses of the authors, it is believed that there are still too many open questions for this paper to be accepted this year ICLR.
Decision: Reject