Embedding models through the lens of Stable Coloring
Authors: Aditya Desai
Shashank Sonkar
Anshumali Shrivastava
Richard Baraniuk
Venue: ICLR 2022 Submitted (2022)

Abstract: Embedding-based approaches find the semantic meaning of tokens in structured data such as natural language, graphs, and even images. To a great degree, these approaches have developed independently in different domains. However, we find a common principle underlying these formulations, and it is rooted in solutions to the stable coloring problem in graphs (Weisfeiler-Lehman isomorphism test). For instance, we find links between stable coloring, distribution hypothesis in natural language processing, and non-local-means denoising algorithm in image signal processing. We even find that stable coloring has strong connections to a broad class of unsupervised embedding models which is surprising at first since stable coloring is generally applied for combinatorial problems. To establish this connection concretely we define a mathematical framework that defines continuous stable coloring on graphs and develops optimization problems to search for them. Grounded on this framework, we show that many algorithms ranging across different domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain.  We show that popular and widely used embedding models such as Word2Vec, AWE, BERT, Node2Vec, and Vis-Transformer can be understood  as instantiations of our general algorithm that solves the problem of continuous stable coloring. These instantiations offer useful insights into the workings of state-of-the-art models like BERT stimulating new research directions.

Under review as a conference paper at ICLR 2022
EMBEDDING MODELS THROUGH THE LENS OF STABLE COLORING
Anonymous authors Paper under double-blind review
ABSTRACT
Embedding- approaches find the semantic meaning of tokens in structured data such as natural language, graphs, and even images. To a great degree, these approaches have developed independently in different domains. However, we find a common principle underlying these formulations, and it is rooted in solutions to the stable coloring problem graphs (Weisfeiler-Lehman isomorphism test). For instance, we find links between stable coloring, distribution hypothesis in natural language processing, and non-local-means denoising algorithm in image signal processing. We even find that stable coloring has strong connections to broad class of unsupervised embedding models which is surprising at first since stable coloring is generally applied for combinatorial problems. To establish this con- nection concretely we define a mathematical framework that defines continuous stable coloring on graphs and develops optimization problems to search for them. Grounded on this framework, we show that many algorithms ranging across dif- ferent domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain. We show that popular and widely used embedding models such as Word2Vee, AWE, BERT, Node2Vec, and Vis- Transfforrrrr can be understood as instantiations of our general algorithm that solves the problem of continuous stable coloring. These instantiations offer useful insights into the workings of state-of-the-art models like BERT stimulating new research directions.
INTRODUCTION
Embedding models are ubiquitous in wide range of real-world applications such as information retrieval (Zuccon et al. 2015), natural language processing (NLP) (Mikolov et al.. 2013a bb. graph classification (Grover & Leskovec (016)) [016) |aamllon et al. (017)) and many more. These models map categorical entities to continuous dense representations (typically in R which provide a continuous measure semantic similarity across categorical entities. Nowadays, there is a heavy dependence unsupervised pre-trained embedding models across domains like Transformers in NLP (Devlin) al. 2019). Visual Transformers (VIT) in Computer Vision (Dosovitskiy et al.) 2020).. Graph Neural Networks (Hamilton et al.) 2017: Xu et al. 2019b) since they learn rich semantic representations of entities from massive amounts of unlabelled data. With little finetuning, these models achieve state- of-the-art results on most of the supervised downstream tasks like sentiment analysis (Xu et al., 2019a), object detection (Beal et al. 2020), and graph classification (Xu et al.) 2019b)).
Historically, embedding models were developed almost independently across structured domains such as NLP, graphs, images, and so on. These algorithms use the neighborhood structure around an entity obtain the embedding for the entity. Interestingly, a popular hypothesis in NLP- Distri- butional Hypothesis states that the "meaning" of the word is determined by its context (neighbors) (Harris 1954, Sahigren| 2008). This hypothesis forms the basis of most unsupervised embedding learning models in NLP (Mikolov et al. [213aa Pennington et al., [2014b Bojanowski et al.) 2017 Sonkar et al. 2020). Similarly, non-local-ment
a denoising algorithm in signal processing, tries to find pixels that should be the same based on the similarity of its neighborhood structure (patch of the image around the pixel in this case) (Awate Whitaker [2066] Buades et al., 2005). Even Graph neural network (GNN) architectures ensure that information of the surrounding neighbors is systematically incorporated in the embedding of node, even in supervised settings (Hamilton et al.)
2017 Maron al. 2019 Xu et al. 2019b). Thus somehow various communities working across

---

Under review as a conference paper at ICLR 2022
diverse domains have narrowed down an entity's neighborhood structure to define the entity's s meaning. In this paper, we try to establish this common principle on mathematically robust grounds.
Structured domains can be easily represented graphs with relations between entities edges in the graphs. For example in NLP words can be treated as nodes of some graph, and co-occurrence relation between words can be represented as edge. As mentioned before that graph embedding architectures capture the topological structure around the node in the node embeddings, and struc- tured domains can be represented as graphs, this raises the question do embedding models from structured domains like NLP and Vision also operate on some domain induced graph and capture neighborhood structural properties in their entity embeddings since we have seen that embedding models across these domains tend to capture "neighborhood" information?
To answer this question, we dive into combinatorial graph theory to understand how to define the notion of structural equivalence a.k.a. isomorphic structures. Weisfeiler-Lehmar (1-WL) algorithm (or color refinement algorithm) is the most popular heuristic used to identify graph isomorphism (Weisfeiler & Leman 1968) and can distinguish broad class of graphs (Babai & Kucera 1979). The fixed point solution of 1-WL is called a stable-coloring and has the property that any two nodes with the same color have the same multi-set of colors in their neighborhood. In essence it means if two nodes have the same color, the graph looks structurally identical from these nodes.
Finally to answer the question if embedding models from structured domains capture the domain induced graph's topological properties in their entity embeddings, one can find links between the mechanics of these models and stable coloring / 1-WL algorithm. In this paper, we establish this connection by providing a general framework linking existing algorithms to stable coloring. We propose more flexible version of stable coloring (SC) called continuous stable coloring (CSC) ) strict generalization of SC. CSC states that the similarity of their neighborhoods determines the similarity of two nodes. Based on this notion, we define a series of optimization problems to solve the problem of CSC We show that various algorithms in NLP like word2vee, AWE, BERT (Devlin et al. 2019). images processing like Visual Transformer (Dosovitskiy al.) (020). graphs like Vec (Grover & Leskovec 2016), etc. are essentially solving different instantiations of this common optimization problem.
Current research already establishes the link between the 1-WL algorithm (Grohe 2020; Morris jet al.) 202T; [hhrvashidze et al.) [201T] MMorrs et al.] 2017) and GNN architectures, which has sparked new line of research in improving GNN architectures (Hamilton et al. [017]] Xu et al.) 2019b] Maron et al. [2119] Morris et al. 2020a b). We hope that the new link we establish between stable coloring and unsupervised embedding algorithms will also stimulate new exciting research in embeddings for other structured domains NLP and Vision.
2 BACKGROUND
In this section, we define stable colored graph, provide an outline for 1-WL (Weisfeiler-Lhhmee graph isomorphism test and General Aggregate and Update (GAU) for Graph Neural Networks (GNNs). We also discuss how domains of NLP and images can be seen as graphs.
2.1 STABLE COLORING
Let a coloring function C be an overloaded function defined on the vertices as well on set of vertices
of G (V. 8) i.e., C V N and 2 + ({N)) } where N is a set of natural numbers
representing colors, and {{}}} is multiset with overloading defined as C(A) = ([C(o)le e A C v]}. We denote neighborhood of a node E V as (w)) {u](E, u) E E).
Definition 2.1 (Stable Coloring). An undirected graph G (V,8) is stable colored w.r.t coloring function C if it holds that C(u) = C(v) if and only if C(N(u)) = C(N())).
We can extend the above definition to directed graphs and graphs with labels. A directed graph G = (V, () is stable colored w.r.t coloring function C V- if it holds that C(u) = C(v) if and only if C(N, = (u)) and C(Nour()) = (()oue(v) where Nu)(u) = [w] u) E E) and Nour(w (u) [w](u, w) E E}. Likewise, an edge-labelled undirected graph g (V,E) is stable colored wint a coloring function C V + N holds that C(u) = C(u) if and only vI, C(N(()) C(M())) where Ni(u) (w](u, w) E E where & C E for edge label I.
2

---

Under review as a conference paper at ICLR 2022
We also define a weak-stable coloring for graph 9 EV, An undirected graph 9 = E) E) is weak-stable colored w.r.t coloring function it holds that C(u) = C(u) if ((N(w) - C(V())) where (a) {w] w) E E}.
2.2 1-WL ALGORITIM AND GENERAL AGGREGATE AND UPDATE FRAMEWORK (GAU) )
I- algorithm: 1-WL is iterative algorithm to achieve stable coloring C for G = E) Let C, denote the coloring at iteration i. It starts with a coloring scheme Co such that Cy(v) is same for all V. In each iteration, it assigns different color and v if (N)()) * (N())) until stable coloring C is reached.
General Aggregate and Update Framework (GAU) For a general multi-layer GNN, the General Aggregate and Update framework to compute node/vertex embeddings (corresponding to colors) of (V, is given iteratively by:
E (u) = ((A)
((k) E Eu) Sag
1E--((w) : (((())))
apdate
where ((gg() and updatte (.) map vertex multiset embeddings to a metric space such as Rd
In GraphSage (Hamilton et al. 2017), authors showed that the iterative procedure in 1-WL algorithm is analogous to General Aggregate and Update procedure in GNNs. This connection has lead to the research direction where 1-WL is being used as a standard to which GNN architectures are being compared to (Xu et al., 2019b Maron al. (2099] Morris et al.) 2020b). Xu et al. (2019b) prove that GAU is as powerful as 1-WL. L. the functions Juddate and fagg are injective.
Various domains such NLP, Images and Graphs be viewed graph their elementary tokens. How we construct graphs is explained in section 6.1 and examples are given in appendix
RELATED WORK
Word embeddings have been popular in NLP since decades (Deerwester al. 1989 Morin & Ben-
gio 2005 Mikolov et al. 2013b Bojanowski et al., 2017). lot of work has been done to under- stand the mathematical underpinning of these models, for instance, relation of embedding models to co-occurrence statistics (Levy & Goldberg 2014 Hashimoto et al., (201;; Allen et al. (019). Study of empirical properties of these embedding models (e.g. analogies) has also attracted theoretical research (Allen & Hospedales] 2019 Ethayarajh et al. 2019))
Recently, graph community has also seen surge in learning node and graph embeddings. The notion of capturing the structural neighborhood around a node inside the node embedding has been the driving principle of these semi-supervised node embedding algorithms like node2vee (Grover & Leskovee 2016), and GraphSAGE (Hamilton et al.) 2017). Hamilton et al. (2017) pointed out that their GraphSAGE node embedding algorithm mimics the aggregate and update procedure of 1-WL algorithm. Subsequently, these neighborhood informative node embeddings found their applications in constructing graph embeddings, and thereby used for classification of structurally equivalent or isomorphic graphs (Chen et al. 2019). [uu et al. (2019b) in their GIN (Graph Isomorphism Network) model modified the aggregate procedure of GraphSAGE to construct graph embedders which were provably as powerful as 1-WL algorithm in distinguishing non-isomorphic graphs. This redirected the research into designing more powerful variants of graph embedders like PPGN (Maron et al., 2019) and k-GNN (Morris et al. 2019) which were provably as powerful 3-WL and k-WL test respectively. Grohe (2020) discusses about these phenomenal works in increasing the expressivity of graph embeddings for supervised graph classification and regression tasks. While this line of research focuses on improving expressivity and generalizability of Graph networks based on its connection to 1-WL, we explore and formalize the unsupervised algorithms under the light of stable coloring / 1-WL algorithm and show that all the current models stem from the common principle that a tokens meaning is derived from its neighbours.
The rest of the paper is organised follows. We begin with discussion of connections between discrete stable coloring and non-local means algorithm in Image processing and distributional hy- pothesis in NLP. We then define continuous version of SC and develop optimization problems to solve for CSC in section 5. In section 6 we show how current state-of the art embedding models are solving the CSC problem in disguise.
3

---

Under review as a conference paper at ICLR 2022
4 ALGORITHMS WITH ROOTS IN DISCRETE STABLE COLORING
Distributional hypothesis NLPP states that words that occur in similar contexts are semantically similar. Analogously, the non-local means image denoising algorithm in CV literature assigns simi- lar intensity values to pixels that have similar patches surrounding them. The essence of both ideas is that the value of an entity is determined by its neighborhood. In this section, we connect these two ideas to the idea of stable coloring which states that if two nodes in a stable colored graph have same colored neighborhood, they have the same color. The proofs of the following theorems provided in appendix.
Theorem 4.1. (Distributional HDp)t)esis(DH) in NLP encodes SC) Let [wi, U2. Wy } be words in vocabulary, and MI R=** is co-occurrence matrix with entries ejj E N containing the number of times wi co-occurs with w within a fixed context window. Let there be function f w - N which takes a word and maps Hypothesis), such that f(w.:)= = I f(wy)) only if row i is same as row j in matrix M. Construct graph GdH with words as nodes and edge labels given by M. Then, function f defines a stable coloring on graph CDH.
a color c E N (assigns meaning in accordance with Distributional
DH defines words to be semantically same if they are substitutions of each other. In true graph ( not the one created from samples) their co-occurrence frequencies with other words is exactly the same. Equating color assigned by / to represent meaning of word as per DH, it is easy see that words that end with same color are semantically same.
Theorem 4.2, Consider discrete signal ((r) sampled at points 2; (i = 1,.... nt), and let the sequence p(a;) = (y( y (rt-1), (I+++)); (.,,(=))))) be a patch of neighborhood values around each v, for some context window length NLM denoises the signal y(z;), i 1,..., with iterative updates. The fixed point denoised version of the signal ya can be written as follows:
va(x;) = D(a;)
K K(pa(rs). P((z))))a))
(2)
where D3x;)= Eÿ= ((ya(za), (a(r))) and K is an arbitrary kernel function. Let the graph Grim = (Vnim: Entmr) where each x; is node u, in Vnlm and each pair (u,, uy) with i o < is represented as directed edge in Enim with label (i j). Then the fixed point solution of NLM with Kronecker delta kernel Ks, ya : N + R defines weak-stable coloring over the graph G.mm-
An image is an example of a discrete signal with pixel intensitics as signal values. With Kronecker delta kernel, NLM terminates with the intensity value of ya(x;) ya(z;) only if pa(x;) pa(2z). Assigning color to node n; as 2;'S 's final denoised intensity value, one can observe that NLM only terminates when weak-stable colored graph is defined over the final image. Note that if the pixel values have high variety, then the probability to get a stable-colored graph is high.
From the above two examples, it interesting to observe that stable colored graph G emerges from the underlying principles used by two different domains. Both in DH and NLM, the words/pixels that end up with the same color have same colored neighborhood around them. In order to find more connections to stable coloring, especially that of embedding models, we need continuous representation of color and hence in next section we start building on a novel notion of color.
GENERAL AGGREGATE LEARNING FRAMEWORK FOR CSC
As shown in the earlier sections, even discrete stable coloring can be connected to various concepts in NLP and image processing. We find even more deep rooted links between stable coloring and unsupervised learning algorithms in structured domains. In fact, we can view embeddings of each node as a continuous 'color' assigned to each node. In order to show these links, we first need to generalize the idea of SC setting where we talk about continuously comparable colors. This section is organised as follows. We first define a CSC problem analogous to SC. We show that this a strict generalization of SC (discrete) problem. Then, inspired by the GAU for stable coloring, we propose a series of learning problems having GAU at their core (called General Aggregate Learning GAL in short). We end this section with generalized algorithm whose parameters, as we show in section can be initialized in various ways to obtain algorithms in varied domains that have been a de facto standard in those domains since a long time.
4

---

Under review as a conference paper at ICLR 2022
5.1 CONTINUOUS STABLE COLORING (CSC)
Let us consider graph G = (V, E) representing the structured domain under consideration. In SC, we assign categorical colors the nodes of the graph. However, for most analytical tasks, including machine learning, we need to assign continuous labels (or embedding) to the nodes. Hence, we introduce the problem of CSC Let the domain of colors assigned be a metric space L (eg. R*) associated with distance metric D. There are various ways to define similarity metrics in liter- ature based on D. For the sake of discussion in this section, we would use a simple definition of ((z, y) e D(z.v), While the theorems with other definitions will change in appearance, they will still maintain the spirit of analysis. We denote the neighbourhood M(u) (x)( v) E}. Let C: V denote the coloring of nodes. We overload the function C, C 2 N to oper- ate on subset of nodes given in the following equation. We use the notation N to denote all multi-subsets of L.
C(V) H[C(v)le e V C}}
We define SN N x N - R. as the similarity metric over the multi-subsets of L via the same similarity S over and a permutation invariant and injective aggregator function Saga NN L as
(4) We refer to this embedding the continuous color under CSC We use this terminology inter- changeably as best for the context. We denote nodes by small case letters (u, u. ...) and the subset/mullt-suuseee of nodes by upper case letters (A, etc. Let now take look at the continuous stable coloring formulation (CSC).
Sv B) ((fegg(A), fagg where, A, B E...
Definition 5.1. (Continuous Stable Coloring
The coloring C V in graph G = (V.E) is called continuous stable coloring parameterised by similarity kernel S L L + R for some metric space L, and an injective aggregator function sagg N if the following holds:
of nodes
S(C(u),C(v) (X/C)))))))((
YuEEE
(5)
In most applications, we look at L = Rd for some d > O. Essentially, CSC states that the similarity between embeddings of two nodes should be equal to the similarity between the neighborhoods of the two nodes. Thus CSC relaxes the Kronecker delta function of comparison over categories in SC to general similarity metric over fact CSC is a strict generalization of SC which we show in the next theorem.
Theorem 5.1. (SC is a special case of CSC ). Stable coloring (discrete) problem is an instance of continuous stable coloring problem with N, S(i. 1 = j) and (s $2) 1(s) = s2) where j E N and $1, 82 E NN. In this case fagg N N function is an injective hash function which maps multi-subsets of N to N.
It is easy verify the validity of this theorem by using correct values for L, S and fago mentioned in the theorem. Next, we define a learning problem GAL which solves the CSC problem.
5.2 GENERAL AGGREGATE LEARNING (GAL) )
Consider G = E) representing the structured domain under consideration. (Xu et al. 2019b). authors showed that GAU framework (equation T with an injective fmerge and Jags operations is equivalent to 1-WL which solves SC problem. We define a solution to the CSC problem that is similar in spirit of GAU. However, instead of providing iterative algorithm like GAU, we pose an optimization objective which, in essence, learns the stable solution directly.
First, we define the notations. Recall that the task is to assign continuous labels to graph nodes in metric space L with distance metric 'D and the similarity metric is given as S(x, v) e eD(,w) Notation for the embedding matrix is defined the same way as that a coloring function C, such that E V L and E 2" {;
E Embedding matrix, E(u) : embedding of node u E V, and E(V) ([E(r)Ju E V)}.
E stores the color assignments of all nodes in V and is learned our setting. Let us look at our first optimization objective which follows naturally from the definition of CSC.
5

---

Under review as a conference paper at ICLR 2022
Definition 5.2. (Global GAL formulation) Let matrix E R store embeddings (or colors) of nodes in graph 9 = E), Then the optimization objective of Global GAL (G-GAL) ) parameter- ized by injective function fagg N + L (where L Rd) is to learn an embedding matrix E that minimizes the following
E arg arg.in min
Σ abs ( E 0.EEV
(II)) + (inn
(7)
The function fago can be as simple as sum operation or as complex as a neural network architecture with learnable parameters. The global formulation follows naturally from the definition of CSC (Rt S, Sagg) problem. Note that we project neighborhood embeddings( or coloring) into the same space as node embeddings. This will be important for subsequent formulations.
Research Question: Can an effective learning strategy be formulated for minimizing G-GAL loss? An effective algorithm for G-GAL formulation can stimulate further research. To the best of our knowledge, we do not know of an algorithm in any domain which uses this formulation. We think possible reason why it is difficult learn is that in most applications, access to subgraph samples of the true underlying graph. Working with a sample of neighbour- hood (u( instead of the complete neighborhood implies we are estimating fage (E))) The errors in S(fagg (E(N(u))) fagg (F((()))) increase super-linearly with error in fagg( (w)).. Nonetheless, this formulation can be of independent interest.
only have
The above mentioned issue of multiplying noisy neighborhood estimates motivates to find simpler learning problem. We formulate a different learning problem and show in theorem 5.2 that this simpler problem, which we call Node-local GAL (L-GAL) problem also solves G-GAL problem.
Definition 5.3. (L-GAL formulation..) Let matrix E R store embeddings (or colors) of nodes in graph = V) Then the optimization objective of L-GAL parameterized by an injective function fagg NL + L (where L R24) is learn embedding matrix E that minimizes the following
E arg min E
(LS(E VEV
In the above L-GAL formulation, we uncoupled the G-GAL formulation and thus eliminated the issue of multiplying noisy neighhorhood estimates. However, we would like to emphasize that the solution to L-GAL is also good solution to G-GAL. We quantify this relation in the next theorem.
Theorem 5.2. (L-GAL solves G-GAL) ). Let the solution E to L-GAL upper- each term in the summation of loss in equation 8 by some >> 0; thus it upper-bounds total loss by [V/e. Then the same solution matrix E is a solution to G-GAL with each term in summation upper-bounded by 2c and thus upper-bounding the total loss by 2
The proof of theorem 5.2 can be found in appendix. Uncoupling works because if you alternately bring E(u) and E(v) closer to (E (e(N())) and fage (E())))) respectively, it forces the dis- tances between (E E(u). E(v)) and (fagg ((N(u))). fage (E))()))) to be nearly equal.
5.3 L-GAL WITH SAMPLES FROM G = (V,E)
Often when trying solve the L-GAL problem, one will be forced to work on sub-graph samples of the graph instead of the entire graph. This can happen for multiple practical reasons (1) True graph is not known and we have access to the graph only through instances of sub-graphs. For example, true NLP graph is not known. But we have access to NLP text which hints at the graph. (2) Performing gradient descent on the entire graph is computationally prohibitive. For example social networking graphs are massive. In this case one need to sample nodes from V and then sample the neighborhood of these nodes to estimate fagy (N())) for all sampled nodes u. To account for this practical scenario, be the data available for a particular domain, G2 = Vz, & be the sub- sub-graph corresponding to the example x X. Let the neighhorhood function restricted this sub-graph be Nr.. We define the
propose an optimization objective for working with sub-graphs. Let X

---

Under review as a conference paper at ICLR 2022
optimization objective as,
arg min Σ ΣE DEV,
where (u) is neighborhood induced by the sub-graph. Essentially, we consider each node in the sub- sub-graph and its induced neighbours N from the sub-graph example and optimize the loss.
5.4 CSC WITH CONSTRAINTS
The problem of CSC, as are most other learning algorithms, is under-specified (particularly when fagg is highly expressive function). Also, sometimes there is additional information that one wants to induct in the loss function which is not present in graph structure. For example, in commercial product search settings, "nike" and "adidas" get similar embeddings due to similar neighborhood graph structure but one might want that the embeddings learned distinguish between these entities. In such cases, a constraint needs to be imposed in CSC to minimize similarity between such pairs. This concept is generalized by using a negative sampling function NS V + 2 which defines set of nodes in 2 on which a constraint needs to be imposed for any given node in V. We write the problem of constrained CSC follows:
C* arg min NEV(DENS(M) S(C(u), SX(C(M(u)) Yu, V.
Σ
In((1 - S(C(w),C(u)) subject to the CSC condition:
(10)
Using the Lagrangian multiplier we can re-write the constrained optimization objective of L-GAL as
arg min
Σ (ln) (E(N())) + BEV
Σ
-In(1 - ((Bu) (11)
VEV(NENS(v)
In most applications, is set to 1. There are variants of inducing negative loss into a system, e.g., softmax (or sampled softmaxx one such popular variant.
5.5 OPTIMIZATION ALGORITHM TO SOLVE L-GAL AND ITS VARIANTS
We have developed series of optimization objectives to solve the problem posed by CSC in the previous subsections. We close this section by discussing the algorithms used to solve these opti- mization objectives. We use any of the standard optimization algorithms: first order algorithms like gradient descent (Courant et al. 1994), stochastic gradient descent (Kiefer & Wolfowitz, 1952) second order algorithms like Adam (Kingma & Ba 2013). Adagrad (Duchi et al.) (011). etc. When solving problem on complete graph, we denote the algorithm as A( 9 (V,&) ,,, Jagg, NS) which is parameterized by underlying graph G (V, E) similarity metric S, aggregation function fagg. and negative sampling function NS V + 2 Whenever working with sub-graph samples of the true graph G, we denote the algorithm as AX, (, NS) where X is the data, each element of which, induces sub-graph G. ( (The precise definition of this in provided in section
6.1
INSTANTIATIONS OF L-GAL IN THE LITERATURE
Firstly, we describe the construction of the graph for a particular domain and sub-graph induction based on samples from the data. Secondly we summarize the graph construction and instantiations L-GAL optimization objective for different embedding models in table I and discuss some aspects of it in section
6.1 GRAPH CONSTRUCTION.
We provide a generie recipe of graph construction on structured domains. Consider for example domain with token set T. Our graph will have these tokens as nodes. However, we can introduce even higher order tokens by combining tokens. For example, the set T will denote a set of all bi-gram tokens in the domain. We can extend this idea to n-grams by considering the set T Thus, the set of all nodes the graph of domain can be written as
V = U
(12)
7

---

Under review as a conference paper at ICLR 2022
Table 1: Examples of instantiation of A, for L-GAL for various state-of-the art embedding mod-
els word2vee (Mikolov et al., 2013a), AWE (Sonkar et al., 2020). BERT (Devlin et al. 2019)
VIT (Dosovitskiy et al. 2020). and Node2vec (Grover & Leskovec 2016). Note that the token in BERT/VIT includes the position.
Parameters of A,
Word2Vec
AWE
BERT
Node22e: nodes
7: Tokens
words
words
word-picees XN BERT depth Lis Le
(16x 16 patches) XN VIT depth Laante
Gram-depth
V: Nodes
co-oecurence in
8: Edges
co-occurence with froq as weights
co-occurence with freq as weights
co-oceurence with freq as weights
co-occurence with froq as weights
random walks with froq weights
S(z.y)
exp ((x,
I-rqp(-(ap)) (((g))
exp(
exp ((z, Vi)
exp (:z,, y
Final loss with Negative Sampling loss
hierarchical
sigmod sampled loss
sampled soltmax
softmax
sofmaa
soltmax
frequency basedl
Negative Sampling
NS(u)
T/ta]
sampling to choose NS ((s.o) B(e)
T/(u]
T/ua}
sample from T/(a)
random
Jage (ut)
Winay Σ
W/w.e B(e)
Ble)
Winn B(e) mEA(
B(r)
MEA(U Σ
(u))
E(e)
Σ (E
din, o) E(r)
Σ
*b, ()(() veV(w)
Σ
F(())
(EA(s)
(EA(s)
2. X
words in sentence
words in
word-pleces with position
patches with position an image
nodes in a random
subgraph induced on by
sentence
in sentence
walk sequencc
where k is the gram-depth. The edges in this graph of gram-depth k depend on the application. This illustrated in figure Ta In most cases, including the models discussed in this paper, the edges are added on basis of co-occurrence statistics in the data. For example, let us consider the case of NLP. The sentence "The tree fell" in a graph with a gram-depth of 3 will include the nodes {the, tree, fell, {the, tree), {the, fell), ftree, fell], {the, tree, fell) If the edges are added on the basis of co- occurrence, then based on the above sentence we add the edges between "The" and "fell," "The tree"' and "fell", "The fell" and "tree," and on, Additionally, edges can have weights corresponding to the frequency of co-occurrence.
While the graph tokens represents the true graph, the algorithm Ag which solves the sampled version of L-GAL problem uses the sub-graph of g = EV,E) which is induced by the sample. Considering the sample x X set of tokens æ C T, the graph induced by this set of tokens, O2 = -Vz, Ex) with neighborhood function N2 where
V2 = (ulv E U UEVV,,
Eg = ou,,,) E Ui (u,v) EJ).
(13)
6.2 CURRENT SENTE-OF-THE-AAE EMBEDDING MODELS ARE INSTANTIATIONS OF L-GAL
Now, we summarize the instantiations of A, for variety of algorithms in domains of NLP, images, and graphs. For details on working of each algorithm, we direct the reader the corresponding original papers. We want to point out the usage of exp(x, y) or I+esp(-(a.g)) metrics. Under the unit-norm assumption on æ and y. a standard assumption for analysis of embed- dings, exp (x, y) = exp (1 (x, /)) which is only a function of the la norm D and inversely proportional to it. Thus, exp(2r, y) ( or 1-xxpp((x is valid similarity metric under unit-norm assumption.
as similarity
6.2.1 WORD2VEC, NODE2VEC, AWE
Theorem 6.1. The algorithm A, for L-GAL initialized with parameters from tarle Word2 Node2 and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2Vec, Node2Vec and AWE
The proof of the above theorem is quite straightforward and is provided in the appendix. The for- mulation of sampled L-GAL problem provides a natural explanation of why AWE performs better than Word2Vec. Both Word2 Vec and AWE operate on the same graph G = (V,E). However, they compute their estimates of fagg (u)) differently. While word2vec performs simple sum, AWE performs weighted sum. One can prove that the MSE error with simple sum is larger than with weighted sum when weights are equal to the co-occurrence frequency. AWE models these weights as these weights are not known apriori and the authors in the paper hint at the learned weights being representative of the these co-occurrence frequencies.
8

---

Under review as a conference paper at ICLR 2022
Construction of Higher Order Graphs
Sags (N(E(u)))
3-gram tnnner
Increasing gram-depth
2-qram
Newer embeddinggs at the level
1-gram
Masked taken (u)
(a) Graph Construction higher order tokens a graph
(b) Illustration how Bert/ VIT follows L-GAL framework
6.2.2 BERT AND VIT
Theorem 6.2. Considering BERT and VIT stacks of attention layers without any non non-line con-linearity and initializing the algorithm A, with the parameters mentioned in the table I Ileads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and VIT.
The proof of the theorem is provided in the appendix. For this analysis, we will assume that BERT is a stack of attention layers without any non- non-linearity. BERT and ViT have similar architecture and hence we provide single proof for them. Let us consider the sub-gra sub-graph induced for BERT by sample æ, which means it essentially has the nodes _ x The sub-graph will have every node connect with every other node. Thus when we look at the embedding of particular masked word, according to definition of Sagg for BERT we will get,
Sego = Σ (w E(v)), (EN(
(Ngo
Σ
E(e)).
(14)
(EN(u))(Ut
Now if this is exactly the computation that BERT (or ViT) performs then with softmax style neg- ative sampling loss, BERT will follow the L-GAL optimization. We claim that BERT does indeed compute this fagg- Ideally, BERT should learn the embedding matrix for all the nodes including tokens and higher order tokens. However, this computationally expensive. Hence, BERT actually models the higher order tokens in terms of its component tokens. It can be seen that if the modelling a weighted linear sum of the components, then BERT essentially ends up computing equation 14 More details can be found in appendix. Also, as in AWE it models the weights in the summation of Jagg (via the attention mechanism). The information propagated through BERT can be visualized shown in figure lb At each layer i {1, k}, BERT computes the token embeddings of order i and this information from each layer flows to the masked word node and gets aggregated.
7 IMPLICATIONS AND CONCLUSIONS
In this paper, we define notion of CSC understand the internals of wide range of embedding models across diverse structured domains. Grounded on this notion, we propose a generalized L- GAL optimization problem that solves for CSC on graphs induced by any structured domain. We thereby prove equivalence between loss functions of popular NI., image, and graph embedding models and our proposed constrained L--AL optimization loss operating on domain-speciff graphs. The methodology of construction of these graphs is also presented for each structured domain.
Our proposed framework with its robust mathematical founding on CSC us graph theoretical per- spective to these unsupervised embedding models. We have already seen great improvements in supervised models based on the connection between 1-WL and GNNs. We believe that our for- mulation will stimulate further research in embedding models. Insights from the underlying graph view of the computation in embedding models can give us new directions to improve quality and scalability and training efficiency of these models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embed- dings. In International Conference on Machine Learning, pp. 223-231. PMLR, 2019.
Carl Allen, Ivana Balazevic, and Timothy Hospedales. What the vec? towards probabilistically grounded embeddings. Advances in Neural Information Processing Systems, 32:7467-7477, 2019.
Suyash Awate and Ross T Whitaker. Unsupervised, information- adaptive image filtering for image restoration. IEEE Transactions on pattern analysis and machine intelligence, 28(3): 364-376, 2006.
László Babai and Ludik Kucera. Canonical labelling of graphs in linear average time. In 20th Annual Symposium Foundations of Computer Science (sfcs 1979), pp. 39 46. IEEE, 1979.
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-based object detection. ArXiv, (bb/20012.09955 2020.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) volume 2, pp. 60-65. IEEE, 2005.
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv preprint arXiv: (rXXiv:905.04577 04579, 2019.
Richard Courant et al. Variational methods for the solution of problems of equilibrium and vibra- tions. Lecture notes in pure and applied mathematics, pp. 1-1, 1994.
Scott C Deerwester, Susan T Dumais, George W Furnas, Richard A Harshman, Thomas K Landauer, Karen Lochbaum, and Lynn A Streeter. Computer information retrieval using latent semantic structure, June 13 1989, US Patent 4,839,853.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171 1171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning reseurch, 12(7), 2011.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analo- gies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pp. 3253-3262, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v 101/3//1/P11-1111 1315. htllhttts///caanttaae http://aac.aaatooooooooooo////-/ 1315
Martin Grohe Word2vec, node2vec, graph2vec, x2vec: Towards theory of vector embeddings of structured data. In Proceedings of the 39th ACM
Symposium on
Principles of Database Systems, PODS'20 20, pp. 1-16, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371087. URL attps ://oc. otg/10..14..
8375339 3387641
10

---

Under review as a conference paper at ICLR 2022
Aditya Grover and Jure Leskovec. Node2vee: Scalable feature learning for networks. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 855-864, New York, NY. USA, 2016. Association for Com- puting Machinery. ISBN 9781450342322. doi: 10.114//939972.2933754 URL https:
7/doi. aiddi.org/10/ 10. 2444233967..2997772
William L Hamilton, Rex Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025-1035, 2017.
Zellig S Harris. Distributional structure. Word, (42-33)146-1160 1954.
Tatsunori Hashimoto, David Alvarez-Melis, and Tommi S Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics, 4: 273-286, 2016.
J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statisties, 23(3):462 466, 1952. ISSN 00034851. http://:://ww
jstor. istor.org/stabllle.eoo. wttpoo///aaebeeee6699/03
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttt,
//arxiv.org abs /1122 6980
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Interna- tional conference on machine learning, pp. 1188-1196. PMLR, 2014.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factoriza- tion. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems. volume 27. Curran Asso-
ciates, Inc., 2014. URL htttps:://proceedings. www.:ne.uiii....m.. wttpuuips.cccpppppeeeeeeeee pcpps/////ppet//////llle///// http://wwwepppppaoo.orooooeeeaea
pdf
Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Topical word embeddings. In AAAI, 2015.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
pdf
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119, 2013b.
Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In International workshop on artificial intelligence and statistics, pp. 246-252. PMLR, 2005.
Christopher Morris, Kristian Kersting, and Petra Mutzel. Glocalized weisfeiler-lehman graph ker- nels: Global-local feature maps of graphs. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 327-336, 2017. doi: 10.1 10.1109/CDMM2017
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gau- rav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural net- works. Proceedings of the AAAI Conference Artificial Intelligence. 33(01):4602 4609, Jul. 2019. doi: 10. 16w9//99.iii...m 33101,33014602, URL https: ptttps://oj... aaai. org/ weogg/idex... php/ AAAT / wtwee//wwwwe/ee 4384

---

Under review as a conference paper at ICLR 2022
Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 21824-21840. Curran Associates, Inc., 2020a. URL bttps: http::::s/prooceeediinooooo ngs neurips. scc/papper/202//flee/eeee 2e
4de Ttt.S//.....11111196682288855511
Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 31824-21840. Curran Associates, Inc., 2020b. URL http:t:s:/pprcceedd..oo ngg. neurips. cc/pappr/2DD0/LLLe//Beeee222B2B2 42g.228888.8011111199992267777.1
Christopher Morris, Matthias Fey, and Nils Kriege. The power of the weisfeiler-leman algorithm for machine learning with graphs. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth Interna- tional Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4543-4550. International Joint
Conferences on Artificial Intelligence Organization, 8 2021. Survey Track.
Korawit Orkphol and Wu Yang. Word sense disambiguation using cosine similarity collaborates with word2vec and wordnet. Future Internet, 1 155:1114, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532- 1543, 2014a.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532-1543, 2014b.
Magnus Sahlgren. The distributional hypothesis. The Italian Journal of Linguistics, 20:33-55 54, 2008
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Shashank Sonkar, Andrew Waters, and Richard Baraniuk. Attention word embedding. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pp. 6894-6902, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguis- tics. doi: 10. 10.186653/1122000coongg http://002..ooiiimmaiiooooome main.608. URL UBLLtttpps///eellatttoooo.....sp
coling- main. 608
Jian Tang, Meng Qu, and Qiaozhu Mei. Pte: Predictive text embedding through large-scale hetero- geneous text networks. In Proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 1165-1174, 2015.
Bruna Thalenberg. Distinguishing antonyms from synonyms in vector space models of semantics.
Technical report, Technical report, 2016.
Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series, 2(9):12-16, 1968.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. BERT post-training for review reading compre- hension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2335.. Minncapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. (0.18653/v//N1-11 1242. URL http:::://aclaanhhllgygg/ 9-122
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019b. URL hotps:
7/ //openzeview, www.:ntettttddiiitonoee...
umm 1t ryGs1155km
Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In Proceedings of the 20th Australasian document computing symposium, pp. 1-8, 2015.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 A WALK THROUGH POPULAR EMBEDDING MODELS AND UNDERSTANDING THEIR DESIGN PRINCIPLES AGAINST THE BACKDROP OF OUR FRAMEWORK
The proposed L-GAL formulation abstracts out different components of embedding models and thus enables design better embedding models by focusing on these components. The components are follows,
Domain Graphs
fagg Neighborhood aggregation function.
NS: Negative Sampling
Estimation of Saga (W()) via sampling-
One can think of the strategies focusing these components as,
1. Better sampling/ estimates of node's neighborhoods.
2. Better design of Sagg function.
3. Construction of more expressive graphs that embed higher order tokens.
4. Importance based negative sampling inject knowledge that best negates the neighbor information given by our proposed domain graph.
With these four strategies in place, next we understand the development NLP models over time by choosing word2vec, one of the most widely-used embedding model, as our starting reference. We demonstrate how our proposed strategies have manifested different embedding models that have improved over word2vec in last decade.
Recall from table that word2vee uses
(V, E) where V is the set single words as nodes and E the edge between words, say and with co-occurrence as the weight.
fogg (u)) weighted sum of neighbour embeddings
NS(u): All words except
Sampling technique sentences
Strategy 1 Better estimation of Jago (N())) Glove vs Word2vec Word2vec is an iterative algorithm over sentences. It uses sentence as a sample to guess word's neighborhood which induces lot of noise. Our framework would suggest to the entire computation of fage instead of sampling if possible for a given fagy and graph. Glove does exactly this. Glove (Pennington et al, 2014a) improves upon word2vee by using global co-occurrence statistics to eliminate noise these neighborhood estimates. Glove's optimization objective (given by equation 12 which is simplified equation 15 in original paper) is as follows:
V
V
J=
J=
Xis log Qij-
i1j11
where V. is the vocabulary, Xij is the co-occurrence count of word with word in the corpus, export
and Qu Σx= exp uf Note that Glove simplifies Qij exp w/ uy which is identical to the similarity kernel we use in our analysis with unit norm assumption.
Strategy 1, thus, helps understand why Glove improves upon word2vec.
Strategy 2 Better fage AWE Word2vee Let consider another word embedding model. AWE (Attention Word Embedding). that improves upon the CBOW model word2vec. CBOW model weighs each word equally compute the context embedding, however, AWE weighs
13

---

Under review as a conference paper at ICLR 2022
each context word differently some context words are more important for prediction of the masked word than others. The context vector in AWE is given by:
Σ
Oog Men; where aw; = exp (kiwoo Twx,
Cowe
KE[-b,b]- (0)
where where is the size of the context window and w, is the index of each word (wo is the index of the masked word; the rest are the indices of the context words), aw, is the attention weight of each context word vector ww calculated using the key matrix K and the query matrix Q. Note that in word2vec Owr, = 1.
Strategy 2 helps us understand why AWE outperforms Word2vec. AWE formulates better fagg function as compared to Werd2Vec.
Strategy 3: More informative domain graphs BERT vs Word2vee As discussed in section ??, BERT uses graph with higher order tokens. In fact, the highest order of token can be interpreted as the number of layers in ther BERT. It also uses the AWE style weighting of the neighbours. Strategy
3 (along with strategy 2) can help us understand how BERT improves over Word2vec embedding
model. Strategy 3 can also help us understand why
(Le & Mikolov 2014
Tang et al.) 2015 Liu
et al. 2015) can improve over Word2Vee.
Strategy 4 : Better NS WSD-W2V Word2Vee Strategy 4 helps understand yet another shortcoming of word2vec. Word2vec struggles with modeling antonyms and often brings antonyms close each other (Thalenberg) 2016). This not surprising since antonyms tend to occur in similar contexts, i.e., identical neighborhoods. With our proposed framework, one can arrive at this conclusion from a graph theoretic perspective. Antonym subgraphs are isomorphic, and thus antonyms get assigned similar continuous stable colors. To overcome this, one needs to introduce information into the setup using variants of negative sampling. Thus external datasets like wordnet Orkkpho & Yang 2019) need to complement word2vec's training strategy for efficient modeling of antonyms.
To conclude, the model design changes that our framework suggest (strategies 1-4) can be found in existing works.
A.2 THEOREM
Let fwi, W.,... 3r be words in vocabulary, and MI E Rax" is co-occurrence matrix with entries eij N containing the number of times w; co-occurs with wy within fixed context window. Let there be a function f w + N which takes a word and maps it to a color E N (assigns meaning in accordance with Distributional Hypothesis), such that f(w:) f ((w;) only if row is same as row j in matrix M. Construct a graph GD with words as nodes and its adjacency matrix given by ML. Then, function f defines stable coloring on graph GDH.
Proof: The proof follows trivially from the definition of f.
A.3 THEOREM
Consider a discrete signal v(z) sampled at n points I; (i = 1,..., n). and let the sequence p(x:) = (y(2,-1), ... (2; (1. y(T++()) be a patch of neighborhood values around each x, for some context window length t. NLM denoises the signal y(z,), i = 1,..., n with iterative updates. The fixed point denoised version of the signal ya can be written as follows:
va(x:) (
(15)
where D(x;)) Σ; (()a(z;), Va(r,)) and K is an arbitrary kernel function. Let the graph Grlm (Vulm: Entm) where each n; is node u4 in Vntm and each pair (u, uy) with i - < t is represented as a directed edge in Entm with label (i - j). Then the fixed point solution of NLM with Kronecker delta kernel Ka, ya : N + R defines weak-stable coloring over the graph Gntm
14

---

Under review as a conference paper at ICLR 2022
Proof:
In order to prove a weak coloring, we need to prove that the neighbourhoods a pixel are equal then its value is equal. Consider any two pixels 2; and 2j with same neighbourhoods, Then Ko (o(pa())) pa(xy) = 1. Let { Th 2kg ...] be the set of size L of pixels whose neighbourhoods that match with neighbourhood of n; and x--- Thus, by equation of final stable solution, both the pixel values are equal to ya(x;) = yu(2,) = El= Ks(pa(z;), pa( (z
A.4 THEOREM 5.1
(SC is a special case CSC) ). Stable coloring (discrete) problem is an instance of continuous stable coloring problem with L N, S(i,j) = 1(2 = =]) and (11 82) ((s1 s2) where i, N and 81, 82 E NN In this case Sagg NN + N function is essentially an injective hash function which maps multi-subsets of N N.
Proof: The proof the theorem trivially follows by using the parameters of CSC as mentioned in the theorem.
A.5 THEOREM
Let the solution to L-GAL upper-bounds each term in the summation of loss in equation 8 by some > O; thus it upper-bounds total loss by [V/e. Then the same solution matrix E is solution to G-GAL with each term in summation upper-bounded by 2e and thus upper-bounding the total loss by [VPe.
Proof:
(N(E(V)))
E(v)
2gg
E(u)
Sagg (N(E(u)))
Figure 2: Illustration of theorem 5.2
Loss(G-GAL)= = Σ lbss- + (,ge:/oo)0)nnoooeelleM/e)))lllld
J.VEV
(16)
Σ cbs(D(F(u),FF()) D(Sagg (Flg (17)
UVVEV
As D is a metric, we have
D(Jogo http::/hh.....ooooooo/oo////.nn/
(18)
D(Sagg
(E(N(t)))
(19)
15

---

Under review as a conference paper at ICLR 2022
By hypothesis, that the each term in the loss of L-GAL is bound by
Dffago (ogg - S 2e
(20)
Thus
Loss(G-GAL)
Σ 2:
PI(M - 1)e s 1216
(21)
U.VEV
A.6
THEOREM 6.1
Theorem A.I. The algorithm A, for L-GAL initialized with parameters from Iable Word2 Vede2Vec and AWE leads to exactly the algorithms (with possibly minor variations) as proposed in the original papers of Word2 Node2Vec and AWE
proof It is easy check that the parameters specified in the table exactly provides the algorithms of corresponding models. Some more details can be found in appendix
A.7 THEOREM
Considering BERT and VIT as stacks of attention layers without any non-linearity and initializing the algorithm A, with the parameters mentioned in the table I leads to the algorithms (with possibly minor variations) as proposed in the original papers of BERT and ViT with the linear modelling on embedding of higher order tokens.
Proof: The parameters of BERT/ VIT such as loss used, tokens can be verified from the table. only remains to prove that BERT computation is same as that of fagg mentioned.
This is done under two assumptions, (1) BERT just a stack of attention layers. (2) each higher order token's ' embedding is modelled as a linear model of it constituent embeddings.
Let us consider the token (u, i) to be masked. Without loss of generality let the position the token be last position. Let each path reaching the level k at masked word be denoted with the locations of the words the path takes at each level. So a path (0,1,0,2 says that takes the route words wo, W1, wo, We to reach the masked word at the final level.
Then we can write the new computation of reaching BERT at level k as
Σ Σ
aip ((wp(()))
(22)
DE PEPATHS(N)
This can be the embedding of the path as BERT models it (linear model assumption). It is easy to see that each path length k maps one-to-one to higher order token in T Under this relation PATHS(K) is exactly the set of all nodes in Sk The information also flows from masked words which aggregates the embeddings of St < Thus the final computation of bert at level K looks like
K
Σ
Σ
Σ
Cip E(w(((())) = Σ B,E(U)
(23)
R=
PE PEPATHS(K) t=
BESK
for some Bs
B EXAMPLES OF GRAPH CONSTRUCTION
B.1 VARIOUS DOMAINS AS GRAPHS
relation Any general set of tokens say T and associated relation R C T T give the graph 9 = (V.E)
with V = T and (u, v) E E iff R(u, v) = 1. While the underlying graph of NLP is not explicitly known examples of relations that can be drawn from the sentence structure are follows.
16

---

Under review as a conference paper at ICLR 2022
Let V be the set of all words f [11 W2, 11] (W1, sentence. Additionally, we can assign weight to the edge equal to the frequency of co-occurrence observed in some natural language corpus.
W2) & iff the words w1 and wa co-occur in
Let be a set of all subsets of T of size less than or equal to k. Let 41 /2 E V then the relation (41,12) 12) & iff 1 Ui2 appear together in some sentence in natural language.
Similarly we can also view the entire image collection as a graph g = (V,E) where set of all patches observed in all images alternatively images of particular category Some examples of relations might be,
R(p1, iff p1 occurs in larger patch K K around p2 in some images P2)
P2) = 1 iff adjacent to p2 in four directions (left, right, top. down)
Graph domain naturally maps the graph data structure. However, can construct more informa- tive graphs from the basic graph structure. For example, we can add hyper-edges to add hyper-graph structure the same graph.
17

---

Under review as a conference paper at ICLR 2022
B.2 WORD2VEC
Word2vec is one of the standard word embedding models (Mikolov et al. b).. The continuous bag-of-words (CBOW) model of word2vec predicts masked word using its context. Let E [U1,..., uN]" E RRNxx be the word embedding matrix where N the size of the vocabulary, and the size of the word vector. CBOW looks the set of sentences drawn from NLP corpus and uses it to learn E. For each sentence of size n, CBOW creates n examples of (u, ctx(u) {ule e x, o u}) pairs by choosing one word and using rest of the words as context ctx. It uses the following formulation,
E = arg min
Σ
(-IIT
1
(24)
1 exp (-(F(u), B(ctx(u))))
(w,etx())E=EX))
Σ In 1 exp exp (-(E(v), E(ctx(u))))
(25)
vENS(u)
where E(clx(u))= = Evectx(() E(v). Note that original formulation of CBOW uses softmax which a variant of the negative sampling loss formulation.
We will show in theorem ?? that Vec is indeed same as A, using some specific initialization of its parameters. We first define the parameters.
Graph Sw2v. Let {wi, W2,... W[vi be words in vocabulary and MI RIVIXIVI is co- occurrence matrix with entries egj E - containing the number of times co-occurs with wy within a fixed context window. Construct graph Gw2v with words as nodes and its adjacency matrix given by M
Data 2w Every example a words (sentence) and the graph induced by these words a sub-graph in Gw2v with every node connecting every other node.
Aggregation function fw2e- Let fagg(ctx((u) Evectx(u) F(v).
Similarity Metric Sxgm- Let S(x. y) be (Ix y)- Under unit-norm assumption, this is a valid similarity metric derived from l2 norm distance metric.
B.3 ATTENTION WORD EMBEDDINGS (AWE)
As can be observed in fw2v. CBOW model equally weights the context words when making prediction. To address this limitation of CBOW, AWE augments CBOW model with an attention mechanism to attend to context words that most relevant for prediction of masked word. AWE also uses the same formulation as word2vec given by equation 25] except that context vector is weighted sum of context word embeddings.
In AWE, E(ctx(u)) Evectx(u) War E(v), where Was = exp (< ku qu >) models the impor- tance of context word u for predicting the masked word u using key and query word embedding matrices given by K [kı,... |N Nxd and Q [g1, ... qN]"" RNNxd respectively.
As we did for word2vec, we will show in theorem ?? that AWE is indeed same as A, using some specific initialization of its parameters. We first define the parameters.
Graph Gawe- Gawe Guzo-
Data X,we- Similar to word2vee, every example is of words (sentence) but unlike the case of word2vec, the graph induced by these words is weighted sub-graph in Gawe-
Aggregation function fawe- Let faga (ctx(u)) Eneetx(w) E(v).
Similarity Metric Sagm. Let S(x, 2) be (fx, y). Under unit-norm assumption, this a valid similarity metric derived from la norm distance metric.
18


Reviews:
[{"review_id": "wh0ujYNGwwL", "reviewer": "Reviewer_mH7f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) The main contribution of the paper is the definition of a new problem CSC. As the authors themselves mention most of their Theorems are straightforward consequences of the definition of the problem. \n\nMajor Weaknesses\n\n1) **No new insights**\n\nThe major weakness of the paper, is despite making a connection between CSC and prior word embedding techniques, the paper does not mention anything new about the techniques or present any insights into how the new connections can be used to better design embedding techniques. \n\nSpecifically, the main connection here is that we embed words by tryin to predict the context of the word and that this is the same as embedding a graph by trying to predict the neighbors of the graph. However, this connection (while not presented as CSC) has been known. See [1] for an example that explores the connection to 1WL in detail. \n\nFurther, my main issue with the paper is that the connection between techniques and word2vec has been made at a surface level in function space. That is, it shows that the embeddings that the two techniques are trying to learn are similar. However, it does not prove any properties  about the embeddings. Maybe say something about its generalization. Or show that if the graph satisfies certain properties then the learned embedding satisfies certain properties. \n\nFurther, the more interesting question in deep learning and the usage of neural networks is not the connection in function space, but the connection in parameter space. That is, how do these networks parameterize the function that we are trying to learn/how do optimization techniques learn different representations in the parameter space. \n\n[1] Martin Grohe. 2020. Word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Data. In Proceedings of the 39th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems (PODS'20). Association for Computing Machinery, New York, NY, USA, 1–16. DOI:https://doi.org/10.1145/3375395.3387641\n\n2) **Missing Details**\n\nI also think the paper is missing details. First, in the problem of CSC, we look at $f_{agg}$ as an *injective* function from the space of all multisubsets of $L$ to $L$. However, the space of *all* multisubsets of $L$ has a strictly bigger cardinality than $L$. (eg. if $L$ is the natural numbers, i.e., is countable, then $N^L$ is uncountable). Hence no injective function can exist. Hence I assume the authors mean that the $f_{agg}$ is defined on the space of all *finite* multisubsets. While this is not too big a deal, it is an issue for a theoretical paper. \n\nThere are also a few vague statement. For example, the authors, say $f_{agg}$ is essentially a hash function in Theorem 5.1. What does this mean exactly? Another is after the box titled research question, the authors say, \"the above mentioned issue of multiplying noisy neighborhoods\". Where are we multiplying neighborhood? What does it mean to multiply neighborhoods?\n\n\n**Minor Details**\n\nThere are typos in the main text. Some of them are\n\n1) Many equations are missing parenthesis. For example, we see $\\mathcal{C}(\\mathcal{N}(u)$ in a few places (def 2.1, top of page 3, last line of 1WL paragraph).\n2) Equation 1, also seems to have a typo, it should be $E^{k-1}(v) : v \\in \\{\\{\\mathcal{N}(u)\\}\\}$. \n3) Theorem 4.2 last line has a typo. \n4) Just before section 3, missing a reference (have ??)\n\n**Improvements**\n\nHere are just a few ideas for improvements. \n\n1) Maybe look into $f_{agg}$ and see how properties of this function effect properties of learned embedding.\n2) Run some experiments. Show empirically, how optimizing for CSC directly using the algorithms presented has similar results to doing the appropriate word embedding technique.\n3) Show if the graph has certain properties then we can say something about the embedding. \n4) Show that using insights from above we can design a new technique. \n\n", "summary_of_the_paper": "In the paper the authors define a new problem called the continuous stable coloring (CSC). This is an extension of the traditional stable coloring problem, however, instead of having discrete color labels, you have continuous color labels. The authors then provide an objective function that could be minimized to obtain a CSC. They then show that if you tweak the similarity measure $S$ and the aggregator function $f_{agg}$, then the CSC problems various word embedding techniques such as word2vec, AWE, Bert (without non-linearity). ", "summary_of_the_review": "Overall, I think the problem of CSC is a neat problem. However, its connection to word embedding techniques has only been explored at a surface level. Hence I feel there is not enough in the paper to merit acceptance as a theory paper. However, the connection could bear fruit if the authors do provide deeper connections either in parameter space or provide results on the properties of the embedding learned. While I cannot vote for acceptance at this time, I do hope the authors revise with new results and resubmit. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "wISghUGIXzB", "reviewer": "Reviewer_K2fA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments\n--------------\n- One of the main issues with this paper is that writing is not clear. In fact, the presentation of the paper leaves the reader with a feeling that this was an initial draft of the work. The paper is pretty difficult to read. There are many typos and the structure is disorganized. I suggest the authors work on improving it.\n\n- Overall, the paper seems to be proposing an interesting contribution. The authors establish a connection between the Weisfeiler-Lehman test of isomorphism and embedding algorithms from structured domains. However, in my view the significance of the paper is low. The authors provide no discussion on potential applications of the proposed methodology, while it is not clear whether it can be useful for domains in which there is already a wealth of embedding approaches such as in natural language processing or in graph representation learning. Furthermore, the authors do not experimentally evaluate the proposed methodology.\n\n- It is not clear to me what is the intuition behind the G-GAL and L-GAL formulations, and how exactly the solutions of those formulations are equivalent to the continuous stable coloring. I would suggest the authors make clear why out of all possible formuations, they have proposed those two and not some other formulation. Furthermore, if we assume that $f_{agg}$ is the sum operator, to my understanding, these two problems admit a trivial solution. Setting the representations of all nodes equal to the all-zeros vector leads Equations (7) and (8) to their minimum value (i.e., a value equal to 0). I would like the authors to comment on this point.\n\n- It is not clear to me how embedding algorithms such as word2vec and node2vec are instances of the L-GAL formulation. In the case of those algorithms, a positive instance cooresponds to a pair of words/nodes that co-occur within a fixed-size window in some text or a random walk. In Equation (8), the representation of a node is encouraged to be close not to the representation of each of its neighbors, but to the sum of them (in case $f_{agg}$ denotes the sum operator).\n\n- The authors present in section 5.5 the different optimization algorithms that one could employ to solve the L-GAL problem and its variants. It would strengthen a lot the paper if the authors could apply the proposed methodology to some real-world problem. For instance, the authors could construct a graph from some text corpus and use the proposed approach to embed the different term to some vector space. Then, they could also evaluate the quality of the generated embeddings in standard tasks such as word analogy and word similarity, and compare them againt those generated by other methods.\n\n- Although I am not aware of any previous studies that have established a connection similar to the one presented in this paper, there are previous studies that have proposed to unify embedding approaches into matrix factorization frameworks [1]. Such works are related to the work presented in this paper and should be discussed in the related work section.\n\nMinor comments\n--------------\n- Function $\\mathcal{C}$ which is used for comparing vectors or multisets of vectors seems quite arbitrary. What are other functions that could potentially be employed and what properties do they need to exhibit?\n\n- In p.2, the following sentence \"As mentioned ... information?\" is very long. It is also not clear and does not read well. I suggest the authors rephrase this sentence.\n\n- In p.3, the authors mention \"Chen et al. (2019) in their GIN ...\". In fact, GIN was proposed by Xu et al. and not by Chen et al.\n\nTypos:\n------\np.2: \"architectures captures capture the\" -> \"architectures capture the\"\\\np.3: \"$\\mathcal{C}_( \\mathcal{N}(u)$\" -> \"$\\mathcal{C}_i (\\mathcal{N}(v))$\"\n\np.3: \"examples are given in appendix ??\" -> \"examples are given in appendix B\"\\\np.4: \"surrounding it\" -> \"surrounding them\"\\\np.4: \"provided in appendix\" -> \"provided in appendix.\"\\\np.5: \"which we show this in the\" -> \"which we show in the\"\\\np.5: \"injective $f_{merge}$\" -> \"injective $f_{update}$\"\\\np.6: \"learn-able\" -> \"learnable\"\\\np.8: \"heirarchial\" -> \"hierarchical\"\n\n[1] Qiu, J., Dong, Y., Ma, H., Li, J., Wang, K. and Tang, J., Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pp. 459-467, 2018.", "summary_of_the_paper": "In this paper, the authors investigate how the solution of the Weisfeiler-Lehman test of isomorphism (WL) is related to embedding algorithms from structured domains. The authors generalize the WL algorithm to the case where the colors assigned to the nodes correspond to feature vectors, and propose a series of optimization problems to solve this problem. Finally, they show that widely used algorithms in natural language processing, computer vision and graph representation learning such as word2vec, BERT, Visual Transformer and node2vec actually solve different instantiations of this common optimization problem.", "summary_of_the_review": "In general, I feel that there is some value in this paper and that the contribution is interesting since,  to my knowledge, no paper so far has established any connection between the Weisfeiler-Lehman test and embedding approaches. However, I have several concerns about this work. The writing is not clear, while the significance of the theoretical results seems to be low. The authors do not perform any experiments, thus there are no empirical results and it is not clear how the proposed approach would work in real-world scenarios. Finally, I am not sure whether some of the claims made by the authors are actually true.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "sZRFDA0j9LF", "reviewer": "Reviewer_EdGh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The writing of the paper can be improved. In particular, the contribution of this paper is unclear to me. The authors repetitively claim that they \"define a mathematical framework that defines continuous stable coloring on graphs and develops optimization problems to search for them\", but there is no validating algorithm at all. \n\n- From a theoretical point of view, how could the proposed mathematical framework help us better understand NLP embedding models? \n\n- It seems the authors try to claim that there exists some similarity between graph coloring problems and NLP embedding problems. However it feels like they are simply repeating the so-called distributional hypothesis. Now say the claim is true. How would that be helpful to researchers using those NLP models, including BERT, ViT, Node2Vec? Again the contribution is not clear.\n\n- I do not understand if Theorem 4.1 is a theorem, or a hypothesis or an assumption. Is G_{DH} a weighted graph? Considering that the entries in M are nonnegative integers (counting co-occurrence), what does it mean by adjacency matrix in this case? \n\n- Many Theorem proofs are just one line like \"the proof follows trivially from ...\" E.g., Theorem 4.1. If they are so obvious, is it necessary to present them as \"Theorems\"?\n\n- The derivation from the assumptions in Theorem 4.1 to Definition 2.1 is not so trivial. How do you define neighborhoods in terms of the co-occurrence matrix?\n\n- Section 2.2, description of 1-WL algorithm \"Ci(N(u)) 6= C(N(u)\". This is not properly typed. Why both sides are u? Also, which function C is being used on the RHS? Please clarify. \n\nEditorial comments:\n\n- Extra whitespace: introduction - \"continuous stable coloring (CSC )\", Section 2 - \"Aggregate and Update (GAU )\", Section 2.2 \"(GAU )\".\n\n- Section A.6: \"with the parameters mentioned in the table ??\"\n\n- Definition 2.1: \"C(N(u)) = C(N(v)\"", "summary_of_the_paper": "The paper tries to relate multiple unsupervised NLP embedding models to the problem of stable coloring. The authors claim that they \"prove equivalence between loss functions of popular NLP, image, and graph embedding models and our proposed constrained L-GAL optimization loss operating on domain-specific graphs\".", "summary_of_the_review": "At the current stage the contribution of this paper is unclear, both theoretically and experimentally.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "nvk66BvGuym", "reviewer": "Reviewer_JX7R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel perspective for understanding unsupervised learning algorithms, which could inspire new research directions. For example, the connection between BERT and stable coloring could help understand the role of different attention layers in BERT. However, the theory of this paper is not proven rigorously and it lacks new method or empirical results.\n\nComments & Concerns:\n - I find the connection between the original 1-WL algorithm defined for the graph isomorphism test and the global GAL formulation a bit over-stretch. In particular, 1-WL is an iterative algorithm that keeps assigning new colors to unseen neighborhood until the overall node partition in a graph by the color is unchanged; 1-WL does not explicitly push the difference between colors in the current iteration to be similar to that of the next iteration. As a result, if one directly use the GAL in equation 9 as the objective function, then there is a trivial minimum where all nodes have the same embedding; therefore, the authors \"proposed\" negative sampling as some sort of regulation. However, negative sampling does not have a correspondence in the original 1-WL algorithm.\n - The connection between local GAL and BERT is also over-stretch. Although there is clear analogous between the multiple attention layers with multiple iterations of 1-WL algorithm. In BERT the weights of multi-head attention in different layers are different. Does this mean that the aggregation function in GAL can be different for different layers? It would be helpful if the authors can provide a rigorous derivation of how the proposed setting leads to the original BERT training mechanism, instead of simply saying it is obvious.\n - In theorem 5.1, the authors claim that the $f_{agg}$ is simply an injective hash function that converts multi-sets to an integer; this is not true since if $f_{agg}$ is injective, then new colors could keep being generated forever, and that the stable coloring algorithm would never converge. It is possible that the authors are referring to a special variant of stable coloring algorithm. In that case I think a more rigorous proof would help understanding.\n - In equation 10, the authors use the coloring function $\\mathcal{C}$, while in equation 11 the embedding $E$ is used. For consistency it is better to use $E$ for both.\n - Theorem 5.2 shows that L-GAL is sufficient to guarantee good solution for G-GAL. However, it may not be necessary. In fact, in 1-WL, the color assigned in current iteration does not need to be the same as the previous iteration for the algorithm to converge, but rather only the partition needs to be the same. I wonder if the author can show the existing algorithms can directly connect to G-GAL.\n\nQuestions:\n - How is the neighborhood $\\mathcal{N}$ in section 5.1 defined?\n - What is the definition of $f$ in equation 8?\n - Are you assuming the kernel function $\\mathcal{S}$ in equation 10 to be strictly < 1?\n - What doe word-pieces ×N in table 1 mean?\n - How is the co-occurance of unigram and bigram defined?\n\nTypos and Grammars:\n - As mentioned before the graph embedding architectures captures capture...\n - $\\mathcal{C}(\\mathcal{N} (u)) = C(\\mathcal{N} (v)$\n - $\\mathcal{C}(\\mathcal{N}_l (u)) = C(\\mathcal{N}_l (v)$\n - Firstly, describe the construction of the graph for a particular domain and sub-graph induction based on the sample in the data.\n - ??? (missing citations)", "summary_of_the_paper": "In this paper the authors proposed continuous stable coloring (CSC) as a new framework to unify the understanding of several existing unsupervised learning algorithms, including Word2Vec, BERT, and Node2Vec. The authors show how the original stable coloring algorithm can be understood as optimizing the CSC objective function. The authors also show how to reduce the existing approaches to CSC.", "summary_of_the_review": "the paper is inspiring but major revision is needed", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new framework to express and analyze embedding methods based on the stable coloring problem. Reviewers highlighted as strengths that the paper provides an interesting perspective for understanding one of the central approaches in NLP, graph learning, and other fields --- and as such could inspire promising research directions. However, reviewers raised concerns regarding the significance of contributions (theoretical insights and analysis, relation to prior work, missing empirical evaluation etc.) as well as the clarity of presentation (also with regard to correctness and scope). All reviewers and the AC agree that the paper is not yet ready for publication at ICLR and would require an additional revision to address the aforementioned issues.
Decision: Reject