Chaining Data - A Novel Paradigm in Artificial Intelligence Exemplified with NMF based Clustering
Authors: Norman J Mapes
Sumeet Dua
Venue: ICLR 2022 Submitted (2022)
Keywords: NMF
clustering
linking data
chaining data

Abstract: In the era of artificial intelligence there is an acceleration of high quality inference from the fusion of data and we have overcome the linking challenge associated with higher order features. We have fundamentally linked together tables of databases for clustering algorithms and expect this paradigm and those related to it to produce many new insights. We propose linked view clustering that is an extension of multi-view clustering by adding complementary and consensus information across linked views of each datapoint. While there are many methods, we focus on non-negative matrix factorization combined with the fusion of linking data in a manner that corresponds to extracting knowledge from the multiple tables of a relational database. It is commonplace to identify hashtag communities on social media by word usage, however there exists troves of data not included but could be. We can incorporate locations by hashtag to improve community detection, this is multiNMF or multiview clustering, but we extend this method to beyond the first link. A general artificial intelligence method to incorporate any table that can be chained backwards has not been done before to our knowledge. We call this linked view NMF or chained view clustering and give the algorithms to perform multiplicative updates and the general solution that can be solved using automatic differentiation such as JAX. We demonstrate how the equations can be interpreted on synthetic data as well as how information flows through the links and as a proof of concept on real data we incorporate word vectors using the method on an authorship clustering dataset.

Under review as a conference paper at ICLR 2022
OPTIMIZING NMF CLUSTERING Loss FUNCTIONS USING RELATIONAL TABLE FLOW OF INFORMATION FOR HIDDEN KNOWLEDGE DISCOVERY
Anonymous authors Paper under double-blind review
ABSTRACT
In the era of artificial intelligence, there is an acceleration of highh-qullity inference from the fusion of data. We propose a novel solution to the linking challenge asso- ciated with higher-order features. We have fundamentally linked together and fac- tored database tables using custom NMF loss functions and expect this paradigm and those related to it to produce many new insights. This algorithm extends multi-view clustering, star based schema and heterogeneous information networks by adding complementary and consensus information across linked views of each data point. While MultiNMF, multi-view clustering and star based schema incor- porates the first link and heterogeneous information networks focus on the graph there exist troves of data that still are not incorporated. Basically we extend het- erogeneous information network analysis to consider not just the edges, nodes and the embeddings of these nodes and edges, but we use the familiar keyed relation- ships of relational databases their tabular format. This optimizes for the struc- ture of databases more specifically than previous works by recreating a relational database from all of its tabular factors. To our knowledge, a general artificial in- telligence method to factor all tables in chain has not been done before. We call this chained NMF or chained view clustering and give the algorithms to perform multiplicatiie updates and the general solution that be solved using automatic differentiation such as JAX. We demonstrate how the equations can be interpreted synthetic data and how information flows through the links. As proof of con- cept on real data, we incorporate word vectors using the method on an authorship clustering dataset.
INTRODUCTION
Until recently, clustering algorithms typically use either a table or at most two relational tables to analyze data. However there is demand to fuse larger quantities of data for greater algorithmic success, we will show one reasonable method to do so. We adopt a simple notation of row x column represent relational table where the rows items and the columns features. Now will begin with an example, community detection based topic modeling using document clustering on social media derived data with NMF, our table term x document table. In this example word vectors can be incorporated by adding a word vector table using MultiNMF via word x word vector table. However this leaves large amounts of available data out of the factoring. We propose chaining several tables together for instance: hashtag word, word x location, location x author, author x topics, as seen Figure []] From this information, we may identify communities that are more broadly informed than simply the words present in the social media posts. At the heart of artificial intelligence is extracting previously unknown and potentially useful information from all available data. However, the problem is that this extent of data fusion using higher-order features has not been done for NMF in prior works.
Although we focus NMF using multiplicative update rules, MUR, Seung & Lee (2001) is en- tirely possible to define a joint loss function, J, as seen in the three subsections below using automatic differentiation such as JAX, Torch, or TensorFlow. The reason to MU over convex optimizers, alternating nonnegative least squares, and automatic differentiation is the reduced computational load and higher empirically observed quality of the results. However, we have implemented MUR

---

Under review as a conference paper at ICLR 2022
Hashtag
Word Location
Location
Word
Author
View 1
View
View
Authors
Embedding or Communities Hashtags
Topics
Duta Fusion
View
Black Hole Metries
Howkkng Rediaten
Bekenszin
Graviaatioal Radin
NMF insetton Point
Temperature
Luminosly
Hawking
Sayiiiectus A
Black Madels
Machematical Modleling
Newtrnan Physies
Black Physies
Thooey Equations
Based Simulations
Data
Inferenct
Physics Engine
Data Mining
Theory
Equeiios
Black Properties
Quantum Mechanics
Size
Trijectures
Coordmaess
Keplecio
blemens
Gureral Reinivay
Superpositon
Unuotaiity Principle
Figure 1: Chaining together views to gain whole view of multiple interrelated datasets with only requirement that features are keyed by other features. While linear chain is shown in the left of the figure, can actually be a radial branching chain as seen in the right of the figure. is shown symmetric, but there is requirement for symmetry. This method differs from graph embeddings because entire relational tables factors can reconstruct the nodes in the graph. (Bottom) Depicted fictitious relational dataset model relating black holes where arrows represent primary keys that are not provided. We propose performing analysis at the NMF insertion point Sagittarius A* analyzing at the datapoint level, this novel approach is different than analyzing the graph structure with graph embeddings or only the immediate neighboring tables. Clusters, factors and other NMF outputs are shown, but only the input structure in all three of the subfigures.
based algorithm that uses automatic differentiation that is faster and provides comparable results MURs. To appreciate the challenges that arise chained view NMF, we must consider that for multi-view NMF, have much easier to solve MUR and can easily incorporate more significant numbers of views than we could incorporate many linked relations. MultiNMF based upon the precept that the primary key has features. The difference between multi-view clustering and tra- ditional table based clustering is that the first treat features as sets of similar types, and the latter treats all features equally. However, linking relations cannot be captured in either form of clustering directly nor graph based embeddings such as heterogeneous information networks, this difference is this work's 's main contribution.
TYPES OF NMF
Here we present the NMF objective functions and its corresponding MUR. Next we show the Mul- tiview NMF that factors multiple matrices related by the primary datatype. In this type give the MUR rule and its generalization to any number of primary datatype interrelated matrices. Finally, illustrate chained view NMF that allows linking of matrices in the database, i.e. feature matrices of features. In linked view we explored the analytic solution but due computational hurdles additionally defined automatic differentiation ascertain the positive and negative terms of the joint loss function thus allowing us to perform the MUR.
way relational
2.1 VANILLA NMF
All division is elementwise and @ elementwise multiplication, finally x represents relational dataset composed of rows whose type is before the and the features whose type is after the x. An
2

---

Under review as a conference paper at ICLR 2022
example of Vanilla NMF is matrix of hashtags x tf-idf of words and yielding two matrices, W and H. W is the hashtag communities or embeddings and the community word usage.
Loss = V W 2
(1)
VH WHH
W - O
(2)
WTV WTWH
H H O
(3)
and for KL Divergence from Burred (2014):
P(i)
Generalized (xz(
Σ
P(i) log Q(i)
P(1) + Q(1)
(4)
VLJ WH i.j
Loss = (KL
= Σ
Vis log
V1 + WHI
(5)
4d
V H" WH Σ, H,
W W
(6)
WT V H WH
Σ W.
2.2 MULTINME
For the case of 2 matrices such as hashtag X word and hashtag X location representing Va and V/:
Loss
|VA
WH 11 + |VB
W
VAH + V B
W WO
(9)
WHA 4 + W
W
HA H WTWH,
(10)
WTVB
H H WTWH
(11)
for the general case we have the following equation:
Loss = V WHA V - WHB + ... + V2 WHz
(12)
V VAHT HT + V + +... + 22H
W W
(13)
WH + W
WH2H
W'VA
HA HA O WTWHA
(14)
WT B
Hg + HB O
(15)
WH
(16)
WTVZ
H H O
(17)
WHW
3

---

Under review as a conference paper at ICLR 2022
2.3 OUR PROPOSED WORK- CHAINED VIEW NMF
What if the relational table that you would like incorporate does not have the same type of rows as V. In star based schema and MultiNMF if V is hashtag rows and word frequency columns then you must use a hashtag row table, such as hashtag rows and location column.
chained view the requirement is relaxed such that any table can be used that has the same column type as V for instance VB has word rows and word vector columns this possible. What becomes apparent is that by relaxing this requirement then any number of possible tables can be joined together for instance: hashtag rows to TF-IDF columns table can join a word row by location column and then join a location row to author column table and finally author row by topics column table. Because this follows the principle of foreign keys in a relational database, is possible to directly use this algorithm to take whole database and gives clusters based on every datapoint in it, this goes heyond a graph based topological view or embedded view of a table such as heterogeneous information networks. This is in effect a feature of features solution using higher order features. Using the example from before in the text we formulate the problem as such:
Loss = V WH
[VB VT Il
IVc VEVVWHHOlE=
(18)
VV V
VV - V. W1
Again using the notation of table as row xcolumn rather than crossproducts, we have W is the hash- tag communities of the database, VA D are hashtag x tf-idf, word location, location x author and finally author x topics respectively. Interestingly this also produces an embedding of the database for hashtags. It may be important decrease the loss assigned to increasingly complex chains so they do not contribute to the same as the most proximal tables. Alternatively one may solve for the last term in the loss function and by doing so incorporate all of the data matrices. Solv- ing for the last term does have one drawback, it disproportionately weighs all data matrices equal, this may not be the intent. It may necessary experiment with different weightings of the terms by scaling the matrices proportional to their weighting, for instance taking the elementwise product of number between 0 and 1 with the data matrix be weighted.
We will begin with the MUR rule using auto differentiation. We must take the loss function and convert it into individual terms that can then be entered into an automatic gradient solver. For instance Frobenius norms produce: Tr((X - W W H)), performing FOIL and using matrix properties you then have terms that can be solved by the autodifferentiation. The reason Frobenius norms are not used directly by the gradient solver is that they do not create the positive and negative terms individually that are needed for the MUR algorithm.
Algorithm Function: Multiplicative Update Auto Differentiation
Input: Joint Loss Function
Output: Matrix to Multiply by W, H, H2, H3, .... Hn during update
Initialisation: Define Positive and Negative Terms of the Joint Loss Function
grad: Providing the Gradients through Auto Differentiation i.e. JAX for Specified Variable
1: Gradients = Absolute Value of All Terms after Application of grad
2: Numerator Sum All Negative Terms' Gradients
3: Denominator Sum of All Positive Term's Gradients
4: Denominator += Epsilon of Machine
5: return Numerator Denominator
Loss
|VA-
WH
IVa
VAWHo
(19)
4

---

Under review as a conference paper at ICLR 2022
VAVEH + VAH
W
(20)
W + WHAHA
WTVA
HA + HAC O
(21)
W
V
VV
H WT 4 WHB
(22)
The following is the KI. divergence chained linkage MUR. The ones matrix in the shape of VB
is found in Equation 23 The ones matrix in Equations 24 and 25 is the shape necessary to allow element-wise division and products.
VD VA VT H + WHA HT
W
(23)
IH A + VALH
WT VA WH
HA HA WT1
(24)
VB WT
VVTW WHR H
HB t HB O
(25)
WT
The general case for Frobenius Loss is (up to the third linkage or VD):
Loss
=|[ A +
|[ V WAWH, 1
(26)
IVC VE W
IVD V.VVVV
This following is the pattern for any number of linkages. The pattern is expected to continue beyond 3 linkages i.e. past VD. |
V V.V.V + V. + VAV.H + +V H
W Wo
VAV HoH VAVBV H + WH, + H
(27)
WTVA
HA + HA O
(28)
WTWHA
V B
H - H Ø W WHE
(29)
WVV
Hc t Ho O WT
(30)
HD + HD ④ WVV.V If there is radial branching linkage from matrix VB that is called VE then the loss term added would be:
(31)
IVF V The chain can then proceed from both VE and Vo. Every time branch is encountered similar term is added.
(32)
Chained view NMF can be combined with Multiview NMF by creating MUR's according this equation:
Experimentally verified to converge for the first linkage
5

---

Under review as a conference paper at ICLR 2022
0 0x V f(0) V f())
(33)
V f0) represents the positive terms in the gradient. Vo f(0) represents the negative terms in the gradient.
3 RELATED WORKS
In Ailem et al. (017) we find an approach to encompass a linked datasets, however this is distinct from approach and does not incorporate multiple chaining. This is formulated identically in Liu (2019) and called multirelational clustering in Equation 34] again this does not incorporate 2nd or beyond chaining of matrices.
Loss 1/211V WH 1/21IM WSW 1
(34)
Where, V,W.H are the same as above and M is the U matrix in SVD for word vectors, S can be an identity matrix.
NMF using MUR has been an effective method decompose data matrices for unsupervised feature representations beginning with [Seung & Lee (2001). This idea was expanded to multiview NMF that when there are more than one set of features for primary key data type Liu al. (2013). has been extended to social network community detection as noted in [hhi et al. (2015) and Twitter political communities in Ozer al. (2016). Extending NMF to hypergraphs was done in Wang et al. (216)). Like proposed approach to find hashtag communities similar work was performed in He et al. (017)). [hen et al. (017)) analyzes the problem associated with social communities of spammers that spread malicious content using multiview NMF, Yang & Wang (2018) published in IEEE Big Data Mining and Analytics a comprehensive survey on multiview clustering including a section MultiNMF.
The modern trend has been to incorporate deep learning capabilities especially the attention mecha- nism seen in, Chen al. (2019) adaptation of MultiNMF the attention paradigm. Additionally the capability to track social media using MultiNME in the recent COVID-19 epidemic was done by Cruickshank & Carley (2020). Star based schemas Sun al. (2009) are similar to MultiNMF in that they analyze the data by the primary key, however the star schema is not like radial tree or cyclical topology that chained NMF capable analyzing. Heterogeneous information networks by (un & Han (2013): Yang et al. (2020) do analyze the higher order topologies however they are concerned with embeddings of the nodes and edges rather than directly analyzing the datapoints of each table and directly optimizing the loss to recreate the each linked table from its matrix factors simultaneously. This simultaneous recreation of all the datasets from each table's factors and the overall cluster factors is our novel contribution.
4 EXPERIMENTS AND CASE STUDIES
4.1 SYNTHETIC DATA LINKAGE INFORMATION PROPAGATION PERFORMANCE, CHARACTERISTICS AND VISUALIZATION
The choice of matrix size in the following experiment section is arbitrary, but the rule when linking matrices the columns of one must correspond the rows of the next matrix. In practice this not problem because if there is a matrix of words on the rows x documents in the columns, the matrices that can link are either properties of the documents or the words; for example word vector or document vector matrices.
We created three matrices VA. VB and Vc. The first matrix, VA. 630 350. The rows represent hashtags and the columns geographic locations. The second matrix, VB has 350 datapoints and 1000 features. The columns or features represent words. So is basically a graph where each geographic location has predominant words used. Finally the thirds matrix is 1000 x 20 representing 1000 "words" and 20 features such as word vectors. Notice the matrix multiplication is possible by 630 x 350 350 1000 1000 x 20, this multiplication is key to our insight.

---

Under review as a conference paper at ICLR 2022
Table 1: Synthetic Dataset Performance Characteristics
CHARACTERISTIC MEAN NMI
STDEV NMI LINKAGES
Third/Goal Matrix Alone
0.90
0.04
Second Matrix Alone*
0.26
0.06
0
First Matrix * **
0.11
0.05
Second Matrix Chained*
0.49
0.03
First Matrix Ch
0.20
0.04
The information that will flow through is from the 7 groups who align with geographic location, word frequency and word vector style.
We synthesized the first matrix by a sparse matrix with 5% noise, the default sparse value is 0 and the noise is a 1. Then we added random I's in specific rectangular pattern along the diagonal, this is visible in the figure and allows the information to flow. If we shuffled the matrix by row these yellow boxes would disappear but we shuffled the true values identically the matrix would still be useful. has not been shuffled so that the process can be inspected. The groups can directly be found by performing vanilla NMF or by chaining to a second or even third matrix. The performance increases and the nature of the clusters changes as the complementary and consensus information is factored with chain view NME We synthesized a second matrix like the first. Finally, we synthesized the third matrix to have blobs in 20 dimensions, this represents 1000 people groups measured in 20 different ways.
Table 3 contains the NMI or normalized mutual information score for the different number of link- ages 0-2. This was provided by the Scikit-Learn module in Python.
4.2 INTERPRETATION LINKED MATRIX INFORMATION PROPAGATION AND VISUALIZATION
To understand how information flows through the data tables we illustrated the dot products as they appear in the equations above and plot them in dimensions on a synthetic Gaussian grouping as fol- lows in Figure 3 and Table In Figure 3 we viewing whether clusters remain intact after chain- ing. While a real dataset would not be as well behaved, we can still demonstrate the preservation of clear clusters through matrix multiplication in the loss functions. For instance, V B VTHHe approximates VD. where V....V VD are simply relational database tabular matrices. When V VA..VV Ve multiplied together and then multiplied with the overall factor matrix W and the factor matrix He we recreate V D. Thus, the decomposition of the relational database can be understood to pre- serve clusters that will occur large row column values in the factor matrices. We will prove that the approximation products don't 't just allow for the same dimensions.
VA = WHA
(35)
VB V WH
(36)
by substitution:
VB H WT WHB
(37)
From this we observe 7 W is the Gram matrix, in data science this is a linear kernel that measures similarity and has rows and columns equal cardinality to the number of clusters NMF. Since we are multiplying the column factors of VA stored in HA by the similarity of the clusters by the column factors of VB stored Hg. This gives us the columns of VA and the columns of VB aka the rows and columns of VB, thus proving that our approximations are adequate or that Equation 37 is reasonable.
2* and ** significantly different level of 0.001
7

---

Under review as a conference paper at ICLR 2022
Chaining together linked views for improved embeddings and clustering.
The First Matric, Location Words
Vorsa. through Second
Third Alone
200
50
1BC
150
200
250
300
The Second Matrixx words blobs with 5% Sparse Noise. Chain to Third Matrix Alone
600
600
The Third Matrix, (shown are the first 2 simensians), as scatter plot of gaussian blabs, (Goal Matrix]
40
35
30-
20-
15-
3 30
15 20
25
30
Figure 2: Chaining together three matrices, 1000x20 to 350 1000 to 630x350. This chain could also have been done real data for instance: hashtag x location graph to location x word topic graph finally to a word topic x continuous feature blob dataset. Chained views improve the ability to discern clusters in each of the linking matrices and give unique interpretation that depends on which information or matrices are included in the chain. (In the blobs, we show the first 2 dimensions of the 20 dimensional dataset.)
Mean Input (Matrix 1)
Mean Input 2 (Matrix 2)
Arithmetic Output Mean (Inner Product)
0.8
3
20 0,8 $ 3,0 = 48
0.8
5
20 0.8
5,0 80
2.8
3
20 2.8 3,0= = 168
2.8
5
20 2.8 * 5.0 280
Table 2: Truth table for multidimensional input signals to output modeling the flow of information for NMF clustering of chained data. The number of features in matrix is 20.
4.3 KL DIVERGENCE CHAINED-JAX vs VANILLA KL FOR 50 AUTHORS
Chained NMF has more usefulness than multiview NMF that cannot handle the case of documentx word and wordx word vector. We incorporated word vectors with chained view using JAX on the 50 authors C50 dataset. We used the Google 300 dimension word vectors all thresholded at 0 that the data was non negategae.v.
Additionally, we analyzed the 20 Newsgroups using chained link KL with word vectors vs Vanilla KI. and observed slight decrease in NMI and ART but better clusters overall found in Table
8

---

Under review as a conference paper at ICLR 2022
Table 3: C50 Dataset Performance Characteristics
Cluster Mode and Metric
RUN
RUN 2
RUN 3
RUN 4
P-VALUE
KI. Chained-Jax and Word Vectors NMT
0.46
0.45
0.47
0.49
< 0.01
KI Vanilla NMI
0.42
0.42
0.40
0.41
See prior row
KL Chained-Jax ax and Word Vectors ARI
0.18
0.17
0.18
0.20
< 0.05
KL Vanilla ARI
0.17
0.16
0.14
0.14
See prior row
ALGORITHM TIME AND SPACE COMPLEXITY ANALYSIS
Inspection of the gradient shows four salient aspects. The first is the radial structure of the input, this an factorial type of complexity. Secondly, because we are factoring radial explosion of matrices, if in data science there is a law of data fusion similar to the law of large numbers in statistics then we can say that if there are enough matrices factored then there are certain aspects of hidden knowledge that change less and less the more complementary and consensus tables are factored. This invariance hidden knowledge discovery is strong motivation to use this method despite its poor complexity analysis for the analytic solutions. Fourth is that the autodifferentiation is not necessarily performing all of the operations that would be done in an analytical solution. While our method performing computational decades on high performance machines we can quickly calculate the analytic terms of the solution using autodifferentiation. Empirically, we observed errors diminishing monotonically using the target matrix and a linking matrix, we see no reason for third matrix and beyond links perform differently as they are based upon the same principle. future work would involve computational proofs because we use the automatic gradient solvers for efficiency. Fourth is that all matrix operations are parallelizable.
modest sized matrix operations will quickly run into TB of memory and
DISCUSSION
The hope of this work is allow researcher to collocate all of the relevant data into the model and see what is previously unknown and potentially useful. Their future task will be what consensus of knowledge can produce, with the hope of a deeper understanding of phenomena that cannot be attained using any single or pair of source data. Artificial intelligence approaches to incorporating wide range of data has the potential to transform what is considered the lack of algorithmic com- mon sense reasoming to something approaching the ability of human reasoning. We are motivated to extend our methods by research that discovers emergent social networks such as Yang et al. (2013). this is complementary method to ours that analyzes graphical topologies and features of nodes, how- ever we extended to the varying features and items a relational database but would like to explore incorporation of the graphical parameters into our method. Additionally, the work by Gong et al. 2013) shows a unique particle swarm methodology that integrates complex networks for clustering, would like incorporate multiobjective optimization into decomposition of relational tables, particle swarm has proven many times to be competitive in results and would be a promising future direction of research.
Multiplying chained matrices together mimics the principle of dilution, for instance if a large matrix multiplies a very small one then most of the information lost going into the smaller, at least becomes more information dense. Appropriately, there is nowhere to store the data in the smaller matrix product. Also large matrix to large matrix allows the flow of information readily. And in the third case small matrix does not have much information to pass through to a larger matrix.
7 REPRODUCIBILITY STATEMENT
In the appendix we provide the code used to compare the vanilla NMF to chained link NMF, we however leave the TF-IDF implementation out because it was from Scikit-Leamm using Python 3.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Melissa Ailem, Aghiles Salah, and Mohamed Nadif. Non-negative matrix factorization meets word embedding. In Proceedings of the 40th international sigir conference on research and devel- opment in information retrieval, pp. 1081-1084, 2017.
Juan José Burred. Detailed derivation of multiplicative update rules for nmf. Paris, France, 2014.
Hongling Chen, Mingyan Gao, Ying Zhang, Wenbin Liang, and Xianchun Zou. Attention-base multi-nmf deep neural network with multimodality data for breast cancer prognosis model. BioMed research international, 2019, 2019.
lain J Cruickshank and Kathleen M Carley. Characterizing communities of hashtag usage on twitter during the 2020 covid-19 pandemic by multi-view clustering. Applied Network Science, 5(1): 1-40, 2020.
Maoguo Gong, Qing Cai, Xiaowei Chen, and Lijia Ma. Complex network clustering by multi- objective discrete particle swarm optimization based on decomposition. IEEE Transactions evolutionary computation, 18(1):82-977 2013.
Chaobo He, Xiang Fei, Hanchao Li, Yong Tang, Hai Liu, and Qimai Chen. A multi-view clustering method for community discovery integrating links and tags. In 2017 IEEE 14th International Conference on e-Business Engineering (ICEBE), pp. 23-30. IEEE, 2017.
Jialu Liu, Chi Wang, Jing Gao, and Jiawei Han. Multi-view clustering via joint nonnegative matrix factorization. In Proceedings of the 2013 SIAM international conference on data mining, pp- 252-260. SIAM, 2013.
Kai Liu. A Study of Non-Negative Matrix Factorizations: Foundations, Methods, Algorithms, and Applications. Colorado School of Mines, 2019.
Mert Ozer, Nyunsu Kim, and Hasan Davuleu. Community detection in political twitter networks using nonnegative matrix factorization methods. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 81-88. IEEE, 2016.
D Seung and L Lee. Algorithms for non-negative matrix factorization. 13:556-562, 2001.
Hua Shen, Fenglong Ma, Xianchao Zhang, Linlin Zong, Xinyue Liu, and Wenxin Liang. Discover- ing social spammers from multiple views. Neurocomputing. 225:49-57, 2017.
Xiaohua Shi, Hongtao Lu, Yangchen He, and Shan He. Community detection in social network with pairwisely constrained symmetric non-negate non-negative matrix factorization. In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015, pp. 541-546, 2015.
Yizhou Sun and Jiawei Han. Mining heterogeneous information networks: structural analysis approach. Acm Sigkdd Explorations Newsletter, 14(2):20-28, 2013.
Yizhou Sun, Yintao Yu, and Jiawei Han. Ranking-basee clustering of heterogeneous information networks with star network schema. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 797-806, 2009.
Wenhong Wang, Yuntao Qian, and Yuan Yan Tang. Hypergraph-regullaizee sparse nmf for hyper- spectral unmixing. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(2):681-69) 1-94, 2016.
Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. Heterogeneous network repre- sentation learning: A unified framework with survey and benchmark. IEEE Transactions on Knowledge and Data Engineering. 2020.
Jaewon Yang, Julian McAuley, and Jure Leskovec, Community detection in networks with node attributes. In 2013 IEEE 13th international conference on data mining, pp. 1151-1156. IEEE, 2013.
Yan Yang and Hao Wang. Multi-viww clustering; A survey. Big Data Mining and Analytics, 1(2): 83-107, 2018.
10

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 CAVEATS
Weakness of this method: The parameters must match exactly to "chain" together tables. However should always be possible to include all of the attributes of relational database to find an embedding of target feature.
A.2 PROOFS AND MUR RULES IN CODE FOR REPRODUCING RESULTS
import jax.numpy numpy as jnp from jax import grad
# for Chained Link NMP using KL and Matrix Operations for epoch in range (200)
num np.divide (Y, np. lelalllllg...................l.l aainalg.mmll lwm.muliiddde dot ( (x, num, H2.T))
num np. wpp.lnalgg.mlti..om
+- np. (x, CH.T
den np.oness (W.shape [0], shape )
den np. np.linalg.multidde ( (x, np. sp..np.oee ones (Y. (Y.shape) H2.T))
den [den == 0] num / den
WdTppp. diniid divide (X, W(H)
den w. W.Tenp. ones ( (W. shape [0] H. shape [1]))
den den 0] eps_maching
H *= num / den
np. divide (Y, np np. lipalgn .multmmdooott (X.T, (, num
( (W.T, X, num)
den dlinalg.multe dot (W.T,X, ones (X. shape [1] H2 shape (1]))))
den [den == 0] eps aps_machine
H2 num / den
# for Chained Link NMF using KL and JAX jnp.sum hwpp.pmllpplyyyco (X, inp. log (jnp divide (x, )
t1 lambda X, W, (-X)
t2 = lambda X: jnp.sum
sum
t3 lambda W, H: jnp. sum (WCH)
t4 = lambda Y, X, W, H2: jnp. sum finp.multiply. (Y, jnp. (jnp. divide (Y, X. T@W0H2) )
t5 lambda Y: jnp.sum (-Y)
t6 = lambda x, W, H2: jnp. (x. (QW02H
terms (21,t2, t3, t5,
Args = lambda: (X (X,), ) HW, (Y, (,, W, H,H (Y,), (X, H2)
Windex [1,None, None,
Hindex = [2, None, 1, None, None, None]
H2index [None, None, None, 3, None,
MU (Xindex)
args Args ()
grads [grad (t, argnums=i) (*a) for a,i,t in zip Xindex, terms) if i is m
numerator
(g) for g in grads if np.any (g 0)
numerator = np.sum (numerator,axis-]
denominator = [g for in grads if ap.any 0)]
denominator np. sum axx return numerator (denominator: eps_machine)
11

---

Under review as a conference paper at ICLR 2022
for epoch in zange (200):
W == MU (Windex)
H MU (Hindex)
H2 == MU (H2index)
###t# for Chained Link NMF using Frobenius and #* or Matrix Operations
min 1 ||X-WH|| + ||Y-XTWH2 s.t. W, H, H2 >=
tr (X-WH) (X-WH) tr ( (Y-XTWH2) T (Y-XTWH2))
t1 lambda X: jnp. jap.trace Tex)
t2 = lambda x, W, Н: -jnp. TQWE))
t3 lambda
Н: -jap.trace [H.TeW. TeX)
t4 = lambda W, н: inp.trace TWW.
t5 = lambda Y: inp. tracerace TeY)
t6 lambda Y, X, W, H2: -inp. trace (Y. TQX. TQWCH2)
TQW. t7 lambda Y, X, W, H2: -jnp. IQW. TБXOY)
t8 lambda X, W, H2: jnp. (H2, Tew T@xex. T@W0H2)
for epoch in range (200):
Wnum = X0H.T + np. diiinalg.multi_do ([X, x. (,
Wden np.
([W H.T]) +
[x, Y, Y, H2.T))
Wden [Wden==0]
aps_machine
#W *= MJ (Windex)
Wnum/Wden
Hnum W.TOX
Hden mp.
N, Hden [Hden 0] eps_machine
H += sinum/Hden
#H MU (Hindex)
H2num np lonallallggll laalll.multidoooo ([W.T,X,
H2den np.
dot (IW.T, X,,
H2den H2den 0] eps_machine
ul H2num / H2den
#u2 MU (H2index)
##Basic Procedure for Testing AutoDiff Gradient Matches Matrix Operation 44 (Proof of Correctness)
from jax import grad
import jax.numpy as jnp
import numpy as np
n=mp.randdo.norma.l..
(15,20))
(20,25))
nemp.ra (0,1,
I=ap.rrando..norma (0,1, (7,20)
r2= (7,1,)),25)
[np.abs (x) for x in H, t4 lambda Y, X, W, H2: jnp.sum (jnp.multiply (Y, log.
H2 = x, H,
(jnp. divi (Y, x. T(WCH2)
gl = grad (t4, argnums==) (Y,X, W, H2)
num ap. divide (Y, np www.llnnn..... wwl.immlttoooooooee
(X.T, )
num = mp.linalg d.iinalg.multii. mulliidoottttto ( (W. (,.T,X,num)
g2 num
12

---

Under review as a conference paper at ICLR 2022
if np. allclose (g1, - : print ("H2 Num is good.")
A.3 RESEARCH FRONTIERS IN MINING HETEROGENEOUS INFORMATION NETWORKS
Quoting from Sun & Han 2013), "Viewing interconnected data an information network and studying systematically the methods for mining heterogeneous information networks is promising frontier in data mining research... How to discover such hidden networks and how to mine knowl- edge (e.g., clusters, behaviors, and anomalies) from such hidden but non-isolated networks (i.e., still intertwined with the gigantic network in both network linkages and semanties) could be an inter- esting but challenging problem." This latest NMF algorithm extended from MultiNMF is proposed to be called chained NMF. We believe the greatest discoveries yet to be found and this papers impact in this interesting and challenging problem pushes the envelope of the possible and future re- search will no doubt unlock doors to finding hidden knowledge in data by analyzing entire volumes of data that have a wide variety of types.
A.4 MATRIX PRODUCT CLUSTER PRESERVATION VISUALIZATION
Information Flows Two Data Matrices Chained 100 Datapoints in clusters
First 25 dot N(3,1)
First 25 dot N(5,1)
Second 25 dot N(3,1)
Second 25 dot N(5,1)
250
200
150
100
50
300
250
200
100
150
150
200
100
250
50
300
Figure 3: Chaining visualization controlled synthetic dataset. Notice the clusters easily visual- ized corresponding to varying the input from either matrix, see Table
A.5 REASONABLE IMPROVEMENTS IN QUALITY OF CLUSTERS USING CHAINED NMF ON NEWSGROUPS.
Typically the best clusters are achieved using KL Divergence Loss for NMF with MUR. We com- pare them qualitatively chained NMF using Google 300 dimension wordvectors for of the newsgroups in the table
13

---

Under review as a conference paper at ICLR 2022
Table 4: 7 Newsgroups Characteristic Cluster Comparison (subset of 20 Newsgroups)
Chained Link NMF KL 7 Cluster Word Choices for Newsgroups
isracl, isracli, government, said, state, palestinian, part, arab, war, during, peace, against, back. year, children, off, agree, soldiers, american, troops
sci.electronies
power, circuit, battery, edu, voltage, light, space, current, radio, sky, elec- tronics, output, thanks, 10, energy, information, msg, tv, off, temperature
sci.med
she, her, medical, doctor, disease, pain, patients, treatment, cancer, go, food, cause, drug, medicine, ago, patient, really, effects, problem, wife
sci.space
space, launch, ground, money, msg, nasa, cost, moon, high radar, re, low, costs, dc, funding, still, safety, buy, long, real
sci.crypt
key, chip, clipper, encryption, government, phone, keys, system, nsa, algo- rithm, law, security, escrow, des, number, public, crypto, data, bit, secure
comp.graphics
thanks, graphics, program, please, image, file, files, software, mail, look- ing. ftp, space, format, hi, advance, images, help, gif, pc, windows
w...eligion.christii
god, jesus, church, jews, christian, believe, christ, christians, bible, israel, him, armenian, faith, jewish, armenians, sin, world, life, turkish, christian- ity
Vanilla NMF KL 7 Cluster Word Choices for 7 Newsgroups 1993, during, april, armenians, argic. 10, armenian, between, 93, ground, 000, center, last, 12, 30, 19, against, city, air, russian
talk.politics.mide mideast
sci.electronics
better, company, around, few, enough, actually, buy, back, going, re, cost, big, 10, off, build, go, long, probably, got, down
sci.med
edu, cause, banks, back, case, cadre, doctor, effect, ago, gordon, medical, chastity, disease, help, soon, anything, deleted, intellect, day, geb
sci.space
thanks, please, mail, advance, space, email, book, nasa, looking, another, information, anybody, articles, look, Il, contact, able, around, copy. help
sci.crypt
clipper, chip, key, government, encryption, system, algorithm, public, keys, data, com, bit, phone, nsa, security, using, chips, secret, already, anything
comp.graphics
thanks, graphics, file, available, looking, please, files, ftp, program, image, advance, help, hi, appreciated, 3d, bit, code, format, etc, color
ww.religion.christian
god, believe, christian, israel, him, church, again, against, come, both, fact, christians, christ, between, agree, another, bible, jesus, however
14


Reviews:
[{"review_id": "owtrxhJ9ZrI", "reviewer": "Reviewer_QKqU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The authors studied the ease of real-world application. They discussed the viability of using commonly available auto-differentiation tools (JAX).\n- The use of two different kinds of dataset to test the algorithm: handwriting and documents.\n\nWeakness\n- Writing needs substantial improvements. It's generally hard to follow the paper. For example there are sudden turns of topic (Abstract: \"..it is commonplace to identify hashtag ...\"), and very easy to spot errors (Introduction, 2nd paragraph, \"A multiNMF is where the primary key ...\" is repeated).\n- Section 2.1. The use of hashtag and words is sudden, Reviewer expect some introduction on the task: recommendation or prediction?\n- Equation 1, what is V?\n- Section 4.3. \"We attained the following results\" but the authors continued to discuss experiment set up, not the results.\n- There are many other similar cases of writing inconsistency. Reviewer won't list them all.\n", "summary_of_the_paper": "The paper studied clustering of data that has multiple dimensions of relation. It proposed a chained non-negative matrix factorization (NMF) technique, which, according to the authors, is different from existing multiple-view NMF. Chained NMF can model data that has chained relations (much like external key in relational database) and produce better clusters.\n\nThe authors evaluated Chained NMF with synthetic data that seems to resemble documents with hashtags and authors. NMI and ARI are both improved. Another illustrative evaluation was done on handwriting characters data.  \n", "summary_of_the_review": "The paper discussed a way to cluster a dataset with multiple chainable joining keys. The authors provided mathematical discussion and evaluation on a small synthetic dataset. However, the paper itself is poorly written, to a point that it's hard for Reviewer to understand how the authors arrived at the main idea and what its strength and weakness are. Clarity is the main weakness.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Z9CgHNmvvu", "reviewer": "Reviewer_k2WG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strong points__: Overall, the paper’s strengths are the technical novelty of the proposed chain-view NMF and the impact of the problem it is addressing.\n\n- The paper presents a new technique for dealing with a real-world data problem that is straightforward in its technical grounding. The extensions the paper proposes to NMF are both conceptually straightforward and a rather elegant way of attacking the data scenario of linked and multi-view data\n- The problem of finding good representations of phenomena, when those phenomena can be described by both multi-view and linked data, is a significant problem. Being able to successfully do this task can lead to better outcomes in fundamental machine learning tasks like clustering.\n\n__Weak points__: Overall, the paper’s weaknesses are in its clarity and its empirical significance. \n\n- The paper has some serious clarity issues that detract from the reader being able to understand the paper and contributions of the paper. The paper needs to be reviewed for grammar and typos, as they are frequent and do detract from understanding the text. One example is at the bottom of page 1, the second paragraph of the introduction, wherein the sentence “A multiNMF is where…” is repeated. Another example is in section 2.1, where it looks like from the text that “Hashtags _X_ tf-idf …” implies that that matrix should be $X$, but it is instead $V$ in the subsequent equation.\nAlso, within the proposed contribution of section 2.3, while it is clear how to deal with the situation of linked data (by multiplying those $V_*$ matrices into the second part of the loss function, it's not clear how to deal with multi-view data (where you have two or more data sets, or tables, with the same row entities). Do you just add an extra term (same as the original term, but with the W being the same across terms) to the loss function for this other table? If so, how do you control for the different impacts of the added together parts of the loss function (i.e. would you weight the different sub-components of the loss function)? This also leads to the intriguing idea of how one should model their data relationships to get the best cluster structure, especially when given the design choice of whether to make a certain table of data chained or multi-view. Also, the intuition idea of chaining the matrices by the cartesian product should be explained. While it certainly makes the matrix math work out, why is that the best way to propagate information from those matrices?\nFigure 2, while a neat image, really doesn’t seem to contribute much actual information to the paper, since the visual inspection isn’t stated as an empirical evaluation test. This space would be better used for either more explanation on the empirical setup (more on that later) or doing more empirical tests.\n\n- The empirical investigations are unclear as to how the experiments were set up and evaluated For example, what clustering was done for the AMI and ARI values? And, why is the synthetic test set up different than the real-world data set up? It seems like both should share the same test setup. For example, why does Table 2 look at NMI and ARI, while Table 1 does not and why does Table 1 consider various different ways of chaining and Table 2 does not? Finally, the word vectors in 4.3, is truncated the negative values really the best way to make sure those vectors are positive? Won’t this cause a loss of critical information, since they are embeddings in a real-valued space (implying negative values contain information)? Perhaps a better way would be to do a transformation of them into a positive space by scaling and/or normalization.\nThe empirical investigations, from Table 2, really do not show much performance improvement over the KL-optimized standard NMF, especially when one considers the chained version requires so many more matrix operations than just a single view NMF. \n\n- Finally, the paper overstates its claim about the proposed technique being a novel paradigm in artificial intelligence. The idea of using more than one view of data to understand phenomena is the foundation of multi-view machine learning, and the idea of using linked data underlies techniques in Heterogeneous Information Networks and complex network clustering. So, while it is a new technique, the paper does not present a new paradigm.\n", "summary_of_the_paper": "The paper presents a new technique, grounded in non-negative matrix factorization (NMF), for unsupervised, representation learning of linked data. The paper presents a technique that can deal with data scenarios whereby a certain phenomenon of interest has multiple views to characterize that behavior (i.e. traditional multi-view clustering set up), but also the data in those views can be linked to further data, as in a relational data. ", "summary_of_the_review": "I recommend the paper be accepted. While it does need some serious work in clarity and maybe some additional empirical investigation, it does present a new technique that is attacking a meaningful problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M8kSwJf9mZ", "reviewer": "Reviewer_RkXc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the idea of incorporating star-schema information is not new, this paper proposes a new approach to model the non-multi-view information. There have been several works in heterogeneous networks such as NetClus [1] and CESNA [2] where the heterogeneous information sources (feature tables) have been clustered. It is not clear how the chained NMF modeling improves over the existing clustering models in the prior literature. \n\nI believe more work needs to be done to understand the convergence of the algorithm, the time and space complexity analysis. Further, more real-world applications should be tried like heterogeneous database-based tasks to understand the utility of chained NMF modeling.  \n\n\n[1] Sun, Yizhou, Yintao Yu, and Jiawei Han. \"Ranking-based clustering of heterogeneous information networks with star network schema.\" Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 2009.\n[2] Yang, Jaewon, Julian McAuley, and Jure Leskovec. \"Community detection in networks with node attributes.\" 2013 IEEE 13th international conference on data mining. IEEE, 2013.", "summary_of_the_paper": "This paper proposes a linked NMF based clustering to tackle the problem of linked dataset. The authors shows that chained NMF provides new way to incorporate information and interpret results in comparison to the multi view clustering approaches which are limited to the same entities across tables (or different views of the same entity). Experimentally, the authors show that chained NMF perform better than vanilla NMF at clustering Newsgroups. ", "summary_of_the_review": "While this paper provides novel idea, it lacks some comparison and justification in comparison to the existing literature. More work needs to be done to justify the modeling choice. Also, more experimental work needs to be done to justify the utility of the above model. In light of the above observations, I have made my recommendation of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach for linked-view clustering based on chained non-negative matrix factorization. Reviewers highlighted that paper proposes a novel and interesting approach to an important problem. However, reviewers raised also significant concerns regarding clarity of presentation (motivation, general approach, contributions, scope) as well as the experimental evaluation. Reviewers raised also concerns regarding justification of the approach being a novel paradigm. After author response and discussion, all reviewers and the AC agree that the paper is not yet ready for publication due to the aforementioned issues.
Decision: Reject