Chaining Data - A Novel Paradigm in Artificial Intelligence Exemplified with NMF based Clustering

1 INTRODUCTION . Until recently , clustering algorithms typically use either a table or at most two relational tables to analyze data . However there is a demand to fuse larger quantities of data for greater algorithmic success , we will show one reasonable method to do so . We adopt a simple notation of row× column to represent a relational table where the rows are items and the columns are features . Now we will begin with an example , community detection based topic modeling using document clustering on social media derived data with NMF , our table is a term × document table . In this example word vectors can be incorporated by adding a word vector table using MultiNMF via a word × word vector table . However this leaves large amounts of available data out of the factoring . We propose chaining several tables together for instance : hashtag × word , word × location , location × author , author × topics , as seen in Figure 1 . From this information , we may identify communities that are more broadly informed than simply the words present in the social media posts . At the heart of artificial intelligence is extracting previously unknown and potentially useful information from all available data . However , the problem is that this extent of data fusion using higher-order features has not been done for NMF in prior works . Although we focus on NMF using multiplicative update rules , MUR , Seung & Lee ( 2001 ) it is entirely possible to define a joint loss function , J , as seen in the three subsections below using automatic differentiation such as JAX , Torch , or TensorFlow . The reason to use MU over convex optimizers , alternating nonnegative least squares , and automatic differentiation is the reduced computational load and higher empirically observed quality of the results . However , we have implemented a MUR based algorithm that uses automatic differentiation that is faster and provides comparable results to MURs . To appreciate the challenges that arise in chained view NMF , we must consider that for multi-view NMF , we have a much easier to solve MUR and can easily incorporate more significant numbers of views than we could incorporate many linked relations . MultiNMF is based upon the precept that the primary key has features . The difference between multi-view clustering and traditional table based clustering is that the first treat features as sets of similar types , and the latter treats all features equally . However , linking relations can not be captured in either form of clustering directly nor graph based embeddings such as heterogeneous information networks , this difference is this work ’ s main contribution . 2 TYPES OF NMF . Here we present the NMF objective functions and its corresponding MUR . Next we show the Multiview NMF that factors multiple matrices related by the primary datatype . In this type we give the MUR rule and its generalization to any number of primary datatype interrelated matrices . Finally , we illustrate chained view NMF that allows linking of matrices in the same way as a relational database , i.e . feature matrices of features . In linked view we explored the analytic solution but due to computational hurdles we additionally defined an automatic differentiation to ascertain the positive and negative terms of the joint loss function thus allowing us to perform the MUR . 2.1 VANILLA NMF . All division is elementwise and is elementwise multiplication , finally × represents a relational dataset composed of rows whose type is before the × and the features whose type is after the × . An example of Vanilla NMF is a matrix of hashtags× tf-idf of words and yielding two matrices , W and H. W is the hashtag communities or embeddings and H is the community word usage . Loss = ‖V −WH‖2F ( 1 ) W←W VH T WHHT ( 2 ) H← H W TV WTWH ( 3 ) and for KL Divergence from Burred ( 2014 ) : Generalized DKL ( P ||Q ) = ∑ i [ P ( i ) log P ( i ) Q ( i ) − P ( i ) +Q ( i ) ] ( 4 ) Loss = DKL ( V ||WH ) = ∑ i , j [ Vi , j log Vi , j WH|i , j − Vi , j +WH|i , j ] ( 5 ) W←W V WHH T∑ iHi , : ( 6 ) H← H W T VWH∑ jW : ,j ( 7 ) 2.2 MULTINMF . For the case of 2 matrices such as hashtag X word and hashtag X location representing VA and VB : Loss = ‖VA −WHA‖2F + ‖VB −WHB‖2F ( 8 ) W←W VAH T A +VBH T B WHAHTA +WHBH T B ( 9 ) HA ← HA WTVA WTWHA ( 10 ) HB ← HB WTVB WTWHB ( 11 ) for the general case we have the following equation : Loss = ‖VA −WHA‖2F + ‖VB −WHB‖2F + ... + ‖VZ −WHZ‖2F ( 12 ) W←W VAH T A +VBH T B + ... +VZH T Z WHAHTA +WHBH T B + .... +WHZH T Z ( 13 ) HA ← HA WTVA WTWHA ( 14 ) HB ← HB WTVB WTWHB ( 15 ) ... ( 16 ) HZ ← HZ WTVZ WTWHZ ( 17 ) 2.3 OUR PROPOSED WORK - CHAINED VIEW NMF . What if the relational table that you would like to incorporate does not have the same type of rows as V. In star based schema and MultiNMF if V is hashtag rows and word frequency columns then you must use a hashtag row table , such as hashtag rows and location column . In chained view the requirement is relaxed such that any table can be used that has the same column type as V for instance VB has word rows and word vector columns this is possible . What becomes apparent is that by relaxing this requirement then any number of possible tables can be joined together for instance : hashtag rows to TF-IDF columns table can join a word row by location column and then join a location row to a author column table and finally a author row by topics column table . Because this follows the principle of foreign keys in a relational database , it is possible to directly use this algorithm to take a whole database and gives clusters based on every datapoint in it , this goes beyond a graph based topological view or embedded view of a table such as heterogeneous information networks . This is in effect a feature of features solution using higher order features . Using the example from before in the text we formulate the problem as such : Loss =‖VA −WHA‖2F+ ‖VB −VTAWHB‖2F+ ‖VC −VTBVTAWHC‖2F+ ‖VD −VTCVTBVTAWHD‖2F+ ... ‖VZ −VTYVTX ... VTBVTAWHZ‖2F ( 18 ) Again using the notation of a table as row×column rather than crossproducts , we have W is the hashtag communities of the database , VA throughVD are hashtag × tf-idf , word × location , location × author and finally author × topics respectively . Interestingly this also produces an embedding of the database for hashtags . It may be important to decrease the loss assigned to increasingly complex chains so they do not contribute to W the same as the most proximal tables . Alternatively one may solve for the last term in the loss function and by doing so incorporate all of the data matrices . Solving for the last term does have one drawback , it disproportionately weighs all data matrices equal , this may not be the intent . It may be necessary to experiment with different weightings of the terms by scaling the matrices proportional to their weighting , for instance taking the elementwise product of a number between 0 and 1 with the data matrix to be weighted . We will begin with the MUR rule using auto differentiation . We must take the loss function and convert it into individual terms that can then be entered into an automatic gradient solver . For instance Frobenius norms produce : Tr ( ( X − WH ) T ( X − WH ) ) , performing FOIL and using matrix properties you then have terms that can be solved by the autodifferentiation . The reason Frobenius norms are not used directly by the gradient solver is that they do not create the positive and negative terms individually that are needed for the MUR algorithm . Algorithm 1 Function : Multiplicative Update Auto Differentiation Input : Joint Loss Function Output : Matrix to Multiply by W , H , H2 , H3 , ... , Hn during update Initialisation : Define Positive and Negative Terms of the Joint Loss Function grad : Providing the Gradients through Auto Differentiation i.e . JAX for Specified Variable 1 : Gradients = Absolute Value of All Terms after Application of grad 2 : Numerator = Sum of All Negative Terms ’ Gradients 3 : Denominator = Sum of All Positive Term ’ s Gradients 4 : Denominator += Epsilon of Machine 5 : return Numerator / Denominator Loss = ‖VA −WHA‖2F + ‖VB −VTAWHB‖2F ( 19 ) W←W VAVBH T B +VAH T A VAVTAWHBH T B +WHAH T A ( 20 ) HA ← HA WTVA WTWHA ( 21 ) HB ← HB WTVAVB WTVAVTAWHB ( 22 ) The following is the KL divergence chained linkage MUR . The ones matrix in the shape of VB is found in Equation 23 . The ones matrix in Equations 24 and 25 is the shape necessary to allow element-wise division and products . W←W VA VB V TA WHB HTB + VA WHA HTA 1HTA + VA1H T B ( 23 ) HA ←HA W T VAWH W T1 ( 24 ) HB ←HB W TVA VB V TA WHB W TVA1 ( 25 ) The general case for Frobenius Loss is ( up to the third linkage or VD ) : Loss =‖VA −WHA‖2F+ ‖VB −VTAWHB‖2F+ ‖VC −VTBVTAWHC‖2F+ ‖VD −VTCVTBVTAWHC‖2F ( 26 ) This following is the pattern for any number of linkages . The pattern is expected to continue beyond 3 linkages i.e . past VD . 1 W←W ... +VAVBVCVDH T D +VAVBVCH T C +VAVBH T B +VAH T A ... + VAVBVCV TC V T B V T A HDH T D + VAVBV T B V T A HCH T C +VAV T AWHBH T B +WHAH T A ( 27 ) HA ← HA WTVA WTWHA ( 28 ) HB ← HB WTVAVB WTVAVTAWHB ( 29 ) HC ← HC WTVAVBVC WTVAVBVTBV T AWHC ( 30 ) HD ← HD WTVAVBVCVD WTVAVBVCVTCV T BV T AWHD ( 31 ) If there is a radial branching linkage from matrix VB that is called VE then the loss term added would be : ‖VE −VTBVTAWHE‖2F ( 32 ) The chain can then proceed from both VE and VC . Every time a branch is encountered a similar term is added . Chained view NMF can be combined with Multiview NMF by creating MUR ’ s according to this equation : 1Experimentally verified to converge for the first linkage θ ← θ ∗ ∇−θ f ( θ ) ∇+θ f ( θ ) ( 33 ) ∇+θ f ( θ ) represents the positive terms in the gradient . ∇ − θ f ( θ ) represents the negative terms in the gradient . 3 RELATED WORKS . In Ailem et al . ( 2017 ) we find an approach to encompass a linked datasets , however this is distinct from our approach and does not incorporate multiple chaining . This is formulated identically in Liu ( 2019 ) and called multirelational clustering in Equation 34 , again this does not incorporate 2nd or beyond chaining of matrices . Loss = 1/2||V −WH||2F + 1/2||M−WSW T ||2F ( 34 ) Where , V , W , H are the same as above and M is the U matrix in SVD for word vectors , S can be an identity matrix . NMF using MUR has been an effective method to decompose data matrices for unsupervised feature representations beginning with Seung & Lee ( 2001 ) . This idea was expanded to multiview NMF that is when there are more than one set of features for a primary key data type Liu et al . ( 2013 ) . It has been extended to social network community detection as noted in Shi et al . ( 2015 ) and Twitter political communities in Ozer et al . ( 2016 ) . Extending NMF to hypergraphs was done in Wang et al . ( 2016 ) . Like our proposed approach to find hashtag communities a similar work was performed in He et al . ( 2017 ) . Shen et al . ( 2017 ) analyzes the problem associated with social communities of spammers that spread malicious content using multiview NMF . Yang & Wang ( 2018 ) published in IEEE Big Data Mining and Analytics a comprehensive survey on multiview clustering including a section on MultiNMF . The modern trend has been to incorporate deep learning capabilities especially the attention mechanism as seen in , Chen et al . ( 2019 ) adaptation of MultiNMF to the attention paradigm . Additionally the capability to track social media using MultiNMF in the recent COVID-19 epidemic was done by Cruickshank & Carley ( 2020 ) . Star based schemas Sun et al . ( 2009 ) are similar to MultiNMF in that they analyze the data by the primary key , however the star schema is not like a radial tree or cyclical topology that chained NMF is capable of analyzing . Heterogeneous information networks by Sun & Han ( 2013 ) ; Yang et al . ( 2020 ) do analyze the higher order topologies however they are concerned with embeddings of the nodes and edges rather than directly analyzing the datapoints of each table and directly optimizing the loss to recreate the each linked table from its matrix factors simultaneously . This simultaneous recreation of all of the datasets from each table ’ s factors and the overall cluster factors is our novel contribution .

Summary: The paper studied clustering of data that has multiple dimensions of relation. It proposed a chained non-negative matrix factorization (NMF) technique, which, according to the authors, is different from existing multiple-view NMF. Chained NMF can model data that has chained relations (much like external key in relational database) and produce better clusters.  The authors evaluated Chained NMF with synthetic data that seems to resemble documents with hashtags and authors. NMI and ARI are both improved. Another illustrative evaluation was done on handwriting characters data.