MODiR: Multi-Objective Dimensionality Reduction for Joint Data Visualisation
Authors: Tim Repke
Ralf Krestel
Venue:  (2020)
Keywords: dimensionality reduction
visualisation
text visualisation
network drawing

Abstract: Many large text collections exhibit graph structures, either inherent to the content itself or encoded in the metadata of the individual documents.
Example graphs extracted from document collections are co-author networks, citation networks, or named-entity-cooccurrence networks.
Furthermore, social networks can be extracted from email corpora, tweets, or social media. 
When it comes to visualising these large corpora, either the textual content or the network graph are used.

In this paper, we propose to incorporate both, text and graph, to not only visualise the semantic information encoded in the documents' content but also the relationships expressed by the inherent network structure.
To this end, we introduce a novel algorithm based on multi-objective optimisation to jointly position embedded documents and graph nodes in a two-dimensional landscape.
We illustrate the effectiveness of our approach with real-world datasets and show that we can capture the semantics of large document collections better than other visualisations based on either the content or the network information.

Under review as a conference paper at ICLR 2020
MODIR: MULTI-OBIECTIVE DIMENSIONALITY RE- DUCTION FOR JOINT DATA VISUALISATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many large text collections exhibit graph structures, either inherent to the content itself or encoded in the metadata of the individual documents. Example graphs extracted from document collections are co-author networks, citation networks, or sereeddennttt-cocccueeee
networks. Furthermore, social networks can be ex- tracted from email corpora, tweets, social media. When it comes to visualising these large corpora, either the textual content or the network graph are used.
In this paper, we propose to incorporate both, text and graph, to not only visualise the semantic information encoded in the documents' content but also the relation- ships expressed by the inherent network structure. To this end, we introduce a novel algorithm based on multi-ohjective optimisation to jointly position embed- ded documents and graph nodes in a -wo-dimensional landscape. We illustrate the effectiveness of our approach with real-world datasets and show that we can cap- ture the semantics of large document collections better than other visualisations based on either the content or the network information.
INTRODUCTION
Substantial amounts of data produced in our modern information society each day. A large portion of it comes from the communication on social media platforms, within chat applications, or via emails. This data exhibits dualtiy in the sense that they can be represented as text and graph. The metadata provides inherent graph structure given by the social network between correspondents and the exchanged messages constitute the textual content. In addition, there are many other datasets that exhibit these two facets. Some of them are found in bibliometrics, for example in collections of research publications as co-author and citation networks.
When it comes analyse these types of datasets, usually either the content the graph structure is neglected. In data exploration scenarios the goal of getting an overview of the datasets at hand insurmountable with current tools. The sheer amount of data prohibits simple visualisations of networks or meaningful keyword-driven summaries of the textual content. Data-driven journal- ism (Coddington 2015) often has to deal with leaked, unstructured, very heterogeneous data, e.g. in the context of the Panama Papers, where journalists needed to untangle and order huge amounts of information, search entities, and visualise found patterns (Chabin (017). Similar datasets are of interest in the context of computational forensics (Franke & Srihari 2007). Auditing firms and law enforcement need to sift through huge amounts of data to gather evidence of criminal activity, often involving communication networks and documents (Karthik et al.) 2008). Users investigat- ing such data want to be able to quickly gain an overview of its entirety, since the large amount of heterogencous data renders experts' investigations by hand infeasible. Computer-aided exploration tools can support their work to identify irregularities, inappropriate content, or suspicious patterns. Current tools") lack sufficient semantic support, for example by incorporating document embed- dings (Mikolov et al. 2013) and the ability to combine text and network information intuitively.
We propose MODIR, scalable multi-objective dimensionality reduction algorithm, and show how it can be used to generate an overview of entire text datasets with inherent network information in a single interactive visualisation. Special graph databases enable the efficient storage of large relationship networks and provide interfaces to query or analyse the data. However, without prior
c.g. https:/ / /www/ nuix ccm/ https://tinkuric.

---

Under review as a conference paper at ICLR 2020
knowledge, it is practically impossible gain an overview or quick insights into global network structures. Although traditional node-link visualisations of graph can provide this overview, all semantic information from associated textual content is lost completely.
Technically, our goal is to combine network layouts with dimensionality reduction of high- dimensional semantic embedding spaces. Giving overview over latent structures and topics in one visualisation may significantly improve the exploration of a corpus by users unfamiliar with the domain and terminology. This means, we have to integrate multiple aspects of the data, especially graph and text, into a single visualisation. The challenge is to provide an intuitive, two-dimensional representation of both the graph and the text, while balancing potentially contradicting objectives of these representations.
In contrast existing dimensionality reduction methods, such as ISNE (Maaten & Hinton 2008), MODiR uses novel approach to transform high-dimensional data into two dimensions while opti- mising multiple constraints simultaneously to ensure an optimal layout of semantic information ex- tracted from text and the associated network. To minimise the computational complexity that would come from naive combination of network drawing and dimensionality reduction algorithms, we formally use the notion of a hypergraph. In this way, we are able to move repeated expensive compu- tations from the iterative secument-centred optimisation to a preprocessing step that constructs the hypergraph. We use real-world datasets from different domains to demonstrate the effectiveness and flexibility of our approach. MODiR- -eeee--geeeatted representations are compared to a series of baselines and state-of- -oee------e-rt dimensionality reduction methods. We further show that our integrated view of these datasets exhibiting duality is superior to approaches focusing on text-only or network-only information when computing the visualisation.
2 RELATED WORK
With MODiR we bridge the gap between text and network visualisation by jointly reducing the dimensionality of the input data. Therefore we subdivided this part into three sections to highlight related work in the areas of text visualisation, representation learning, as well as dimensionality reduction. Other work that tries to jointly model text and networks but without dimensionality reduction and without focus on visualisation is LINE (Tang et al., 2015). They generate information networks consisting of different types of nodes, e.g. words from document content and authors from document metadata. Another tool that investigates combining graph structure with textual elements is VOSviewer ( (Van Eck & Waltman 2014). They construct and visualise bibliographic networks that provide a multi-view interface to explore and filter keywords and network aspects of such datasets. In our work we go beyond building a network from textual data but instead project the textual data into a latent space.
Document visualisation aims to visualise the textual content, such that users gain quick insights into topics, latent phrases, or trends. Tiara (Wei et al. 2010) extracts topics and derives time-sensitive keywords to depict evolving subjects over time as stacked plots. Another line of work projects documents into a latent space, for example by using topic models or embeddings: Creating scatter- plots of embedded documents of a large corpus may result in very dense and unclear layout, so Chen et al. (Chen et al., 2009) developed an algorithm to reduce over-full visualisations by picking representative documents. A different approach is taken by Fortuna et al. (Fortuna et al. 2005), ) who do not show documents directly, but generate a heatmap of the populated canvas and overlay it with salient phrases at more densely populated areas from the underlying documents in that region. Friedl et al. (Fried & Kobourov] 2014) extend that concept by drawing clear lines between regions and colouring them. They also add edges between salient phrases based on co-occurrences in the texts. A map analogy can be used to visualise the contents of documents by embedding them into a high dimensional semantic space (Le & Mikolov, 2014)) and projecting it on a two-dimensional canvas as a document landscape. Most recently Cartograph (Sen et al.) (017)) was proposed, which is visually very similar to previous approaches, but pre-renders information at different resolution and uses a tiling server with (geographic) map technology to deliver responsive interactions with the document landscape. Regions are coloured based on underlying ontologies from a knowledge-baig base. Networks are traditionally visualised using so-called node-link graphs. This way, any additional information related to nodes and edges are lost. The layout of nodes usually follows a force-based analogy first proposed by Fruchterman & Reingold (199].. Newer approaches optimise the compu-
2

---

Under review as a conference paper at ICLR 2020
tational complexity and include local metrics to better represent inherent structures for example ForceArlas2 (Jacomy et al. 2014), which is the default network algorithm for the network visualisa- tion tool Gephi.
The text and network visualisation methods discussed above primarily use structural properties of the data to generate their layout. Although we focus on the visualisation of text data with inherent graph information, MODiR can work with arbitrary kinds of data. Our model only requires a way to project the data into a high-dimensional Euclidean vector space so that the distance between two points can be interpreted as their (semantic) similarity. Traditionally, text can be represented as bag- of-words vector that optionally is weighted by respective tf-idf scores. In recent years, embeddings became more popular as they conserve semantic meaning in their vector representation. [Mikolov] et al. 2013) introduced neural architectures to learn high-dimensional vector representations for words and paragraphs (Le& & Mikolov] 2014). Similar methods are used to learn representations for nodes in a network based on either the structural neighbourhood (Faerman et al., 2018) or additional heterogeneous information (Chang et al. 2015 Schlötterer et al. (217)) attempted to learn joint rep- resentations of network structure and document contents but saw no improvement over conventional models in series of classification tasks. We only use the structural information of the network for better control over fine-grat fine-grained adjustments in our layout algorithm.
The goal of dimensionality reduction is to represent high- nigh-dimensional data in a low-dimensional space while preserving the characteristics of the original data as sound as possible. A very common application of dimensionality reduction is project high-dimensional data into two dimensions for the purpose of visual interpretation. Generally, these methods follow one of three mathemat- ical models. Linear models, such as Principle Component Analysis (PCA) (Pearson [901] can be calculated very efficiently and have proven to reduce input spaces to improve the performance of downstream tasks. Thus, they are often indirectly used for feature extraction. Although reduc- tions to two dimensions for visualisations are appropriate for quick initial data exploration, other approaches are able to better preserve data characteristics in two dimensions. For example, the non- linear Sammon mapping [Sammon] 1969) tries to preserve the structure of inter-point distances in high-dimensional space in low-dimensional space. The resulting visualisations are generally better then PCA to show relatedness of individual data points. Lastly, there are probabilistic models like Stochastic Neighbour Embeddings (SNE) (Hinton & Roweis 2003). They are similar to a Sammon mapping in that they use inter- inter-point distances but model these distances probability distributions. The t-distributed SÑE has proven to produce competitive results for visualising datasets while pre- serving characteristics (Maaten & Hinton 2008), however its nondeterministic nature may produce greatly varying results. Recently, FIISNE was proposed, an optimisation of tSNE that significantly reduces the computational complexity (Linderman al., 2019). Other newer dimensionality reduc- tion algorithms like LargeVis (Tang et al., 2016) and UMAP (McInnes et al. 2018) scale almost linearly by using efficient nearest neighbourhood approximations in the high-dimension space and spectral embeddings to initialise positions of points in the low-dimensional space to reduce the num- ber of fine-tuning iterations.
3 MULTI-OBJECTIVE DIMENSIONALITY REDUCTION
Visualisations of complex datasets are restricted to two or three dimensions for users to grasp the structure and patterns of the data. We integrate multiple entities (i.c., documents and persons) into a joint visualisation, which we call landscape. This landscape consists of a base-layer containing all documents depicted as dots forming the document landscape; nodes and their connections placed on top of this base-layer as circles connected by lines forming the graph layer. In this section, we propose the MODiR algorithm which integrates multiple objectives during the layout process to find an overall good fit of the data within the different layers. Our approach is derived from state-of- tae-art methods for drawing either the network layer or the document landscape.
We assume that documents are given as high- sigh--imensional vectors and entities are linked among one another and the documents. These links are used as restrictions during the multi-objective dimen- sionality reduction of document vectors. Let (6) X C Rd be the set of n documents in their d- d-dimensional representation and ( Y C R² the respective positions scape. Let H(V,E) E) be a hypergraph based on the network information inferred from the document corpus, with vertices V X XUP, P., where X are the documents and p; E TP are the entities in the
the document land-
3

---

Under review as a conference paper at ICLR 2020
network and hyperedges Gk & describing the relation between documents and entities. For each pair of entities Pm.Pa E IP that are connected in the context of documents x1() ,.... X, there is hyperedge e; = {pmm Pn, x(*), ,..").... Analogously, the same definition applies to Y. Further, H or H= is used explicitly state the respective document representation used. The position in the graph
layer IP + IR² of an entity Pm is defined as
T (Pm: H²)
Σ Σ Npm ekEEpm y'U UP
y
(1)
where E C 7 is the set of hyperedges containing Pm and Np.. is the number of documents Pm is associated with More elaborate methods like a density-based weighted average are also applicable mitigate the influence of outliers. For simplicity we will abbreviate т (Pm: H as
| This effectively places an entity at the centre of its respective documents.
Let 1/: X + V be the projection (xx(1);; W) = W = ()) where W E <R2xn the projection matrix leant by MODiR based on multiple objectives (1112233 using gradient descend, as defined later in this section. The objectives are weighted by manually set parameters f212. to balance the effects that favour principles focused on either the graph layer or the document landscape, as they may contradict one another. Given a high-dimensiones hypergraph H² the matrix W, and entity projection x, we define the resulting multi oulti-objeectve dimensionality reduction function as
T (HL W. T) = H..
In the following paragraphs, we will formally introduce MODiR's objectives. Objectives (1) and (2) are inspired by tSNE and use the neighbourhood context of documents in X to position similar documents near one another and unrelated ones further apart in Y. Objective (3) attracts documents based on co-occurrence in hyperedges so that the resulting Tm will be closer if they are well con- nected in the graph. This third objective also implicitly brings documents closer their respective entities.
another. Semantically similar documents should be closer on the document landscape and dissimilar ones further apart. To measure the semantic sim- ilarity of documents, Maaten & Hinton (2088) used naïve bag-of-words representation. Although ISNE preserves the inherent semantic structure in two-dimensional representations from these sparse vectors (Pezzotti et al.) 2017), we opted to use document embeddings. This has the advantage that, when only part of the data visualised, the embedding model can still be trained on larger set of documents and thus retain the additional information. Objective (1) is inspired by the efficient usage of context words in word2vec (Mikolov et al. 2013). Corresponding to the skip-gram model, we define the context ( C X of document ( by its k nearest neighbours in the embedding
Objective (1): Similar documents
space. The first objective is defined as
4p (xr 0
Σ ((
|| 200 (ly y
with a being the sigmoid function. Distances are normalised based on the context to make them comparable between the high-dimensional and two-dimensional space and rescaled by the sigmoid.
Objective (2): Dissimilar documents are apart from one another. The optimal solution to the previously defined objective would be project all documents onto the same point on the two- dimensional canvas. In order to counteract that, we introduce negative examples for each pair of context documents. We do so by sampling a set of documents that are not in the ie neighbourhood of ( Let (
C X
x
be the set of negative samples for (+),, then the second objective
defined as
(1) o
Σ
x0011-1b>' y yonl)
(3)
(9
Npw. = (0(0) &le E (x:(() Ee pm Eac
4

---

Under review as a conference paper at ICLR 2020
Objective (3): Connected entities are serves two purposes: All documents ( associated with a person Pm are placed near its nm position in the graph layer and two people Tm and mn are forced near one another if they are connected.
another and their documents. This object
Let Eyer C be the set of hyperedges in the hypergraph H. containing the document y and yyt = UeaeS
(6) ck \ IP all documents that are linked to y through an entity, then the third
objective is deñined
()( =o
Σ yy 1y'0 10yy001 1))
which, when minimised, attracts documents that are related through entities. This has two implicit effects: An entity Pm gets closer to its documents they are attracted to Tm without having to explicitly compute this position using Equation| []] Also, related entities Pm. Pr are attracted to one another since they appear in the same hyperedges. The computational complexity of this objective is strongly related to the connectedness of entities in the graph. For dense graphs, we propose a heuristic by only using a subset of s documents from the context Ey), of y An objective modelling a repulsive force as in force-directed graph layouts is not needed as the first two objectives ((122) provide enough counteracting force.
Algorithm. The positions of entities and documents on the landscape are calculated using the previously defined objectives follows. First, construct the hypergraph 71 with document contexts including the set of k-neighbourhoods xt.=") Relevant pairwise distances can be stored in adjacency matrix reduce computational overhead in Equations 2 and B
For more efficient
training, the randomly sampled negative neighbourhoods X!.x( can prepared ahead of time and then only masked during later. The s-neighbourhoods for entities in Equation 4 4 yet can only be prepared with references, W. y,0 updates with each iteration. We designed the algorithm to move as much repetitive computations to pre-processing ahead of time or each epoch. Creating these sets is very efficient using Hierarchical Navigable Small World graphs (HNSW) (Baranchuk al.) 2018) for approximate nearest neighbour search. Overall we are able to reduce the pre- pre-processing complexity to On log n) and for each iteration Okln), with k < n near linear. After generating the context sets, we use gradient descend to update the projection matrix W with learning rate reducing the overall error & as defined by
p(xx)) = 01 (x:) + 02522(1;)) + Ozspa(x;).
(5)
Selecting appropriate values for the hyperparameters k, s, and (12223) critical to produce mean- ingful results. We found / Ao in all experiments produce the best results this way for every similar document the model has one dissimilar document to compare. Inspired by tSNE (Maaten & Hinton 2008), we limit hyperparameters by setting k and dynamically for each document based on a user-defined perplexity. With these adaptations, the only parameters to be set are the perplexity that roughly determines the context size, the learning rate 7), and the objective weights, which can often stay at a default setting. A reference implementation including a modular processing pipeline for different datasets, approaches, and experiments is available on GitHub")
4 EXPERIMENTS
Our approach is mainly motivated to explore business communication data (namely emails), such as the Enron corpus (Klimt& & Yang 2004). However, due the lack of ground truth, we will focus our evaluation on research publications and their co-authorship network. Results of dimensionality reduction can be subjective, so as in prior work on dimensionality reduction (McInnes et al. 2018 Sen et al.) 2017 Maaten & Hinton 2008), we will qualitatively compare our approach to a variety of baselines and provide some quantitative experiments.
Experimental Setup. Here, we are using the Semantic Schola Open Corpus (S2) (Ammar et al. 2018 with over 45 million articles covering a range of scientific fields and the AMine network
(link will be published in camera ready version)
hutrass//all.smmaniiicccoooaoo.. https://aiiner..orgb1llooo
5

---

Under review as a conference paper at ICLR 2020
Table Quantitative evaluation of different landscapes (AM / S2 // ENR]
CLUSTER
DOC2VEC
T-SNE
PCA
MODiR
Data Mining
0.49 / 0.39
0.30/0.55
0.52 / 0.55
0.39 / 0.42
Database
0.49 0.82
0.64 0.34
0.47 0.34
0.69 / 0.32
ML
0.51 / 0.35
0.21 0.21/0.25
0.38 0.23
0.35 / 0.23
NLP
0.58 0.76
0.73 0.34
0.81 /.334
0.73 0.68
Comp Vision
0.51 / 0.67
0.56 0.39
0.49 0.39
0.54 0.29
HCI
0.64 0.68
0.47 0.41
0.61 0.41
0.39 0.38
Avg.
0.54 / 0.61
0.49 1.49/0.37
0.54 0.38
0.53 0.39
AtEdge
5.32/4.09/3.89
5/3.91/3.6
4.79/2.94/2.59
(AM) (Tang et al. 2008) published in 2008 with over two million papers by 1.7 million authors. Unlike DBLP however, they not only contain bibliographic metadata, such authors, date, venue, citations, but also abstracts to most articles, that we use to train document embeddings using the Doc2Vec model Gensim | Similar Carvallari et (Cavallari et al. 2017). we remove arti- cles with missing information and limit to the six communities Data Mining, Databases, Machine Learning, NLP, Computer Vision, and HCL. This way we discard clearly unrelated computer science articles and biomedical studies for more fine grained analysis. For in-depth comparisons an even smaller subset from S2 of 24 hand-picked authors, their co-authors, and their papers (S2b).
To our knowledge, there of high-dimme nigh-dimenssonal data. Popular approaches for traditional dimensionality reduction are tSNE and PCA. As baselines, we the original optimised implementation of tSNF written in C as provided by the authors. The quantitative evaluation is two-fold: MODiR can simulate their behaviour by setting 0g- = 0 and thus ignoring the objective that incorporates network information.
algorithms that multiple objectives for dimensionality reduction
In our experiments we use the following parameter settings. For USNE we set the perplexity to Perp(P.)
0.5 and run it for 1,000 iterations. In MODiR we the neighbourhood size to k: = 10, the negative context size to I = 20, and all objective weights (112233) = 1.0. For discussion on the influence of hyperparameters, we refer to the supplemental material. The speed of convergence depends on the learning rate 1) and thus dictates the number of maximum iterations. Early stopping with a threshold on the update rate could be implemented. Depending on the size of the dataset and fixed learning rate of ' 0.01, MODiR generally converges after 10 to 200 iterations, for larger and more connected data it advisable to use a higher learning rate the first epoch for initialisation and then reducing to very small updates. For better comparability, use a constant number of iterations T = 100.
Quantitative Evaluation. As Maaten & Hinton (2008) state, it is by definition impossible to fully represent the structure of intrinsically high-dimensional data, such as set of document embed- dings, in two dimensions. However, stochastic neighbour embeddings are able to capture intrinsic structures well two dimensional representations (Kobak et al. 2019). 3) To measure this capability, compare the ability of k-means++ (Arthur & Vassilvitskii 2007) to cluster the high- and two- dimensional space. We set the number of clusters to the number or research communities (k = 6) and calculate the percentage of of papers for each community per cluster. Therefore we assign each community to the cluster with most respective papers and make sure to use clustering with an even distribution. Results are listed in Table for (SNE, PCA, MODiR, and the original high dimensional embedding averaged over five runs. We see, that as expected due to topical overlap of communities, even original embeddings can't tan't be accurately clustered. Interestingly though, there seems be significant difference between AM and S2 although the sets of papers intersect, which we assume due the fact, that S2 is larger and additionally contains more recent papers. Although PCA
'ttpps: hhttp:://radirrehrreee www.wrriwieeeee.....m wttpsshrre.coom/genee....
embedding size: dimensions, vocabulary size: 20k
tokens, trained for 500 epochs hhttps://Ivdne httpi/attttnnigtttuooo.o. github. io/tane/

---

Under review as a conference paper at ICLR 2020
MIL
NLP
HL
NLP
DB
DB
DB
DB
NL
(a) Baseline network-firss edges
(b) Bascline network-first nodes
(c) Bascline documents as nodes
DB
ML
DB
(d) Baseline documents-first
(e) Baseline document ISNE
(f) MODiR
Semantic Scholar co-authorship network (S2b), subsampled for readability; (a) the net- work is laid out first, documents randomly placed along edges; (b) the network is laid out first, documents are randomly placed around nodes; (c) documents are part of the network layout as nodes in the graph that replace author-author edges; (d) the document landscape is laid out first, nodes are positioned at the centre of their associated documents; (e) iSNE is applied on papers and authors together, where documents are aggregated to represent authors
Figure I:
often does not generate visualisations in which classes can be clearly distinguished, the clustering algorithm is still able to separate them with competitive results compared to tSNE and MODiR.
MODIR not only aims to produce good document landscape, but also good layout of the network layer. Graph layouts are well studied, thus we refer to related work on aesthetics (Purchase) (2022) and readability (Nguyen et al. 2017). While these are very elaborate and consider many aspects, we decided to use Noack's 's normalised ArEdge-length (Noack) 2007):
AtEdge = Σ, E, - Fl E, Σ, | F
E
|P/2
describes how well the space utilisation is by measuring whether edges are as short as possible with respect to the size and density of the graph. Table contains containss the results. Although the AtEdge metric is comparable for layouts the same graph, is not comparable between datasets as can be seen by the fact, that larger number of edges causes overall lower score. The AtEdge length produced by PCA is generally better than that of iSNE while MODiR outperforms both as our approach specifically includes an optimised network layout. The better performance of PCA over ISNE can be explained by the resulting layouts being more densely clustered in one spot. Although the AtEdge length aims to give a lower score for too close positioning, is not able to balance that to the many very long edges in the layout produced by iSNE.
Qualitative Evaluation. Apart from purely quantitative evaluation, we use the hand-selected Semantic Scholar dataset (S2b) to visually compare compare network-centric baselines (a-c), focusent-focused baselines (d-e) and MODiR (f) in Figure [ Papers are depicted as circles where the stroke colour corresponds to the communities, black lines and dots are authors and their co- authorships, size corresponds to the number of publications. For better readability and comparabil- ity, the number of drawn points is reduced and three communities marked.
In Figure la we use the weighted co-authorship network drawn using (Fruchterman & Reingold 1991) and scatter the papers along their respective edges after the graph is laid out. We see, that active collaboration is easy to identify as densely populated edges and research communities se- lected areas are mostly coherent and unconnected researchers spatially separated from others.
7

---

Under review as a conference paper at ICLR 2020
Although it is possible to distinguish the different communities in the graph layer,the document landseape isn't as clear. The ML researchers are split apart from the rest of the NLP community, which in turn is overcrowded. Figure Ib uses the same network layout but places articles randomly around their first author, which makes it easy to spot the scientific communities by colour. Lastly, we include papers nodes and co-authorship edges are connected through them during the network layout in Figure Ic] This produces a very clean looking layout compared with the other baselines, however papers lump together and are not evenly distributed. Furthermore, semantic nuances be- tween papers are mostly lost which becomes most apparent in the now separated database clusters. Also, the semantic overlap between the ML and NLP communities is not noticeable.
Figure Id positions documents using tSNE and places researchers using Equation | We see that articles are positioned on the landscape so that research areas are distinctly recognisable by colour. Papers that could not be assigned to specific area are scattered across the entire landscape. The collaboration network is laid out surprisingly good. The research interests of the authors are coherent between the network and the document landscape, it even shows the close relation between NLP and ML, while showing a clear separation to database related topics. Nonetheless, the network should be loosened for better readability, for example members of the same research group who frequently co-author papers tend to collide. Unconnected authors are almost not visible as they drift toward densely populated areas in the middle. In Figure Tel we included authors as virtual documents the sum of their papers during the tSNE reduction. This shows some improvement, as the network layout more loose and fewer edges overlap and the issue with collapsing research groups also mostly mitigated. The semantic overlap of ML and NLP is niceley captured along with the difference to the database papers. However, the network is not clearly readable.
With MODiR the three research communities become clearly distinguishable, both in the graph layer and in the document landscape. Nodes of well connected communities are close together, yet are not too close locally, and separate spatially from other communities. The document landscape is laid out more clearly, as papers from different fields are grouped to mostly distinct clusters. Obviously there is still a slight overlap as result of semantic similarities. As previously pointed out, this visualisation also correctly reveals, that the ML and NLP communities are more closely related to each other (both use machine learning) than to DB. The authorship of documents however can only be conveyed through interaction, so this information is not present in the static visualisations shown here. Based on these results we argue, that the network information improves the (visual) community detection. The document embeddings of articles can only reflect the semantic similarities, which may overlap. In conjunction with information from the co-authorship network, the embeddings are put into their context and thus are more meaningful in a joint visualisation.
5 CONCLUSIONS
In this paper we discussed how to jointly visualise text and network data with all its aspects on a single canvas. Therefore we identified three principles that should be balanced by a visualisation algorithm. From those we derived formal objectives that are used by a gradient descend algorithm. We have shown how to use that to generate landscapes which consist of base-layer, where the embedded unstructured texts are positioned such that their closeness in the document landscape re- flects semantic similarity. Secondly, the landscape consists of a graph layer onto which the inherent network is drawn such that well connected nodes are close one another. Lastly. both aspects can be balanced so that nodes are close to the documents they are associated with while preserving the graph-induced neighbourhood. We proposed MODIR, a novel multi-objective dimensionality reduc- tion algorithm which iteratively optimises the document and network layout to generate insightful visualisations using the objectives mentioned above. In comparison with baseline approaches, this multi-objective approach provided best balanced overall results as measured by various metrics, In particular, we have shown that MODiR outperforms state-of-the algorithms, such as tSNE. We also implemented an initial prototype for an intuitive and interactive exploration of multiple datasets.
REFERENCES
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Se- bastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam
8

---

Under review as a conference paper at ICLR 2020
Skjonsberg, Lucy Lu Wang. Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Et- zioni. Construction of the literature graph semantic scholar. In NAACL-HLI, pp. 84-91. ACL, 2018.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. In SIAM, pp. 1027-1035. SIAM, 2007.
Dmitry Baranchuk, Artem Babenko, and Yury Malkov. Revisiting the inverted indices for billion- scale approximate nearest neighbors. In ECCV, pp. 202-216. Springer, 2018.
Sandro Cavallari, Vincent W Zheng, Hongyun Cai, Kevin Chen-Chuan Chang, and Erik Cambria. Learning community embedding with community detection and node embedding on graphs. In CIKM, pp. 377-386. ACM, 2017.
Marie-Anne Chabin. Panama papers:
study for records management? Brazilian Journal of Information Science: Research Trends, 11(4), 2017.
Shiyu Chang, Wei Han, Jiliang Tang, Guo-Jun Qi, Charu C Aggarwal, and Thomas S Huang. Het- erogeneous network embedding deep architectures. In KDD, pp. 119-128. ACM, 2015.
Yanhua Chen, Lijun Wang, Ming Dong, and Jing Hua. Exemplar-based visualization of large doeu- ment corpus. TVGC, 15(6):11611166 2009.
Mark Coddington. Clarifying journalism's quantitative turn: A typology for evaluating data jour- nalism, computational journalism, and computer- assisted reporting. Digital Journalism, 3(3): 331-348, 2015.
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. Using structured events to predict stock price movement: An empirical investigation. In EMNLP, pp. 1415-1425. ACL, 2014.
Evgeniy Faerman, Felix Borutta, Kimon Fountoulakis, and Michael W Mahoney. LASAGNE: Lo- cality and structure aware graph node embedding. In WI, WIC, pp. 246-253. ACM, 2018.
Blaz Fortuna, Marko Grobelnik, and Dunja Mladenic. Visualization of text document corpus. Infor- matica, 2944)4997-502, 2005.
Katrin Franke and Sargur Srihari. Computational forensics: Towards hybrid-intelligent crime investigation. In IAS, pp. 383-386. IEEE, 2007.
Daniel Fried and Stephen G Kobourov. Maps of computer science. In PACVIS, pp. 113-120. IEEE, 2014.
Thomas MJ Fruchterman and Edward M Reingold. Graph drawing by force-directed placement. Software: Practice and experience, 21(11): 1991.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In NIPS, pp. 857-864 NIPS Foundation, Inc., 2003.
Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian. Forceatlas2, a continuous graph layout algorithm for handy network visualization designed for the gephi soft- ware. PLOS ONE, 96):e98677 2014.
Mukundan Karthik, Mariappan Marikkannan, and Arputharaj Kannan. An intelligent system for semantic information retrieval information from textual web documents. In IWCF, pp. 135-146. Springer, 2008.
Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. In ECML, pp. 217-226. Springer, 2004.
Dmitry Kobak, George Linderman, Stefan Steinerberger, Yuval Kluger, and Philipp Berens. Heavy- tailed kernels reveal a finer cluster structure in t-sne visualisations, In ECML. Springer, 2019.
Quoc Le and Tomas Mikolov. Distributed representations sentences and documents, ICML, pp. 1188-1 1966. PMLR, 2014.

---

Under review as a conference paper at ICLR 2020
George C Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation- based t-sne for improved visualization of single-cell ma-seq data. Nature meth- ods, 16(3):243, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using tsne. JMLR, 9(11)2559-2605, 2008.
Leland MeInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXin:1802.03346, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In N/PS, pp. 3111-1119999 NIPS Foundation,
Inc., 2013.
Quan Hoang Nguyen, Peter Eades, and Seok-Hee Hong. Towards faithful graph visualizations. CoRR, abs/ 1sss17000009,,, 2017.
Andreas Noack. Unified quality measures for clusterings, layouts, and orderings of graphs, and their application as software design criteria. PhD thesis, Brandenburg University Technology, Cottbus- Sertbb-Senfttnberg. Germany, 2007. Chapter 6, pp97.
Karl Pearson. On lines and planes of closest to systems of points in space. Philos. Mag., 2(11): 559-572. 1901.
Nicola Pezzotti, Boudewijn PF Lelieveldt, Laurens van der Maaten, Thomas Höllt, Elmar Eisemann, and Anna Vilanova. Approximated and user steerable tsne for progressive visual analytics. TVCG, 23(7):179-1152, 2017.
Helen C Purchase. Metrics for graph drawing aesthetics. JVLC, 13(5):001-516. 2002.
Tim Repke and Ralf Krestel. Bringing back structure to free text email conversations with recurrent neural networks. In ECIR, pp. 114-126. Springer, mar 2018.
John W Sammon. A nonlinear mapping for data structure analysis. IEEE Transactions on computers, 100(5):401-499 1969.
Jörg Schlötterer, Christin Seifers, and Michael Granitzer. On joint structure and document content. In CD-MAKE, pp. 237-251. Springer, 2017.
joint representation learning of network
Shilad Sen, Anja Beth Swoap, Qisheng Li, Brooke Boatman, Ilse Dippenaar, Rebecca Gold, Monica Ngo, Sarah Pujol, Bret Jackson, and Brent Hecht. Cartograph: Unlocking spatial visualization through semantic enhancement. In IUI, pp. 179-190. ACM, 2017.
Dominic Seyler, Tatiana Dembelova, Luciano Del Corro, Johannes Hoffart, and Gerhard Weikum. A study of the importance of external knowledge in the named entity recognition task. ACL, pp. 241-246. ACL, 2018.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In WWW, pp. 1067-1077. ACM, 2015.
Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. Visualizing large-scale and high- dimensional data. In WWW, pp. 287-297. ACM, 2016.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Aretminer: extraction and mining of academic social networks. In KDD, pp. 990-998. ACM, 2008.
Nees Jan Van Eck and Ludo Waltman. Visualizing bibliometric networks. In Measuring scholarly impact, pp. 285-320. Springer, 2014.
Furu Wei, Shixia Liu, Yangqiu Song. Shimei Pan, Michelle X Zhou, Weihong Qian, Lei Shi, Li Tan, and Qiang Zhang. TIARA: visual exploratory text analytic system. In KDD, pp. 153-162. ACM, 2010.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ALGORITHM
In Algorithm D we formalise our approach for MODiR described in Section For the intuition, descriptions, and definitions, we refer to Section
Algorithm 1: Simplified MODiR Algorithm
Data:
Hypergraph 7/- = (V,E) with nodes V/ and hyperedges & as sets of high-dimensional document representations X and people P; number of iterations T N; objective weights (1.2,3) E IR; context sizes k; 1, 8 E N; update size v
Result: document positions 7("
begin
// initialise landscape (projection matrix W)
W (g() R2|a((0 X);
// pre-processing of contexts
x + (a
(x0) (200 e x Heo e x]:
x ((x10, (zll) 2060 & xx ))(((0) eX);
Y + (()(0, (100)0 E Hy y0 6) E Y):
for 1 = 11... ....Td do
foreach sampled pairs x () in x Ux U Y do
())))= + vxx(
20 + (x((()));
(1233)991233 (1))) ا (0))) .0)))
update W by 1) to reduce & with gradient descend;
end end
calculate all ı with Equation D
end
A.2 DETAILS ON DATASETS
The motivation for this paper is to visualise social networks along with their respective text docu- ments, especially email corpora, for exploring and understanding large datasets. We argue, that our approach is applicable to any given dataset with inherent graph structures, so we include variety of examples for evaluation. We apply MODiR to the Enron corpus (Klimt & Yang 2004) which orig- inally consists of around 600,000 messages belonging to 158 users and Quagga (Repke & Krestel, 2018) to extract individual emails from quoted conversations, remove duplicates, extract additional correspondents from inline metadata, and try to combine the aliases of people. Assessing the quality of given layout requires very specific domain knowledge including deep understanding of seman- tic structure across all documents and a close familiarity with entity relations. Email corpora, such the aforementioned one, lack of gold standards and domain knowledge on our side, we con- sider additional sources. Thus we use named entities extracted from business news articles. From the corpus of 448.395 Bloomberg- and 106,519 Reuters news articles (NEW) published by Ding et al (Ding et al. 2014), we select those that contain the search term "commerzbank" as central entity and consider co-oceurrences of organisation entities extracted with AmbiverseNI.U (Seyler et al., 2018). This results in a graph where almost all entities are connected to single central entity that appears in all articles.
Academic co-authorship networks including respective publications have well defined labels pro- vided by venues or communities, so there are no ambiguities or additional annotations. The two processed and publicly available corpora of research articles, the AMMnne network (AM) (Tang ct al. 2008) published in 2008 with over two million papers by 1.7 million authors and the recently
https://aminer..e http:///11oo/////oooomo aobard/aminnrnetworr. nernetwork

---

Under review as a conference paper at ICLR 2020
Table 2: Number of documents, people, and their connections in filtered datasets used in this paper
DATASET
DOCUMENTS
NODES
# EDGES
AMiner (AM)
49,670
56,449
110,146
SemanticScholar (S2)
170,098
183,198
701,442
SmallScholar (S2b)
489
24
39
Enron (ENR)
189,437
32,353
950,100
News (NEW)
3,734
2,944
5,240
Table Number of articles in selected communities from Semantic Scholar (S2) and AMiner (AM)
LABEL
VENUES
IN AM/ S2
Data Mining
KDD, ICDM, CIKM, WSDM SIGMOD, VLDB. ICDE, EDBT
4,728 / 13,699
Database
7,155 / 14,888
MI.
NeurIPS, AAAL, ICMI,, IJCAI
10,374 41,815
NLP
EMNLP, ACL., CONLL, COLING
41,815 22,523
Comp Vision
CVPR, ICCV, ICIP, SIGGRAPH CHI, IUI, UIST, CSCW
11,898 43,5558
HCI
8,608 33,615
published Semantic Scholar Open Corpus (S2) (Ammar et al. 2018) with over 45 million articles. Both corpora cover range of different scientific fields. Semantic Scholar for example integrates multiple data sources like DBLP and PubMed and mostly covers computer science, neuroscience, and biomedical research. Unlike DBLP however, $2 and AM not only contain bibliographic meta- data, such authors, date, venue, eitations, but also abstracts to most articles, that we use train document embeddings using the Doc2Vec model in Gensim 10 Similar to Carvallari et al. (Cav- allari al. 2017) remove articles with missing information and limit to six communities that aggregated by venues as listed in Table This way we reduce the size and also remove clearly unrelated computer science articles and biomedical studies. For in depth comparisons we reduce the S2 dataset to 24 hand-picked authors, their co-authors, and their papers (S2b).
Note, that the characteristies of the networks differ greatly as the ratio between documents, nodes, and edges in Table 2 shows. In an email corpus, larger number of documents is attributed to fewer nodes and the distribution has high variance (some people write few emails, some lot). In the academic corpora the other hand, the number of documents per author is relatively low and similar throughout. Especially different the news corpus, that contains one entity that is linked to all other entities and to all documents.
A.3 HYPERPARAMETER SETTINGS
In Section we gave brief overview of the settings used in the experiments presented above. Here, we provide additional insights from our experiments on different hyperparameter settings for MODiR. The context sizes are the most important parameters. The first two objective weights can be ignored, as the context size has similar effects, so we set 0 = 0₂ 1.0 in all our experiments. Generally, small numbers for 7,5 perform better. This is in line with expectations, as each item æ(i) will also be in the context of its respective neighbours and will therefore amplify its attractive force. A large number for k for example will force all points towards the centre of the canvas or even larger, produce random scatter as the gradients amplify. In our experiments we use & = 10, for datasets with few thousand samples, i should usually be below We also found, that the negative context is best with = 20 for all sizes. Furthermore, we set both 0 = 0₂ = 1.0 for all experiments because the influence on selecting k; 1 much larger. The graph context is also set to & 10 (in our dataset the number of entities is close to the number of documents), the objective weight can be
https::/aap www.nt iemanttiscchollltt http:///gg.....eee......e.eeoorr https:: htt///rddimmhuureeee com/gensim/ embedding size: 64 dimensions, vocabulary size: 20k tokens, trained for 500 epochs
12

---

Under review as a conference paper at ICLR 2020
freely adjusted between around 0.8 v 8g I 1.2 set the influence of the entity network. Similar to the semantic neighbourhoods the first and second objective, the choice is significantly more influential than 03-
A.4 LANDSCAPE VISUALISATIONS
Serfiesiee
Algorthrm
Re eeee Hegatase management system
Aigocton
"Arlaii fetwork
Gatera model gontim
Domcunor Calo
niuurll network
Expentment Zomes HARgGpaaCH fileork
Algmmihm
Algoritm
Algrttinn Brocroin
Algne
Figure MODiR visualisation of Semantic Scholar (S2), all six communities become clear.
York
AAlommotiv industry
New York City
Spain
George soms
Spain
Area
Portugal
Autamonyye Industry
Raly
ven
Spain
gouge
Incusrry
Pronua
Josef
Spain
Spaine
Ne City
Span
Track gauge
Petroleun motallu mncrab
Investientt
Spain
Petroisum Indyst
New York City
City
Aiganettvv enducayy.
Jialy
Froight traasport
lntor agency
Italy
Pentleeum Incusny
Gold an invest
yen
Aggela Marical
Co
Figure 3: MODiR visualisation business news about Commerzbank, highlighted on articles about Volkswagen. All nodes (companies) the network are connected the centre node (Commerzbank) and our algorithm still manages to retain the semantic areas.
13


Reviews:
[{"review_id": "Skx-6E86tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors introduce a novel algorithm based on multi-objective optimization to jointly position embedded documents and graph nodes in a 2-d landscape. \n\nFirst, the multi-objective optimization used in this paper is quite confusing. Since the overall error defined in (5) is a weighted summation of three objectives, it is not a real multi-objective optimization where multiple objectives are optimized simultaneously. \n\nAuthors define a hypergraph to jointly model documents and entities. However, there lacks the novelty on the objectives defined in this paper. The optimization using gradient descent is also trivial. \n\nIt is hard to understand why equations (2) and (3) share the same objectives but the two sets of samples have totally different meanings. Moreover, both objectives are summed uniformly as shown in the experimental setting. Please explain it in details.\n\nFor the quantitative evaluation, Table 1 shows the percentage of papers for each community per cluster. There are many numbers for a single clustering problem. It might be preferred to use a single measurement such as normalized mutual information or accuracy for clustering evaluation. In this way, it is easy to judge the compared methods.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgoG_OpYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Contribution summary]\nAuthors propose MODiR, a dimensionality reduction approach that utilizes both semantic and structural features. Authors report quantitative (metric: precision of k-means++ clustering & AtEdge-length) and qualitative analysis (visualization). \n\n[Comments]\nExperimental analysis could improve e.g. by including more baselines (e.g. other baselines the authors cite in the paper), or by providing more quantitative analysis - e.g. performance at varying degree of completeness / artificial perturbation of the graph. In the current manuscript, the quantitative analysis does not immediately show the benefit or the characteristics of the proposed approaches. In the main section of the manuscript, the analysis is limited to citation networks. \n\nWhile the idea of leveraging both semantic and structural information for the purpose of dimensionality reduction seems to be new and interesting, similar approaches have been explored for text classification tasks (e.g. via graph convolutional networks). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a consensus among reviewers that the paper should not be accepted. No rebuttal was provided, so the paper is rejected. 
Decision: Reject