Title: MULTI-TASK NETWORK EMBEDDING WITH ADAPTIVE LOSS WEIGHTING

Abstract: Network embedding is to learn low-dimensional representations of nodes which mostly preserve the network topological structure. In real-world networks, however, nodes are often associated with a rich set of attributes and labels which are potentially valuable in seeking more effective vector representations. To properly utilize this information, we propose a Joint Autoencoders framework for Multitask network Embedding (JAME), which aims to encode a shared representation of local network structure, node attributes, and available node labels. Jointly embedding via multi-task learning is strongly dependent on the relative weighting between each task's loss function. Tuning these weights by hand is an expensive and difficult process, making multi-task learning prohibitive in practice. Therefore, we define an adaptive loss weighting layer capable of learning an optimal combination of loss weights during representation learning. Empirical evaluations on real-world datasets show effectiveness and efficiency of our JAME model compared to relevant baseline methods.
Network embedding techniques project nodes in a network to a low-dimensional vector space while preserving network structure and inherent properties. It has attracted increasing attention recently due to significant progress in downstream graph mining tasks such as node classification (Grover & 

:

To overcome the problems of the existing methods, we need to design an efficient multi-task learning framework with adaptive loss weighting. It has been widely shown that multi-task learning improves the overall performance of each task relative to learning them separately (Caruana, 1997; Chen et al., 2009; 2011) . Specifically, combining autoencoders as a unified framework has demonstrated promising performance to solve multiple classification and regression tasks simultaneously (Meir et al., 2017; Banijamali et al., 2017; Cadena et al., 2016; Zhuang et al., 2015) . The structure of autoencoders facilitates the integration of multiple information sources towards an efficient joint representation learning. Therefore, we propose a Joint Autoencoders model for Multi-task network Embedding, abbreviated as JAME, which encodes shared representations for nodes in the network via optimizing multiple objectives. In detail, our end-to-end model consists of three autoencoders: a graph convolutional network (GCN) (Kipf & Welling, 2016) autoencoder and two standard autoencoders. The GCN autoencoder captures graph local connectivity by applying spectral filters followed by nonlinear activation functions. Two standard autoencoders perform supervised classification tasks which infer the embeddings of nodes involving both labels and attributes. Overall, our joint autoencoders handle three embedding tasks by optimizing three respective objective functions. However, the performance is highly dependent on an appropriate choice of weighting between each task's loss. To search for an optimal weighting, we define an adaptive loss weighting layer where inputs of the layer are loss values. During joint representation learning, weights within the loss weighting layer are contributed to the gradient and get updated. In summary, the main contributions of this paper are as follows:

• We propose a novel end-to-end model for labeled attributed networks, JAME, which aims at encoding joint representations of nodes via multi-task learning. • We define an adoptive loss weighting layer which simultaneously optimizes the weighted combination of losses during representation learning. • We empirically evaluate effectiveness and efficiency of JAME on real-world networks,

showing its superior performance to the state-of-the-art baselines.

The rest of the paper is organized as follows. Section 2 presents related work. We present notations and the proposed model in Section 4. Section 5 presents empirical evaluation. In Section 6, we conclude and discuss future work.

2. RELATED WORKS:

Network embedding via deep neural networks has become an efficient tool to deal with today's large-scale networks. Recently, there has been numerous embedding methods which focused on preserving the topological structure of plain networks (Goyal & Ferrara, 2018) along with various real-world applications (Jia & Saule, 2018; Zhou, 2019; Rizi & Granitzer, 2019; Nikolentzos et al., 2017) . However, today's networks are often associated with a rich set of attributes and abundant label information, which can be encoded within a joint embedding framework. Gao & Huang (2018) proposed DANE, a joint autoencoders model which takes graph adjacency matrix and attributes as input to learn a shared representation via optimizing four objective functions, i.e. first-order proximity loss, high-order proximity loss, semantic proximity loss, and complementary loss. Their final embeddings preserve first-order and high-order proximity of the network which result superior performance in typical graph mining tasks. Another work by Meng et al. (2019) named CAN employs two variational autoencoders containing an inference and a generative model. The inference model encodes features of nodes and attributes into Gaussian distributions while the generative model reconstructs the original edges and attributes. During representation learning, a reparameterization method is applied to transform embeddings from the Gaussian random variables to the deterministic variables which helps to measure affinities between nodes and attributes. CAN finally learns separate embeddings for attributes and nodes in the same semantic space, mapping nodes nearby their relevant attributes.

Some recent works like BANE (Yang et al., 2018) and LANE (Huang et al., 2017b) applied matrix factorization techniques to attributed networks to jointly embed nodes and attributes into a shared vector space. BANE first applies Weisfeiler-Lehman proximity to aggregate adjacency and attribute matrices into a unified proximity matrix. The proximity matrix then is factorize by Weisfeiler-Lehman factorization approach to obtain final binary representations. LANE first integrate attributes and labels in to same proximity matrix applying a spectral technique persevering node and attribute proximities. Then, it uses graph Laplacian (Chung & Graham, 1997) to smoothly embed label matrix and proximity matrix uniformly into the final vector representations. Although, LANE observes abundant labels during representation learning, the performance is relatively low with an expensive runtime complexity.

Multi-task learning conducts multiple related learning tasks simultaneously such that useful information can be shared among tasks. During multi-task learning usually multiple objectives are jointly optimized to improve efficiency and accuracy of all relevant tasks. Here, the main challenge is how to automatically obtain optimal weights for these objectives within model training. Kendall et al. (2018) 

3. PROBLEM DEFINITION AND PRELIMINARIES:

Let G = (V, E) be a labeled attributed network, where V is a set of n nodes and E is the set of edges. Each node in the network is associated with a set of attributes and label information indicating its affiliation group. We denote A ∈ {0, 1} n×n as adjacency matrix, X ∈ {0, 1} n×f as the attribute matrix with f attributes, and Y ∈ {0, 1} n×k as label matrix with k categories. In the attribute matrix X, every row x i describes the attributes associated with node i. In the label matrix, Y ij = 1 indicates node i belongs to category j. The final embedding matrix is denoted as U ∈ R n×d where d n is the embedding dimension size.

Based on the terminologies described above, we formally define the problem of multi-task network embedding as follows: Given a labeled attributed network G, the goal is to learn a mapping function Θ : {A, X, Y} → U in an supervised manner such that the network structure A, attribute X, and label information Y are preserved running the following learning tasks:

• Network structure embedding encodes network topological structure into vector representation:

EQUATION

• Attributed network embedding integrates attributes of the nodes into vector space:

EQUATION

• Labeled network embedding incorporates node labels into vector embeddings:

Θ y : { A, Y} → U (3) 4 PROPOSED APPROACH 4.1 FRAMEWORK

In this paper, we propose JAME, a Joint Autoencoders model for Multi-task network Embedding with adoptive loss weights. The framework of JAME is shown in Figure 1 . In general, JAME consists of three major parts: network structure autoencoder, attributed network autoencoder, and labeled network autoencoder. These three autoencoders handle three embedding tasks which are optimized at the same time. The network structure autoencoder employs GCN (Kipf & Welling, 2016) layers to encode local links by reconstruction of the adjacency matrix A. The attributed network autoencoder receives the adjacency matrix as input and tries to predict their associated attributes X. The labeled network autoencoder aims at predicting nodes labels Y during joint training. For a multi-task learning problem containing multiple objective functions, JAME offers an adaptive loss weighting layer L capable of learning optimal weighted combination of several objectives. In the following subsections, we will introduce our proposed JAME model in detail. 

. . . . . . Ŷ Ŷ X ̂ X ̂ U U L L Figure 1:

The framework of our end-to-end JAME model. The model is fitted with the adjacency matrix A as input and outputs adjacency matrix A, attribute matrix X and binary label matrix Y, respectively. Shared Embedding layer aggregates information from structure, attribute and labels while Loss Weighting layer learns optimal weights for each embedding task.

4.2. NETWORK STRUCTURE EMBEDDING:

We employ GCN (Kipf & Welling, 2016) layers into basic autoencoders to encapsulate non-linear structure of the input graph. In Figure 1 , the GCN encoder gets latent representations for each node while the decoder computes the pair-wise distance between node latent representations produced by the encoder. After applying a non-linear activation function, the decoder reconstructs the original adjacency matrix. More formally, our GCN encoder is defined as:

EQUATION

where

Ã = D -1 2 (A + I n )D -1 2

) is a symmetrically normalized adjacency matrix with selfconnections, I n is the identity matrix, and D ii = j A ij . H gcn ∈ R n×s is the hidden representation from the GCN encoder with dimension size s. W (l) is the trainable weight matrix and b (l) is the bias of l th layer. Sigmoid σ(.) and ReLU are non-linear element-wise activation functions.

In Figure 1 , the bottleneck features in the shared embedding layer constitutes an informative compact representation of the data. More formally, shared embedding is the summation of representations from the GCN encoder H gcn , attribute encoder H x and label encoder H y :

EQUATION

The graph decoder reconstructs the graph adjacency matrix:

EQUATION

)

where Â is the reconstruction of the adjacency matrix, and U ∈ R n×d is the shared representation matrix with dimension size d. Inspired by Tran ( 2018), we learn model parameters by minimizing a masked cross-entropy loss formulation:

EQUATION

EQUATION

)

where a i ∈ R n is an adjacency vector of A, and âi ∈ R n is the reconstruction output vector of Â.

is the Hadamard (element-wise) product, and m i is a Boolean mask:

m i = 1 if a i = 1, else m i = 0.

4.3. ATTRIBUTED NETWORK EMBEDDING:

To capture attribute proximity associated to nodes, we employ a standard autoencoder. The encoder gets latent representations for each node while the decoder reconstructs the attributes of nodes. The hidden representation of the attribute encoder is defined as:

H x = σ(ReLU(AW (0) + b (0) )W (1) + b (1)

), (9) here A is the adjacency matrix, and H x ∈ R n×s is the hidden representation from the attribute encoder with dimension size s.

The attribute decoder rebuilds the attributes associated to nodes:

EQUATION

)

where X is the reconstruction of the attribute matrix, and U ∈ R n×d is the final shared representation with dimension size d. Due to the binary valued representation of the attribute matrix, we optimize the binary cross-entropy to learn model weights:

EQUATION

where, x i ∈ R n is an attribute vector of X, and xi ∈ R n is a reconstruction output vector of X.

4.4. LABEL INFORMED NETWORK EMBEDDING:

Label information plays an essential role in determining the inscape of each node with strong intrinsic correlations to network structure and node attributes. We propose to model labels information via a supervised autoencoder. The encoder gets latent representations for each node while the decoder predicts the labels of nodes. The label encoder transformation is defined as:

EQUATION

where H y ∈ R n×s is the hidden representation from the label encoder with dimension size s.

The label decoder predicts the abundant labels of nodes:

EQUATION

)

where Ŷ is the reconstruction of the binary label matrix, and U ∈ R n×d is the final shared representation matrix with dimension size d. In multi-class setting, we minimize the categorical cross entropy loss to update model parameters:

EQUATION

here y i ∈ R n is a label vector of Y, ŷi ∈ R n is a reconstruction output vector of Ŷ, and k is the number of label categories.

4.5. MULTI-TASK WEIGHTING:

JAME handles three embedding tasks through optimizing three objective functions. To learn joint vector representations, we need to combine multiple loss functions where each loss has a weight. Since searching for the optimal weightings is expensive and difficult, we propose to learn a weighted linear sum of losses using a loss weighting layer:

EQUATION

where w a , w x , and w y are weights for the network structure, attribute, and label embedding loss respectively. The parameters in the layer L are trained jointly with autoencoders, hence the weights are contributing to the gradient and get updated.

4.6. JAME TRAINING:

Our goal is to minimize the multi-task loss function L = i w i L i , where the sum runs over all T tasks. We thus propose an adaptive method, in which w i can vary at each training step t : w i = w i (t). To optimize the weights w i (t) in L(t) = i w i (t)L i (t), we propose a simple algorithm that w i directly involved to the backpropagated gradient from each task. The detail of our method is described in Algorithm 1. At each training step t, the algorithm receives a mini-batch of nodes and updates loss weights and model weights until convergence.

Algorithm 1: JAME Training with Adaptive Loss Weights Initialize loss weights w i (0) = 1 ∀i Initialize model weights W (l) ∀l Pick weights of the shared layer W (U ) for t = 0 to max iterations do Input batch to compute L i (t) ∀i and L(t

) = i w i L i (t) [standard forward pass] Compute G (i) W (U ) (t) = ∇ W (U ) w i (t)L i (t) ∀i Compute L grad = i G (i) W (U ) (t) Compute loss weights gradients ∇ wi L grad Compute standard gradients ∇ W (l) L(t) Update w i (t) → w i (t + 1) using ∇ wi L grad Update W (l) (t) → W (l) (t + 1) using ∇ W (l) L(t) [standard backward pass] end 4.7 TRAINING COMPLEXITY

It is not difficult to see that the runtime complexity of JAME is O(ncdI), where n is the number of nodes in the graph, d is the maximum dimension of the hidden layers, I is the number of iterations. c is the average number of non-zero entries in each row of matrix A (average degree of the network) which is a constant in scale-free networks (Barabási & Bonabeau, 2003) . Parameter d is related to the dimension of embedding vectors but not related to the number of nodes. I is also a constant and independent from n. Therefore, cdI is independent from n, and thus the overall training complexity is O(n) which linear to the number of nodes in the network.

5. EMPIRICAL EVALUATION:

In this section, we conduct experiments to evaluate the effectiveness of JAME on node classification, link prediction and attribute inferring tasks. Single task evaluation can reveal whether our shared embeddings retain certain type of information. We first introduce the datasets, baseline methods, and experimental settings before presenting the main experiments. We then investigate how loss weighting effects the performance of each individual task. Last, we study the efficiency of JAME framework compared to the baseline methods. Our reference code and data are available at https://www.dropbox.com/sh/zyuwgi067ojn427/ AAAzfwpfml3BpMI9vbY7TGzOa?dl=0.1 

5.1. DATASETS:

We conduct our experiments on the following labeled attributed network datasets: Cora (Sen et al., 2008) , Citeseer (Sen et al., 2008) and Facebook (Leskovec & Mcauley, 2012) . Cora and Citeseer are citation networks, where nodes are publications and edges are citation links. Attributes of each node are bag-of-words representations of the corresponding publications. The labels indicate research topics of the publications. The Facebook network is built from relation data of 10 ego networks, while attributes are constructed from profile information. We set the ego number of each node as its group label. The statistics of the datasets are provided in Table 1 . We compare JAME with the following relevant network embedding baselines:

• LANE: It is a labeled attributed network embedding model that learns node representations based on eigen-decomposition of the graph affinity, attribute affinity and label affinity matrices (Huang et al., 2017b ). • CAN: It learns network representations via a variational auto-encoder algorithm, which consists of an inference model for encoding attributes into Gaussian distributions and a generative model for reconstructing both real edges and attributes (Meng et al., 2019) . • attri2vec: It integrates the network structure and node attributes together seamlessly by a defined transformation function (Zhang et al., 2019) . To preserve the network structure, DeepWalk (Perozzi et al., 2014) mechanism is employed, which makes nodes sharing similar neighbors embedded closely in the attribute subspace. • DANE: It captures the network non-linearity and preserves the proximity of both topological structure and node attributes via a joint autoencoder model (Gao & Huang, 2018) .

5.3. PARAMETER SETTINGS:

For all of the baselines, we follow the authors suggested hyperparameter settings available in the original papers as follows. DANE needs to construct a high-order adjacency matrix where the window size is w = 10, the walk length is l = 80, and the number of walks is γ = 10. For CAN, the hyperparameters of prior distributions, σ V , σ A , are set as 1, and the model is trained in 200 iterations. In attri2vec, w = 10, l = 100, γ = 40, and the number of iterations is 100 million. LANE has three objective functions which are manually weighted as 1, α 1 > 1 and α 2 > 10 to achieve the optimal performance. We train our JAME model for 8 iterations using the Adam optimizer (Kingma & Ba, 2014) with the learning rate being 0.01. The dimension size of each hidden layer is s = 300.

5.4. NODE CLASSIFICATION:

Node classification is a widely adopted task for validating quality of network embeddings (Goyal & Ferrara, 2018) . To investigate the contribution of integrating node labels, we partially observe node labels during representation learning, then we conduct node classification. Among baselines, only LANE (Huang et al., 2017b ) is able to leverage node labels into vector representations, hence we set it as our main baseline. For both methods, we choose embedding dimension d = 128, and we employ Macro-F1 to measure the performance of node classification. To make a comprehensive evaluation, we randomly observe 10% to 100% of the labels during representation learning. From Table 2 , we can find that our proposed JAME consistently outperforms LANE on Citeseer and Facebook. In Cora, observing more than 50% of labeled nodes shows better performance. It shows JAME fed with more labels achieves higher performance since multiple tasks potentially compensate each other by learning from the shared information. In Table 3 , once more labels are contributed in multi-task learning, more attributes and links are also involved in shared representations. Overall, the presented results indicate effectiveness and robustness of JAME to handle and encode additional label information associated to the network. We initialize all weights in the loss weighting layer to 1.0 then the model adjusts these weights to the optimal values during representation learning. Table 4 demonstrates the training steps for learning optimal weights on the Cora dataset. We observe that the loss weighting layer learns the optimal combination of weights to have the highest performance on each task.

5.9. EFFICIENCY EVALUATION:

To study the efficiency of JAME, we compare it with all the baselines on the Facebook dataset. Among baselines, attri2vec (Zhang et al., 2019) 

6. CONCLUSION AND FUTURE WORK:

Labeled data is an essential information source available in a variety of attributed networks, which is beneficial to network representation learning. Incorporating labels into network representation learning is promising however challenging. To this end, we propose a novel framework JAME, which jointly projects labels and attributes into a unified vector space by running multiple embedding tasks. Our proposed model learns a weighted combination of multiple objective functions via a loss weighting layer. Via experiments on three real-world networks, we demonstrate (1) JAME can effectively incorporate the network structure, attributes, and labels into final representations, (2) learning embeddings by JAME is more efficient than the baselines as training complexity is linear to the number of nodes. As to future work, we aim to extend our JAME model to a heterogeneous one, where the network contains multiple types of nodes and edges.

We will share the code on Github in future.

Figures and Tables:

Figure 4: Attribute inference performance of different methods for different embedding sizes

Figure 5: Running time comparison of baselines with respect to the number of nodes on Facebook.

Statistics of datasets.

Table content: <table><tr><td>Dataset</td><td colspan="4">Nodes Edges Attributes Labels</td></tr><tr><td>Cora</td><td>2,708</td><td>5,429</td><td>1,433</td><td>7</td></tr><tr><td>Citeseer</td><td>3,312</td><td>4,660</td><td>3,703</td><td>6</td></tr><tr><td>Facebook</td><td>4,049</td><td>88,234</td><td>576</td><td>10</td></tr><tr><td>5.2 BASELINES</td><td/><td/><td/><td/></tr></table>

Performance of node classification observing different percentages of labels during representation learning.

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td>Macro-F1</td><td/></tr><tr><td/><td>10%</td><td>30%</td><td>50%</td><td>80%</td><td>100%</td></tr><tr><td>Cora</td><td colspan="5">JAME 0.621 ± 0.002 0.663 ± 0.002 0.729 ± 0.002 LANE 0.663 ± 0.001 0.684 ± 0.001 0.726 ± 0.001 0.769 ± 0.001 0.808 ± 0.001 0.892 ± 0.002 0.981 ± 0.002</td></tr><tr><td>Citeseer</td><td colspan="5">JAME LANE 0.538 ± 0.001 0.607 ± 0.001 0.616 ± 0.001 0.635 ± 0.001 0.677 ± 0.001 0.616 ± 0.002 0.651 ± 0.002 0.731 ± 0.002 0.810 ± 0.002 0.936 ± 0.002</td></tr><tr><td>Facebook</td><td colspan="5">JAME LANE 0.522 ± 0.001 0.534 ± 0.001 0.559 ± 0.001 0.578 ± 0.001 0.581 ± 0.001 0.740 ± 0.002 0.769 ± 0.002 0.813 ± 0.002 0.902 ± 0.002 0.983 ± 0.002</td></tr></table>

Impact of observing labels on multi-task weighting during representation learning.To further show capability of JAME for label encoding, we learn representations d = 128 observing full labeled networks, and visualize them by t-SNE(Maaten & Hinton, 2008). The visualization results for all datasets are shown in Figure2. Different node colors indicate different users groups. We see learned embeddings from different classes are clearly compact and separated. We construct edge features from node representations by the Hadamard product(Grover & Leskovec, 2016) to feed a logistic regression. Figure3shows the link prediction performance of JAME and the baselines for different embedding sizes. As shown, in the Citeseer dataset, JAME consistently performs better than any of the baseline models. In Cora and Facebook datasets, JAME obtains competitive AUC scores as CAN, since both employ GCN to encode local connections in the network.Attribute inference aims at predicting the value of attributes associated to the nodes in the network. We follow the CAN(Meng et al., 2019) experiments for attribute inference which conduct a binary classification task using a logistic regression. To evaluate our learned embeddings, we randomly divide nodes into a training (80%), and a test set (20%). We employ the AUC to measure the attribute inference performance since attributes of the nodes are 0/1-valued. Figure4presents the performance of JAME against the baseline models on the three attributed networks. We can find that JAME and CAN show competitive performance in the Citeseer and Facebook datasets as both employ joint autoencoders. CAN employs two variational autoencoders to simultaneously reconstruct both adjacency and attribute matrices that leads to more accurate attribute persevering in final vector embeddings.

Table content: <table><tr><td>Dataset</td><td/><td/><td/><td/><td>Task Weights</td><td/><td/><td/></tr><tr><td/><td colspan="2">50%</td><td/><td/><td>80%</td><td/><td/><td>100%</td></tr><tr><td/><td>wa</td><td>wx</td><td>wy</td><td>wa</td><td>wx</td><td>wy</td><td>wa</td><td>wx</td><td>wy</td></tr><tr><td>Cora</td><td colspan="3">0.163 0.206 0.633</td><td colspan="3">0.225 0.316 0.652</td><td colspan="3">0.333 0.383 0.678</td></tr><tr><td>Citeseer</td><td colspan="3">0.120 0.109 0.528</td><td colspan="3">0.166 0.135 0.556</td><td colspan="3">0.251 0.237 0.588</td></tr><tr><td colspan="4">Facebook 0.153 0.138 0.346</td><td colspan="3">0.182 0.150 0.368</td><td colspan="3">0.281 0.224 0.398</td></tr><tr><td colspan="4">5.5 NETWORK VISUALIZATION</td><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>0 1 2 3 4 5 6</td><td/><td/><td/><td>0 1 2 3 4 5</td><td/><td/><td>1 0 3 7 6 2 8 9 4 5</td></tr><tr><td/><td>(a) Cora</td><td/><td/><td colspan="2">(b) Citeseer</td><td/><td/><td colspan="2">(c) Facebook</td></tr><tr><td colspan="10">Figure 2: Network visualization of JAME embeddings observing full labeled networks.</td></tr><tr><td colspan="2">5.6 LINK PREDICTION</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">Link prediction is a commonly used task to demonstrate the meaningfulness of the vector repre-</td></tr><tr><td colspan="10">sentations for preserving local connections in the network. To carry out the link prediction, we</td></tr><tr><td colspan="10">follow Grover &amp; Leskovec (2016) experiments which gather positive and negative edges. The pos-</td></tr><tr><td colspan="10">itive examples are obtained by randomly removing 50% of the existing edges from the original</td></tr><tr><td colspan="10">graph whereas negative examples are node pairs which are not connected by edges (non-existing</td></tr><tr><td>edges).</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

requires around 10944.35 sec to learn embeddings for the Facebook network. Due to the high runtime of attri2vec, we compare JAME to the other Performance of Link Prediction (LP), Attribute Inference (AI), and Node Classification (NC) applying weighted combination of multiple embedding tasks. Results are reported on the Cora network dataset. baselines which learn representations in more reasonable time. From the results in Figure5, we observe that JAME takes much less running time than LANE, DANE and CAN consistently. As the number of input nodes increases, the performance difference in time also raises up. Although DANE and CAN employ autoencoders, they need a lot more iterations to converge which increases the computation time. In summary, all these observations illustrate the efficiency and scalability of our proposed joint model.

Table content: <table><tr><td>Epoch</td><td/><td>Task Weights</td><td/><td>LP</td><td>AI</td><td>NC</td></tr><tr><td/><td>wa</td><td>wx</td><td>wy</td><td>AUC</td><td>AUC Macro-F1</td></tr><tr><td>1</td><td colspan="3">0.920 0.930 0.916</td><td colspan="2">0.746 0.811</td><td>0.615</td></tr><tr><td>2</td><td colspan="3">0.837 0.856 0.831</td><td colspan="2">0.829 0.895</td><td>0.818</td></tr><tr><td>3</td><td colspan="3">0.755 0.777 0.763</td><td colspan="2">0.887 0.929</td><td>0.913</td></tr><tr><td>4</td><td colspan="3">0.671 0.698 0.733</td><td colspan="2">0.922 0.938</td><td>0.960</td></tr><tr><td>5</td><td colspan="3">0.587 0.619 0.714</td><td colspan="2">0.937 0.946</td><td>0.964</td></tr><tr><td>6</td><td colspan="3">0.503 0.542 0.704</td><td colspan="2">0.944 0.948</td><td>0.972</td></tr><tr><td>7</td><td colspan="3">0.418 0.459 0.689</td><td colspan="2">0.947 0.954</td><td>0.978</td></tr><tr><td>8</td><td colspan="3">0.333 0.383 0.678</td><td colspan="2">0.951 0.956</td><td>0.981</td></tr></table>

Peer review (iclr2020):
This paper proposed a joint network embedding model to incorporate the information of network structures, node attributes and labels simultaneously. Specifically, three auto-encoders, which are responsible for the three kinds of information, respectively, are developed, and are then combined in an intuitive but direct way to consider the mutual influences among different types of information. The weights of combining different types of losses are claimed to be learned from data, although it is achieved in the most direct way. Strengths: The paper proposed an auto-encoder based framework, which can handle the information of network structure, node attributes and node labels simultaneously Weaknesses: The novelty of this paper is very limited. The proposed model seems to be constructed by simply combining several existing methods mechanically. For example, the authors follow the work of GCN (Kipf & Welling, 2016) and use Eq. (4) for network structure embedding. However, the adjacency matrix (structural information) is used to convolute node features (attribute information) in GCN, it is not convincing that the authors apply this method by just removing the node feature matrix. The number of baselines is not enough, and lacking baselines with only structural information. The experimental results are not satisfactory. In the experiments of link prediction and attribute inference, the baseline `CAN` performs better than the proposed model in most cases. In the experiment of efficiency evaluation, the experimental environment is not explained and the number of iterations of compared models are different, which is not rigorous enough. Questions to the Authors: In Eq. (4), (9) and (12), why the parameters of the network (W^((0) ),b^((0) ),W^((1) ),b^((1) )) are the same? Are these formulas wrong? If they are correct, then what is the difference between H_x and H_y? A similar problem exists in Eq. (6), (10), (13). In Eq. (9) and (12), why the attribute encoder and label encoder do not utilize attribute matrix X and label matrix Y respectively, why they only encode the structural information?