Cooperation of Experts: Fusing Heterogeneous Information with Large Margin
Authors: Shuo Wang
Shunyang Huang
Jinghui Yuan
Zhixiang Shen
zhao kang
Venue: ICML 2025 poster (2025)
Keywords: Heterogeneous Information; Multiplex Network; Expert Mechanism
Primary area: deep_learning->graph_neural_networks
TL;DR: We propose CoE, a collaborative expert framework that effectively fuses heterogeneous information sources through adaptive cooperation and large-margin optimization.

Abstract: Fusing heterogeneous information remains a persistent challenge in modern data analysis. While significant progress has been made, existing approaches often fail to account for the inherent heterogeneity of object patterns across different semantic spaces. To address this limitation, we propose the **Cooperation of Experts (CoE)** framework, which encodes multi-typed information into unified heterogeneous multiplex networks. By transcending modality and connection differences, CoE provides a powerful and flexible model for capturing the intricate structures of real-world complex data. In our framework, dedicated encoders act as domain-specific experts, each specializing in learning distinct relational patterns in specific semantic spaces. To enhance robustness and extract complementary knowledge, these experts collaborate through a novel **large margin** mechanism supported by a tailored optimization strategy. Rigorous theoretical analyses guarantee the framework’s feasibility and stability, while extensive experiments across diverse benchmarks demonstrate its superior performance and broad applicability.

Cooperation of Experts:
Fusing Heterogeneous Information with Large Margin
Shuo Wang" Shunyang Huang"
Jinghui Yuan"
Zhixiang Shen Zhao Kang 11
Abstract
the underlying phenomena; in a social network, people linked with different types of ties: friendship, family rela- tionship, professional relationship, etc. To characterize the complexities of this heterogeneity. we encode multi-typed information into a unified heterogeneous multiplex network, where each layer contains the same number of nodes (pos- sibly with different attributes) but different type of links, enabling a comprehensive and effective representation of the diverse relationships inherent in real-world data (Sun & Han, 2013).
Fusing heterogeneous information remains a per- sistent challenge in modern data analysis. While significant progress has been made, existing ap- proaches often fail account for the inherent het- erogeneity of object patterns across different se- mantic spaces. To address this limitation, we pro- pose the Cooperation of Experts (CoE) frame- work, which encodes multi-typed information into unified heterogeneous multiplex networks. By transcending modality and connection differences, CoE provides powerful and flexible model for capturing the intricate structures of real-world complex data. In our framework, dedicated en- coders act domain-specific experts, each spe- cializing in learning distinct relational patterns in specific semantic spaces. To enhance robustness and extract complementary knowledge, these ex- perts collaborate through novel large margin mechanism supported by a tailored optimization strategy. Rigorous theoretical analyses guarantee the framework's 's feasibility and stability, while extensive experiments across diverse benchmarks demonstrate its superior performance and broad applicability. Our code available at https: / //github. com/ stnggea an/CoE.
Inspired by the success of graph neural networks (GNNs) (Kipf & Welling, 2016; Wang et al., 2019), numerous ap- proaches have been proposed for multiplex network rep- resentation learning (Jing et al., 2021; Shen et al., 2024b). However, these methods often rely on training a single pre- dictor across the entire network, which can lead to sub- optimal results by neglecting the intrinsic heterogeneity and varying characteristies of different link patterns (Wang et al., 2024). As shown in Figures la and 1b, classifier performance, which is independently trained on each net- work, differs significantly across different layers, high- lighting the importance of accounting for this variability, To effectively capture diverse node patterns, dedicated ex- componeco designed to focus on specific aspects of the network, are introduced (Shi et al., 2024). Each expert specializes in learning from a particular type of interaction, thereby reducing cross- coos-interference between relationships and enabling a more nuanced representation of complex structures.
1. Introduction
Most real-world objects and data are heterogeneous in the form of multiple types or diverse forms of interactions, which pose significant challenges for modeling their in- tricate relationships (Han, 2009; Shen & Kang, 2025). For instance, multimodal data combines diverse sources, such as images and text, where each modality contributes unique features that collectively offer a richer understanding of
When utilizing multiple experts, effective collaboration among them becomes crucial. Existing Mixture of Experts (MoE) approaches employ gating mechanism that activates only a subset of experts during both training and inference (Li et al.; Shi et al.). While this strategy improves effi- ciency, inherently limits the ability to fully exploit the rich and diverse information embedded in heterogeneous data, This highlights the importance of fostering collaboration rather than competition among experts to leverage cross- layer knowledge effectively. However, this shift introduces two key challenges that must be addressed.
'Equal contribution 'Corresponding author 'niiversity of Electronic Science and Technology of China, Chengdu, Sichuan Province, China NNrttwwestem Polytechnical University. Corre- spondence Shuo Wang <runner2ist gom, Zhao Kang <zkang @ uds.c..du.en....
First, given the challenges capturing the diverse and in- tricate patterns within networks, key question arises: how can we design a framework that effectively extracts and in-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Cooperation of Experts: Fusing Heterogeneous Information with Large Margin
Clasa
$1.78 90.0
25.19
06.78
NetworkA
Natork.A
00:67
Network
Numwokk
Class
Class
Natwork
ACM
(b) Yclp
Impact of fusion procedure on Yelp.
Figure 1. (a) and (b) present the classification results on different networks from the ACM and Yelp datasets, representing the diverse and intricate patterns within networks. (c) "+" symbol denotes directly adding the networks, while "&" represents the fusion procedure used in CoE.
tegrates such complex information across all networks? To address this, we propose Cooperation Experts (CoE), novel architecture specifically designed enable efficient division of labor among experts. The framework adopts two-level experts framework: low-level experts focus on learning patterns from individual networks, while high-level experts operate on a fused representation to capture cross- network relationships. Unlike simple concatenation, the fused representation optimized by maximizing mutual information across networks. As shown in Figure 1c, this approach enhances performance by encoding complemen- tary information, enabling more robust and comprehensive representations.
Our work further advances the utilization multiple experts. To the best of our knowledge. this the first framework to emphasize expert cooperation rather than competition. We introduce a novel large margin opti- mization strategy that maximizes the effectiveness of each expert's 's specialized expertise, fostering a more holistic and in-depth understanding of the underlying knowledge within networks.
The effectiveness of our method is validated through rigorous theoretical analysis and extensive experimen- tal evaluation. We conduct comprehensive experiments with various state-of-the-ard methods on both multi- relational and multi-modal tasks.
Second, how can well-trained experts collaboratively con- tribute to the final prediction? promote cooperation rather than competition, influence in the decision-mall makinn process, determined by its proficiency in capturing distinct patterns. This is achieved through a learnable confidence tensor that dynamically ad- justs the authority of each expert based on its specialization. To ensure effective collaboration, we introduce an inno- vative optimization mechanism for the confidence tensor, which strategically maximizes the margin between the two confidently predicted outcomes. This mechanism not only balances the contributions of different experts but also addresses disparities in their expertise, ultimately boosting overall predictive performance.
assign each expert weighted
2. Related Work
2.1. Multiplex Network Learning
Multiplex network learning focuses on capturing diverse structural patterns in multi-relational networks support tasks such node classification, clustering, and link predic- tion (Qian al., 2024; Xie et al., 2025). With the rise of GNNs, HAN (Wang et al., 2019) integrates structural and attribute information using attention mechanisms at both the node and semantic levels, while MAGNN (Fu et al., 2020) and MHGCN (Yu et al., 2022) leverage graph convolution to extract relational features across multiple layers. MGBO (Huang al., 2024) employs self-expression learning for homophily modeling, whereas HDMI (Jing al., 2021) and DMG (Mo et al., 2023) utilize contrastive learning to enhance self-supervised representations. However, these methods heavily rely on reliable network topology struc- tures, limiting their effectiveness in real-world scenarios with noisy or incomplete data.
Our primary contributions can be summarized as:
We propose the Cooperation of Experts (CoE), novel expert coordination framework designed tract stable and nuanced node patterns from multiplex networks. By tailoring experts to capture both special- ized and shared features, CoE effectively disentangles and integrates heterogeneous information, uncovering intricate connections across diverse relationships.
2

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
2.2. Graph Structure Learning
homogeneous processing of the entire graph, which over- looks structural and feature diversity. To address this, recent studies integrate MoE with graph learning (Wang et al., 2024; Kim et al., 2023) to improve adaptability. Further- more, Mowst (Zeng et al., 2024) refines this approach by training experts to specialize in both feature and structure modalities. However, existing expert mechanisms primarily focus on expert competition, often neglecting the effective utilization of inter-relational dependencies- prrecccseeessppeciseelll the gap our work aims to bridge.
Graph Structure Learning (GSL) has become a crucial area of research in GNNs, particularly for optimizing unreliable graph structures with significant noise. Supervised GSL methods often generate new adjacency matrices with learn- able components, optimized using label information. These methods include probabilistic models (c.g., LDS and GEN (Franceschi et al., 2019; Wang et al., 2021)), similarity- based techniques (e.g., GRCN and IDGL (Yu et al., 2021: Chen et al., 2020)), attention mechanisms (e.g.. NodeFormer (Wu et al., 2022)). or by treating all elements in the adja- cency matrix as learnable parameters (e.g., ProGNN (Jin et al., 2020)). In contrast, methods like SUBLIME, STA- BLF, and GSR (Liu et al., 2022; Li et al., 2022; Zhao et al., 2023) leverage contrastive learning to construct graph struc- tures in self-supervised manner. While most previous studies have focused on single, homogeneous graphs, recent efforts have expanded GSL to heterogeneous graphs in pervised settings (Li et al., 2024a; Pan & Kang. 2023), as demonstrated by GTN and HGSL (Yun et al., 2019; Zhao et al., 2021). More recently, InfoMGF (Shen et al., 2024b) has explored unsupervised fusion of graphs from multiple sources, advancing research in multiplex graph structure learning. Despite these strides, there remains lack of research addressing the challenge of effectively enabling the integration of complex information across multiple net- works.
3. Preliminaries
Heterogeneous Multiplex Networks. It represented by G (G1,.., Gv} composed of V networks, where G, [A,, is the v-th network. X" Nxd, is the attribute matrix for N nodes, so that X, E Rdd represents the feature vector of node i. Ag E 1),1) is the corresponding adjacency matrix and D. is diagonal matrix denoting the degree matrix of A.. Y denotes the node label. We summarize the notations in Appendix A.
Experts Definition. During the training phase, each expert is assigned distinct task by focusing on specific layers, re- ferred to as their knowledge fields. For instance, classifier F is exclusively trained on layer G₁ if G constitutes the knowledge field of F. Consequently, experts are defined as the combination of classifiers and their corresponding layers within the designated knowledge fields, denoted as E,,
2.3. Multiple Learners Learning
4. Our Proposed Method
Multiple Learners Learning is powerful machine learning paradigm that leverages multiple models to enhance predic- tive performance. A classical approach within this paradigm is ensemble learning, which aggregates diverse models to reduce variance and bias, thereby improving generalization. Common ensemble methods include bagging (Zhou & Tan, 2024) and boosting (He al., 2024). However, while boost- ing relies on sequential training where each model builds upon the previous one, it fails to independently capture infor- mation from diverse networks (Emami & Martinez-Muñooz, 2023). In contrast, advanced bagging techniques emphasize diversity among learners. Building on the classical Random Forest strategy (Rigatti, 2017), recent work introduces multi- weighted bagging, such as the WRF method (Winham al., 2013). Although ensemble learning facilitates information fusion, its advantages are predominantly observed in tabu- lar data (Chen& & Guestrin, 2016), making it challenging to extract complex patterns inherent in multiplex networks.
In this section, we introduce the CoE framework using a top-down approach. We begin by detailing the encoding of heterogeneous information into multiplex networks, fol- lowed by the two-level design of experts. Next, we describe the collaborative strategy employed by the experts, and finally, we conclude with the optimization of the critical confidence tensor. The overall framework of the proposed approach is shown in Figure 2.
4.1. Encoding Heterogeneous Information
Modeling arbitrary forms of heterogeneous information through multiplex networks presents critical challenge: ensuring the reliability of the network structure. Real-world networks are often sparse and contain significant noise, which can degrade downstream performance. Therefore, we apply a Graph Structure Learning (GSL) strategy to refine and optimize the network structure.
Another notable specialization within this domain is the expert mechanism. The classical Mixture of Experts (MoE) framework enhances adaptivity by assigning specialized submodels (i.e., experts) to distinct regions of the input space (Li et al.). A key limitation of many GNNs is their
Specifically, we employ network learner, which not only generates refined networks but also ensures the preservation of crucial node features and structural information. We achieve this by utilizing the commonly employed Simple
3

---

Cooperation of Experts: Fusing Heterogeneous Information with Large Margin
Multimodal Data
Graph
Insge
Cres Eatrang Loss
M
D
litution
Graph
lane
High-level experts
Leurning
Andio
Love-level experts
K G,
Heterageneoms Multiplex Newwrrs
E
Confidence Towser
Fused Networks
Large Margin Mechanism
Encoding
Multiplex Network
Heterogeneous Information
Two--evee Experts
Figure The overall framework of proposed CoE. Specifically, CoE first encodes various information into heterogeneous multiplex networks, followed by network fusion through mutual information maximization. Subsequently, two-level experts trained single fused networks respectively. Expert collaboration is enabled by confidence tensor, which optimized via a large margin mechanism.
Graph Convolution (SGC) (Wu et al., 2019), which is a two-layer attentive network:
IG; G2) + [(;;; Gj) (Federici et al.). The loss function can formulated as:
(1) where D. Dr and A. = A, + represents the order of graph aggregation. (-) is the non-linear activation function and @ denotes the element-wise multiplication. For multi-modal data where Ay is not provided, we use the original X to replace H..
H1 = A.
o Wi) W
-E= = Σ (G)Y) - Σ
Σ
V
1= ==i=+11
(2)
Σ [G Giot) Σ
IG
E= s3
where we fuse G¹, and GjJ by optimizing the loss function above to generate the fused network Gij- Additionally, an extra high-level expert is trained on Gzot. which is obtained by fusing all refined single networks following the same fu- sion procedure. Through mutual information maximization, Gig effectively captures the intricate information shared across the single networks. Since directly computing mu- tual information is impractical due to the complexity of data, we reformulate the network-level mutual information in terms of node-level representations.
H, is subsequently utilized reconstruct the network ing the K-nearest neighbors (KNN) method. In practice, we employ KNN sparsification with its sensitvveee approximation to enhance efficiency (Fatemi et al., 2021). I)- Once the reconstructed adjacency matrix is obtained, we perform post-processing operations, like existing GSL paradigm (Li al., 2023), to ensure that the final adja- cency matrix A', adheres to key properties, including negativity, symmetry, and normalization. This process ulti- mately results in the construction of the heterogeneous multi- plex networks G (G', Giy, }, where G2 {A,, X*).
Theorem 4.1. Given a network G with label Y, the cross- entropy loss Lets Y) the upper bound of oT; where % denotes the node representations of all nodes in network G. (Sun al., 2022; Li et al., 2024b)
4.2. Two-level Design for Task-driven Experts
As described above, the two-level expert framework is de- signed to operate across distinct knowledge fields. Specifi- cally, low-level experts focus on individual networks, learn- ing network-specific link patterns, while high-level ex- perts capture shared information across networks iden- tify higher-order dependencies that cannot be uncovered when networks are analyzed independently. To enhance the task relevance of low-level experts, we guide their learn- ing by formulating it as the maximization of (G);; Y). For high-level experts, we incorporate I[;;; G)) as a maxi- mization objective during the training phase, which has been theoretically shown to provide a lower bound for
Following Theorem 4.1, the original LE could be eventually transformed follows:
ŻE Σ La, Σ
Σ
lu(Z';
j=
(3)
V
Σ Iib (Z': Zlot)
ΣΣ
;
2b))
Here, [1,(2';2 is the lower bound of the mutual infor- mation T 1(2 25) between networks and j (Liang et al.,

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
2024):
and the second most confidently predicted outcome among experts. By doing so, it ensures greater consistency in pre- dictions made by different experts, enhancing the overall robustness and reliability of the system. This objective can be formalized as follows:
lis(Z'; y + p() log ( (pp())) En expf (2) ,2,)
(4) where ((--) is score critic approximated using a non- linear projection and cosine similarity. The joint distribution of node representations from networks and j denoted by p(z", ,²)), while the marginal distribution is represented by p(=z). The representations z' and 2 are positive samples of each other, indicating that they correspond to the same node in networks and j. respectively. Implementation details are summarized in Appendix E.
Y (Y o S(Og)) mar2(8 (Og:))
(6)
with the symbol maxz(1) (v) representing the second largest element in vector v, Y; is the i-th column of Y, while the symbol @ represents the Hadamard product. It is worth noting that the definition of margin itself is given by max; (S(9g))) max (S (S((g:)), where maxi (v) is the highest value in vector It is proved that using our defined margin (Eq. 6) leads to the same optimization direction as the original definition while avoiding approximations (Yuan et al., 2024).
4.3. Large Margin Mechanism
Since each expert demonstrates a distinct knowledge pref- erence for different node patterns, it is crucial to bridge the information gap across various knowledge fields. Unlike tra- ditional expert coordination mechanisms (Chen et al., 2022), which rely on a subset of experts, we advocate leveraging all experts in the decision-making process. This approach reduces the risk of amplifying noise caused by the reliance on limited number of experts. To promote collaboration rather than competition among experts, we introduce con- fidence tensor, defined as follows.
However, the max function is non-smooth, and the marz function is both non-convex and non-smooth. Therefore, use the logsumexp function approximate the max function, which is expressed as f f()) : log Ei=1e where a quite large constant. By setting the maximum value of the vector be operated upon to zero and then applying the max operator, marz function can be effectively computed by:
Definition 4.2. A confidence tensor E Rexexk is defined, where с denotes the number of categories and k represents the number of experts. The (r, 8, t)-th element of O denoted as ree quantifies the credibility assigned by the t-th expert to the prediction that sample belongs the r-th category when it actually belongs to the s-th category.
marz(8 (Og:))
1 -lo a
((5(0g)-Yo5((m)
By adding up the loss term for each node, the generalization loss is:
For the convenience of calculation, we empirically expand the confidence tensor θ into Recke Therefore, the final collaborative prediction result of the experts is obtained according to the following formula
M Σ Y, (e)))
(8)
U argmar (S (Sg.)))
(5)
log (lssee))
j=1...e
where g is the collection of various expert opinions. For any expert Fy and given node Us the corresponding decision could be represented
vector E, (v:) E Rex1, thus şi is (ET (vx), E (w:),-.. ..... E (w:))T. S(-) represents the softmax operation on the internal vectors.
On the other hand, also crucial to ensure that predic- tions from all experts always tend be correct. We use cross-entropy loss to measure the distance from the correct opinion.
By leveraging the confidence tensor, all experts contribute to the final decision decision-making process with varying levels of influence. However, in a multi-expert collaborative sys- tem, significant emrors within the cognitive framework of particular expert have the potential to mislead the entire system. To mitigate this risk and improve generalization, it is crucial to design a loss function to encourage consistency among experts. This loss function is designed to maximize the margin between the most confidently predicted outcome
N C -E-XXTTe log (Og.))))
Thus two types of losses are incorporated during the tensor optimization phase, where C and M are specifically de- signed to enhance the correctness and consistency expert decisions. Accordingly, the overall loss function can be
5

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
formulated as
i.i.d. training sample S (() A CoE clas- sifier f X - A produces a probability vector over C classes, denoted f(x) = f.tt)),....f.(t!) T. We define the margin of f at (z, y) as
v C=C - MM + Σ
V
V
Lo (Z.
Σ
Σ
Ix( 22)
i=1
i=
j=++1
V
v
75 () = fu(x) max (r),
(12)
Iu Zlot)
I&b 2
y'
j#i
large positive Tr y) implies a strong preference for the correct class y. The usual 0-1 loss is lo.1 (f:z, ==
(10)
where 7) is given hyperparameter to balance C and M. additional experiments are conducted to show the impact of all parameters appeared in the optimization phase.
Iarg max, fc( # y]. We also define eg (f:r.y) == Th v) S 0] and the ramp loss:
5. Theoretical Analysis
(1(11) 2 n
71 75(2,1)
In this section, we establish that our proposed model sat- isfies key properties, including partial convexity, Lipschitz continuity and robust generalization ability. Additionally, present the optimization algorithm and prove its conver- gence to critical points of the model's loss function. Partial convexity indicates that the loss function remains convex with respect to certain intermediate variables, ensuring a smoother optimization landscape and reducing the risk of getting trapped in local optima.
(,(f;
=
1
0 (5 v) 7, (13)
7
1,
7)(,,1) S..
One has lo. (f:(f;:; v) S C. (f; < (J; v)-
Let by o be the set of ramp-loss functions induced by a hypothesis class F, where each f E F is CoE classifier. Then for any 5 > 0, with probability at least 1 ō over an i.idd. sample S = { v)) the following holds for all I F:
Theorem 5.1. [(Og,) is a convex function with respect to (Ogi)-
B B((0-1(f)) & B (0 v)]
Theorem 5.2. C is Lipschitz continuous, and the Lipschitz
constant T. S 2//ck 7
1 n
2 RR, + +3/ 4ff vi)
())
(14)
Consider performing gradient descent for T iterations, yielding a sequence of points {Gu, .,.....6r O Or-1)- Due to the Lipschitz continuity of the loss function, the norm of the gradient at the point with the smallest gradient con- verges to zero. This implies that, with sufficiently large number of iterations, there must be at least one critical point within the set of points visited during the process.
2n
where RF, is the Rademacher complexity of the CoE waarginffnnctinn class.
In CoE framework, we have experts E Eg each outputs a probability vector Ex(x) E RC, Besides, we have a confidence tensor O and we form g(x:) = [E) ,-.., Fx(=) then f(x) (oftmax(Og(z))) And the margin is (1 [Og(z)), max, ()0((())))
Theorem 5.3. For step size S 4. the gradient descent algorithm generates a sequence {0,) such that
min 12 2(((O))- - (() nfT
(1))
Here we assume ||0|| S Bg and (1) S Ge Vj, where Be and Ge could be considered as two bounds. Hence [|g(x))]2 S VT: G.. Let Fe be the set of CoE margin functions 7/-
where L is the minimum value of (()).
The above theorem proves that CoE is theoretically conver- gent. All proofs for theorems in this section are given in Appendix D.
Then (Fo) S CMC BoGLvK vn where CMC is a constant
the order of vin(C), reflecting the multi-class max operation. With probability at least 1 - all f E Fe satisfy
Moreover, as generalization capability serves critical metric for evaluating model performance, we conduct systematic analysis of the proposed model's generalization behavior in this study.
[[(or(()] < - Σ e, (f:I. v.) GB V/
log(
+31
We begin by formalizing the learning setup. Let X x y be the lnput-label space with D = C classes. We have
7/n
2n
(15)
6

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
Table 1. Accuracy + STD comparison (%) under the setting of network node classification task. The highest result is highlighted with red boldface and the runner-up is bolded. The symbol "OOM" means out of memory.
Methods
ACM
DBLP
Yelp
MAG
Amazon
GCN
89.04_0.62
74.03 161 00.70
74.60.10.13 OOM
93.12:10.28
HAN
91.30++0.33
81.28+0.23
52.04+1.555 1.55
OOM
LDS
88.55=+0.42
87.17+00.81
87.11
OOM
OOM
GRCN
92.00=10.20
91.10:1:.43
78.30
OOM
95.27-10.11
IDGL
92.73=1.11
79.29=40.58
88.28-5.08
OOM
93,68=00.45
ProGNN
9..09
1..1
1..43++1.3
OOM
93.12:+0.47
GEN
90.82-12.69
91.33-1:0.7
69.22-13.04
OOM
9.05
NodeFormer
90.73=0088
80.26:10.66
90.76=+1.03
77.1410.19 19
97.72-10.38
SUBLIME
91.81=0026
91.49-10.32
90.91=10.75
67.45:10.2
97.02-10.93
STABLE
89.18 1..0
89.77=3336
OOM
97.76=10.51
GSR 9..7
77.83++0.38
84.46:10.79
OOM
94.10=10.79
HDMI
91.45=0.43
90.14:10.38
73.75=0087
69.25:10.12
95.08=10.47
InfoMGF
92.81=0.29
91.45:+00.37
92.01=00422
77.32++0.09
97.78-+0.21
GMoE
90.29==0.41
91.18++0.43
91.92-H0377
77.27+00.54
97.78==0.4
Mowst
1.01
89.69+00.99
91.31+1.17
77.40+00.11
97.89-0.44
CoF:
94.21+0.14
92.27+00224
93.40+0.12
78.37+00.16
98.01-
shows that ensuring a large margin y and controlling the norms Be (confidence-tensor magnitude) and G. (expert- output scale) leads to generalization guarantee. Increasing the number of experts le has a vk impact, illustrating the trade-off between model capacity and margin- based guar- antees. Due to CoE mechanism limits the number of k and Bg, while G. is fixed, thus our model has remarkable generalization ability.
tailed descriptions of these datasets can be found in Ap- pendix C.
Baselines. We evaluate the performance of our method by comparing it against range of existing approaches. In the network data scenario, we select two supervised structure-fixed GNNs-GCN (Kipf& & Welling, 2016) and HAN (Wang et al., 2019) as well as six supervised GSL methods: LDS (Franceschi et al., 2019), GRCN (Yu et al., 2021), IDGL. (Chen et al., 2020), ProGNN (Jin et al., 2020), GEN (Wang et al., 2021), and NodeFFormer (Wu al., 2022). Additionally, we include three unsupervised GSL methods:
6. Experiments
In this section, we present comprehensive experiments to thoroughly evaluate the effectiveness and robustness of our proposed CoE model. Specifically, we address the follow- ing research questions: RQ1: Does CoE outperform SOTA models on multiplex and multimodal network datasets? RQ2: What is the impact of the key components on model performance? RQ3: How robust the CoE algorithm when subjected to structural attacks or noise?
SUBLIME (Liu et al., 2022), STABLE (Li et al., 2022), and GSR (Zhao et al., 2023), and two unsupervised multiplex methods: HDMI (Jing et al., 2021) and InfoMGF (Shen et al., 2024b). Finally, we also compare against two state- of-the-art Graph-MoE methods: GMoE (Wang et al., 2024) and Mowst (Zeng et al., 2024).
For multimodal data, select four representative GSL methods, including two supervised approaches (ProGNN and GEN) and two unsupervised approaches (SUBLIME and InfoMGF). Additionally, we incorporate three multi- view methods (i.e., DCCAE (Wang et al., 2015), CPM-Nets (Zhang et al., 2020), and ECML. (Xu et al., 2024)) and two multimodal methods (i.e., MMDynamics (Han al., 2022) and QMF (Zhang et al., 2023)). Classification accuracy is used as the evaluation metric. To ensure a fair comparison, we adopt GCN as the backbone encoder for all models. For methods that cannot support multiplex networks or multi- modal data learning, we conduct training on each network or modality separately and report the maximum accuracy results. Implementation details can be found in Appendix
6.1. Experimental Setup
Datasets. To thoroughly evaluate the effectiveness of CoE, we conduct experiments on five benchmark network datasets, including citation networks (ACM (Yun et al., 2019) and DBLP (Yun et al., 2019)), review networks (Yelp (Lu et al., 2019) and Amazon (McAuley & Leskovec, 2013; Gao et al., 2023)), and a large-scale citation network MAG (Wang et al., 2020). Additionally, we perform experiments on four multimodal datasets: ESP, Flickr, IAPR, and NUS (Xia et al., 2023), which lack structural information. Thus, we first build network structures with KNN for them. De-
7

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
B.
Table 3. Performance (% + σ) CoF and variants.
6.2. Node Classification Performance (RQ1)
Variants
ACM
DBLP
Yelp
MAG
Amazon
The experimental results, presented in Table 1, show that our proposed CoE achieves the highest accuracy across all datasets, consistently outperforming existing approaches. Notably, CoE surpasses recent Graph-MoE and multiplex methods. Furthermore, our method exhibits the lowest stan- dard deviation in most cases, highlighting its stability.
RF WRF
93.39-0.19
91,48:0.14
91.61#0.21
76.91=0.16
97.56:20.12
93.64-0.15
91,97+0.22
93.05+0.16
77.45-0.18
97.78-0.26
wo HE w/o GSI.
91.25=0.17
90,71:0.26
68.27:0.32
78.05=0.27
94,50:00.16
93.60 0.29
91.13+0.244
93.14+0.33
77.97-0.34
97.87-0.22
CaB
94.21=0.14
92.27:40.24
93:40=0.12
78.37=0.16
98.01=00.99
nism, we replace it with two variants: the original random forest (RF) (Rigatti, 2017) and the weighted random for- est (WRF) (Winham et al., 2013). Experimental results confirm the superiority of our large-margin mechanism. Notably, WRF outperforms RF, suggesting that different experts demonstrate varying levels of proficiency for differ- ent node patterns. This finding highlights the importance of designing a tailored expert collaboration mechanism to accommodate specific scenarios.
The multimodal data classification results are summarized in Table 2, where the topology first inferred from the initial feature matrix. As shown in the table, our pro- posed CoE significantly outperforms baseline models, in- cluding both network learning frameworks and multimodal approaches. These results highlight CoE's stability and adaptability across diverse settings.
Table 2. Accuracy + STD comparison (% + σ) under the setting of multimodal data classification task.
6.4. Robustness Analysis (RQ3)
Methods
ESP
Flickr
IAPR
NUS
To assess the robustness of our proposed method, we in- troduce perturbations to the network structure by randomly adding or removing edges in the ACM dataset. The propor- tion of modified edges varied from 0 to 0.9 to simulate different levels of attack intensity. For comparison, we evaluate several baseline methods, including the traditional fixed-structure model (GCN), the graph structure learning method (SUBLIME), the multiplex learning method (In- foMGF), and the expert-mixing approach (Mowst).
PmGGN
78.52+0.16
67.77++0.24
67.63 0.11
62.78++0.39
GEN
79.05+0.24
63.33-+0.22
67.75-
64.26++0.19
SUBLIME
77.53-20.14
64.97:+0.18
62.06-0.111
52.54:10.16
InfoMGF
79.23=0.24
68.79:10.22
68.75-0.18
63.80:10.19
DCCAE
78.48=0.45
67.98:100.2
0.64
62.39:10.27
CPM-Nets
80.09=0.59
69.49:10.46
67.33=0.45
65.34:10.54
ECML
71.21=0.11
62.41:10.16
58.89=0.18
64.34:0.14
MMDynamics
0.51
65.44:10.73
66.59=0.41
63.64:10.68
QMF
00.34=0.34
69.24:10.34
69.08=0.45
65,42:10.37
CoE
81.11=0.05
70.24:10.09
71004=0.04
66.80:10.32
g,s
a2-
6.3. Ablation Studies (RQ2)
To assess the contribution of each component in the CoB framework, we design four variant models by systematically modifying or removing specific elements and evaluate their performance on the node classification task. The experimen- tal results, presented in Table 3, provide insights into the impact of each component on overall performance.
Hond
76
Prrtt
73
6.0
02
0.8
0.5
0.2
0.6
Add edge rabe
Delete edge rate
(a) Adding edges
(b) Deleting edges
Figure 3. Robustness analysis on ACM.
Effectiveness of the expert learning components. Recall that CoE employs GSL with a two-level expert design, incor- porating both low-level and high-level experts. To analyze their contributions, we construct two variant models: one without high-level experts (w/o-HE) and another without the GSL process (w/o-GSL). The results show that both variants exhibit degraded performance, with the absence of high-level experts having more pronounced negative impact compared to the removal of GSL.
The experimental results, presented in Figure 3, clearly show that as perturbations increase, the performance of methods degrades to varying extents. However, graph structure learning methods (CoE, InfoMGF, and SUBLIME) exhibit strong resilience. In particular, CoE benefits signifi- cantly from GSL, allowing it to accurately detect and correct disrupted network connections. Even under extreme per- turbations, CoE consistently outperforms other methods, demonstrating exceptional robustness in maintaining stable performance and effectively mitigating the adverse effects of structural distortions.
Effectiveness of expert collaboration mechanism. To evalu- ate the effectiveness of the proposed large-margin mecha-
8

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
Furthermore, we investigate the impact of two key hyperpa- rameters in CoR: the optimization-phass parameters and a. We set these parameters to {50, 100, 200, 500, 1000). As illustrated in Figure 4, CoE shows low sensitivity to variations both and at demonstrating the remarkable stability of our model. Additional sensitivity analyses for other hyperparameters are provided in Appendix F.
Chen, T. and Guestrin, C. Xgboost: A scalable boosting system. In Proceedings of the 22nd acm sigkdd inter- national conference on knowledge discovery and data mining, pp. 785-794, 2016.
Chen, Y., Wu, L., and Zaki, M. Iterative deep graph learn- ing for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems, 33: 33:19314-19326 2020.
$4-
Chen, Z., Deng, Y., Wu, Y., Gu, Q.. and Li, Y. Towards layer in deep learning.
Accrrsyyt
Arcuneyy
92
derstanding the mixture-of-expee Advances in neural information processing systems, 35: 23049 23049-23062,22 2022.
20
28
50
1.50
250
1000
Emami, S. and Martinez-Muñoz, G. Sequential training of neural networks with gradient boosting. IEEE Access, 11: 42738 42750, 2023.
(a) Sensitivity on y
(b) Sensitivity on
Figure 4. Sensitivity analysis critical hyper- syper-parameters.
Fatemi, B., El Asri, L., and Kazemi, S. M. Slaps: Self- supervision improves structure learning for graph neural networks. Advances in Neural Information Processing Systems, 34:22667-2268) 2021.
7. Conclusion
In this work, we propose and analyze a novel paradigm for fusing diverse information through heterogeneous multiplex networks. Our approach introduces an expert mechanism into network learning and provides a tailored optimization strategy. We present a comprehensive theoretical analysis alongside extensive experiments, comparing our method with various baselines. To the best of our knowledge, this is the first study to explore expert learning in multiplex networks. Unlike traditional approaches that emphasize expert competition, our work pioneers shift toward expert cooperation, advocating for further research on applying the Collaboration Experts mechanism to more complex scenarios.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view in- formation bottleneck. In International Conference Learning Representations.
Franceschi, L., Niepert, M., Pontil, M., and He, X. Learning discrete structures for graph neural networks. Interna- tional conference on machine learning, pp. 1972-1982. PMLR, 2019.
Fu, X., Zhang. J., Meng, Z., and King, Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of the web conference 2020, pp. 2331-2341, 2020.
Impact Statement
Gao, Y., Wang. X., He, X., Liu, Z., Feng, H., and Zhang, Y. Addressing heterophily in graph anomaly detection: A perspective of graph spectrum. In Proceedings of the ACM Web Conference 2023, pp. 1528-1538. 2023.
This paper aims to advance the field of Machine Learning. While our work has potential societal implications, we do not identify any specific concerns that require particular emphasis this stage.
Han, J. Mining heterogeneous information networks by exploring the power of links. In International conference on discovery science, pp. 13-30. Springer, 2009.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. U24A20323).
Han, Z., Yang, F., Huang, J., Zhang, C., and Yao, Multi- modal dynamics: Dynamical fusion for trustworthy mul- timodal classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 20707-20717, 2022.
References
Baranwal, A., Fountoulakis, K., and Jagannath, A. Effects of graph convolutions in multi-layer networks. In The Eleventh International Conference on Learning Repre- sentations.
He, Y.X., Wu, Y.C.,, Qian, C., and Zhou, Z.-H. Mar- gin distribution and structural diversity guided ensemble pruning. Machine Learning, 113())3555-3357, 2024.

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
Huang, Y., Mo, Y., Liu, Y., Nie, C., Wen, G., and Zhu, X. Multiplex graph representation learning via bi-level optimization. In Proceedings of the Thirty-Third Inter- national Joint Conference on Artificial Intelligence, pp. 2081-2089 2024.
Liu, Y., Zheng, Y., Zhang, D., Chen, H., Peng, H., and Pan, S. Towards unsupervised deep graph structure learning. In Proceedings of the ACM Web Conference 2022, pp. 1392-1403, 2022.
Lu, Y. Shi, C., Hu, L., and Liu, Z. Relation structure-aware heterogeneous information network embedding. In Pro- ceedings of the AAAI conference artificial intelligence, volume 33, pp. 4456 4463, 2019.
Jin, W., Ma, Y., Liu, X., Tang, X., Wang, S., and Tang, J. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 66-74, 2020.
Mao, Y., Yan, X., Guo, Q., and Ye, Y. Deep mutual informa- tion maximin for cross-modal clustering. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 35, pp. 8893-8901, 2021.
Jing. B., Park, C., and Tong, H. Hdmi: High-order deep multiplex infomax. In Proceedings of the Web Conference 2021.
2021, pp.
McAuley, J. J. and Leskovee, J. From amateurs to connois- seurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference World Wide Web, pp. 897-908, 2013.
Kim, S., Lee, D., Kang, S., Lee, S., and Yu, H. Learning appology-speciffe experts for molecular property predic- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8291-8299, 2023.
Mo, Y., Lei, Y., Shen, J., Shi, X., Shen, H. T., and Zhu, X. Disentangled multiplex graph representation learning, In International Conference on Machine Learning, pp. 24983-25005. PMLR, 2023.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations, 2016.
Pan, E. and Kang. Z. Beyond homophily: Reconstruct- ing structure for graph-agnostic clustering. In Interna- tional conference on machine learning, pp. PMLR, 2023.
Li, B., Pan, E., and Kang, Z. Pc-conv: and heterophily with two-fold filtering. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 13437-13445, 2024a.
Unifying homophily
Qian, X., Li, B., and Kang. Z. Upper bounding barlow twins: A novel filter for multi-relational clustering, In Proceedings of the AAAI conference on artificial intelli- gence, volume 38, pp.
Li, J.. Su, Q. Yang, Y., Jiang, Y., Wang, C., and Xu, H. Adaptive gating in oftture-f--eppert based language models. In The 2023 Conference on Empirical Methods in Natural Lunguage Processing.
2024.
Rigatti, S. Random forest. Journal of Insurance Medicine, 47(1):31-39, 2017.
Li, K., Liu, Y., Ao, X., Chi, J., Feng, J., Yang, H., and Hc, Q. Reliable representations make stronger defender: Unsupervised structure refinement for robust gnn. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 925-935, 2022.
Shen, Z. and Kang, Z. When heterophily meets heteroge- neous graphs: Latent graphs guided unsupervised rep- resentation learning. IEEE Transactions on Neural Net- works and Learning Systems, 2025.
Shen, Z., He, H., and Kang, Z. Balanced multi-relational graph clustering. In Proceedings of the 32nd ACM In- ternational Conference on Multimedia, pp. 4120-4128 2024a.
Li, S., Song, J., Zhang, B., Ruan, G., Xie, J., and Wang, C. Gagsl: Global-augmented graph structure learn- ing via graph information bottleneck. arXiv preprint 2xXi::4111.04366, 2024b.
Shen, 7.,, Wang, S., and Kang, 7. Beyond redundancy: Information-awwrc unsupervised multiplex graph struc- ture learning. Advances in neural information processing systems, 2024b.
Li, Z., Sun, X., Luo, Y., Zhu, Y., Chen, D., Luo, Y., Zhou, X., Liu, Q. Wu, S., Wang, L.. et al. Gslb: The graph structure learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
Shi, C., Yang, C.,, Zhu, X., Wang, J., Wa, T., Li, S., Cai, D., Yang, Y., and Meng, Y. Unchosen experts can contribute too: Unleashing moe models' power by celf-contrast. In The Tiihty- Annual Conference on Neural Informa- tion Processing Systems.
Liang, P. P.. Deng, Z., Ma, M. Q., Zou, J. Y.. Morency, L.-P., and Salakhutdinov, R. Factorized contrastive leam- ing: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
Shi, Y., Wang, Y., Lang, W., Zhang, J., Dong, P., and Li, A. Mixture of experts for node classification, arXiv preprint arXiv:2412.00418, 2024.
Xie, X., Li, B., Pan, E., Guo, Z., Kang. Z., and Chen, W. One node model: Featuring the missing-hallf for graph clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1168821696,, 2025.
Sun, Q.. Li, J., Peng, H., Fu, Ji, C., and Philip, S. Y. Graph structure learning with variational informa- tion bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4175-4174, 2022.
Xu, C.,, Si, J., Guan, Z.., Zhao, W., Wu, Y., and Gao, X. Reliable conflictive multi-view learning. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 38, pp. 16129-16137, 2024.
Sun, Y. and Han, J. Mining heterogeneous information networks: a structural analysis approach. ACM SIGKDD explorations newsletter, 14(2):20-25 2013.
Yu, D., Zhang, R., Jiang, 7., Wu, Y., and Yang, Y. Graph- revised convolutional network. Machine Learning and Knowledge Discovery in Dutabases: European Confer- ence, ECML PKDD 2020, Ghent, Belgium, September 14- 18, 2020, Proceedings, Part III, pp. 378-393, Springer, 2021.
Wang, H., Jiang, Z., You, Y., Han, Y., Liu, G., Srinivasa, J., Kompella, R., Wang, Z., et al. Graph mixture of experts: Learning large-scale graphs with explicit diversity moddling. Advances in Neural Information Processing Systems, 36, 2024.
Yu, P., Fu, Yu, Y., Huang, C., Zhao, Z., and Dong, J. Multiplex heterogeneous graph convolutional network. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2377-2387, 2022.
Wang, K., Shen, Z., Huang, C.,, Wu, C.-H., Dong, Y., and Kanakia, A. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 111):396-43, 2020.
Wang, R., Mou, S., Wang, X., Xiao, W., Ju, Q. Shi, C., and Xie, X. Graph structure estimation neural networks. In Proceedings of the web conference 2021, pp. 342-353, 2021.
Yuan, J., Chen, H., Luo, R., and Nie, F. A margin- maximizing fine-grained ensemble method. arXiv preprint arXiv:2409. 12849, 2024.
Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
Wang, W., Arora, R., Livescu, K., and Bilmes, J. On deep multi-view representation learning. In International con- ference on machine learning, pp. 1093-1092. PMLR, 2015.
Zeng, H., Lyu, H., Hu, D., Xia, Y., and Luo, J. Mixture weak and strong experts on graphs. The Twelfth International Conference on Learning Representations, 2024.
Wang, X., Ji, H., Shi, C., Wang, B., Ye, Y., Cui, P., and
Yu, P. S. Heterogeneous graph attention network. In The world wide web conference, pp. 2022-2032, 2019.
Zhang, C., Cui, Y., Han, Z., Zhou, J. T., Fu, H., and Hu, Q. Deep partial multi-view learning. IEEE transactions on pattern analysis and machine intelligence, 44(5):2402- 2415, 2020.
Winham, S. J., Freimuth, R. R., and Biernacka, J. M. A weighted random forests approach to improve predictive performance. Statistical Analysis and Data Mining: The ASA Data Science Journal, 6(6):496-505 2013.
Zhang, Q., Wu, H., Zhang. Hu, Qu, H., Zhou, T., and Peng, X. Provable dynamic fusion for low-quality multimodal data. In International conference machine learning, pp. 41753-41769. PMIRR 2023.
Wu, F., Souza, A., Zhang. T., Fifty, C., Yu, T., and Wein- berger, K. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861- 6871. PMLR, 2019.
Wu, Q. Zhao, W., Li, Z., Wipf, D. P., and Yan, Node- former: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35:27387-211 -24401, 2022.
Zhao, J., Wang, X., Shi, C., Hu, B., Song, G., and Ye, Y. Heterogeneous graph structure learning for graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 4697-4705, 2021.
Xia, W., Wang, T., Gao, Q. Yang, M., and Gao, X. Graph embedding contrastive multi-modal representation learn- ing for clustering. JEEE Transactions on Image Pro- cessing, 32:1170-1183, 2023. doi: 10.1109/TIP.202. 3240863.
Zhao, J., Wen, Q., Ju, M., Zhang, C., and Ye, Y. Self- supervised graph structure refinement for graph neural networks. In Proceedings of the Sixteenth ACM Interna- tional Conference on Web Search and Data Mining, pp. 159-167, 2023.

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
Zhou, Z.-H. and Tan, Z.-H. Learnware: Small models do big. Science China Information Sciences, 67(1)2 2024.
12

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
A. Notations
Table Notations used in our paper.
Notation
Description
V.N.dy
The number of newworks/nnddsffsttrc...
G (Gi..,, Gv}
The heterogeneous multiplex network.
G. X*)
The y-th original network.
A. (0,1)NxM
The adjacency matrix of y-th original network.
D.
The diagonal matrix denoting the degree matrix of Ag.
X c CNNa,
The attribute matrix of e-th network.
X; e Rd
The feature vector of node i.
G (4,
The v-th refined network.
G
The network fused by all refined single networks. The expert integrates the node predictor and the corresponding networks in its knowledge field. The label information.
E;
H" ERNx,
The node embeddings of the original network from the attention learner. The order of graph aggregation. The number of neighbors in ANN.
K
The number of categories. The collection of various expert opinions.
k
The number of experts.
The confidence tensor.
G,
The mutual information between the i-th and i-th refined networks.
M
The generalization loss contributed hy all nodes.
The cross-entropy loss to measure the distance from the correct opinion.
Cat
The cross-entropy loss.
Ce
The training loss of all experts.
The overall loss function.
(az
The largest element in vector
(a
The second largest element in vector Constant used in computing mazz.
A given hyperparameter in L.
((.))
The score critic approximated using non non-linear projection and cosine similarity. The softmax operation the internal vectors.
Si-)
a()
The non-linear activation function.
The Hadamard product.
B. Hyper-parameters Settings and Infrastructure
Table 5. Details of the pareeerss settings.
Dataset
E
Ir
dp
d
K
r
L
ACM
800
0.001
128
64
15
2
0.2
100
100
DBLP
400
0.005
64
32
10
2
0.2
100
100
Yelp
400
0.0001
128
64
15
2
0.2
100
100
MAG
400
0.001
64
15
3
0.2
100
Amazon
400
0.001
256
64
15
3
0.2
100
100
All experiments are conducted on a platform equipped with an Intel(R) Xeon(R) Gold 5220 CPU and an NVIDIA A800 80GB GPU, using PyTorch 2.1.1 and DGL 2.4.0. Each experiment run five times and the average results are reported.
Our model is trained using the Adam optimizer. The hyper-paranen settings for all datasets are presented in Table 5. Here, E represents the number of training epochs, which is tuned within the set {100, 200, 400, 500, 800, 1000} and Ir is the learning rate which is searched within the set {0.0001, 0.005, 0.001, 0.005. 0.01 The hidden layer dimension d, and the representation dimension d of the graph encoder GCN are tuned within the set {32, 64, 128, 256}. The number of neighbors K for KNN is searched within the set (5. 10, 15, 20, 30). The order of graph aggregation r and the number of layers L in GCN are set to either 2 or 3, which is line with the common layer count of GNN models (Baranwal et al.). The temperature parameter Te in the contrastive loss is fixed at 0.2. the optimization phase, the parameters and 7y are
13

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
both 100. For the large-scale datasets MAG and Amazon, when computing the contrastive loss for estimating mutual information, use batch size of 2560 and process the data in batches.
C. Datasets
Table 6. Statistics of multi-relational network datasets.
Dataset
Nodes
Relation type
Edges
Features
Classes
Training
Validation
Test
Paper-Author Paper (PAP)
26,416
ACM
3,025
Paper- Paper-Subject-Pagee (PSP)
2.197,556
1,902
600
300
2,125
Author Paper-Author Author (APA)
2,398
DBLP
2,957
334
600
300
2,057
Author-Papers Paper-Conferere Conference- Cowferrencce-apper-uuuiie Paper-Author (APCPA)
1,460,724
Userrsssss (BUB)
525,718
Yelp
Business Business (BSB)
2,475,108
2,014
Business-Rating Ratinn Levels-Business (BLB)
1,484,692
Paper- Paper (PP)
1,806,596
MAG
113,919
128
40,000
10,000
63,919
Paper-Author- Paper (PAP)
10,067,799
Pree-Prrduutt (UPU)
175,608
Amazon
11,944
User Star rating. User (USU)
3,566,479
25
4,777
2,388
4,779
User- User-Review-UUe Review-User (UVU)
1,036,737
Table 7. Statistics of multimodal datasets.
Datasel
Classes
Total
Features
Modal
Training
Validation
Test
IAPR
6
7,855
100
Image,texi
3,926
1,961
1,968
ESP
7
11,032
100
Image,text
5,514
2,754
2,764
Plickr
12,154
100
Image,text
6,076
3,037
3,041
NUS
20,000
100
Image,text
10,000
5,000
5,000
choose multi-relational network benchmark datasets in total. The statistics the datasets are provided in Table Following (Shen et al., 2024a), here we extract MAG from the original OGBN-MAG (Wang et al., 2020), remaining the nodes from the four largest classes.
We also select 4 multimodal datasets where the topological structure is not given. For fair comparison, adopt the identical data processing approach as described in (Mao et al., 2021) on all datasets, where the 4096-dimensional (D) image features are extracted by VGG-16 net and 768D text features are extracted by BERT net. To further improve efficiency, Principal Component Analyses (PCA) is employed to reduce the dimensions of both image features and text features to 100. The examples and descriptions of the aforementioned multimodal datasets are shown in Table 7.
D. Proof
D.I. Proof of theorem 5.1
Proof. Assume that L(Og.) can be divided into three parts: L1, L2, and Lz. which are expressed as follows:
C1 =
(Y log(S (9g:))) L2 Σ (TYTS(())))
f=1
(16)
L3 log
1=1
C, and L2 are evidently convex with respect to (Og.). Considering L3. the exponential term a(S(Og:) Y, o S(Og:)); a linear transformation with respect to S(Qg,). Since the log-sum-exp function is well known to be convex, and the composition of convex function with linear function preserves convexity, the proof is complete.
14

---

L Cooperation of Experts: Fusing Heterogencous Information with Large Margin
D.2. Proof of theorem 5.2
Proof. is not difficult to prove that, through a variable substitution, the derivative
&Ca can be equivalently written as: 60
0Ls Ej=1 I(j # m 00
- S
(17)
7- ( (a0()0)m + 1)
(0)))))
Where I(j m) is the indicator function, which is 0 when j = m and 1 otherwise. It is evident that 1gyl S 1, 10mpl S 1, and pS(4g) S Therefore, the following inequality holds:
aC1
1g + (8(0g)) 2
(18)
80
Similarly, we have:
0L2
S s7(((gg)
1gg0mp s( S 27
(19)
pmpm
Based the fact that Ej=1 (0S())) (0 m +1| S c, is not difficult prove that
8Ls 80
I
2ye"
For any 2 and
Ô, by the mean value theorem, there exists a such that:
(()) L(ê) VL(o) - -)
(20)
Applying the Cauchy-Schwand inequality, we obtain:
(C(e) C(ô) s [V[(())== 611
(21)
where / = Ep.
( (ac ac pm A=o
From the bounds on 30..'
acg
we know:
0C
2 + 76a
(22)
00 pg
θ=a
Thus, summing over all p, gives:
|V < Σ 2 (2(1+
76a
))
Pig
(23)
3ea
= c
1+ 1
Taking the square root, the Lipschitz constant is therefore bounded by:
2 1 1 7
(24)
D.3. Proof of theorem 5.3
Proof. Since the gradient of (()) is Lipschitz continuous with constant L, we have the inequality:
((0+1)) < ((O,) (V 0,) + 1 0
(25)
Substituting the update rule 0,+1 O, ())..)))), we get:
2(0++)) & (().) // ( + L
2 |VL(),))
(26)
Rearranging terms yields:
L(0 s L())) - (1 L
[V((),)]
(27)
2
15

---

Cooperation of Experts: Fusing Heterogencous Information with Large Margin
To ensure decrease in the objective function, choose 1 S I For such 7,, the term v D 2
is positive, and thus:
((0(+1) " "((](()))1
(28)
Summing over = =0,1,... 1, we have:
T-1
((0o) - L(0,) 2
[V(())
(29)
1=
Since ((0) lower-bounded by L*, it follows that:
((06) - --- 7 T-1
(30)
Dividing through by 7/ gives:
7 |C(O)) 2(C(O)) L*)
(31)
T/T
Thus, there exists some & (0.1.....
, T
1} such that:
V2(0))) 2(C(On) L*))
(32)
n/
This proves the result.
E. Methodology Details
Following the approach proposed in previous works (Liu et al., 2022; Shen et al., 2024b), after constructing the cosine similarity matrix of H" we implement post-processing techniques ensure that A' exhibits the characteristics of sparsity, non-negativity, symmetry, and normalization. For the convenience of discussion, the subscript v is omitted hereinafter.
KNN for sparsity. In most applications, fully connected adjacency matrix often has limited practical significance and incurs high computational costs. Therefore, we employ the K-Nearest Neighbors (KNN) operation to sparsify the learned graph. For each node, we retain the edges with the top-K values and set the others to O, thereby obtaining the sparse adjacency matrix AtP, Note we employ efficient KNN with locality-sensitive hashing (Fatemi et al., 2021) to enhance the model's scalability. This approach avoids the resource-intensive computation and storage of explicit similarity matrices, reducing the complexity from O(N2) to B(NB), where N is the number of nodes and B is the batch size of the sparse KNN.
Symmetrization and Activation. Since real-world connections are typically bidirectional, we symmetrize the adjacency matrix. Additionally, the weight of each edge should non-negative. Given the input A these operations can be expressed as follows:
(AAp)) + (A*P)T
(33)
where a()) represents a non-linear activation implemented by the ReLU function.
Normalization. The normalized adjacency matrix with self-loops be obtained follows:
A' = (D>vm)= A. (Damm) -
(34)
where Da the degree matrix of Asym with self-loops. Subsequently, for each view, we can acquire the adjacency matrix A',, which possesses the desirable properties of sparsity, non-negativity, symmetry, and normalization.
16

---

Cooperation of Experts: Fusing Heterogeneous Information with Large Margin
F. Additional experiments
94
Accuracy (%)
92
90
ACM
Yelp
10
15
20
25
Figure Sensitivity analysis K.
Subsequent to the research on a and 7, an investigation into the sensitivity of the number of neighbors is conducted. The values of K span of (5, 10, 15, 20, 25). As depicted in Figure 5, despite the significance of K model parameter, the CoE demonstrates minimal sensitivity its variations. This finding suggests that the performance of the model remains relatively stable across broad spectrum of K values, thereby indicating its robustness in response to changes in this parameter.
17


Reviews:
[{"review_id": "y6PJH0UlPe", "reviewer": "Reviewer_LQq1", "summary": "This paper presents the CoE framework, a groundbreaking method for extracting knowledge from diverse and multi-layered networks. Its core novelty lies in a hierarchical expert coordination mechanism, where specialized low-level experts focus on capturing unique relational patterns, while high-level experts integrate insights from across these networks. The framework is further refined through the incorporation of a large margin mechanism, which optimizes expert collaboration, enhancing both robustness and generalization capabilities. Theoretical assessments validate the feasibility of the proposed method, and comprehensive experiments conducted on benchmark datasets demonstrate its superiority compared to existing techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback. We are especially grateful for the positive recognition of our novel expert coordination strategy, solid theoretical foundation, and strong empirical performance. Below, we provide responses to the specific suggestions and questions. Figures and Tables are summarized in this link: **https://anonymous.4open.science/r/ICML_rebuttal-7D0E**\n\n**Weakness & Suggestion1: Interpretability of Expert Decisions**\n\nWe appreciate your suggestion regarding the interpretability of expert contributions. In fact, we have emphasized the relative contribution of each expert using confidence scores in Figure 1(a) and 1(b) of the main paper, which demonstrate how different experts participate dynamically across semantic contexts.\n\nIn addition, to enhance interpretability more explicitly, we further compute SHAP values for each expert across different datasets. For better visualization, we normalize the SHAP values within each dataset and display them in tables for small-scale expert settings. For datasets with a larger number of experts, we provide heatmaps to show expert influence across different classes, enabling a more fine-grained interpretation of their roles.\n\nThese analyses confirm that our confidence tensor indeed reflects meaningful and diverse expert specializations. We include these SHAP-based visualizations and interpretation discussions **in the link above**.\n\n**Suggestion2 & 3 & 4: Corrections of writings**\n\nSorry for the mistake we made. We thank the reviewer for pointing out the writing issues and ambiguous expressions. We will carefully revise the identified sentences to improve clarity and precision. Specifically, we will rephrase the description of the “+” and “etc.” symbols for better readability, and correct the noted grammatical issues. These improvements will be incorporated into the revised version to enhance overall presentation quality.\n\n**Q1: Attention-based approaches**\n\nWe apologize for not clearly highlighting the attention-based baselines in the original submission. Thank you for raising this point. In fact, among our baselines, NodeFormer (Wu et al., 2022) and HAN (Wang et al., 2019) are attention-based methods. To strengthen our comparison and address this oversight, we additionally include SGFormer [1], a recent Transformer-based graph structure learning method that applies self-attention over both features and learned graph topology. We compare its performance with CoE on all datasets, and present the results **in the link above**. This updated comparison confirms that CoE consistently outperforms attention-based methods across all datasets.\n\n[1] Wu et al. \"SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations.\" NeurIPS (2023).\n\n**Q2: How does CoE handle conflicting expert opinions?**\n\nThe confidence matrix $\\Theta$ we learn represents the authority of each expert. To explain with a simple example: if expert $i$ believes the sample belongs to class $p$, the distribution vector of their judgment is $\\Theta(:,p,i)$. If another expert $j$ believes it belongs to class $q$, the distribution vector is $\\Theta(:,q,j)$. The combined opinion of the two experts is $ \\Theta(:,p,i) + \\Theta(:,q,j)$. The final classification result is then obtained through Eq. (5): $\\hat{y}_i=\\underset{j = 1...c}{argmax}\\ \\left(\\mathcal{S}\\left(\\Theta g_i\\right)\\right)_j$.\n\nEach expert provides an opinion, which we consider as a probability distribution vector. By summing the probability distribution vectors, normalizing the result, and performing an argmax operation, we can handle both similar and opposing opinions among the experts.\n\nOnce again, we sincerely appreciate your time and effort in reviewing our paper. Your constructive criticism has been invaluable in refining our work, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!", "claims_and_evidence": "Yes, they possess a convincing persuasive force.", "methods_and_evaluation_criteria": "Yes, the methodologies and assessment standards put forth should be pertinent and suitable for the problem or application in question.", "theoretical_claims": "Yes, I have checked the validity of the proofs underpinning the theoretical claims discussed.", "experimental_designs_or_analyses": "Yes, I have checked the validity of the experimental designs and analyses.", "relation_to_broader_scientific_literature": "In this work, unlike prior Mixture of Experts (MoE) models that rely on gating mechanisms (activating a subset of experts), CoE promotes cooperation instead of competition. The application of MoE to heterogeneous information fusing is new.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1) Novel Expert Coordination Strategy: (a) Unlike prior Mixture of Experts (MoE) models that rely on gating mechanisms (activating a subset of experts), CoE promotes cooperation instead of competition. (b) The introduction of high-level experts allows for cross-network knowledge fusion, enhancing model flexibility.\n2) Strong Theoretical Foundation: (a) The paper provides convexity and Lipschitz continuity proofs, ensuring convergence and stability. (b) The mutual information maximization strategy enhances the fused representation’s effectiveness. (c) The confidence tensor mechanism allows all experts to contribute dynamically, preventing over-reliance on a small subset.\n3) State-of-the-Art Performance: CoE outperforms all baselines on multiplex network and multimodal classification tasks.\n\nWeaknesses:\n1) Interpretability of Expert Decisions: While CoE optimizes expert collaboration, it does not provide an explicit mechanism to interpret the contributions of individual experts. A visualization or an explainability analysis (e.g., SHAP values) would be useful.", "other_comments_or_suggestions": "1) A visualization or an explainability analysis (e.g., SHAP values) would be useful.\n2) Remove the \"..., etc.\" as \"etc.\" already implies continuation in Line 48.\n3) “(possible with different attributes) but a different type of links” should be changed to “(possibly with different attributes) but different types of links”.\n4) \"The symbol “+\" denotes directly add up the networks\" can be changed to \"The symbol “+” denotes directly adding the networks\".", "questions_for_authors": "1) The paper primarily compares CoE with graph structure learning (GSL) and MoE models. Are there any attention-based approaches (e.g., Transformer-based graph fusion)?\n2) How does CoE handle conflicting expert opinions? Does the large-margin mechanism still work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HDxXMS71Ls", "reviewer": "Reviewer_s8EY", "summary": "This paper proposes the Cooperation of Experts (CoE) framework, which aims to address the challenge of fusing heterogeneous information in modern data analysis. The CoE framework encodes multi-typed information into unified heterogeneous multiplex networks and allows dedicated encoders, or \"experts,\" to collaborate rather than compete. The authors claim that this approach captures the intricate structures of real-world complex data and outperforms existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback. Your constructive criticism is invaluable in refining our work. Below, we give point-by-point responses to your comments. Figures and Tables are summarized in this link: **https://anonymous.4open.science/r/ICML_rebuttal-7D0E**.\n\n**Weakness 1 & Question 1: Model scalability**\n\nWe appreciate your important concern about scalability. CoE is designed to be modular and extensible, and we agree that scalability becomes critical for real-world deployment. In fact, the following characteristics of CoE are particularly favorable for large-scale data:\n\nParallelizable Expert Modules: Unlike boosting or sequential learning schemes, experts are trained in parallel, which makes the framework highly suitable for distributed training and parallel inference.\n\nExpert Fusion via Lightweight Confidence Tensor: Rather than using complex hierarchical gating or stacking mechanisms, CoE fuses the predictions of all experts through a simple yet effective linear tensor-based confidence fusion mechanism. This module has negligible memory and computational overhead, and scales linearly with the number of experts.\n\nReformulating Mutual Information (MI): network-level MI is replaced by node-level representations to reduce complexity.\n\nScalability-Aware Experiments: On the largest datasets in our experiments (e.g., MAG (113,919 nodes) and Amazon (11,944 nodes)), CoE demonstrates not only strong performance but also comparable training time to state-of-the-art baselines (see the next part).  \n\nNew Large-Scale Benchmark: We additionally include a new large-scale experiment on the DGraph dataset, with 111,310 nodes and 430k+ edges. CoE outperforms other scalable baselines such as InfoMGF and other baselines which get relatively high scores on the previous datasets. These results provide strong empirical evidence that CoE can scale to realistic large-scale graph settings. To directly show the performance, we summarize the results on MAG, Amazon and DGraph **in the link above**.\n\n**Weakness 2 & Question 2: Empirical computational complexity**\n\nWe appreciate the reviewer’s concern regarding the computational efficiency of CoE. Although CoE adopts a two-stage structure where experts are trained individually before being fused, we emphasize that:\n\n- **The number of experts is limited** and does not grow with data size. In practice, we only train limited number of experts, which makes the overall expert training cost highly manageable.\n- **Fast convergence is achieved**. This is achieved due to the design of our confidence tensor and the optimization strategy. Specifically, the confidence tensor is a lightweight linear transformation trained jointly with the final fusion stage, and it converges rapidly in practice. \n- **Linear scalability is observed** on large-scale datasets. Despite the two-stage structure, CoE exhibits empirically near-linear runtime scaling with dataset size. For example, on large-scale datasets like MAG and DGraph, CoE trains even faster than several single-encoder GSL baselines, as detailed **in the link above**. This demonstrates that the proposed architecture does not incur significant overhead compared to standard GNN-based models.\n\n**Suggestion 1: Limitation of proposed method**\n\nCurrently, CoE is designed for heterogeneous multiplex networks, the current formulation assumes static input structures. Extending the framework to accommodate more complex graph settings like dynamic or hierarchical networks remains an interesting direction for future research. Additionally, although we adopt GCN as the base encoder in this work for fairness in comparison, incorporating other architectures such as GAT or graph transformers remains future work.\n\n**Suggestion 2: \"While boosting relies on sequential training where each model builds upon the previous one\"->\"While boosting relies on sequential training, where each model builds upon the previous one\"**\n\nWe sincerely appreciate the suggestion. We will revise the corresponding phrase in the refined version.\n\n**Suggestion 3: Notation confusion**\n\nSorry for the trouble caused by our symbols, we apologize for the confusion caused by inconsistent notation. As you correctly noted, some symbols such as $\\mathcal{L}$ are overloaded in the supplementary, mainly occurred in Section D.  We will unify and revise all instances of $\\mathcal{L}$ in the final version of the paper. Additionally, we will conduct a thorough pass of the paper to ensure precise and consistent mathematical expression. \n\nYour concerns about scalability and efficiency are well-justified and have helped us greatly improve the clarity and practicality of our method. We hope the new large-scale experiments, added analysis, and revisions in the final version will adequately address your concerns. We sincerely hope this response convinces you that CoE is scalable, practical, and broadly applicable, and that our efforts in addressing your suggestions merit a higher overall recommendation.", "claims_and_evidence": "Theoretical analysis and extensive experimental on multi-relational graphs and multi-modal data verify the claims.", "methods_and_evaluation_criteria": "The proposed method makes sense to me. The experiments are conducted on two types of data, verifying the generalizability of the proposed method.", "theoretical_claims": "Theoretical justification is provided to show the convergence property of the proposed method, which makes sense to me.", "experimental_designs_or_analyses": "The experiments are conducted on two types of data and compared with different categories of methods, which is sufficient.", "relation_to_broader_scientific_literature": "The proposed CoE framework encodes multi-typed information into unified heterogeneous multiplex networks, which transcends modality and connection differences. It provides a unified approach to handle real-world complex data. To my best knowledge, this is the first application of expert learning idea to multiplex networks.", "essential_references_not_discussed": "The references are complete.", "other_strengths_and_weaknesses": "Strengths:\n1. The CoE framework introduces a unique approach to handling heterogeneous information by transcending modality and connection differences, which is a significant advancement in the field.\n2. The paper provides rigorous theoretical analyses to support the feasibility and stability of the CoE framework, adding credibility to the proposed method.\n3. The extensive experiments across diverse benchmarks demonstrate the superior performance of the CoE framework, indicating its broad applicability and effectiveness.\n\nWeaknesses\n1. While the paper verifies the effectiveness of the CoE framework, it lacks a deeper discussion on the scalability of the approach, especially for very large datasets. Training multiple experts and computing mutual information across networks introduces computational complexity.\n2. The paper could benefit from some experimental analysis in terms of computational efficiency, as this is an important aspect of practical applications.", "other_comments_or_suggestions": "1. The limitation of the proposed method should be discussed.\n2. \"While boosting relies on sequential training where each model builds upon the previous one\"->\"While boosting relies on sequential training, where each model builds upon the previous one\"\n3. Some notations are confusion. For example, L represents different meanings in the supplementary.", "questions_for_authors": "1. How does the CoE framework perform in terms of scalability when applied to very large datasets? Are there any specific challenges or limitations?\n2. Can the authors provide more details on the computational efficiency of the CoE framework in the experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otOYusZJ6M", "reviewer": "Reviewer_t4sH", "summary": "This paper proposes the Cooperation of Experts (CoE) framework, which solves the problem of multimodal heterogeneous information fusion by constructing a heterogeneous multiplexing network. The research focuses on the challenge of pattern heterogeneity across semantic spaces, designing specialized encoders as domain experts, combining large interval collaboration mechanisms and optimization strategies to achieve robust modeling and knowledge complementary extraction of complex data structures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback. Your constructive criticism is invaluable in refining our work. Below, we give point-by-point responses to your comments. \n\n**Weakness 1**\n\nWe fully agree that deeper theoretical analysis strengthens the credibility of a new learning framework. To clarify, we would like to emphasize that **Theorem 5.5 in the main paper provides a mathematical convergence analysis** of the optimization procedure. This theorem rigorously proves that the gradient norm of training objective is guaranteed to converge to zero at a sublinear rate, which establishes the optimization stability and convergence of our training process. Following your advice, we further prove a **generalization error bound** ---- a probabilistic upper bound on the 0-1 loss of the CoE classifier:\n\nLet $\\mathcal{X}\\times\\mathcal{Y}$ be the input-label space with $|\\mathcal{Y}|=C$ classes. We have an i.i.d. training sample $S=\\{(x_i,y_i)\\}_{i=1}^n$.  \n\nA CoE classifier $f$ produces a probability vector over $C$ classes, denoted $f(x)=(f_1(x),\\dots,f_C(x))^\\top$. We define the margin of $f$ at $(x,y)$ as\n$$\n\\gamma_f(x,y):=f_y(x)- \\max_{y'\\neq y} f_{y'}(x),\n$$\na large positive $\\gamma_f(x,y)$ implies a strong preference for the correct class $y$. The usual 0-1 loss is $\\ell_{\\mathrm{0\\text{-}1}}(f;x,y):=\\mathbb{I} [\\arg\\max_{c}f_c(x)\\neq y].$ We also define $\\ell_{\\gamma}^{\\mathrm{0\\text{-}1}}(f;x,y):=\\mathbb{I} [\\gamma_f(x,y)\\leq0]$ and the ramp loss:\n\n$$\n\\ell_{\\gamma}(f;x,y) :=\n\\begin{cases}\n  0, & \\gamma_f(x,y)\\ge \\gamma,\\\\\\\\\n  1-\\dfrac{\\gamma_f(x,y)}{\\gamma}, & 0<\\gamma_f(x,y)<\\gamma,\\\\\\\\\n  1, &\\gamma_f(x,y)\\le0.\n\\end{cases}\n$$\nOne has $\\ell_{\\mathrm{0\\text{-}1}}(f;x,y)\\le \\ell_{\\gamma}^{\\mathrm{0\\text{-}1}}(f;x,y) \\le \\ell_{\\gamma}(f;x,y).$\n\nLet $\\ell_\\gamma\\circ\\mathcal{F}$ be the set of ramp-loss functions induced by a hypothesis class $\\mathcal{F}$, where each $f\\in\\mathcal{F}$ is a CoE classifier. Then for any $\\delta>0$, with probability at least $1-\\delta$ over an i.i.d. sample $S=\\{(x_i,y_i)\\}_{i=1}^n$, the following holds for all $f\\in\\mathcal{F}$:\n\n$$\n\\mathbb{E}[\\ell _{0-1}(f)]\\le \\mathbb{E} _{(x,y)\\sim\\mathcal{D}}[\\ell _\\gamma^{0-1}(f;x,y)]\\le \\frac{1}{n}\\sum _{i=1}^n\\ell _\\gamma(f;x _i,y _i)+\\frac{2}{\\gamma}\\mathfrak{R} _n(\\mathcal{F})+3\\sqrt{\\frac{log(\\frac{2}{\\delta})}{2n}},\n$$\n\nwhere $\\mathfrak{R}_n(\\mathcal{F})$ is the Rademacher complexity of the CoE margin-function class. Due to space limitations, the detailed proof will be included in the final version and is omitted here.\n\nIn CoE framework, we have $k$ experts $E_1,\\dots,E_k$, each outputs a probability vector $E_j(x)\\in \\mathbb{R}^C$. Besides, we have a confidence tensor $\\Theta$ and we form $g(x)=[E_1(x)^\\top,\\dots,E_k(x)^\\top]^\\top$, then $f(x)=\\mathrm{softmax}\\bigl(\\Theta g(x)\\bigr).$ And the margin is $\\gamma _f(x,y)=[\\Theta g(x)] _y-\\max _{y'\\neq y}[\\Theta g(x)] _{y'}$.\n\nWe assume $||\\Theta|| _F\\le B _\\Theta$ and $||E _j(x)|| _2\\le G _e ,\\forall j,x$. Hence $||g(x)|| _2\\le \\sqrt{k}G _e$. Let $\\mathcal{F} _\\Theta$ be the set of CoE margin functions $\\gamma_f$. \n\nThen $\\mathfrak{R} _n(\\mathcal{F} _\\Theta)\\le C _\\mathrm{MC} \\frac{B _\\Theta G _e \\sqrt{k}}{\\sqrt{n}}$, where $C _\\mathrm{MC}$ is a constant on the order of $\\sqrt{ln(C)}$, reflecting the multi-class max operation. \n\nWith probability at least $1-\\delta$, all $f \\in \\mathcal{F}_ \\Theta$ satisfy\n\n$$\n\\mathbb{E}\\bigl[\\ell_{\\mathrm{0\\text{-}1}}(f)\\bigr]\\le\\frac1n \\sum_{i=1}^n \\ell_{\\gamma}\\bigl(f;x_i,y_i\\bigr)+\\frac{2B_\\Theta G_e\\sqrt{k}}{\\gamma \\sqrt{n}}+3\\sqrt{\\frac{\\log(\\tfrac2\\delta)}{2n}}.\n$$\n\nIt shows that ensuring a large margin $\\gamma$ and controlling the norms $B_\\Theta$ (confidence-tensor magnitude) and $G_e$ (expert-output scale) leads to a generalization guarantee. Increasing the number of experts $k$ has a $\\sqrt{k}$ impact, illustrating the trade-off between model capacity and margin-based guarantees. Due to CoE mechanism limits the number of $k$ and $B_\\Theta$,  while $G_e$ is fixed, thus our model has remarkable generalization ability.\n\n**Weakness 2**\n\nWe would like to clarify that our main experiments already include evaluations on **large-scale network datasets** such as Amazon and MAG in Table 1, all of which involve diverse views and heterogeneous information types. Besides, we conduct experiments on four **multi-modal datasets** in Table 2. To further strengthen our claims, we conduct robustness experiments on large-scale datasets and main experiment on an additional large-scale dataset DGraph, results are concluded in **https://anonymous.4open.science/r/ICML_rebuttal-7D0E**.\n\nOnce again, we appreciate your valuable suggestions to help us improve our work. We will include the missing citations in the revised version and ensure all relevant work be properly cited. With the newly added theoretical analysis and experiments, we hope your concerns have been addressed. We sincerely hope this response and the revised improvements can earn your stronger recommendation.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "* Inspired by the MoE (Mixture of Experts) framework, but innovatively introducing a learnable confidence tensor, it solves the limitation of expert competition rather than cooperation in traditional MoE (such as the unactivated expert utilization problem proposed by Shi et al., 2024a).\n* By maximizing the margin of the predicted results, the theory of ensemble learning (such as Boosting's weight adjustment) is combined with graph neural networks, distinguishing it from static expert weight allocation methods (such as Liu et al., 2022's fixed gating mechanism).", "essential_references_not_discussed": "> Common ensemble methods include bagging (Zhou & Tan, 2024) and boosting (He et al., 2024). \n\nIn recent years, deep forests [Zhou and Feng, NSR 2019] have emerged in the field of ensemble learning, combining bagging and boosting techniques. Especially the theoretical analysis of its large margin property [Lyu et al., NeurIPS 2019] is highly relevant to this paper.\n\n1. Zhou, Z. H., & Feng, J. (2019). Deep forest. National Science Review, 6(1), 74-86.\n2. Lyu, S. H., Yang, L., & Zhou, Z. H. (2019). A refined margin distribution analysis for forest representation learning. Advances in Neural Information Processing Systems, 32.", "other_strengths_and_weaknesses": "Strengths:\n* For the first time, a framework emphasizing expert cooperation rather than competition was proposed, breaking through the paradigm limitations of expert competition in traditional MOE models. Introducing the large margin optimization mechanism into expert collaboration scenarios provides a new perspective for model optimization (the theoretical innovation of this mechanism is validated by comparing it with traditional RF/WRF methods).\n* The experimental design covers 0-90% of the network structure disturbance intensity and systematically verifies the robustness of the method in extreme attack scenarios, making up for the shortcomings of existing research on high-intensity disturbance testing.\n\nWeaknesses:\n* Theoretical depth limitation: No mathematical convergence proof or generalization error boundary analysis provided for the large margin mechanism.\n* Doubtful generalization of scenarios: The current experiment is only based on the ACM dataset and has not validated its effectiveness on larger scale graph data or cross modal scenarios.", "other_comments_or_suggestions": "NAN.", "questions_for_authors": "NAN.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ahp0hKbWi2", "reviewer": "Reviewer_nSKz", "summary": "The authors propose the Cooperation of Experts (CoE) framework to fuse heterogeneous information in multiplex networks. Authors design a two-level expert system where low-level experts focus on individual network layers while high-level experts capture cross-network relationships. The framework employs a large margin mechanism to optimize the collaboration among experts, ensuring that each expert's specialized knowledge is effectively utilized. The experiments demonstrate the superiority of their approach over state-of-the-art methods in both multi-relational and multi-modal tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and constructive feedback. We greatly appreciate your recognition of our motivation, expert coordination design, and robustness to structural perturbations. Below we provide detailed responses to your suggestions. Figures and Tables are summarized in this link: **https://anonymous.4open.science/r/ICML_rebuttal-7D0E**.\n\n**Weakness & Suggestions: Writing and clarity**\n\nThank you for pointing out the writing issues. We agree that improving readability is important for accessibility and impact. In the final version, we will:\n\n1. Thoroughly proofread the entire manuscript for grammatical correctness and clarity.\n\n2. Refine awkward phrasings, shorten overly long sentences, and ensure each technical statement is precisely and clearly expressed.\n3. Review the supplementary materials and figure captions to maintain consistency and readability.\n\nWe apologize for the writing quality issues in our manuscript, which may have caused some difficulty in following the arguments. Once again, we sincerely appreciate your careful reading, and we have made efforts to clarify the text in the revised version. We are confident that these improvements will significantly enhance the presentation quality of the paper.\n\n**Question1: Numbers of experts**\n\nThanks for the concern, we appreciate the opportunity to clarify this point. In Section 4.2, we emphasize that the number of experts is fixed to $\\frac{V(V+1)}{2}+1$, where $V$ is the number of views. This is mainly because CoE train $V$ low-level experts on each network view and $C^{2}_V= \\frac{V(V-1)}{2}$ high-level experts. \n\nAdditionally, we pointed out that an extra high-level expert is trained on $G_{tot}$. Thus, we have $V+ \\frac{V(V-1)}{2}+1=\\frac{V(V+1)}{2}+1$ experts in total. It is remarkable that the number of experts is limited, thus the overall expert training cost highly manageable. We analyze the model's efficiency **in the link above**.\n\n**Question 2: Loss convergence plot in the supplementary**\n\nFollowing your advice, we provide the convergence plots **in the link above**, on common-scale and large-scale datasets respectively. \n\nAcross all datasets, the training loss decreases smoothly and steadily, without oscillation or divergence, reflecting stable optimization dynamics.  A significant loss drop is observed within the first 100–200 epochs, indicating that the confidence tensor and expert fusion strategy are effective at capturing informative gradients early in training. The convergence behavior is consistent across datasets of different scales and domains, which shows that our training process is not sensitive to specific data distributions or graph modalities. These properties collectively confirm that CoE is not only theoretically convergent (as shown in Theorem 5.5), but also empirically stable and efficient in practice.\n\nOnce again, we sincerely appreciate your thoughtful comments and are encouraged by your support. Your feedback has helped us further improve the clarity and completeness of the paper. We hope the final version will fully meet your expectations, and we are more than happy to add clarifications to address any additional recommendations and reviews from you!", "claims_and_evidence": "Clear and convincing evidence are given to support the claims.", "methods_and_evaluation_criteria": "The proposed method is well-suited for the heterogeneous information fusion problem. The datasets are representative of heterogeneous data, demonstrating the broad applicability of the method.", "theoretical_claims": "I reviewed the proofs and they appear to be correct.", "experimental_designs_or_analyses": "I carefully reviewed the experimental setup and analyses to assess their rigor.", "relation_to_broader_scientific_literature": "1. The paper introduces a novel framework that emphasizes expert cooperation rather than competition, which is a significant departure from traditional Mixture of Experts (MoE) approaches. This is a fresh perspective in the field of graph neural networks (GNNs) and multiplex network learning.\n\n2. The large margin mechanism is an innovative optimization strategy that ensures robust expert collaboration, leading to improved predictive performance.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Pros：\n1.\tThe motivation is clearly presented by Figure 1. The observed phenomenon is interesting and important. \n2.\tThe expert perspective for multiplex network learning is interesting and novel. \n3.\tCoE is highly resilient to structural perturbations, which is crucial for real-world applications where data may be noisy or incomplete.\n\nCons：\nThe writing quality of the paper could be improved for better readability and clarity. Several grammatical errors, awkward phrasings, and unclear sentences make it difficult to follow the arguments.", "other_comments_or_suggestions": "1.\"The cross-entropy loss Lcls(Z,Y ) is the upper bound of −I(G;Y ), where Z denotes the node representations of all nodes in network G.\"\nCorrection: \"The cross-entropy loss Lcls(Z,Y) serves as an upper bound for −I(G;Y), where Z represents the node embeddings of all nodes in network G.\" \n\n2. \"We select two supervised structure-fixed GNNs—GCN (Kipf & Welling, 2016) and HAN (Wang et al., 2019)—as well as six supervised GSL methods.\"\nCorrection: \"We select two supervised, structure-fixed GNNs—GCN (Kipf & Welling, 2016) and HAN (Wang et al., 2019)—along with six supervised GSL methods.\" \n\n3.\"L(Θgi) is a convex function with respect to (Θgi).\"\nCorrection: \" L(Θgi) is a convex function with respect to Θgi.\"", "questions_for_authors": "1.\tThe number of experts is given in Line 187. How do you get it? It should be explained in more detail.\n\n2.\tThe supplementary material suggests that the authors add a loss convergence plot.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a groundbreaking method for extracting knowledge from diverse and multi-layered networks. All reviewers give positive scores. The motivation is clear. The writing is smooth. The experiment is sufficient.
Decision: Accept (poster)