Bridging Gaps: Federated Multi-View Clustering in Heterogeneous Hybrid Views
Authors: Xinyue Chen
Yazhou Ren
Jie Xu
Fangfei Lin
Xiaorong Pu
Yang Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-view learning
Clustering
Federated learning
Primary area: machine_vision

Abstract: Recently, federated multi-view clustering (FedMVC) has emerged to explore cluster structures in multi-view data distributed on multiple clients. Many existing approaches tend to assume that clients are isomorphic and all of them belong to either single-view clients or multi-view clients. While these methods have succeeded, they may encounter challenges in practical FedMVC scenarios involving heterogeneous hybrid views, where a mixture of single-view and multi-view clients exhibit varying degrees of heterogeneity. In this paper, we propose a novel FedMVC framework, which concurrently addresses two challenges associated with heterogeneous hybrid views, i.e., client gap and view gap. To address the client gap, we design a local-synergistic contrastive learning approach that helps single-view clients and multi-view clients achieve consistency for mitigating heterogeneity among all clients. To address the view gap, we develop a global-specific weighting aggregation method, which encourages global models to learn complementary features from hybrid views. The interplay between local-synergistic contrastive learning and global-specific weighting aggregation mutually enhances the exploration of the data cluster structures distributed on multiple clients. Theoretical analysis and extensive experiments demonstrate that our method can handle the heterogeneous hybrid views in FedMVC and outperforms state-of-the-art methods.

Bridging Gaps: Federated Multi- Clustering in Heterogeneous Hybrid Views
1.2,+ Xinyue Chen' Yazhou R.n Jie Xu
Fangfei Lin' Xiaorong Pu Yang Yang'
School of Computer Science and Engineering, University of Electronic Science and Technology of China, China; 2henzzen Institute for Advanced Study, University of Electronic Science and Technology of China, China
Abstract
Recently, federated multi-view clustering (FedMVC) has emerged to explore clus- ter structures multi-view data distributed on multiple clients. Many existing approaches tend to assume that clients are isomorphic and all of them belong either single-view clients or multi-view clients. While these methods have suc- ceeded, they may encounter challenges in practical FedMVC scenarios involving heterogeneous hybrid views, where a mixture of single-view and multi-view clients exhibit varying degrees of heterogeneity. In this paper, we propose novel Fed- MVC framework, which concurrently addresses two challenges associated with heterogeneous hybrid views, i.e., client gap and view gap. To address the client gap, we design a local-synergistic contrastive learning approach that helps single-view clients and multi-view clients achieve consistency for mitigating heterogeneity among all clients. To address the view gap, we develop global-specine weighting aggregation method, which encourages global models learn complementary features from hybrid views. The interplay between local-synergistic contrastive learning and global-specific weighting aggregation mutually enhances the explo- ration of the data cluster structures distributed on multiple clients. Theoretical analysis and extensive experiments demonstrate that our method can handle the heterogeneous hybrid views in FedMVC and outperforms state-of-the-art methods. The code available at hettttt://// htpp:://githan /ttp//gtttttttt/e/ com/ /tt.//MartaiiMMSSS
Introduction
Recent advancements in sensors and the internet have allowed many distributed clients to collect unlabeled data from multiple views/modalities([) 17 25 Utilizing these unlabeled data while considering the need for data privacy among clients has given rise to an emerging field of federated multi-view clustering (FedMVC) [8), which enables multiple clients to collaboratively train consistent clustering models without exposing private data. The clustering models across distributed clients can be applied in many applications (e.g., recommendation [1] and medicine (5)) and thus FedMVC
attracts increasing research interest [4 26 35).
Existing FedMVC methods tend to assume that clients are isomorphic and all of them belong to either single-view clients or multi-view clients. For instance, FedDMVC X assumes that a dataset with V views is distributed across V single-view clients, each having the same set of samples. FFPM 14 assumes that the data are distributed among multiple multi-view clients, with each client having V views and non-overlapping samples among clients. Despite the achieved success, they may encounter challenges when handling some practical FedMVC scenarios involving heterogencous
"Corresponding author @yazhou.ren@ uestc.edu.cn).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

hybrid views, where different clients are heterogeneous such that single-view clients and multi-view clients are hybrid.
This scenario is prevalent in real world situations 6 for example, hospitals in metropolitan
areas employ CT, X-ray, and EHR for disease detection, whereas remote areas usually rely single detection method. Similarly, smartphones can simultancously capture audio and images, but recording devices are limited to collecting audio data only.
The presence of heterogeneous hybrid views might limit the applicability of previous FedMVC methods. We decompose these challenges into two issues. (a) Client gap. Multi-view clients collect multiple views that have the opportunity to learn comprehensive cluster partitions by leveraging multi-view information. Conversely, single-view clients only own single views and easily obtain biased cluster partitions if the single view only contains marginal information. (b) View gap. Hybrid views collected by different clients have quality differences (e.g., images might contain richer visual information than texts, but texts could have semantic information), and extracting complementary information from different views across all clients is not trivial.
In this paper, we introduce a novel FedMVC method, namely Federated Multi-view Clustering via Synergistic Contrast (FMCSC), which can simultaneously leverage the single-view and multi-view data across heterogeneous clients to mine clustering structures from hybrid views. Figure I shows an overview of our FMCSC framework. First, note that lacking unified information supervision, naive aggregation of local models may lead to model misalignment. Therefore, we propose cross- client consensus pre-training to align the local models on all clients to avoid their misalignment. Second, to address the client gap, we design local-synergistic contrastive learning that helps to mitigate the heterogeneity between single-view clients and multi-view clients. In particular, leverage feature-level and model-level contrastive learning to align multi-view clients and single- view clients, respectively. Third, tackle the view gap, we develop the global-specific weighting aggregation which encourages global models to learn robust features from hybrid views, further exploring complementary cluster structures. The local-synergistic contrastive learning and global- specific weighting aggregation promote each other to explore the data cluster structures distributed on multiple clients. Overall, FMCSC effectively facilitates all clients in bridging the client gap and view gap within the heterogeneous hybrid views through theoretical and experimental analysis.
Our main contributions are follows:
We propose a novel FedMVC method that can handle the heterogeneous hybrid views and explain the success mechanism of the proposed method through theoretical analysis from the perspective of bridging client and view gaps.
We design local-synergistic contrastive learning and global-specific weighting aggregation, using mutual information as a bridge connect local and global models, together to explore the cluster structures in multi-view data distributed on different clients.
Theoretical and experimental analyses verify the effectiveness of FMCSC, which shows excellent clustering performance under various federated learning scenarios.
2
Related Work
Federated multi-view clustering (FedMVC) has emerged to explore cluster structures in multi-view data distributed on multiple clients. Existing FedMVC methods can be classified into two categories based the partitioning of multi-view data among clients. (1) Vertical FedMVC assumes that dataset with V views is distributed across V single-view clients, each having the same set of samples. Robust federated multi-view learning (FedMVL) [16] addresses high communication costs, fault tolerance, and stragglers. Federated deep multi-view clustering (FedDMVC) [8] focuses on addressing the challenges of feature heterogeneity and incompleteness. Existing vertical FedMVC methods still rely on the idealistic assumption that different views of the same sample can be aligned across clients, which warrants further investigation. (2) Horizontal FedMVC assumes that the data are distributed among multiple multi-view clients, with each client having V views and non-overlapping samples among clients. Federated multi-view fuzzy c-means consensus prototypes clustering (FedMVFPC) 14 utilizes federated learning mechanisms to perform fuzzy c-means clustering on multi- vietw data. Horizontal federated multi-view learning (H-FedMV) 5 aims to improve the local disease prediction performance by sharing training models among clients. Although existing approaches have been

---

Multi-view clients
Server
(s ... f (sw
A (inl
4 Giwill
C
Mmn
21
foor
-2"
Distribute models
Sa (m)
fi (swl) fx (1wh)
(sw))
f (swe))
(a) Cross-Client Consensus Pre-training
(b) Locul-Synergistic
Aggregate
Heterngetcoos global models
((w!) 16
Sp
(1w))
Global- Global-Specific Weighting Aggregation
Single-view clients
The framework of FMCSC. Initially, each client conducts cross-client consensus pre- training to alleviate model misalignment (Section 3.2). Then, all clients begin training using the designed local-synergistic contrast (Section 3.3) and upload their local models to the server. The server performs global-specific weighting aggregation and distributes multiple heterogeneous global models all clients (Section 3.4). Finally, leveraging global models received from the server, clients discover complementary cluster structures across all clients.
Figure 1:
successful, they may encounter challenges in practical FedMVC scenarios with heterogeneous hybrid views. Specifically, heterogeneity refers to the differences among clients, where single-view and multi-view clients coexist. The hybrid views indicate the uncertainty the number and quality of views involved in training. Our proposed FMCSC is a variant of horizontal FedMVC, and can handle such scenarios by bridging the client gap and the view gap among clients.
3
Methodology
The key goal of FMCSC is to bridge client and view gaps. On the one hand, multi-view clients have the opportunity to learn comprehensive cluster partitions by leveraging multi-view information. Single- view clients aim to bridge the client gap between themselves and multi-view clients, thereby avoiding obtaining biased clustering partitions. On the other hand, considering the inherent discrepancies in data quality among different views, our objective is to extract complementary information from hybrid views across all clients.
3.1 Problem Definition
We consider a heterogeneous federated learning setting where M multi-view clients and S single-view clients collaborate to train and mine complementary clustering structures using multiple heteroge- neous global models {JG w.......,, fy (:: w Sg (5 w)). Here, fğ (:; w") represents the global model capable of handling the v-th view type, and fg (+: w) represents the global model capable of handling multi-view data. Each single-view client p [S] has its private dataset Sp {xx !SS. where x represents the i-th sample of the p-th single-view client collected from the v-th view type. adopts a small model fp (:; (:)) De + ddd based on its local view types. The multi-view client
m E [M] has its local dataset Mm = {x! x x x [MM.l. where (x! (x,.x? xY) represents the i-th sample of the mith multi-view client collected from model fm (:3 w..) REY, De - R&d based on its local data. For simplicity, we assume that the output features of all models have the same dimension d.
5=1 V. different view types. trains small

---

3.2 Cross-Client Consensus Pre-training
Multi-view datasets often contain redundancy and random noise. Current mainstream methods employ self-supervised autoencoder models, such as autoencoder (AE) [13] and variational autoencoder (VAE) [20] to learn high-level features from raw data. In FMCSC, we employ an encoder-decoder pair, denoted E (-) and (+) with learnable parameters ø and 0", for each view in each client.
For multi-view client m. we define z - (xf) Rd as the d dg-dimensional latent feature of
the i-th sample, and the output of the autoencoder is x = D2 (z") E RDD We calculate the reconstruction loss between the input x and the output x for all samples in this client. Additionally,
the local model of this client consists of V encoder-decoder pairs, which can be pre-trained by minimizing the reconstruction objective:
Mm
|
Dÿ- (E (xx)
(I)
M m
Similarly, for single-view client p that contains type of view data locally, is sufficient to construct an encoder-decoder pair. Pre-training can be performed using the same objective as in Eq. (
1spl Spl
1
C =
x
Dz (E (x
(2)
Key Observations: federated learning, diverse local data distributions frequently lead to model drift, causing slow and unstable convergence [19 47]. In FMCSC, single-view clients have never en- countered data from other view types, thus intensifying the issue model drift due to heterogeneous hybrid views. Moreover, the absence of uniformly labeled data across all clients allows the recon- struction objective of autoencoders to optimize from multiple different directions. In feature space, this problem manifests itself as angular deviations among features [48), and from the perspective of model aggregation, it manifests itself model misalignment. In other words, direct aggregation of models can blur the feature distinctions captured by local models, leading to inseparability among features, as shown in Figure 3
A direct strategy to alleviate model misalignment is through alignment. Based on this naive idea, we propose that the multi-vies client that finishes training first should distribute its network pa- rameters { [B (X ()} and (D), (.),... Djv (.)} to the remaining clients. Each client then performs pre- trainiain based on this model, thereby alleviating the model misalignment caused by unsupervised training. Notably, as pre- pre-training solely involves training the autoencoder, the construction of local models is inherently dependent on the view type. Therefore, single-view clients can still refer the network parameters of multi-view clients. This process facilitates consensus pre-training among the clients, ultimately leading to the uploading of pre-trained model parameters to the server. The server initializes global models based the models uploaded by clients.
3.3 Lynel-SSeerristie Contrast
During pre- pre-training. the features extracted through the reconstruction objective usually contain both common semantics and view-private information. The latter is often meaningless even misleading, leading to poor clustering effectiveness. To mitigate the adverse effects of view-private information, each client also needs design consistent objective during training learn common semanties.
For multi view client m, possesses information from multiple views. Inspired by many previ- ous works of MVC [40 42 44 45), we employ feature contrastive learning to achieve consis- tency objectives. Considering the conflict between consistency and reconstruction objectives, we opt to operate in distinct feature spaces. We refer to the features obtained through the autoen- [M..l
coders { (z) 2....., 2 i=1 as low-level features. Moreover, we stack V non-linear mappings [7C' (z): 1)) ....) ..V (z"
VV) to obtain high-level features { (bb, h,....,h," ,.,)), M Addi- tionally, a non-linear mapping 7 (Z; V) EY d. + nd is constructed by:
H = H [Z; v H. [z' z ..... z v),
(3)

---

where (h,) H E IR. Ma/xx and Z E R RR..M d., We aim to preserve the representative
capacity of low-level features to prevent model collapse while learning the common semantics H among views in the high-level feature space.
In the high-level feature space, the common semantics H are learned from all views and should be very similar to the common semantics learned from individual views. Based on this, we define (h,, | v=l..... as V positive feature pairs, and the remaining {h, h2j j negative feature pairs. Then, we use cosine similarity to measure the similarity of feature pairs:
== are V[I.mll - 1)
sim (h,, hj) = (hi. hy) [b byj
where (:.)) is dot product operator. We introduce the temperature parameter Tm to moderate the effect of similarity. Subsequently, the feature contrastive loss is formulated as:
V
[Mmll
e ///
C
log
(5)
Mm
Σ jzts e sim (1).))//m
Moreover, multi-view clients aim assist single-view clients in bridging client gaps and discarding view-private information detrimental to clustering. They optimize multiple heterogeneous global models using local data, ensuring that even the global model designed for single-view processing ac- quires generalized common semantics. Such common semantics are also advantageous for uncovering complementary clustering structures across clients. Concretely,
Σ 1. (s.) Jm (s w
min (we, V
where J (: w represents the local model that can handle the v-th type of view, which is initialized by the global model f (: w"). fm (:; (m) represents the local model of multi-view client m, where multi-view clients possess data of all view types.
For single-view client p, where each sample only has a single view, we design a model contrastive learning achieve consistency objectives. Specifically, client p contains data of the v-th view type {x: 1Spl with its local model as fp (=ww). To explore common semantics, we adopt the same approach multi-view clients by constructing two non-linear mappings 7. (Z"; v²) and H (Z; V)) to obtain high-level features H and common semantics H. The global modelf, (:5 w") after aggregation further enhances its ability learn generalized common semantics. To encourage the local model of client to approach the more generalized global model, we formulate the model contrastive loss:
Spl
sim (im(h,,h))/
C
log
Spl
sim hb by //T e sim (riimhh,c (h,,,))
Spl where (h?),r] represents the output of the local data after being processed by the global model, and Tp denotes the temperature parameter. The significance of Eq. ? lies treating {h,, (n) as negative pairs. This allows the local model of client p to converge towards the global model while amplifying the differences between the reconstruction and consistency objectives in the local model.
positive pairs and [h, z!" z²" |Syl !5pl
During training, the respective total losses for multi-view client m and single-view client are:
L = L. + L. = CT + CP.
In the optimization of FMCSC, LT"h and LT; utilized reconstruction losses learn representations for each view individually. Meanwhile, L and L' are employed to discover common semantics across views, facilitating the exploration of complementary clustering structures across clients.
3.4 Global-Specifie Weighting Aggregation
To bridge the view gap and aggregate heterogeneous models, we design a weighted specific aggrega- tion strategy on the server, yielding multiple heterogeneous global models.
5

---

Theorem 1. Assume 6m. E (0.1) such that p (b, h.) > 8m i = 1.2,... Mmmml and
p (h? h.) pz h.) > 6p. 1.2. SSp hold. The following inequality establishes the relationship between the consistency objectives and the mutual information of the multi-view client m and single-view client p. respectively:
Σ (H, 2 V log Mm 6m
(9)
T THH (H, -
Proofs of all the theorems in this paper are provided in Appendix Cdue to space limit. Theorem) | Eq. (5) and Eq. 7 indicate that minimizing contrastive loss L and L are equal to maximizing mutual information. Such connection has also been discussed in [4 51]
Through theoretical analysis, use mutual information Σ (H, and (H. H") (H, Zo)
as weights to evaluate the quality of the models from multi-view clients and single-view clients, respectively. A higher level of mutual information indicates better model quality, leading to higher weights during aggregation.
Considering the heterogeneity of the models, we aggregate client models with the same architecture on the server, referred to as specific aggregation. Specifically,
M
(:
OmJm (:: (m),
m==1
(10)
M
S
(:ww)) f =
w + a (::
> Eapt (:w²)) p=1
where v 1,2..... Om and Opp represent the weights for model aggregation of multí-view client m and single-view client p respectively. In this scenario, there are total of M multi-view clients and S single-view clients, resulting in (V + 1) heterogeneous global models.
For global models that handle multiple view types simultaneously, the expected risk is defined as (f') and optimized by minimizing the empirical risk (f) Similarly, for global models dealing with processing single view type, such as the v-th view type, the expected and empirical risks are
defined as Ls- (f) and Ls- (f) respectively. Inspired by previous works (23 37 the generalization bound of clustering approaches, we obtain the following theorem by analyzing the generalization bound of the proposed FMCSC method.
Theorem 2. Suppose that for any x E X and ff E F, there exists D < oo such that ||x|| fl (x)Il 11.f" (x)]l Ilf% (x)l 1158 (x)]l E [0, DJ hold. With probability 8 for any F.
the following inequality holds
12V D
log 1 8
(M) sLM())
+ +VV2
V
V M
2M [M
10D2
log
Ls (f) st.s- (f) +
+ +D2
(11)
SSe Spl
2.5 |Spl
4
2cM [Mml d
+ +log - + dr (D.. Nos
4
d
M Mm
where M the number of multi-view participating clients. S" the number of single- clients with v-th view type that participated in the training, [Mml and SSp) are the mumber of samples in each multi-viev client and single-view client, respectively. dy (D., D) measures the difference between data from the w-th view distribution D.,, and the milti-view data distribution D.
Three key implications can be derived from Theorem 2 i) For global models capable of handling multi-view data, having more samples from multi-view clients, such as increasing the number of samples per client or adding multi-view participating clients, contributes to the improvement of

---

generalization performance. ii) For global models dealing with single-view data, having more samples from both multi-view and single-view clients enhances their generalization performance. which is a reflection of multi-view clients helping single-view clients to bridge the client gap. iii) Although the view gap is mitigated, the high dissimilarity among views still leads high distribution divergence dz (D,:D) which impairs the quality of the global model.
Finally, each client applies the K-means [30 on the common semantics H obtained from the corresponding global model to calculate the cluster centroids and obtain their local clustering results. For example, for multi-view client m, letting {e
K denote the K cluster centroids, we have:
1m ΣEb. - -c
min CL
(12)
The clustering result for the i-th sample is y, = arg min, h: - c,l1². By concatenating clustering results from all clients, we obtain the overall clustering results.
4
Experiments
4.1
Experimental Settings
Our experiments are carried out on four multi-view datasets. Specifically, MNIST-USPS comprises 5000 samples collected from two handwritten digital image datasets, which are considered as two views. BDGP 4 consists of 2500 samples across 5 drosophila categories, with each sample having textual and visual views. Multi-Fashion [4] contains images from 10 categories, where we treat three different styles of one object as three views, resulting in 10000 samples. NUSWIDE ( consists of 5000 samples obtained from web images with views. Considering the sample quantities, we allocate BDGP to 12 clients, MNIST-USPS and NUSWIDE to 24 clients respectively, and Multi-Fashion to 48 clients to simulate the federated learning settings.
Datasets.
34
Comparison Methods.
We select 9 state-of-the-art methods, including HCP-IMSC (24), IMVC-CBG
DSIMVC (37) LSIMVC [27] Prolmp 222] JPLTD [29],, CPSPAN [8]. FedDMVC [8] and Among them, apart from FedDMVC and FCUIF, which are FedMVC methods. all the other comparison methods are centralized incomplete multi-view clustering methods. To ensure fair comparisons, we concatenate the data distributed among the clients and use them as the input for centralized methods. Among these, the data from multi-view clients can be regarded as complete data, while the data from single-view clients can be considered as missing data.
39
FCUIF 36
Implementation Details. For encoder- pair, the encoder structure is Input- FC500 FC500 - FC2000 - Fc and the decoder is symmetric with the encoder. Also, we set temperature parameters 7m = Tp 0.5 and use the batch size of 256. The output dimension d set to 20 for all local and global models and communication rounds R is set to 5. All experiments in the paper involving FMCSC that are mentioned are performed when M/S- 1:1.
4.2 Results and Analysis
Clustering Results. Table presents a quantitative comparison under various heterogeneous hybrid view scenarios. Each experiment is independently conducted five times, reporting average values and standard deviations. We construct different scenarios by adjusting the ratio multi-view clients to single-view clients. Remarkably, FMCSC achieves exceptional performance in various heterogeneous hybrid view scenarios, surpassing recent methods. This indicates our ability to achieve satisfactory clustering performance while preserving data privacy. Furthermore, with the increasing proportion of single-viww clients, all methods exhibit varying degrees of performance decline, aligning with common expectations. Even when the number of single-view clients is twice that of multi-view clients, FMCSC still achieves superior clustering performance, which is encouraging.
Ablation Studies. Components A and D represent the consensus pre-training process and weighted aggregation, respectively. Component B indicates that multi-view clients bring the global models closer as Eq. (6) Component C indicates that model comparison in single-view clients as Eq. (7). Table 2 shows that the impact of component A on clustering performance dependent on the dataset. Additionally, note that the results in Item-l I are obtained after running four times
7

---

Table 1: Clustering results (meanistd%) of all methods four datasets. The best and second best results are denoted in bold and underline.
Mulii /ingle clients
M/S=21
M75= 1:1
M/S = 1:2
Evalustion metrics
ACC
ACC
NMI
ARI
ACC
NMI
ARI
TCCF (2022)
24
80.230.0
74.820.0
69.820.1
79.080.2
71.600.2
66.920.2
76,230.1
71.020.1
63.620.1
IMVC-CRG (2022)
46.6+0.1
40.3+0.2
22.2+0.4
38.9+0.1
35.3+0.3
14.9+0.2
37.3+0.6
31,7+0.4
10.6=0.2
DSIMVC (2022)
17
55.140.1
27.6a0.1
25.040.1
54.540.1 1
26. 740.1
24.4
54.1+0.2
26.6:4.22
24.0=0.3
MNIST-USPS
1.SIVVC (2022)
59.3+0.2
55.2+0.9
38.2+1.7
52.740.4
46.5+0.2
25.8+0.5
41,8+0.4
37.1
14.240.5
Prolmp (2023)
22
91.2+009
84.4+1.0
80.8
87.340.6
77.940.7
73.1+10
84.8+0.9
75.940.9
66.62188
JPL (2023)
40.740. 1
22.6+0.1
17.3+0.0
40.0+0.2
19.8+0.1
15.2+0.1
32.3+0.1
13.7+0.1
10.1
CPSPAN (2023)
[8
79.5 59.5+22.
77.3+2.0
70.7+2.2
76.5+2.7
73.742.1
66, 6...6+3.0
74,6+3.6
74.5+3,3
64.5+3.5
FedDMVC (2023)
11.9
81.9+0.9
73.7+1.4
69.9+1.5
72.5+2.9
58,9+2.6
63.1+0.4
62.6+0.5
48,4+0.3
FCUIF (2024)
36
85.3+0.3
83.2=0.3
75.7+0.5
72. 84t
70.3+1.5
64,4+18
67.2+008
64.4+0.9
53.5+0.8
FMCSC (Ours)
95.1+0.8
87.8+1.2
88.8 1.5
92.9+1.2
84.2+2.2
85.0=2.5
90.1+1.2
79,4+2.6
79,5+3..
ICC--MMCC (2022)
24
93.130.0
81.9.0.0
83.640.0
89. 8.0.0
73.4
76.4
89.5+0.0
72.540.1
76.7=0.0
IMVC-CBG (2022)
39
37.910.4
21.0=1.0
10.4+0.1
37.2+s0.1
21.0=0.0
7.8+0.0
36,911.11
20.420.1
6,4+0.0
DSIMVC (2022)
37
92.5a0.44
81.7
84.910.9
89.5+2.0
76.5+2.0
77,8
86.143.3
70.0.3.9
76.623.4
LSIMVC (2022)
44.110.5
23.720.4
5.710.2
39.2 39.3
19.740.5
4.840.2
35.3
14.9+1.3
2.840.4
BDGP
Prolmp (2023)
22
91.640.3
82.4+3.8
80.0
90.4+1.5
76.2:0.5
79.3+1.6
75.640.5
$2.342.0
44.621.
JLLTD (2023)
56.510.2
41.320.1
31.7
49.410.1
33.340.0
18.5+0.0
51,010.2
34.140.1
21.520..
CPSPAN (2023)
18
78.7
38.
18.6
37.3+1.3
50.3+2.3
39.413.7
52.4
34.711,4
27.142.1
FedDMVC (2023)
8
92.0+0.1 1
80.2+0.2
84,7+0.1
91 5+0.5
77.1+0.4
80.3+0.7
82.2+0.2
63.4+0.3
61.9+0.3
FCUIF (2024)
36
93.840.1
82.2+0.1
85.1
90.340.2
75.2:0.3
78.4
85.7.0.2
67.5
63.240.2
FMCSC (Ouns)
94_5+0.8
1.2
86.
91.
77.3+2.2
81.0+2.5
90.0+1.2
73.3+2.66
76.8+3.2
HCP-IMSC (20022)
24
70.6#0.1
07.4
37.7
67. 17011
64.7
50.1
39,9+0.7
56,4
1..
IMVC-CBG (2022)
80
46.3+0.0
49.420.0
26.3+0.0
43.240.1
42.720.1
19.2+0.1
38 9.0.2
39.4
13.5
Muki- Muki- Fishiin
DSIMVC (2022)
1.
13.6=1.11
1
77.7+1.4
76.7+0.8
86.8
76.7+1.7
1.5
66.4=1
LSIMVC (2022)
27
51.120.5
49.920.1
31.5
50.240.6
$2.240.1
35 25.11
99.920.2
48 48.620.0
28.2
Prolmp (2023)
22
69. 10..4
3
55.2+0.8
69.0+0.1
64.6+0.2
52.5+0.2
53.9+2.4
10.7
17.8=1.6
JPLTD (2023)
29
44.620.0
43.4-0.0
36.9
37.2+0.1
36.610.1
29.5
25 8
25.1
16.520.1
CPSPAN (2023)
64. 1.2
1.
55 15
61
19.7+1.3
54.4 8
59, 39.342.0
08.0+1.8
13.3+1.9
FvDDMVC (2023)
8
67.720.3
74.620.8
58.020.6
66. 66.6400.4
63.3:0.7
54.3
57 620.77
58.5
43.2+0.7
FCUIF (2024)
70.740.5
79.5
63.1
420.6
71.540.4
29.5
62.5400.6
61.340.5
45.620.5
FMCSC (Ours)
92.4+0.1
85.820.2
84.720.3
40
82.810.7
80.9=1.0
87.5+0.6
79.1=0.1
76.3+1.0
HCP-IMSC (2022)
24
36. 10.0
8,5+0.0
6.7
3=0
8,250.0
6.320.0
51 310.0
6.0=0.1
4.7+0.1
IMVC-CRG (2022)
19
30.8+0.1
4,8+0.0
1.10
30.4+0.0
4.6+0.00
2.5+00.0
29. -+000
4.0=0.0
1.9+0.1
DSIMYC (2022)
51.1+1.3
25.3=0.8
23.4+0.8
6+0.8
22.2+0.7
20,4=0.6
46.7+0.5
18.340.6
16.0=0A4
NUSWIDE
LSMMVC (2022)
kk
37.2+0.2
10.8=0.1
6.8+0.1
36.4+0.3
11.8+0.1
7.0+0.4
33,9+0.4
9,2+0.3
5.8+0.2
ProTmp (2023)
MM
38.4+0.1
11.1=0.0
8.3+0.1
1
10.5+0.1
7.6+0022
34 3+0.7
8.0+0.0
6.1+0.0
JDL (023)
39
53.0+0.2
25.9=0.2
23.7+0.1
51 5+0.5
22.5+1.0
21.5+0.8
50.0+0.2
19.4+0.1
14,1+0.1
CPSPAN (2023)
33.740.2
9,0+0.9
6.2+0.1
33.340.3
6.6+0.9
4,4+0.1 I
29, 4+0.6
5.4+1.1
3,2+0.4
FedDMVC (2023)
8
41.7+0.2
14.4=0.1
12.3+0.1
37.5+0.7
9.8+0.9
7.8+00.
32.6++02
5.8+0.2
4.3+0.2
FCUIF (2024)
16
45.24
15.0+0.3
14, 1+0.2
40.2+005
10.0#0.6
9.2+0.5
38 240.4
9.6+0.3
8,2+0.3
FMCSC (Ours)
56.1+0.2
26.320.5
23.920.4
52.740.2
23.0=1.3
21.8:0.4
50.8:0.9
20.1+0.7
18.8+0.8
Table 2: Ablation studies on four datasets when M/ M/S=1:1.
Components
MNIST-USP
BIGP
Milti-Fashiin
NUSWIDE
A
c
D
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NM1
ARI
ACC
NMI
ARI
1tem-1
V
91.12
82.24
81.63
69.84
48.71
46.29
87.80
79.86
76.61
42.16
12.37
11.16
Item-2
v
60.82
39.74
34.28
61.04
30.76
30.14
56.04
34,22
27.75
42.08
10.15
9.06
Item-3
7
88.66
76.85
76.84
87.40
68.96
71.09
82.58
74.29
68.82
45.86
15.06
12.64
Item-4
89.52
78.12
78.31
85.44
64.15
67.26
88.14
79.61
76.91
47.54
15.20
13.79
Iten-5
v
v
92.93
84.18
85.02
91.92
77.29
80.95
90.36
82.81
80.89
52.74
22.97
21.81
the number of communication rounds compared to FMCSC. AL- though the initial misalignment of the model on MNIST-USPS and Multi-Fashion can be mitigated through multiple communication rounds. Component A still plays crucial role in our training process, facilitating consensus among clients during pre-training to alleviate model misalignment and accelerate convergence effectively. Item-3, which lacks component C, involves both single-view and multi-view clients carrying out the same operation of replacing their local mod- els with global models. We observe substantial improvement in Item-3 compared to Item-2, indicating that the success of model com- parison in component C is attributed to high-quality global models. This achievement requires collaborative efforts from both multi-view and single-view clients. Additionally, the inclusion of the weighted aggregation in component D enhances the benefits of collaborative training among all clients.
18
23
Figure 2: ACC vs. Tm and Tp.
8

---

Parameter Analysis. We investigate the sensitivity of our clustering performance on MNIST-USPS
dataset to two primary hyperparameters in lyce--syni contrastive learning: Tm and Tpx as shown in Figure 2 The values of Tm and Tp are tuned within the range of 0.1, 0.3, 0.5, 0.7, 1. Our observations include: i) When Tm and Tp are set to small values, such as 0.1, the clustering performance of the proposed FMCSC decreases. This may be attributed to excessive emphasis on view consistency, potentially resulting in an inseparable intrinsic feature space. ii) As the values of Tm and T increase, the clustering results gradually recover, and they exhibit insensitivity within the range of 0.3 to Empirically, we set Tm = Ty = 0.5 for datasets.
Qualitative Study on Model Misalignment. To further quantify the impact of model misalign- ment and the effectiveness of our proposed strategy, we visualize the model outputs, i... the consensus semantics H, both without consensus pre-training and with consensus pre-training. F displays the t-SNE 38 visualizations generated from a randomly selected multi-view -iew client, where different colors represent different classes. In Figure (a), (a), observe that the out- put of the global model without consensus pre- training shows mixed features that cannot be clearly distinguished. This confirms our view- point that direct parameter aggregation leads to model misalignment, manifested as feature confusion in the feature space. In contrast, Fig- ure 3 (b) demonstrates that FMCSC produces more distinct and separable features, mitigating the negative impact of model aggregation.
(a) Without consensus.
(b) With consensus.
Figure Visualization on model misalignment.
ACC
100
AR2
KOEP
SUSVIce
100
1E0UP
91277 Pachess
(a) ACC us. Samples per client.
(b) ACC es. Number of clients.
(c) Privacy analysis.
Figure 4: (a) Effect of samples per client on generalization performance. (b) Scalability with the number of clients Multi-Fashion. (c) Sensitivity under privacy constraints when M/S= 2:1.
4.3 Attributes of Federated Learning
Generalization Analysis. To check the validity of our theory for the proposed method, we
investigate the impact of the number of samples per client and client participation rate
the generalization performance in the clustering task, as shown in Figure (a) and Table 3
In this setting, the total number of clients fixed. We find that: i) Increasing the number of samples per client effectively raises the total number of samples in-
Table Effect of participation rates on generalization performance.
Client types
Participating clients
Non Non Non-participating clients
Participation rate
50%
70%
90%
50%
70%
90%
MNIST-USPS
90.64
91.71
92.97
89.11
91.67
92.06
Dataset
BDGP
88.32
89.48
92.73
85.92
86.63
87.65
volved in training, which en-
Multi-Fashion
87.58
89.46
89.81
87.31
87.48
88.73
hances the model's general- ization performance. ii) As the proportion of participat-
NUSWIDE
53.12
53.18
53.58
46.08
51.46
52.51
ing clients increases, the accuracy of pontpaatiii clients also improves, and the performance gap between participating and non-participating clients narrows. These observations are consistent with the theoretical understanding in Theorem 2 i.e., the generalization performance of the model
enhanced as both the number of samples per client and the number of participating clients increase.

---

Number of Clients. We next consider the effects of changing the number of clients as shown in Figure 4 (b). It is observed that as the number of clients increases, the performance of FMCSC experiences a slight decline but remains generally stable. Only when the client number reaches 100, does a noticeable decline in performance occur, which is attributed to the insufficient number of samples within each client.
Privacy. FMCSC, by design, does share any raw data between clients and the server. Only the model parameters on each client are shared with the server. To further protect client privacy, we adopt differential privacy [I] by adding noise to the model parameters uploaded from the client to the server. Figure 4 (c) illustrates the clustering accuracy of FMCSC under different privacy bounds We observe that FMCSC achieves both high performance and privacy at 50. However, as the level of noise increases at E 10, the performance of FMCSC unavoidably degrades.
5
Conclusion
In this paper, we propose FMCSC that can handle practical scenarios with heterogeneous hybrid views and explore the data cluster structures distributed multiple clients. First, we propose cross-client consensus pre-training to align the local models on all clients to avoid their misalignment. Then, local-synergistic contrast and global-specific weighting aggregation gap and the view gap across distributed clients and explore the cluster structures in multi-view data distributed on different clients. Theoretical analysis and extensive experiments demonstrate that FMCSC outperforms state-of-the-art methods across diverse heterogeneous hybrid views and various federated learning scenarios. In future work, we will the method for more downstream tasks and some real-worll situations, such as medical analysis and financial forecasting.
designed bridge the client
Acknowledgment
This work was supported in part by National Natural Science Foundation of China (No. 62476052), Sichuan Science and Technology Program (No. 2024NSFSC1473), Central Guidance for Local Science and Technology Development Fund Projects (No. 2024ZYD0268), and Shenzhen Science and Technology Program (No. JCY1202308071 1595544441.
References
[1] Martin Abadi, Andy Chu, Ian Goodffllow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM CCS, pages 308-318, 2016.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
[3] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurlPS, pages 1-8, 2006.
[4] Xiao Cai, Hua Wang, Heng Huang, and Chris Ding. Joint stage recognition and anatomical annotation of drosophila gene expression patterns. Bioinformatics, 28(12)i16-i2) 2012.
[5] Sicong Che, Zhaoming Kong, Hao Peng, Lichao Sun, Alex Leow, Yong Chen, and Lifang He. Federated multi-view learning for private medical data integration and analysis. TIST, 13(4):1-25 2022.
[6] Jiayi Chen and Aidong Zhang. Fedmsplit: Correlation-adaptive federated multi-task learning across multimodal split networks. In KDD, pages 87-96, 2022.
[7] Man-Sheng Chen, Ling Huang, Chang-Dong Wang, and Dong Huang. Multi-view clustering in latent embedding space. In AAAI, pages 3513-3520, 2020.
[8] Xinyue Chen, Jie Xu, Yazhou Ren, Xiaorong Pu, Ce Zhu, Xiaofeng Zhu, Zhifeng Hao, and Lifang He. Federated deep multi-view clustering with global self-superviiion. In ACM MM, pages 3498-3506 2023.
10

---

[9] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng, Nus- wide: worl-world web image database from national university of singapore. In ACM CIVR. pages 1-9, 2009.
[10] Chenhang Cui, Yazhou Ren, Jingyu Pu, Jiawei Li, Xiaorong Pu, Tianyi Wu, Yutao Shi, and Lifang He. A novel approach for effective multi-view clustering with ieformation-theoreee perspective. In NeurlPS, pages 1-13, 2023.
[11] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how easy is it to break privacy federated learning? In NeurIPS, pages 16937-16947, 2020.
[12] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks, In AISTATS, pages 315-323, 2011.
[13] Geoffrey E. Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. NeurIPS, 6, 1993.
[14] Xingchen Hu, Jindong Qin. Yinghua Shen, Witold Pedrycz, Xinwang Liu, and Jiyuan Liu. An efficient federated multi-view fuzzy e-means clustering method. TFS, (114))1886-188) 2023,
[15] Mingkai Huang, Hao Li, Bing Bai, Chang Wang, Kun Bai, and Fei Wang. federated multi-view deep learning framework for privacy-preserving recommendations. arXiv preprint arXiv:2008. 10808, 2020.
[16] Shudong Huang, Wei Shi, Zenglin Xu, Ivor W Tsang, and Jiancheng Lv. Efficient federated multi-view learning. Pattern Recognition, 131:108817, 2022.
[17] Weitian Huang, Sirui Yang, and Hongmin Cai. Generalized information-theorrtiic multi-view clustering. In NeurlPS, pages 1-13, 2023.
[18] Jiaqi Jin, Siwei Wang, Zhibin Dong, Xinwang Liu, and En Zhu. Deep incomplete multi- view clustering with cross-view partial sample and prototype alignment. In CVPR, pages 11600-11609) 2023.
[19] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pages $132-5143, 2020.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[21] Rafat Latala and Krzysztof Oleszkiewicz. On the best constant khinchin-kahane inequality. Studia Mathematica, 109(1):101-104, 1994.
[22] Haobin Li, Yunfan Li, Mouxing Yang, Peng Hu, Dezhong Peng, and Xi Peng. Incomplete multi-view clustering via prototype-basee based imputation. In IJCAI, pages 3911-3911 2023.
[23] Shaojie Li and Yong Liu. Sharper generalization bounds for clustering. JCML, pages 6392 6402, 2021.
[24] Zhenglai Li, Chang Tang, Xiao Zheng, Xinwang Liu, Wei Zhang, and En Zhu. High-order correlation preserved incomplete multi-view subspace clustering. TYP, 31 31:2067-2080, 2022.
[25] Paul Pu Liang, Zihao Deng. Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. In NeurlPS, pages 1-28, 2023.
[26] Yi-Ming Lin, Yuan Gao, Mao-Guo Gong, Si-Jia Zhang, Yuan-Qiao Zhang, and Zhi- Yuan Li. Federated learning on multimodal data: A comprehensive survey. MIR, pages 1-15, 2023.
[27] Chengliang Liu, Zhihao Wu, Jie Wen, Yong Xu, and Chao Huang. Localized sparse incomplete multi-view clustering. TMM, 25:53 2022.

---

[28] Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong, Li Liu, Marius Kloft, and En Zhu. Efficient and effective regularized incomplete multi-view clustering. TPAMI, 43(8):2634-266) 2020.
[29] Wei Lv, Chao Zhang, Huaxiong Li, Xiuyi Jia, and Chunlin Chen. Joint projection learning and tensor decomposition-basee incomplete multiview clustering, TNNLS, pages 1-12, 2023.
[30] J MacQueen. Classification and analysis of multivariate observations. In BSMSP, pages 281-297, 1967.
[31] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style. -ereforerroeece deep learning library. In NeurIPS, 2019.
[34] Xi Peng, Zhenyu Huang, Jiancheng Lv, Hongyuan Zhu, and Joey Tianyi Zhou, Comic: Multi- view clustering without parameter selection. In ICML, pages 5092-5101 101, 2019.
[35] Dong Qiao, Chris Ding, and Jicong Fan. Federated spectral clustering via secure similarity reconstruction. In NeurlPS, pages 1-36, 2023.
[36] Yazhou Ren, Xinyue Chen, Jie Xu, Jingyu Pu, Yonghao Huang, Xiaorong Pu, Ce Zhu, Xiaofeng Zhu, Zhifeng Hao, and Lifang He. A novel federated multi-view clustering method for unaligned and incomplete data fusion. Information Fusion, 108:1-10, 2024.
[37] Huayi Tang and Yong Liu. Deep safe incomplete multi-view clustering: Theorem and algorithm. In ICML, pages 21090-211 221110 10, 2022.
[38] Van, Laurens der Maaten, and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
[39] Siwei Wang, Xinwang Liu, Li Liu, Wenxuan Tu, Xinzhong Zhu, Jiyuan Liu, Sihang Zhou, and En Zhu. Highly-efficient incomplete large-scale multi-view clustering with consensus bipartite graph. In CVPR, pages 9776-9785, 2022.
[40] Song Wu, Yan Zheng, Yazhou Ren, Jing Hc, Xiaorong Pu, Shudong Huang. Zhifeng Hao, and Lifang He. Self-weighted contrastive fusion for deep multi-view clustering. TMM, 2024.
[41] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 11Xhv:108..0774,220 2017.
[42] Jie Xu. Shuo Chen, Yazhou Ren, Xiaoshuang Shi, Heng Tao Shen, Gang Niu, and Xiaofeng Zhu. Self-weighted contrastive learning among multiple views for mitigating representation degeneration. In NeurlPS, 2023.
[43] Jie Xu, Yazhou Ren, Xiaolong Wang, Lei Feng. Zheng Zhang, Gang Niu, and Xiaofeng Zhu. Investigating and mitigating the side effects of noisy views for self-supervised clustering algorithms in practical multi-view scenarios. In CVPR, pages 22957-22966, 2024.
[44] Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi- view clustering. In CVPR, pages 16051-16060, 2022.
[45] Weiqing Yan, Yuanyang Zhang, Chenlei Lv. Chang Tang, Guanghui Yue, Liang Liao, and Weisi Lin. Gefagg: Global and cross-view feature aggregation for multi-view clustering. In CVPR, pages 19872, 2023.
[46] Mouxing Yang, Yunfan I.,, Peng Hu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Robust multi-view clustering with incomplete information. TPAMI, 1055) 2022.
12

---

[47] Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via contrastive representation ensemble. ICLR, 2023.
[48] Fengda Zhang, Kun Kuang, Long Chen, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Fei Wu, Yueting Zhuang, et al. Federated unsupervised representation learning. FITEE, 24(8):1181-119) 2023.
[49] Zheng Zhang. Li Liu, Fumin Shen, Heng Tao Shen, and Ling Shao. Binary multi-view clustering. TPAMI, 41(7):1774- 1782, 2018.
[50] Handong Zhao, Hongfu Liu, and Yun Fu. Incomplete multi-modal visual data grouping. In IJCAI, pages 2392-2398, 2016.
[51] Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. arXiv preprint arXiv:2008 03030, 2020.
13

---

We provide more details and results about our work in the appendices. Here are the contents:
Appendix
Framework proposed algorithm.
Appendix
More related work.
Appendix
Proofs of Theorem and Theorem 2.
Appendix
More details about experimental settings.
Appendix
Additional experiment results.
Appendix
Broader impacts of proposed method.
Appendix
Limitations of our proposed method.
A
Framework of the Proposed Algorithm
Algorithm outlines the execution flow for both clients and the server in FMCSC. Initially, cross- client consensus pre-training aligns the unsupervised local models of each client (Lines 1-2). During the training, feature contrastive learning is employed for multi-view clients to explore common semantics among different views (Lines 5-12). For single-view clients, model contrastive learning designed between local models and global models, promoting the extraction of more generalized common semantics from client models (Lines 13-19). Subsequently, the server develops global- specific weighting aggregation to aggregate multiple high-quality models (Lines 21-23). Finally, complementary cluster structures are discovered using the global models across clients (Line 25).
Algorithm Federated Multi-view Clustering via Synergistic Contrast (FMCSC)
Input: Data with V views distributed among M multi-view clients and S single-view clients, communication rounds R, Local epoch E.
Output: Overall clustering results.
Pre-train all client models by Eqs. --
2: server receives pre-trained models from all clients and initializes the global models.
3: while not reaching R rounds do
4: for = 1 to (M + 5) do in parallel
5: if c E [M] then
> Multi clients
6:
while not reach the maximum iterations E do
7:
Learn common semantics by Eqs.
8:
Optimize the total loss by Eq.
9:
end while
10:
Optimize multiple global models by Eq. (6).
11:
Upload (f (: ) and fm (:; w..) to the server.
12:
else if E [S] then
b> Single-view clients
while not reach the maximum iterations E do
14:
Learn common semantics by Eq.
15:
Optimize the total loss by Eq.
16:
end while
Upload fp (=:")) to the server. end if
18:
end for
Server
20:
Assigning weights to local models by Eq. (9
Aggregate models among all clients by Eq. (0).
22:
Distribute multiple global models to each client.
23:
end while
24: Calculate the clustering results by Eq. 12
B More Related Work
Multi-view clustering (MVC) methods leverage consistency and complementary information between multiple views to enhance clustering efffcctiveness. Based their ability handle missing data,
14

---

existing MVC methods can be classified into two categories. Complete multi-view clustering 4
which uncovers hidden patterns and structures by leveraging complete multi-view data for clustering. The success of existing complete multi-view clustering relies on the assumption of sample integrity across multiple views. However, in real-world scenarios, samples of multi-view are partially available due to data corruption or sensor failure, which leads to incomplete multi-view clustering studies [28 46 50). Incomplete multi-view clustering via prototype-base basee imputation (ProImp) [22] employs a dual attention layer and dual contrastive learning loss to learn view-specific prototypes and recover missing data. Cross-vied partial sample and prototype alignment network (CPSPAN)
49
18 employs complete data to guide sample reconstruction and proposes shifted prototype alignment to calibrate prototype sets across views. However, the aforementioned MVC methods assume that multi view data are stored within single entity, thus lacking the concept of heterogeneous clients, and only addressing the hybrid views scenarios we mentioned.
C Theoretical Analysis
C.1 Proof of Theorem
In this part, we want to prove that minimizing contrastive loss D. and D are equal to maximizing mutual information. The proof is motivated by [32 51
Proof. 2 and C as the contrastive loss, denoting the consistency objectives of the multi-view client mn and single-view client p respectively, i.e.,
V Mm
eim((a //
log
Mm
Σ jzts e sim (h., Tmm"
1Spl
e (im(hi
1
C
log
[Spl
sim b²)/== +
When j # assume that p (h;, hz) ph:) (hz), which means that p(n,hh))
1. Let
p(b))(bū))
N₁ - M Mmdl p(b..)t)
we can then
p(hi)p(h;)'
Mm [Mmll
P (h,, h1)
I (H:
Σ p (h, h2) log
p ph:;) p h1))
Mmmll
p (h, h²)
[Mml
p (h; h²)
Σ
p (h, h
log
+
Σ
Σ
p (b,, h2)) log
p (h:) (h,)
(h.) p (hz)
i=1
iti
Mml
(h, h²)
Σ t=1
(h, h²) log
(h:) (h.) No Ni.
p(h..b))
[mml
[Mml
p(b))p(h))
p (h, h²) log
p (h, h²)) log
N
Since positive pairs are correlated, we have the estimate: p (h; h²) 2 p (h.) p (b.). In addition,
we assume that there exists a constant 5, (0, 1) such that (h) h.) > 6m, i = 1,2, ... [MMl
holds. According to and with the estimation, we have (h:) x [.... = 1,2. ... Mm and
sim b/).
p (h..h)) the following inequality holds: (bg))
15

---

plh, p(b,,h))
Σ
(H.
M
p (h h²)) log
p(())
M.ml
p (b, h,)) log
M
Σ
p (b h5)
N₂
j=1 p (h:) p (h))
==1
i=1
b(b,,)))
Mmml
p (h)
hi) log
p(h,) (p(;))
log (Mmml
p (h, h,²) 1 + p (h,) (h,)
Mm
N₂
==1
8m
I.m
e h//m
log
V log Mmm
Mm
Σj
sim b1.l1f///
(iin(b,
2 V log [mml
5m cm
Similarly, we assume that there exists constant 6p E (0, 1) such that p (h, h:) - p (x) lh:) > 6p.
i 1.2. SSpl holds. And we have p (h;) = |7p esim eaim(he (h.:h)//r. x p p(b,,19)
and p(hs)p(ng)
caim(haaf)/r & p(h,.?) the following inequality holds: p(he)n(z))*
s log
Spp
p ph, h²) p (hh)) p (h?)
p ph;,z?)
1 (H, - 1 (H,Z*)
p (h,, h
Σ
p (h, 23 log
p (h:) p (2"))
SSp
[Sp
2
p ph;; h²)
h,)log (hi, hg) p (h:) p (h
(h h:) log
Spl p (hς
Spl
p (h,) p (h.)
S
p (h;; 2,))
Spl P (x.
h:) log
p (h,, (z,²)
1=1
S
26p
(h, h
26p
p (h, h²)
p (h,, 2 p (b:) p (z?)
log
log
+
Ssp
(h,)) p (h²)
Spl
p (h.))(p?)
SSpl
sin (h..h,)/+
28p
log
S
(hm + ebim(b.,;)/rr
26p
C.2 Proof of Theorem
We consider a heterogeneous federated learning setting where M multi-view clients and S single-view clients. For each multi-view client, we define the empirical risk and its expectation as E. (f) and (f), respectively. Similarly, for each single-view client, the empirical risk and its expectation are denoted as (f) and (f). Let f X + R x Rd. x Rd denotes the function that maps input samples into reconstruction samples, low-level features and high-level features. Then, the reconstruction samples, low-level features and high-level features are given by x := fa (x") E D.,,,
2 := S (x+) E Rdd., and h == JK (x²) E Rdd, respectively. Additionally, we define (xt) J==1 == x;, then common semantics are denoted h; f (x.) E Rd. To prove Theorem we first introduce the following three lemmas.
Lemma 1. We define the empirical risk Enn(f) (f) for vieti-view client m as [Mml
V
Σ jike e crmm (h, bi
()) = Mm
- - (x))1 log
\/m
u=l
V [Mmil
sim (1i((1((s))
|xx - fa (xx))12 -log
Mm
Ej e sim (f))
16

---

Let Cm(f) ) be the expectation of (f). Suppose that for any x E X and f E F, there exists D < oo such that ||x||, (x)] 11fk (x)II, 1112 (x)] E [0, DJ hold. With probability 1 8 for any
F, the following inequality holds
12VD2 MMM + DVD2
log 2/ 2 2|
(f) S (f)+
Proof. This proof is inspired by
37 For simplicity, we define Jn xx)
sim (fh (x:). fk (x))) /mm and fn (x:) = sim (fh ( (xx) /Tm. respectively. Then, the empirical risk and its expectation can be formulated as
Mm
(
1
|x] - Ja (x))11 log
1m
Ejee (x((x
Mim
|xx - Ja Js (x+))1² + log
Eexp exp (Sh (xx;
Sn (xi)
Mm
M.Mal
Mm
||x2 fa (x))11 fn (xi)
+
1
log
exp [sn (x1,
Mm
[mmm
and
Cm(
Σ
Ex |ay fa (0)))) fn (xx)
Σ
Ex
log
exp
fuh (xx,
Let Mn the sample set that different from Mm by only one set of data x, == (x;) The empirical risk of the function f on Mn, is denoted as L. (f). We have
sup (f) - -mn(f) sup (5) - E, (f) feF
FEF
Em (f) -En
sup fer
v (12x;) - fz (xx)]12 -18"" - fz (xx))2
(fr (x,) - fn (xx.))
< <sup fer
[Mml ==1
+ sup fEF
2
log Σ
exp [nf (x,, - log Σ
exp {fr (Xc, (3,1
[Mm w=1
V
< sup fEF
||x;||1 - || + fl (xx))1- -1fa (xx)
+ I1 (x || + 2 (x))i)
+ 2
[mm|
p=l
+ sup feF
In (x.) Sn (8+) + sup
2
Σ
Σ
(Jn (x., Sn (Rr, xi))
Mm pml
feF Mm (IMm 1)
uml
9V
Mm
Then, we analyze the upper bound of the expectation term E Esup [m)(f) Lm). Let oI....,, MMM be i.i.d. independent random variables taking values in (-1, 1} and Mm be the independent copy of Mm We have
17

---

E sup FEF
(f) (m)
Mml
1
sup S Mm /EF
[|x] Ia (xx))] -|x] Ia (x:))
Mmml
1
+ E Mm. Mm sup JEF
(fn (x,) - (s.
[m
v=l
Mmml
sup + B Mm Mm FEF
2
log Σ exp (Jnn (xi, - log Σ exp (sr (Rs: {ززز
Mm
D=l
1=1
jz
jzl
V
Mmmml
v
Mm
< Mmno sup /EF
|xx - fa (xx))1
+ 21E M...
sup fEF
oifn (xi)
Mm
Mm
w=1
1=1
o=l
1=1
V 1M
+ Mm.. sup fer
Jish xj)
[m ([.Ml -1) 1) mm
1=1 A
3
I.mal Σ
|xx fa (022)192)
<2V max Emm, sup FEF
+ 2V max E.M. sup FEF
Σ
[fn (x:))
[mm
Mm
i=1
1
1.mll
+ 4V max E Em
sup Mm. fEF
1
Σ
lif (xx))
[ml (IMm 1)
D2V
Wm
where the second last inequality is obtained by the Khintchine-Kahang inequality (21). Thus, according to the McDiarmid inequality [31]. with probability at least 1 - for any ʃ E F, we have
12V
log 1
(
+ DV
VM
2 2|
Due to the presence of view gaps among different views, let the data from the u-th view follow the dis- tribution D., and the multi-view data follow the distribution D. We analyze the generalization bounds for Im (:: wh)) in multi-view client m. which is built upon prior works from domain adaptation.
Lemma 2 (Generalization Bounds for Domain Adaptation 2 Bl). Consider the v-th view data domain DD and the multi-view data domain D, respectively. Given feature extraction function R. X + H that shared between DD and D. Let F be set hypothesis with VC-dimension d. Then, for every f E F, with probability at least 1 - 8:
4
2c Mm log +)+ ds (D.: + Aus
Em(J) S Lm (f) Mm
d dlog
where dz (D., D) denotes the divergence measured over a symmetric-difeett
hypothesis space.
D. and D are the induced distributions of DD and D under R. respectively, S.I. BnaD [B(h)] EynDD [B)))))) given probability event B, and so for D. Xo = miny Lmv (f) + Lm denotes an oracle performance.
18

---

Lemma 3. We define the empirical risk (x( for single-view client p as
e sim (b.,
(p(()
Σ
|x - foe (x(')1 log
p
@sm TD + @im(b.
sim simm (f
S Σ
|xx fo (xx))) - log e (xim((((xx))ff(xx))))
Let [f) be the expectation of Lf) Suppose that for any x E X and f E F, there exists D < oo
such that ||X||, flfe (x)] .llf" (x)]l [f (x)]I 1155 (x)] [0, D) hold. With probability 1 - 8 for any f E F, the following inequality holds
10D2
log 1
f,(f) S +
+ DD²
Vis
2 S
Proof. For simplicity, we define fg (xx)
sim (f% (x,) Sg (xx) TTp and fu.z (x.)
sim (fh (xx)) TTp, respectively. Then, the empirical risk and its expectation can be formu- lated as
15
ela())) |x - fa (x,)))? log edo(x)) + (xi))
E/( = Spl
|
||x - fz (xf))1 + log ( (efalxe) + - fo (x:)
Spp
S
S
||x] fa (x:)11 fo (xxi)
1
log (efolar)) + elhnatsi))
+
Spl
SSpl
and
c,() = Ex || fa (x²) - Sn (xx)) + Ex [log (efslai) + edhnstor))
Let Sp be the sample set that different from Sp by only one sample point &,. The empirical risk of the function f on Sp is denoted as (J). We have
(P) - E,(0) - sup (c((0) -,,1) 2,
sup FEF
fEF
E)( - 2,00
< sup fEF
1
(Ix;) Sn (xx))1 - || - Sa (xt)1 - (Js (x+) - Sg (x.)))
< sup fe"
[Sp]
[log (falxx)) + edh.,co.)) llog (cfaler) + cl
+ sup /EF
|Sp]
S sup [
11.- ||*|||² || + 51 (xx) -1.fa (xx)11 + 2|| I1 (x |1 + | (x)I)
/e3
+ sup
15g (x,) - ff (x+) + sup
(Kf (x,) + friz (x+)) - (Fo (xx) + fr.. (x+))
feF [Spl
feF 2|5
8D2 [Spl
19

---

Then, we analyze the upper bound of the expectation term se (2p(f) - Lp(f), Let O1....,0|, Ssp be ii.d. independent random variables taking values in {-1, 1} and Sp be the in- dependent copy of Sp. We have
E sup [x(l) Lp(f) FEF
S
S
S,
Sp sup fEF
[|x - (x -|x; - fz (27)11))
+s sup JEF
(fo (x.) - fo (xi))
|Sp]
Spp
i=1
18y
sup + E
log (eb(xx) + elhaa(xi)) log (obl88)) elhna(88i)
SSpl
fEr
|Sp1
SS
1
<2E Sp.a sup JEF
oi |x
fa (xx))² + +2Bs
sup fet
JzfS (xi)
Spp
Spl
|
+ 2Rs, sup fe3
0i (Jg (xx) + fh, (xx))
Sp
i=1
- for (622)1999 1
|spl
|8
<2Es sup p. JEF
1
Σ
||
+ 2E sup leF
Σ i=1
Ifg
(xx))2
|spl
SSp
|
sup + 2Es
1
Σ i=1
(1s (xx)) [fn (xx))²)
S
FEr
10D
SSSp
Thus, according to McDiarmid inequality
31 with prohability at least 1 - 5 for any f E F, we have
10D2
log
s < 2,(++ +
+ SD2
Vspl
V 2 S
Our objective is collaboratively train all clients obtain multiple heterogeneous global models capable of handling different view types. This manifested in the optimization of multiple global objective functions. For the global objective of handling multiple view types simultaneously, expected risk is defined as (f), typically optimized in the form of empirical risk minimization, defined as:
ΣM(
1 M
Σ () m2
Similarly, for handling individual view types, such as the v-th view type, the global objective entails defining the expected risk Ls- (f), with empirical risk defined
Zs-(0)
1 Σ 50
+
1 M m=1
Em Em-(f) (f).
Now give the proof of Theorem 2
20

---

Proof. According to Lemma]
| with probability at least 1 5 for any f E F, we have
12
log
(M) S (n)
9VD2,
VM MM
2M [M
where M is the number of multi-view clients and [M is the number of samples in each multi-view client. Additionally, we have
Σ dy
Du. D) =ds ds (D..
2
Σ sup fer
Pro. [f] - Prp [f]|
M
M
According to Lemma
2 and Lemma 3 with probability at least - 5 for any f E F, we have
10
log 2 SSpl
Ls- <Ls (f)
+ +D2
/ Spl
+
M Mm
4
d'log
2cM [Mml
d
+ log
4
8)
+ dr (D., D) + Ao,
where S" the number of single-view clients with w-th view type and Spl the number of samples in each single-view client.
D Experimental Settings
D.1 Datasets
We conduct the experiments the following public datasets. MNIST-USPS [4] is widely- dataset for handwritten digits (0-9) and consists of 5000 examples with two views of digital images. The MNIST feature size is 28 x 28, while the USPS feature size is 16 x 16. BDGP 4 examples related to drosophila embryos, each represented by a 1750-dimensional visual feature and 79-dimensional textual feature. Multi-Fashion [41 is image dataset featuring products like Coats, Dresses, and T-shirts, with images sized at 28 x 28. Following the approach in [10], which constructs three-view version using 30,000 images, each instance includes three different images belonging to the same class. Consequently, the three views of each instance represent the same product with three distinct styles. NUSWIDE [9] is a web image dataset offering multiple views. including 65-dimensional color histogram, 226-dimensional block- wise color moments, -45-dimensional color correlogram, 74-dimensional edge direction histogram, and -imeeennii wavelet texture. We utilize a total of 5000 samples for the evaluation of our proposed method.
comprises 2500
Table 4: The statistics of experimental datasets.
Dataset
Clients
Sample
View
Class
Dimension
MNIST-USPS
24
5000
2
10
[[28, 28), [16,16]]
BDGP
12 2500
2
[1750.79]
Multi-Fashion
10000
[784, 784, 784]
NUSWIDE
24
5000
5
5
[65, 226, 145, 74, 129]
D.2 Implementation Details
The models of all methods are implemented on the PyTorch 33
platform using NVIDIA RTX-3090 GPUs. For an encoder-decoder pair, the encoder structure follows Input- Fcsuu FC FC Fczu, and the decoder is symmetric to the encoder. The non-linear mappings H v and (Z; v\ adopt network architectures of Fo2o - Fe25e Fezo and Fo2ov - FC258 - Fez0. respectively. The activation function is ReI.U [12), and the optimizer uses Adam. For all the datasets used, the learning rate fixed at 0.0003, the batch size set to 256 and the temperature parameters Tm and Tp both set to 0.5. Local pre- pre-training performed for 250
[H² (z
21

---

epochs on all datasets. After each communication round between the server and clients, local training conducted for 10 epochs for the BDGP dataset and 25 epochs for other datasets on each client. The communication rounds between the server and clients are set to R = 5. All experiments in the paper involving FMCSC that are not mentioned are performed when M/S = 1:1.
We report some results that number of parameters and runtime by FMCSC to give the reader some information about the computational resources used by the method. Table 5 that the number of parameters and runtime by FMCSC are small and easy to reproduce.
Table 5: Number of parameters and runtime by FMCSC.
Dataset
MNIST-USPS
BDGP
NUSWIDE
Number of parameters per client Runtime
3.4M-6.7M 7.7M
3.5M-7M
3.4M-10.IM
2.8M-13.7M
763.8s
108.5s
1183.5s
954.7s
D.3 Comparison Methods
We select state-of-the-ar methods, including HCP-IMSC 24), IMVC-CBG (39, DSIMVC 37.
LSIMVC (27). Prolmp [22). JPLTD (29). CPSPAN [8]. FedDMVC 8 and FCUIF 36 Among them, apart from FedDMVC and FCUIF, which are FedMVC methods, all the other comparison methods are centralized incomplete multi-view clustering methods. To ensure fair comparisons, we simplify the heterogeneous hybrid views scenario in our paper into hybrid views scenario. Specifi- cally, we concatenate the data distributed among the clients and use them as input for centralized Among these, the data from multi-view clients can be considered
methods, shown in Figure
Mulli-view clients
Single-view clients
View
View
View V
Figure 5: Comparison strategies.
complete data, while the data from single-view clients can be regarded as missing data. It's worth noting that in our reported results, our method operates under the heterogeneous hybrid views sce- nario, whereas the other comparative methods operate under the hybrid views scenario. Although existing solutions exposure of raw data, due to privacy concerns, may cause more data owners to refuse to participate in collaborative training. In contrast, our method can extract complementary clustering structures across clients without exposing their raw data, offering better privacy protection and performance improvement than current state- toite-of-the-att methods.
bypass the challenge of heterogeneous clients by simply concatenating data, the
E
Additional Experiment Results
E.1
Convergence Analysis
Convergence analysis of the reconstruction loss, consistency loss, and total loss for multi-view clients and single-view clients is conducted using MNIST-USPS, BDGP, Multi-Fashion, and NUSWIDE. As illustrated in Figure 6 the increasing number of epochs leads to a gradual convergence of all loss functions, ultimately reaching stable state. This clear observation serves as compelling evidence for the stability and effectiveness of our proposed model.
22

---

0.150
L000 10000
0.07
0.000
90
103
150
200
50
100
150
200
250
300
Eooch
(a) MNIST-USPS
(b) BDGP
C3
LEGS
611
150 Epech
200
250
Epoch
(c) Multi-Fashion
(d) NUSWIDE
Figure 6: Convergence analysis on four datasets.
E.2 Attributes of Federated Learning
In this section, we present additional experimental results of FMCSC in various federated learning scenarios, including the number of clients and privacy. Figure 7 presents the impact the number of clients on clustering performance for MNIST-USPS, BDGP, and NUSWIDE. It observed that, with an increasing number of clients, the performance of FMCSC shows a slight decline but remains generally stable. However, on MNIST-USPS, as the number of clients reaches 50, the clustering performance experiences an unavoidable decrease due to the insufficient number of samples for each client.
ACC
2012
ACC
304
A22
The raaber chume
of
(a) MNIST-USPS
(b) BDGP
(c) NUSWIDE
Figure Scalability with the number of clients.
100
1:10
E-
E-50
c-50
00
Privacy
No Privacy
60
40
BOCP
Mult-tashion
NUSWIDE
0-
MNISTUSPS
Mutti Fastion
MUSWIDE
Detmne
Dataset
Figure 8: Sensitivity under privacy constraints when M/S 2:1.
23

---

In FMCSC, we assume that all participating parties are other. An attacker faithfully executes the training protocol but may launch privacy attacks to infer the private data of other parties. Previous studies have demonstrated that sharing gradients can leak information about raw data [1]. To address this concern we utilize differential privacy to further enhance the privacy guarantee of FMCSC. Differential privacy algorithms aim to introduce noise during individual data processing to protect against the disclosure of private information [. Formally, let A be random mechanism that takes dataset D input and belongs to set S. Assuming Dı and D2 are two neighboring datasets differing in only one point, A is (e, 6) differentially private if:
and do not collude with each
Pr (Aii) E S] < exp(E) Pr (D22) S] 6
(13)
Here, e and 8 quantify the individual impact the overall output in differential privacy. e represents the strength of differential privacy, with smaller values indicating stronger privacy protection at the potential cost of increased noise. 8 O number of samples is used handle probabilistic exceptional cases. We implement Laplace noise achieve differential privacy, and Figure reports the clustering performance of FMCSC under different privacy strengths E. Our observations reveal that FMCSC achieves both high performance and privacy at 50, especially the BDGP and Multi-Fashion datasets, However, as the level of noise increases at = = 10, the performance of FMCSC unavoidably degrades.
rigurtrreport
F Broader Impacts
Heterogeneous hybrid views are prevalent in real-world scenarios. Our proposed method extends the application domain of existing FedMVC approaches to address complex scenarios such as healthcare and the Internet of Things (IoT). For example, hospitals in metropolitan areas use CT, X-ray, and EHR for disease detection, whereas remote areas often rely on single detection method. Similarly, smartphones can capture both audio and images simultaneously, while recording devices are limited collecting audio data only. Furthermore, this research not expected introduce any new negative societal impacts beyond those already known.
G
Limitations
Our model addresses heterogeneous hybrid views in FedMVC, but idealistically categorizes clients into two types: single-view clients and multi-view clients. In more realistic scenarios, multi-view clients can be further classified into full-view clients and partial-view clients based on the number of view types they have. Such detailed categorization could encourage more heterogeneous devices to participate federated learning, thereby enhancing the model's 's generalization ability and accelerating its application in fields like healtheare and finance. We will continue to explore this problem in future work and apply our findings to world scenarios.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we first introduce the relevant concepts of VedMVC and summarize existing methods, analyzing their shortcomings. Notably, existing methods assume isomorphic clients, prompting to propose the concept of heterogeneous hybrid views. We focus addressing the client gap and view gap that arise in this scenario. Our proposed method, which designs seal-synergistic contrastive learning and global- specific weighting aggregation, aims to bridge these gaps and explore cluster structures. Additionally, we analyze the method's generalization performance and its effectiveness across various federated learning scenarios through theoretical and experimental evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix G Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
25

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please Appendix
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer Appendix D.2 for information on reproducing the main experi- mental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
26

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have uploaded the code for the new proposed method and the datasets used in the experiments in the Supplementary Material. The baseline methods compared in the paper are open source and be reproduced directly after following the comparison strategy mentioned in Appendix D.3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca cc/
public/guides/ Sottass//ooeSubbiiisiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Appendix for more experimental setting/details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report error bars in Table I and Figure 4 (b), which are standard deviations of the results obtained after five independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have read the NeurIPS Code of Ethics and confirm that the paper complies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cite the original paper that produced datasets see Appendix D.I
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "9bLkMFs3ov", "reviewer": "Reviewer_14Lq", "summary": "This paper proposes a novel Federated Multi-View Clustering method capable of handling heterogeneous hybrid views. By designing local-synergistic contrastive learning and global-specific weighting aggregation, the proposed method explores clustering structures across different clients. The effectiveness of the proposed method is demonstrated both theoretically and empirically.", "questions": "Please see ‘Weaknesses’.", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe motivation behind the paper is clear, and the proposed heterogeneous hybrid view scenario is more applicable to real-world situations compared to other FedMVC methods.\n2.\tThe paper conducts extensive experiments, demonstrating the effectiveness of the proposed method.", "weaknesses": "1.\tSeveral key observations are presented in Section 3.2 on cross-client consensus pre-training; however, the purpose of these observations is not very clear.\n2.\tThe paper describes the transfer of multiple model parameters between the client and the server, but does not analyze the communication overhead.\n3.\tThere are some inaccuracies in the descriptions, such as \"select 10 state-of-the-art methods\" in Lines 239 and 578, which should be 9？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer 14Lq:\n\nWe sincerely appreciate your constructive comments and suggestions. \n\n**Q1: Several key observations are presented in Section 3.2 on cross-client consensus pre-training; however, the purpose of these observations is not very clear.**\n\nThank you for your feedback. In Section 3.2, we have two key observations: (a) the presence of single-view clients exacerbates the issue of model drift; (b) the absence of uniformly labeled data across all clients leads to the reconstruction objective of autoencoders optimizing from multiple different directions, resulting in model misalignment.\n\nFor observation (a), we use the local-synergistic contrastive learning designed in Section 3.3, which helps single-view clients bridge client gaps and mitigate model drift. Theorem 2 demonstrates the effectiveness of this strategy by analyzing the generalization bounds of the proposed method. For observation (b), we propose cross-client consensus pre-training to align the local models on all clients and avoid their misalignment. Figure 2 visualizes the model outputs, further quantifying the impact of model misalignment and the effectiveness of our proposed strategy. Additionally, the ablation study results in Table 2 show that the proposed strategy plays a crucial role in our training process, facilitating consensus among clients during pre-training, effectively alleviating model misalignment, and accelerating convergence. We will revise our manuscript to make this point clearer. \n\n**Q2: The paper describes the transfer of multiple model parameters between the client and the server, but does not analyze the communication overhead.**\n\nBelow, we use the MNIST-USPS dataset as an example to calculate the total communication overhead required by FMCSC. Table 5 reports the number of parameters per client by FMCSC. Suppose the data are distributed among 24 clients, with an equal number of multi-view and single-view clients. In this case, in each communication round, the total data volume that all clients need to transmit to the server is 485.6MB, and the data volume that the server needs to transmit to all clients is 54MB. Due to the proposed cross-client consensus pre-training strategy accelerating convergence, the communication rounds are set to 5. Therefore, the total communication overhead required to reach convergence for this dataset is calculated to be 2.6GB. \n\nSimilarly, we further calculate the total communication overheads required for FMCSC to reach convergence on other datasets. The results indicate that these overheads are acceptable, holding for both clients with powerful computing capabilities (such as large institutions) and lightweight clients (such as mobile devices).\n\n| Dataset  | MNIST-USPS | BDGP  | Multi-Fashion | NUSWIDE |\n| :--------------------: | :--------: | :---: | :-----------: | :-----: |\n| Communication overhead |   2.6GB    | 1.5GB |     6.4GB     |  4.4GB  |\n\n**Q3: There are some inaccuracies in the descriptions, such as \"select 10 state-of-the-art methods\" in Lines 239 and 578, which should be 9？**\n\nThank you. We will correct the mistakes and further polish our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GpQjQZCfVg", "reviewer": "Reviewer_s4wh", "summary": "This paper proposes a novel method i.e. Federated Multi-View Clustering in Heterogeneous Hybrid Views (FedCSC), which introduces a locally collaborative contrastive learning algorithm to achieve consistency between single-view and multi-view clients, thereby mitigating heterogeneity among all clients. Furthermore, a global-specific aggregation algorithm has been used to address the gaps between different views. Benchmark experiments validate the effectiveness of this approach.", "questions": "1.How were $\\alpha_m$ and $\\alpha_p$ derived in Eq. 10 of this paper?\n\n2. In the experimental results on the MNIST-USPS dataset in Figure 8, why does the performance with 24 clients outperform that of both 16 clients and 50 clients?", "limitations": "Given.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: The paper introduces a novel global weighted aggregation method, encouraging the global model to learn complementary features from mixed views, demonstrating a certain level of innovation.\n2: The paper introduces a novel federated multi-view learning framework that considers the scenario of a mix of single-view and multi-view clients, followed by experimental analysis.\n3: The paper conducts a comprehensive theoretical analysis and validation of the proposed method.\n4: The code is provided for reproduction", "weaknesses": "1. The dataset size of 10,000 is insufficient; validating the proposed methods requires a significantly larger dataset to ensure robustness and broader applicability.\n\n2. The computational complexity of global-specific weighting aggregation is estimated to be high. Therefore, it may not be suitable for large-scale data tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer s4wh:\n\n**Q1: The dataset size of 10,000 is insufficient; validating the proposed methods requires a significantly larger dataset to ensure robustness and broader applicability.**\n\nThanks for the suggestion. We conduct further experiments on the large-scale YoutubeVideo dataset [1], which contains 101,499 samples across 31 classes, where each sample has three views of cuboids histogram, HOG, and vision misc. Below are the clustering results of FMCSC and several comparison methods when the number of multi-view clients and single-view clients are equal ($M/S = 1:1$):\n\n| Method | IMVC-CBG (2022) | DSIMVC (2022) | ProImp (2023) | FedDMVC (2023) | FCUIF (2024) | FMCSC (Ours) |\n| :----: | --------------- | ------------- | ------------- | -------------- | ------------ | :----------: |\n|  ACC   | 18.32           | 15.01         | 22.45         | 21.52          | 23.04        |    26.42     |\n|  NMI   | 11.83           | 8.11          | 17.48         | 16.96          | 18.46        |    20.74     |\n|  ARI   | 2.04            | 1.20          | 3.43          | 3.42           | 3.72         |     5.82     |\n\nThe YoutubeVideo dataset is 10 times larger than the Multi-Fashion dataset with 10,000 samples. The results demonstrate that FMCSC adapts well to large-scale datasets and outperforms other methods, ensuring the proposed method's robustness and broader applicability.\n\n[1] Omid Madani, Manfred Georg, and David A. Ross. On using nearly-independent feature families for high precision and confidence. Machine Learning, 92:457–477, 2013.\n\n**Q2: The computational complexity of global-specific weighting aggregation is estimated to be high. Therefore, it may not be suitable for large-scale data tasks.**\n\nThank you for raising your concerns. We specifically analyze the computational complexity of global-specific weighting aggregation as follows.\n\nEq. (10) shows the detailed process of global-specific weighting aggregation performed by the server. During this process, the server receives $M$ local model parameters capable of handling multi-view data and $(VM + S)$ local model parameters capable of handling a single specific view. We define the number of these local model parameters as $N_m$ and $N_p$, respectively.  When aggregating to obtain $f_{g}(\\cdot; \\mathbf{w})$, the server performs a weighted sum of the $M$ multi-view client parameters, resulting in a computational complexity of $O(N_m M)$. For $V$ models $f_{g}^v(\\cdot; \\mathbf{w}^{v})$, the server performs a weighted sum of the single specific view models, with a complexity of $O(VN_m M + N_p S)$. Therefore, the total computational complexity is $O(N_m M + VN_m M + N_p S)$.\n\nThrough this calculation, we observe that the computational complexity is related to the number of local model parameters and the number of participating clients. Table 5 presents the number of parameters per client for different datasets, such as 3.4M-10.1M for the Multi-Fashion dataset, corresponding to our definitions of $N_m$ and $N_p$, while the number of participating clients is typically below 100. The computational complexity derived from the above analysis is considered acceptable and confirms that our proposed method applies to large-scale data tasks, as addressed in the response to Q1. Additionally, Table 5 reports the running time of the proposed method on all datasets, e.g., 763.8s for the MNIST-USPS dataset, indicating low overall time complexity.\n\n**Q3: How were $\\alpha_m$ and $\\alpha_p$ derived in Eq. 10 of this paper?**\n\nTheorem 1 shows that optimization objectives in multi-view and single-view clients can be measured by different mutual information metrics, with proximity to these objectives reflecting model quality. Based on this, we use mutual information to evaluate the quality of models from multi-view and single-view clients. These metrics are calculated locally and sent to the server with the model parameters. The server assigns aggregation weights based on these values. Specifically, in Eq. (10), $\\alpha_m$ are derived by normalizing $\\sum_{v=1}^{V}I\\left(\\mathbf{H}, \\mathbf{H}^{v}\\right)$, which are calculated by $f_{m}\\left(\\cdot; \\mathbf{w}_m\\right)$ from different multi-view clients. \n\n$\\alpha_p$ are derived by normalizing $I\\left(\\mathbf{H}, \\mathbf{H}^{g}\\right) - I\\left(\\mathbf{H}, \\mathbf{Z}^{v}\\right)$, which are calculated by $f_{p}\\left(\\cdot; \\mathbf{w}_p^{v}\\right)$ from different single-view clients. Higher mutual information values indicate better model quality, leading to higher weights during aggregation and achieving high-quality global aggregation.\n\n**Q4: In the experimental results on the MNIST-USPS dataset in Figure 8, why does the performance with 24 clients outperform that of both 16 clients and 50 clients?**\n\nThank you for your detailed observation. We believe that clustering performance is closely related to the number of clients, but it does not exhibit a simple linear relationship. For example, results on the BDGP dataset in Figure 8 show that the performance with 12 clients outperforms that of both 8 clients and 16 clients.\n\nThis phenomenon has two main reasons. First, a moderate increase in the number of clients promotes diversity in local models, benefiting the high-quality aggregation of the global models. Second, with a fixed number of samples, significantly increasing clients makes the data for each client sparse, negatively affecting local model training and clustering performance. The experimental results in Figure 8 highlight this issue: when the number of clients reaches a critical point, the severe insufficiency of samples within each client leads to a significant decline in FMCSC's clustering performance. This indicates that global clustering performance is highly correlated with the training quality of local models. This finding also motivates us to further explore high-quality training with few samples per client, encouraging more devices to participate in training and collaboration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "abrxHVrqbM", "reviewer": "Reviewer_jVcj", "summary": "The authors introduce a novel method called Federated Multi-view Clustering via Synergistic Contrast (FMCSC), to simultaneously leverage the single-view and multi-view data across heterogeneous clients to discover clustering structures from hybrid views. This method bridges client and view gaps through a combination of theoretical and experimental analysis to discover the cluster structures in multi-view data distributed across different clients.", "questions": "I am curious whether the proposed method can be applied to both vertical FedMVC and horizontal FedMVC scenarios?", "limitations": "Limitations and societal impact have been discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Federated multi-view clustering (FedMVC) is a recently proposed and increasingly popular research direction within the multi-view learning community. This paper addresses a novel issue in FedMVC, termed ‘heterogeneous hybrid views,’ where a mixture of both single-view and multi-view clients exhibit varying degrees of heterogeneity.\n\nThrough theoretical and experimental analysis, the paper clearly shows how the proposed method bridges client and view gaps. The proposed method performs well across different federated settings, with reproducible code.\n\nThe paper is also well-written, with clear explanations, extensive experiments, and detailed theoretical proofs.", "weaknesses": "The paper does not mention the data partitioning strategy of the proposed method. The authors need to provide details on how data are distributed among different clients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer jVcj:\n\nWe thank the reviewer for valuable comments and suggestions that have greatly improved our paper.\n\n**Q1: The paper does not mention the data partitioning strategy of the proposed method. The authors need to provide details on how data are distributed among different clients.**\n\nThank you for pointing out this issue. The proposed method adopts the common IID partition, where multi-view data are randomly and uniformly distributed across all clients, ensuring that each client's data distribution is similar to the overall data distribution. In implementation, we achieve this by setting a large Dirichlet distribution parameter, ensuring that the proportion of different classes of data allocated to each client are nearly equal, thereby achieving the effect of independent and identically distributed data. \n\n**Q2: I am curious whether the proposed method can be applied to both vertical FedMVC and horizontal FedMVC scenarios?**\n\nThank you for your insightful comments. We believe the proposed method can be applied to both vertical FedMVC and horizontal FedMVC scenarios. In the paper, we mention that existing FedMVC methods usually assume that clients are isomorphic and belong to either single-view clients or multi-view clients. In the heterogeneous hybrid views scenario applicable to FMCSC, vertical FedMVC scenarios can be viewed as having only single-view clients, with the number of clients equal to the number of views; horizontal FedMVC scenarios can be seen as having only multi-view clients. When facing horizontal FedMVC scenarios, FMCSC can be directly applied by setting the number of single-view clients to zero. Additionally, for vertical FedMVC scenarios, where only single-view clients exist, the method cannot bridge gaps with the help of multi-view clients. Therefore, the local-synergistic contrast module in Section 3.3 needs to be frozen before applying FMCSC.\n\nThe above analysis demonstrates that FMCSC, suitable for heterogeneous hybrid views scenarios, can still be applied to both vertical FedMVC and horizontal FedMVC scenarios. The heterogeneous hybrid views scenario complements and serves as an alternative to the current FedMVC assumption, making it better aligned with real-world situations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UY2emvJ6Ea", "reviewer": "Reviewer_uC6c", "summary": "A variant of horizontal FedMVC is proposed to address more realistic scenarios involving heterogeneous hybrid views. It develops specific strategies and conducts theoretical analyses from the perspective of bridging client and view gaps. The proposed method demonstrates promising experimental results on several datasets.", "questions": "1) The experimental results reported in the paper show a large discrepancy for DSIMVC on the MNIST-USPS dataset compared to the original results, while the differences are much smaller for the BDGP and Multi-Fashion datasets. Why does this phenomenon occur?\n2) In the reported experiments, the number of clients varies across different datasets. How were these numbers chosen?", "limitations": "see the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The scenario of heterogeneous hybrid views assumed in the paper is interesting and merits further exploration.\n2) The appendices are thorough and well-organized, including theoretical proofs and additional experiments.", "weaknesses": "1) Contrastive learning strategies have been widely used in multi-view clustering methods; thus, the synergistic contrast strategy proposed in this paper may not offer significant novelty.\n2) The paper dedicates considerable effort to describing how common semantics H is extracted from the raw data of each client, but it lacks an explanation of why this approach is suitable for clustering tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer uC6c:\n\nThank you for your valuable feedback. \n\n**Q1: Contrastive learning strategies have been widely used in multi-view clustering methods; thus, the synergistic contrast strategy proposed in this paper may not offer significant novelty.**\n\nWe would like to emphasize that the local-synergistic contrast strategy comprises both feature contrastive learning and model contrastive learning, aimed at addressing the client gap and mitigating the heterogeneity between single-view clients and multi-view clients.\n\nWe acknowledge that feature contrastive learning used in multi-view clients has already been applied in some multi-view clustering methods. However, our goal is that multi-view clients can help single-view clients bridge client gaps. Thus, the innovation of this module does not lie in multi-view clients using feature contrastive learning to train local models, but rather in single-view clients using model contrastive learning to bridge client gaps and discard view-private information detrimental to clustering. By establishing a unified goal of extracting common semantics H in both single-view and multi-view clients, a communication bridge is built. Meanwhile, the extraction of common semantics helps in discovering complementary clustering structures across clients. Furthermore, model contrastive learning allows the local single-view clients to converge towards the global model while amplifying the differences between the reconstruction and consistency objectives in the local models.\n\n**Q2: The paper dedicates considerable effort to describing how common semantics H is extracted from the raw data of each client, but it lacks an explanation of why this approach is suitable for clustering tasks.**\n\nThank you for your feedback. As we mentioned in our response to Q1, the extraction of common semantics H aims to unify the training objectives of single-view clients and multi-view clients. This allows single-view clients to use model contrastive learning to bridge client gaps and discard view-private information that is detrimental to clustering. We believe that focusing on common semantics, which eliminates the adverse effects of view-private information, is more conducive to discovering subsequent clustering structures. We will revise our manuscript to make this point clearer. \n\n**Q3: The experimental results reported in the paper show a large discrepancy for DSIMVC on the MNIST-USPS dataset compared to the original results, while the differences are much smaller for the BDGP and Multi-Fashion datasets. Why does this phenomenon occur?**\n\nThank you for your detailed observation. We have carefully reviewed the replication code for DSIMVC and confirmed that the experimental results reported in Table 1 are accurate. In DSIMVC, incomplete samples are generated by randomly removing views under the condition that at least one view remains in the sample. In contrast, our comparison strategy considers data from multi-view clients as complete data, while data from single-view clients are regarded as incomplete data. Unlike the random removal in DSIMVC, our approach results in incomplete data that tends to have consecutive same missing views. This inconsistency in handling incomplete scenarios leads to the discrepancy between the reported results and our replicated results.\n\n**Q4: In the reported experiments, the number of clients varies across different datasets. How were these numbers chosen?**\n\nFigure 3 (b) and Figure 8 show that as the number of clients increases, the performance of FMCSC experiences a slight decline but remains generally stable. However, when the number of clients reaches a certain threshold, the clustering performance of FMCSC drops significantly. We believe this occurs because samples within each client become insufficient, which hinders local model training and negatively impacts clustering performance. Therefore, for different datasets, we aim to ensure that each client has more than 200 samples to maintain the stability of FMCSC during training. In the future, we will further explore high-quality training with fewer samples per client, encouraging more devices to participate in training and collaboration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a  federated multi-view clustering method for handling heterogeneous hybrid views. It introduces local-synergistic contrastive learning to  achieve consistency between single-view and multi-view clients, and eliminates the gaps between views via a global-specific weighting aggregation strategy. Experimental results and theoretical analysis demonstrate its effectiveness and strengthes.  After rebuttal, all reviewers give postivie and recongrize its novelty, good-structure, extensive experiments, and detailed proof. Based on these, I recommend accepting this paper.
Decision: Accept (poster)