Federated Training of Dual Encoding Models on Small Non-IID Client Datasets
Authors: Raviteja Vemulapalli
Warren Richard Morningstar
Philip Andrew Mansfield
Hubert Eichner
Karan Singhal
Arash Afkanpour
Bradley Green
Venue: Submitted to ICLR 2023 (2023)
Keywords: dual encoding models
federated learning
representation learning
self-supervised learning
federated self-supervised learning

Abstract: Dual encoding models that encode a pair of inputs are widely used for representation learning. Many approaches train dual encoding models by maximizing agreement between pairs of encodings on centralized training data. However, in many scenarios, datasets are inherently decentralized across many clients (user devices or organizations) due to privacy concerns, motivating federated learning. In this work, we focus on federated training of dual encoding models on decentralized data composed of many small, non-IID (independent and identically distributed) client datasets. We show that existing approaches that work well in centralized settings perform poorly when naively adapted to this setting using federated averaging. We observe that, we can simulate large-batch loss computation on individual clients for loss functions that are based on encoding statistics. Based on this insight, we propose a novel federated training approach, Distributed Cross Correlation Optimization (DCCO), which trains dual encoding models using encoding statistics aggregated across clients, without sharing individual data samples. Our experimental results on two datasets demonstrate that the proposed DCCO approach outperforms federated variants of existing approaches by a large margin.

Under review as a conference paper at ICLR 2023
FEDERATED TRAINING OF DUAL ENCODING MODELS ON SMALL NON-IID CLIENT DATASETS
A Anonymous authors
Paper under double-blind review
ABSTRACT
Dual encoding models that encode a pair of inputs are widely used for representa- tion learning. Many approaches train dual encoding models by maximizing agree- ment between pairs of encodings on centralized training data. However, in many scenarios, datasets are inherently decentralized across many clients (user devices organizations) due to privacy concerns, motivating federated learning. In this work, we focus on federated training of dual encoding models on decentralized data composed of many small, non-IID (independent and identically distributed) client datasets. We show that existing approaches that work well in centralized settings perform poorly when naively adapted to this setting using federated aver- aging. We observe that, we can simulate large-batch loss computation on individ- ual clients for loss functions that are based on encoding statistics. Based on this insight, we propose novel federated training approach, Distributed Cross Corre- lation Optimization (DCCO), which trains dual encoding models using encoding statistics aggregated across clients, without sharing individual data samples. Our experimental results on two datasets demonstrate that the proposed DCCO ap- proach outperforms federated variants of existing approaches by large margin.
INTRODUCTION
Dual encoding models (see Fig. are class of models that generate a pair of encodings for a pair of inputs. either by processing both inputs using the same network or using two different networks.
These models have been widely successful in self-supervised representation learning unlabeled
unimodal data (Chen et al.) 2020a] Chen & He [021 Zbontar et al. 2021 He et al. [2020; Grill
ct al. 2020). and are also a natural choice for representation learning of paired multi-modal data (Jia
al.
2021 Radford et al., 2021 Bardes et al. 2022).
While several approaches exist for training dual encoding models in centralized settings (where the entire training data is present
central server), training these models decentralized datasets is less explored. Due to data privacy concerns, learning from decentralized datasets is becoming increasingly important. Federated learning (McMahan et al.) 2017] is widely-used approach for learning from decentralized datasets without transferring raw data to a central server. In each round of federated training, each participating client computes a model update using its local data, and then central server aggregates the client model updates and performs global model update.
In many real-world scenarios, individual client datasets are small and non-IID (independent and identically distributed), e.g., in cross-device federated settings (Kairouz et al. 2021 Wang et al., 202T). For example, in the context of mobile medical apps such as Aysa (AskAysal 2022) and DermAssist (DermAssist 2022), each user contributes only a few (1-3) images. Motivated by this, we focus on federated training of dual encoding models on decentralized data composed of large number of small, non-lII client datasets.
Recently, several highly successful approaches have been proposed for training dual encoding models in centralized settings based contrastive losses (He et al. 2020 Chen et al. 2020a cutistics-basee losses (Zbontar et al., 2021 Bardes et al., 2022). 2). and predictive losses combined with batch normalization (loffe & Szegedy 2015) and noo-gradieent operation (Grill et al. 2020 Chen & He (202T). One way to enable federated training of dual encoding models is to adapt these existing approaches using the Federated Averaging (FedAvg) strategy of McMahn et al. 2017). As

---

Under review as a conference paper at ICLR 2023
Loss
Loss
Loss
G
M
M,
X
Figure 1: Dual encoding models. Two inputs generated using random data augmentations (T ~ T) are encoded by (a) the same encoder network or (b) two different networks. In (c) aligned inputs from two different modalities are encoded by two different modality-specific networks.
described in Section 2 all of these approaches require large and diverse training batches to work well, and their performance degrades when trained on small, non-IID client datasets (see Sec. 4.4).
While Zhuang et al. (2021) Makhija et al. (2022), Zhuang et al. (2022) and He et al.] (202) ex- tend predictive loss approaches such BYO (Gril al. 2020) and SimSiam (Chen He 2021)
to federated settings, they consider only cross-silo settings (with thousands of samples per client)
where large-batch training is possible.
In this work we observe that, in the case of statistics-based loss functions, we can simulate large- batch loss computation on each individual (small) client, by first aggregating encoding statistics from many clients and then sharing these aggregated large-batch statistics with all the clients that contributed to them. Based on this observation, we propose a novel approach, Distributed Cross Correlation Optimization (DCCO), for federated training of dual encoding models small, non-III client datasets, The proposed approach simulates large-batch training based with the loss function of Zbontar al. (2021), which refer as Cross Correlation Optimization (CCO) loss. This is achieved without sharing individual data samples or their encodings between clients.
MAJOR CONTRIBUTIONS
We observe that large-batch training of dual encoding models be simulated on decentral- ized datasets by using loss functions based on encoding statistics aggregated across clients, without sharing individual samples.
Building on this insight, we present Distributed Cross Correlation Optimization (DCCO), novel approach for training dual encoding models decentralized datasets composed of large number of small, non-IID client datasets.
We prove that when we perform one step of local training in each federated training round, one round of DCCO training is equivalent one step of centralized training on large batch composed of all samples across all clients participating in the federated round.
We evaluate the proposed DCCO approach CIFAR- 100 and dermatology datasets, and show that it outperforms FedAvg variants of contrastive and CCO training by significant margin. The method also significantly outperforms supervised training from scratch, demonstrating its effectiveness for decentralized self-supervised learning.
2 PROBLEMS WITH EXISTING APPROACHES
Contrastive loss functions Contrastive losses explicitly maximize similarity between two encod- ings of a data sample while pushing encodings of different samples apart. This is highly effective when each sample is contrasted against large set of diverse samples (Chen al. 2020a He et al.)
2020 Radford et al. 2022 Jia et al. 2021). Contrastive learning approaches can be extended to fed- erated settings by combining within crchin-clientt contrastive training with the FedAvg strategy of McMa-
han et al. (2017). But this reduces performance as each sample is contrasted against small set of within-client samples, which may be relatively similar.
2

---

Under review as a conference paper at ICLR 2023
Dependence on batch normalization Approaches such as BYOL Grill al. 2020) and Sim- Siam (Chen & He 202I) train dual encoding models using a predictive loss that encourages one encoding of data sample to be predictive of another encoding of the same sample. Unlike trastive losses, predictive losses do not explicitly push the encodings of different samples apart. Despite this, these approaches work well when trained with large batches in centralized settings. Importantly, both B BYOL and SimSiam use batch normalization (loffe & Szegedy) 2015). whose efficacy decreases rapidly when batches become smaller (Toffe 2017: Wu & He 2018). When train- ing on small, non-lID client datasets in federated settings. group normalization (Wu & He 2018) is typically used instead of batch normalization (Hsieh et al. 2020 Hsu et al.) 2020). When we ex- perimented with BYOL and SimSiam by replacing batch normalization with group normalization, the models did not train well (see Appendix G suggesting that batch normalization is important for these approaches to work wel[] This dependence on batch normalization suggests that these approaches are not a good fit for federated training on small, non-IID client datasets.
Statistics- loss functions While the above contrastive and predictive losses directly use in- dividual sample encodings in their computation, the CCO loss introduced by Zbontar et al.) (2021) is a function of encoding statistics computed over batch of samples. This loss function maximizes the correlation coefficient values of matching dimensions and minimizes the correlation coefficient values of non-matching dimensions of the encodings of dual encoding model. Since CCO loss is a function of batch statistics, its efficacy decreases with smaller batch sizes. While it works well in centralized settings where large batches are used for training, it performs poorly in federated settings when used for training on small, non-III client datasets (see Sec. 4.4..
3 PROPOSED APPROACH
3.1 CROSS CORRELATION OPTIMIZATION Loss
Let X and Y be the two inputs dual encoding model, and F [F]] E Rd and G |G E Rd be their encodings, respectively. The CCO loss used for training dual encoding models in Zbontar al. (2021) is given by
d Loco = C1- + 2d-12
d
Σ 2
(1)
1m1
3
where Cty represents the correlation coefficient between the jth component of F and the jth com- ponent of G:
(F.G,) (G,) -
Ci = V ((B))) - (F.)2 ((G))))-(())
(2)
Here, () represents average values computed using a batch of samples. The first term in the CCO loss (Eq. [ encourages the two encodings of data sample be similar by maximizing the corre- lation coefficient between the matching dimensions of the encodings, and the second term reduces the redundancy between output units by decorrelating the different dimensions of the encodings.
3.2 MOTIVATION: AGGREGATING AND REDISTRIBUTING ENCODING STATISTICS
Both contrastive and CCO losses are highly effective when used with large training batches, and their efficacy decreases as batch size decreases. Contrastive losses directly use individual sample encodings. and it is unclear how large-batch contrastive loss can be computed small clients with- out sharing individual sample encodings between clients. Unlike contrastive losses, CCO loss only ((G)))) (F.G,)- Although CCO loss is nonlinear
encoding statistics (F)) ((F.))).
Fetterman Albrecht (020) also observes that batch normalization crucial for BYOL to work. Though Richemond et al. (2020) refutes this and claims that BYOL works well with group normalization, we were not successful in attempts to train BYOL with group normalization.
Zbontar et al. (2021) do not refer this loss CCO loss; we this more general term instead of Barlow/wins loss to emphasize the loss function's applicability to cases where the two encoder networks differ. added normalization factor --r the loss function reduce the sensitivity of tradeoff parameter À the dimensionality of the encodings.
3

---

Under review as a conference paper at ICLR 2023
Local
Model update
Client
Client
Current model
Local stats
Aggregated stats
Model update
Updated model
Server
Client
Server
Client
Server
Local
Model
Client K
Client K
stats
update
Local stats (LS)
Combined stats (0,= Qw) (Fi)e ((F)))
Encoding model
Compute stats
(F)) g. (6)). (IG))) (F,9))
loss
(1G)) (F,)))
Aggregated stats from server (AS) Ar ()) (G ((G)), (F,G,)
AS - LS
Forward propagation Backward propagation
8g
local training client k. Here refers the stop gradient operation, and ① denotes addition. The combined statistics ()o used for loss computation are equal to the aggregated statistics ( () from the server. Note that gradient backpropagation happens only through the local statistics ()x-
Figure 2: Top: one round of DCCO training. Bottom:
function of these statistics, the statistics themselves are average values, i.e. linear combinations of individual sample values. Hence, we can compute sarge-batch statistics as weighted average of small-batch statistics computed on individual clients, without sharing individual sample encodings. This makes it possible to simulate large-batch CCO loss on individual clients by first aggregating encoding statisties from many clients and sharing these aggregated large-batch statistics with all the clients that contributed to them. Based on this insight, we propose our DCCO approach that simulates large-batch training using small, non-IID client datasets.
The strategy of aggregating and redistributing statistics can also be used to implement multi-client synchronous batch normalization federated settings. However, since batch normalization typi- cally used every layer of the encoder network, this will add multiple additional rounds of server- client communication (as many as the number batch normalization layers in the network) which undesirable. In contrast, the proposed DCCO approach munication by aggregating statistics at the end of the network before computing the loss function.
only additional round of com-
3.3 DISTRIBUTED CROSS CORRELATION OPTIMIZATION (DCCO)
Figure presents the proposed DCCO training approach. each training round, a central server samples K clients from the pool of available clients and broadcasts the current model to these clients. Each client & uses this model to encode its local data and compute local encoding statistics (Fi), ((F.)²) (Gz) (G,)²²) (F.G)), based Ne samples. These local encoding statisties are then aggregated by the central server using weighted averaging (Eq. 3) and the aggregated statisties shared with the participating clients.
K No N
No N
K N
(FA) =
(Fi) ((EA)) =
((FA))) (G₂)
Σ
(G))
N
k=1
k=
(3)
Σ No k=1
K Σ 1=
N N (RG,),, N
K
((G,)
N ((G)) (RGG)
N
k=
Then, each client computes a local model update by minimizing the CCO loss function computed using aggregated statistics () During local training, while aggregated statistics are used for com- puting the loss function, gradients client has access only to its local data. Finally, the local model updates are aggregated by the central server and used update the current model. To aggregate client model updates,
backpropagated only through local statistics ( )xx since each
weighted
4

---

Under review as a conference paper at ICLR 2023
average, weighted by examples per client. During the entire training process, individual data sam- ples never leave the clients either in raw or encoded formats.
Our work focuses on training with small client datasets (for example, no more than six images per client in the case of DERM dataset). Hence, in each federated training round, we only perform one each participating client. In this setting,
step of local training equivalent to one centralized training step on batch composed of all data samples across all clients participating in the federated round. See Appendix for the proof. This equivalence between fed- erated and centralized training holds only because of the additional step of aggregating and sharing encoding statistics. Naively applying FedAvg with one within-client CCO loss-based training step per round is not equivalent to centralized training, as we show in Section 4
round of DCCO training is
While this work focuses on the CCO loss of Zbontar et al. (2021). the proposed distributed learn- ing strategy (statistics aggregation and redistribution followed by aggregated cartistics-based local training) can also be used with other statistics- loss functions such as Baardes et (2022).
EXPERIMENTS
In this section, we evaluate the proposed DCCO approach in the context of self-supervised learning where the two inputs dual encoding model are generated by applying random augmentations to single unlabeled input, and both the inputs are processed using the same encoder network (see Fig. Ta)).
Experimental setup First, the encoder network is pretrained on an unlabeled, decentralized dataset. Then, linear classifier is added on top of the encoder network, and either only the newly added classifier (linear evaluation protocol) or the entire network (full finetuning protocol) trained using a small labeled classification dataset on central sever. The accuracy of final model is used as the evaluation metric.
Comparative approaches We compare the proposed DCCO approach with FedAvg variants of within-client contrastive and CCO loss-based training. We use centralized training based on CCO loss an upper bound for the proposed approach. We also report results for fully-supervised training from scratch (using the limited laheled data) to demonstrate the effectiveness of DCCO as a self- supervised pretraining strategy.
4.1 DATASETS
This dataset consists of 50K training and 10K test images of size 32 x 32 from 100 object categories. We use all 50K training images as unlabeled data for pretraining, and a small fraction of them (10% or 1%) as labeled data for supervised finetuning. To study the effect of non-identical client data distribution, we generated III) and non-III client datasets using the Dirichlet distribution- baseu sampling process of et al. (2019) with a = 1000 and a 0, respectively. We generated multiple decentralized datasets by varying the total number of clients and the number of samples per client (see Table
CIFAR-100 (Krizhevsy & Hinton, 2009
Dermatology (DERM) (Liu et al., 2020) This dataset consists of de-identified images of skin conditions captured using consumer-gre and exhibit significant variations in terms of pose, lighting, blur, and body parts. Each case in this dataset includes one to six images of a skin condition from single patient. A portion of the cases this dataset are labelled with 419 unique skin conditions by a panel of US corri--ertiffed dermatologists. Following Liu et al.] (2020) and Azizi et al.) (2021) (022), we focus on the most common 26 skin conditions and group the rest into a common "Other" class leading to a final label space of 27 categories. Following Azizi et al. (0022), we use total of 207,032 unlabeled images (59,770 cases) for pretraining the encoder network. For federated pretraining, we consider each case separate client. For supervised finetuning, classification accuracy is reported on a test set of 4, 4,146 labeled cases. This dataset also provides a validation split with 1,190 labeled cases which use for tuning some hyperparameters.
digital cameras. The images are heterogeneous in nature
up to 4.592 labeled cases. The final
5

---

Under review as a conference paper at ICLR 2023
4.2 NETWORKS
Residual networks (He et al. 2016) with 14 and 50 layers are used as encoder networks for ex- perimenting with CIFAR- 100 and DERM datasets, respectively. Following Zbbontar al. (2021). a three layer fully-connected projection network is used to increase the dimensionality of the en- codings before computing the CCO loss during pretraining, and the projection network is discarded while training the final classifier. The projection network configurations are [1024, 1024, 1024] and [2048, 2048, 4096] for CIFAR-100 and DERM datasets, respectively. For the contrastive loss, following CCen et al. (2020a b). fully-connected projection network is used to reduce the dimen- sionality of the encodings before computing the loss. The projection network configurations are [256. 256, 128] and [2048, 2048 128] for CIFAR-100 100 and DERM datasets, respectively. Weight standardization (Qiao et al. 2019) and group normalization (Wu & He 2018) with 32 groups are used at every layer except the last projection layer. For the DERM dataset, since each case consists of multiple images (up to six), the final classification network performs average pooling of individ- ual image encodings and uses linear classifier top of the average-pooled feature predict the label for a case. Following (Azizi et al. (2021). we use 224 x 224 images as input while pretraining the encoder, and 448 x 448 images as input while training the classifier.
4.3 TRAINING DETAILS
CIFAR-100 We experimented with several decentralized versions of CIFAR-100 dataset by vary- ing the total number of clients and the number of samples per client (see Table []. During federated pretraining, all models were trained for 100K rounds. For FedAvg variants of CCO and contrastive loss-based training, we observed overfitting. For these approaches, we evaluated multiple pretrained checkpoints under linear evaluation protocol and report the results for the best checkpoints. We did not observe such overfitting for the proposed DCCO approach and report results based on 100K rounds of training. Each client is visited around 1000 times during federated pretraining of 100K rounds. So, for centralized CCO loss-based pretraining, we used 1000 epochs of training with batch size of 512.
For federated experiments, each case in the dataset is considered as client dataset. We conducted experiments by varying the number of clients sampled each round (see Tab. (2). All models were trained for 75K rounds during federated pretraining. For this dataset (59,770 clients), each client is visited around 320 times if we perform 75K rounds of federated training with 256 clients per round. So, for centralized CCO loss- training, used 320 epochs of training with a batch size of 512.
DERM
We used value of 20 for the tradeoff parameter À in CCO loss (Eq. I1 and value of 0.1 for the temperature parameter in contrastive loss (Chen et al. 2020a). In all federated experiments, we used gradient descent with learning rate 1.0 as the local optimizer on clients. Please refer Appendix B for further details about the training hyperparameters and data augmentations used in all settings.
4.4 RESULTS
4.4.1 CIFAR- DISCUSSION
Table shows the performance of various approaches under linear evaluation and full finetuning protocols for multiple decentralized versions of CIFAR-I00 dataset. When only 500 images are labeled, training only the linear classifier performed better than finetuning the entire network. So, in this case, we only report results under linear evaluation protocol. The proposed DCCO approach outperforms the FedAvg variants of contrastive and CCO loss-based training by a significant margin for both IID and non-III client datasets. Specifically, the case of non-IlI client datasets, the performance gains are in the range of 11-18% under linear evaluation protocol and 6-10% under full finetuning protocol, and in the case of IID client datasets, the gains are in the range of 4-10% under linear evaluation protocol and 4-6% under full finetuning protocol. The proposed DCCO approach also outperforms fully-supervised training from scratch by a significant margin (10-20%) demonstrating the effectiveness of DCCO federated self-supervised pretraining strategy.
In the case of non-IID client datasets, for a fixed global batch size (i.e., the total number of samples participating in round), the performance of DCCO approach increases the number of samples

---

Under review as a conference paper at ICLR 2023
NON-IID CLIENT DATASETS
CLIENT DATASETS
TOTAL CLIENTS
50,000
12,500
6,250
3,125
12,500
6,250
3,125
SAMPLES / CLIENT
16
4
8
16
CLIENTS/ ROUND
512
128
64
32
128
64
32
LINEAR EVALUATION PROTOCOL (SK LABELED TRAINING SAMPLES)
CCO + FEDAVG
FAILED
28.7
30.7
FAILED
41.2
44.1
CONTRASTIVE FEDAVG
31.1
32.4
31.8
41.1
44.1
46.4
PROPOSED DCCO
51.7
49.6
48.1
45.9
51,4
52.0
51.9
CENTRALIZED CCO
52.6
SUPERVISED FROM SCRATCH
42.4
FULL FINETUNING PROTOCOL (5K LABELED TRAINING SAMPLES)
CCO FEDAVG
FAILED
36.3
38.9
FAILED
43.0
46.0
CONTRASTIVE FEDAVG
40.8
41.8
41.3
44.9
46.8
47.4
PROPOSED DCCO
51.5
50.5
49.3
47.7
51.5
52.
52.0
CENTRALIZED CCO
52.5
SUPERVISED FROM SCRATCH
42.4
LINEAR EVALUATION PROTOCOL, (500 LABELED TRAINING SAMPLES)
CCO + FEDAVG
FAILED
13.5
14.5
FAILED
26.2
27.6
CONTRASTIVE + FEDAVG
14.8
15.6
15.0
24.7
27.8
29.3
PROPOSED DCCO
33.3
31.2
29.5
26.7
33.3
33.5
33.5
CENTRALIZED CCO
34.0
SUPERVISED FROM SCRATCH
13.1
Table 1: Comparison of various approaches CIFAR-100 dataset. Results for CCO + FedAvg and Contrastive + FedAvg are not reported when each client has only one sample, because we need at least samples compute these losses. Results for CCO FedAvg not reported when each client has four samples, because training with CCO loss based on four samples was unstable.
per client decreases and the number of clients per round increases, approaching the performance of centralized CCO training when each client has only one sample This is because the proposed DCCO approach optimizes a loss function based on aggregated global batch statistics and using more clients increases the diversity of samples in the global batch. However, we do not see such trend in the of IID client datasets because each sample in every client dataset is already sampled in an IID fashion from the full CIFAR- 100 dataset.
We do report results for FedAvg variants contrastive and CCO losses when each client has only one sample because we need at least two samples to compute these loss functions. The proposed DCCO approach can still be used in this case since it uses statisties aggregated from multiple clients to compute the loss. We do not report results for FedAvg variant of CCO loss when each client has only four samples because training was unstable this case.
Ideally, when each client has only sample, the performance of DCCO with 512 clients round should be close to the performance of centralized CCO with batch size of 512. However, notice small (~1%) performance gap between the two approaches. We used the TensorFlow framework for running centralized training experiments and the TensorFlow Federated framework for running federated training experiments. This may account for the observed discrepancy. For example, the random data augmentation pipeline used during pretraining differs between these two setups in terms of whether they stateful random operations or stateless random operations with explicitly provided seeds.

---

Under review as a conference paper at ICLR 2023
LABELED DATA (FINETUNING)
1,524 CASES
3,057 CASES
4,592 CASES
NUM. CLIENTS (PRETRAINING)
59,770
59,770
59,770
CLIENTS/ ROUND
64
128
256
64
128
256
64
128
256
CCO FEDAVG
FAILED
FAILED
FAILEED
CONTRASTIVE FEDAVG
38.9
41.8
43.9
45.3
46.7
47.8
47.7
49.1
51.5
PROPOSED DCCO
42.4
46.3
47.8
46.7
50.3
51.1
48.3
52.8
52.5
CENTRALIZED CCO
48.0
52.0
53.6
SUPERVISED FROM SCRATCH
30.6
35.9
39.2
Table 2: Comparison of various approaches on DERM dataset under full finetuning protocol. Results reported because training using within-client CCO loss unstable as
for CCO FedAvg each client in this dataset has maximum of six samples.
4.4.2 DERM DISCUSSION
Table shows the performance of various approaches under full finetuning protocol for different amounts of labeled data. The proposed DCCO approach clearly outperforms the FedAvg variant of contrastive loss-based training and achieves a performance close that of centralized CCO training. The proposed approach also results in significant performance gains (13-17%) when compared to fully-supervised training from scratch demonstrating its effectiveness in leveraging unlabeled de- centralized datasets. We do not report results for FedAvg variant of CCO loss because training was unstable in this case as each client this dataset has maximum of six images.
5 RELATED WORK
Self-supervised learning Contrastive methods, motivated by the InfoMax principle (Linsker) 1988), aim to maximize the mutual information between an instance and its encoded representation. Since reliable estimate of mutual information is hard to obtain, methods often resort optimiz- ing the InfoNCE loss (van den Oord et al. 2018). Among contrastive methods, MoCo (He et al.) 2020) uses a queue of examples to construct the contrastive pairs. also uses momentum encoder for training stability. MoCo v2 (Chen et al. 2020c) improves MoCo by adding a MLP projection head and more data augmentations, SimCLR (Chen al. 2020a) removes the momentum encoder and uses the same network to encode both inputs. It also replaces the queue with large batch sizes. SimCLRv2 (Chen et al. 2020b) improves the original model by increasing the size of the encoder network and the depth of the non non-linear projection module.
Different from contrastive learning, methods such as Barlow Twins (Zbontar et al. 2021 and VI- CReg (Bardes et al. 2022) define their objective functions based certain statistics of the encodings such as cross correlation and variance. ARB (Zhang et al.) 2022) learns representations by mapping the embeddings onto an orthonormal basis space. BYOL Grill et al.) (2220) and SimSiam (Chen & He. (021)) use asymmetric two-tower architectures that uses a small non-linear prediction network on one side and the stop-gradient operation the other side. TTan al. (202T) studies the effect of the non-linear predictor and stop-grm noopggradient in these methods and observe that both components are essential prevent collapse.
Another line of self-super learning works use clustering for representation learning. DeepClus- ter (Caron et al. 2018) and SeLa (Asano al., 2020) alternates between clustering the encod- ing space and training the encoder network using a cluster prediction loss. SwAV (Caron et al.)
2020) simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations of the same image. MSN (Assran et al.
2022
combines
SwAV and BYOL while using masking to generate different views of the same image. Recently,
several works (He et al., 2022 Fang et al.] 2022 Wei al. 2022. Chen al.
2022 Xie et al.
2022 El-Nouby et al. 2021: 1021:13 Bao et al. 2021) also explored masked content prediction tasks for self-supervised representation learning.
8

---

Under review as a conference paper at ICLR 2023
Federated dual encoding models To enable large-batch contrastive training in federated settings, Wu et al. (2022b) and Zhang et al. (2020) propose to share individual data sample encodings be- approach only shares encoding statistics
tween clients, raising privacy concerns. In contrast, aggregated across several clients. /Zuuang et al. 2022 2022) extend BYOL (Grill et al. 2020) to cross-silo federated settings by using separate target encoder on each client. The online encoder on each client is updated by combining both local and global online encoders taking into account the divergence between them. [He et al. (202I) extends SimSiam to cross-silo federated settings by using separate personalized model on each client in addition to a common global model shared across clients. The global model is trained using the standard FedAvg strategy and the local models are trained to optimize the SimSiam loss function on local data while pushing the encodings generated by local models to be close to the encodings generated by the shared global model. Makhija et al. (202)) goes a step further and completely removes the shared global model. To enable co collaborative learning, they use a public dataset and regularize client model training by forcing the encodings of the public dataset generated by each client model in round to be close to the average of the en- codings generated by all client models in the previous round. All these methods focus on cross-silo settings with small number of clients and thousands of samples per client. Makhija et al.] (2022) showed that the performance of these methods degrades significantly as the number of clients in- creases. Different from all these approaches, we focus on decentralized datasets composed of large number of small, non-IID client datasets.
Federated training of dual encoding models has also been explored in the context of recommendation systems where the goal is to predict next item from a sequence of previous items. In this setting, sequence of previous items and next item form the two inputs
dual encoding model, and we learn a finite lookup table of item encodings while training a network that encodes sequences of item encodings. Ning et al. (202T) assumes that the entire item lookup table is accessible by each client and optimizes spreadout loss (Yu et al., 2020) on the item lookup table while maximizing the agreement between the encodings of previous items sequence and next item. Wu et al. (2022a) assumes that the full item lookup table is available only the central server and proposes strategy to sample items relevant for each client. The encodings of these items are transferred to the clients and used negatives in contrastive loss-based training. Different from these works, we are focusing on continuous signals such as images where the concept of finite item lookup table is inapplicable.
6 CONCLUSION AND FUTURE WORK
In this work, we proposed an approach for training dual encoding models on decentralized datasets composed of a large number of small, non-IID client datasets. The proposed approach optimizes a loss function based on encoding statisties and simulates large-batch loss computation on individual clients by using encoding statistics aggregated across multiple clients. When each client participat- ing in a training round performs only one local training step, each federated round of DCCO training is equivalent to centralized training step on a batch consisting of all data samples participating in the corresponding round. Our experimental results on two image datasets show that the proposed approach outperforms FedAvg variants of within-client contrastive and CCO loss- training. Our proposed approach also outperforms supervised training from scratch by significant margin; demonstrating its effectiveness as a federated self-supervised learning approach.
This paper focused on self-supervised pretraining on unimodal datasets and the CCO loss function of |bbontar et al. (021). In future work, we plan experiment with multi-modal datasets, and evaluate the proposed aggregated statistics-based distributed learning strategy with other statistics- based loss functions such as Bardes et al. (2022). There are also several other interesting research directions we plan to pursue in the future. How can we extend the proposed approach to large client datasets? When training on large client datasets, we may need to perform multiple steps of local training on each client within one federated round. When we perform multiple training steps on client, in each step, only a small subset of samples that are contributing to the loss function participate in gradient computation. Also, while the model weights change after each local step, the aggregated statistics used in the loss function remain constant. How can we address the effects of these partial gradients and stale statistics? The proposed DCCO approach uses two communication rounds between server and clients within one federated training round. How can we reduce this to one round of communication?

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. ICLR, 2020.
AskAysa. com/
Skin condition questions? Al-enabled answers, 2022. URL https://askaysa
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07141, 2022.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesslingam, Simon Kornblith, Ting Chen, Vivek Natarajan, and Mo- hammad Norouzi. Big self-supervise models advance medical image classification. In ICCV, 2021.
Shekoofch Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Korbblith, Ting Chen, Patricia MacWilliams, S. Sara Mahdavi, Ellery Wulczyn, Boris Babenko, Megan Wilson, Aaron Loh, Po-Hsuan Cameron Chen, Yuan Liu, Pinal Bavishi, Scott Mayer McKinney, Jim Winkens, Abhijit Guha Roy, Zachary Beaver, Fiona Ryan, Justin Krogue, Mozziyar Etemadi, Umesh Telang, Yun Liu, Lily Peng. Gregory Corrado, Dale R. Webster, David Fleet, Geof- frey E. Hinton, Neil Houlsby, Alan Karthikesalingam, Mohammad Norouzi, and Vivek Natarajan. Robust and efficient medical imaging with supf-super CoRR, abs/2205.09723. 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.000 0825444 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variaace-ivaarannc--cvaaannn regulariza- tion for self-supervised learning. In ICLR, 2022.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi semi-supervised learners. In NeurIPS, 20200.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. arXiv preprint arXiv:2202.0330 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04220 2020c.
DermAssist, Identify skin conditions with DermAssist, 2022. URL https://health:
google/ Satogle/consumesee
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint 12. 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. arXiv preprint arXiv:2202.03335 2022.
Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with bootstrap your own latent (byol), 2020. URL /ttp:s://niiteeddaaaaa github. 1o/ understanding--eee superv wwe.conttaatttteeeeeeereee. T
10

---

Under review as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In NeurlPS, 2020.
Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. SSFL: tackling label deficiency in federated learning via personalized self- supervision. CoRR, abs/2110.024700 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. In JCML, 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. CoRR, abs/1909.06335, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In ECCV, 2020.
Sergey loffe. Batch remormalization: Towards reducing minibatch dependence in batch-normalized models. In NeurlPS, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Sung, Zhen Li, and Tom Duerig. Scaling up visual and saiiunla representation learning with noisy text supervision. In ICML, 2021.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Olivera, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Gar- rett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang He. Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi. Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür,, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Scbastian U. Stich. Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1-21) 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
Yuan Liu, Ayush Jain, Clara Eng, David H. Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guil- herme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, Vishakha Gupta, Nalini Singh, Vivek Natarajan, Rainer Hofmmann-Wellenhot Gregory S. Corrado, Lily H. Peng. Dale R. Webster, Den- nis Ai, Susan Huang, Yun Liu, R. Carter Dunn, and David Coz. A deep learning system for differential diagnosis of skin diseases. Nature Medicine, 26:900- 908, 2020.
Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. CoRR, abs/2205. 10000.1249.. 12493, 2022.
11

---

Under review as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--ffiitn learning of deep networks from decentralized data. In AISTATS, 2017.
Lin Ning, Karan Singhal, Ellie X. Zhou, and Sushant Prakash. Learning federated representations and recommendations with limited negatives. CoRR, abs/2108.07931. 2021.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan L. Yuille. Weight standardization.
CoRR, abs/1903. 10520, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Pierre H. Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel L. Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. BYOL works even without batch statistics. CoRR, abs/2010. 10241, 2020.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICMI., 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1800 507.03348, 2018.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat,
Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide feder- ated optimization. arXiv preprint #rXiv:2107.69117 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self- superviuee visual pre-training In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Fedel: Federated contrastive learning for privacy-preserving recommendation. CoRR, abs/2204.09850) 2022a.
Yawen Wu, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, and Jingtong Hu. Decentralized unsupervised learning of visual representations. In IJCAL, 2022b.
Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. CoRR, 01111708.888888 2017.
Felix X. Yu, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Federated learning with only positive labels. In JCML, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yucting Zhuang. and Xiaolin Li. Federated unsupervised representation learning. CoRR, abs/2010.08982. 2020.
Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-supervised learning. In CVPR, 2022.
Weiming Zhuang, Xin Gan, Yonggang Wen. Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In ICCV, 2021.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In ICLR, 2022.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF EQUIVALENCE TO CENTRALIZED TRAINING
Claim: When we perform one step of local training on each participating client in federated training round, one round DCCO training is equivalent to one centralized training step on a batch composed of all data samples participating in the round.
Proof: Let PT and Gj respectively denote the 1t and jth components of encodings F and G of th sample client. Based on the definition of (F get
θ (
8 ((F))
(Fi) = ( StopGradient (FF) (Ft)
(4)
OF
OF.
Similarly, by definitions of (G)) (RGG)) get 0 (((P))))
0 (((R)))
0 (G2)c
a( ((G)) 0((G))²)
OPT
DF
8G
8G7
0 (FAG))c
0 (PPG)))
0 (PGG) (FRG))
(5)
OF
OF.
DG}
By chain rule, we have
@Ccco
aCcco 0(A)c
OCcco (((P)))
aCcco 0 (F.Gg)c
+
OPT
(Fiic
Or:
((F))) OF.
(RRG) ar.
dCcco
dCcco a (G,)c
aLcco a(G))))
dCcco @(RG)) (F.GGi OGT (6)
8G7 (Gjic 8G3
((G))
807
1=1
Substituting Eq. and in Eq.
we get
dLcco
aLcco 0 (Fi)
eccco (((F))))
accco ((AG))
Σ
OF
((Fic OFT
((F))
OF
@(FGg)cc OF j=1
1 aLcco
2
aLcco F. No.
1
Σ
dCcco Cj
Nv ((Flc
NA ((E)))
O
aCcco
aLoco (
aLcco ((G,))
aLcco (F,G)))
0G;
(Giic OGT
a(GG))
0G7
d(FGG)ic OG,
aCcco F. 0(F,G))c 1 iLoco 2 ôLcco Gj No i=1
Σ
N₂ (G,)c
N₂
((G)))
By chain rule, the gradients for encoding model parameters on 4thh client are given by
aLoco
aLcco OF
aCcco OG
ΣΣ
(8)
a0
8G 00
where N₂ is the number of samples on the client that contributed to Leco- Substituting Eq.
Eq.
we get
aLcco No
aLoco +2 +2
aLoco F. +
Σ aCcco
dp"
G
00
N
( ((RP))
(
00
k
j=1
N
aLcco +2 +2 @GG
aLcco a((())
aLoco
0G;
G
F
(9)
N
@(FGG)cc ==
00
13

---

Under review as a conference paper at ICLR 2023
The values of stats ((Fi)) {((E))))
((G))ol {(G) {(F.G))c)
and loss Loco are same on all the clients participating in DCCO training round. So, when each client performs one step of local training and the server averages the model updates from the clients by weighing them according to the number of contributing samples on each client, the equivalent global model gradient update is given by
aLcco 00
1
K
aLcco
N
00
N Σ
d
aLcco OCcco + 0 (
a
aCoco @(
dF
F. Σ (((F))
Gj
+
N
00
R=
j=1
N
d
aLcco + i(G))c ((G))
aLcco Gy + +
eCcco (FFG))
dG
FT
(10)
00
where N is the total number of samples participating in the round.
If all the N samples participating in DCCO round are present on a single client, then according Eq. 9 the gradients computed on that client using these N samples will be same the gradients in Eq. [O Hence one round of federated DCCO training is equivalent to one step of centralized training on a batch composed of all N samples participating in the federated round.
B HYPERPARAMETERS AND DATA AUGMENTATIONS
CIFAR- DATASET
During self-supervised pretraining, used all the data augmentations from Grill et al. (2200) except blur, and during supervised finetuning we only used flip augmentation.
Augmentations:
Federated prefraining: For updating the model on server, used Adam optimizer (Kingma & Ba 2015) with cosine learning rate decay. We experimented with initial learning rates of 5e and Te and report the best results. All models were trained for 100K rounds.
Centralized pretraining: We trained for 1000 epochs using Adam optimizer with batch size 512, initial learning rate of 33, and cosine learning rate decay.
Full network finetuning with 5K samples: We trained for 100 steps using Adam optimizer with batch size 256, initial learning rate 5e and cosine learning rate decay.
Full network finetuning with 500 samples: We trained for 40 steps using Adam optimizer with batch size 128, initial learning rate - and cosine learning rate decay.
Supervised training from scratch: We trained for 100 epochs using Adam optimizer with batch size 256, initial learning rate 0.01, and cosine learning rate decay.
Linear classifier training with 5K samples: We trained for 1000 steps using LARS optimizer You al. 2017) with batch size 512, initial learning rate of 2.0, cosine learning rate decay, and mentum of 0.9.
Linear classifier training with 500 samples: We trained for 400 steps using LARS optimizer with batch size 128, initial learning rate of 2.5, cosine learning rate decay, and a momentum of 0.9.
DERM
Augmentations:
During self-supervised pretraining, we used random rotation and all the data aug- mentations from Grill et al. (2020) except solarization. During supervised finetuning, used all the data augmentations from Azizi et al. (202T).
Federated pretraining: For updating the model server, used LARS optimizer with cosine learning rate decay and a momentum of 0.9. For DCCO pretraining, the initial learning rate was set to 0.15, 0.9 and 1.8 while using 64, 128 and 256 clients per round, respectively. For FedAvg training
14

---

Under review as a conference paper at ICLR 2023
BYOL training
SimSiam training
0.00
Group
-0.25
Group
-0.25
Batch normalization
Betch noraaiiztiin
-0.75
Teming loss
-0.75
-1.00
-1.00
-1.25
-122
-1.50
-2.00
-200
800
100
200
300
400
500
Epoch
(a) BYOL training
(b) SimSiam training
Figure ResNet14 training on CIFAR-100 dataset.
with contrastive loss, the initial learning 256 clients per round, respectively. All models were trained for 75K rounds.
0.15, 0.3 and 0.6 while using 64, 128 and
Centralized pretraining: We trained for 320 epochs using LARS optimizer with batch size 512, initial learning rate of 0.6, cosine learning rate decay, and a momentum of 0.9
Supervised training from scratch and full network finetuning: We used Adam optimizer with batch size of 128. For each experiment, we used the validation split of this dataset to search for the best learning rate (among 3e and 1e 4) and number of training steps (up 100k).
C BYOL AND SIMSIAM WITH GROUP NORMALIZATION
When we experimented with BYOL and SimSiam by replacing Batch Normalization (BN) with Group Normalization (GN), the models did not train well. Figure 3a shows the training loss when R encoder is trained with BYOL approach on CIFAR-100 dataset. When we use GN instead of BN, the loss quickly drops close to its lowest possible value and the model does not learn after that. When we evaluated the BN and GN-based BYL encoders after training them for 800 epochs with batch size of 512, we achieved 41 % and 3% accuracy, respectively, under linear evaluation protocol. This clearly shows that BN is crucial for BYOL to work well. Similar behavior was observed in the case of SimSiam (see Fig. 3b
15


Reviews:
[{"review_id": "s9bD0aRUQq", "reviewer": "Reviewer_G6CA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a distributed implementation of Cross Correlation Optimization (DCCO) loss for contrastive learning, experiments shows the proposed implementation of loss function outperforms some baseline algorithms in the federated learning setting.", "summary_of_the_review": "Overall I feel the novelty is limited and some more metrics in experiments should be reported.", "strength_and_weaknesses": "Strengths is the proposed method outperforms the baseline methods by a large margin in terms of accuracy.\n\nWeaknesses:\n1. The limited novelty of the proposed method, it is simply a distributed way to calculate a loss function involving sum and product operations.\n\n2. I may missed some details but I did not find comparison of communication of the proposed algorithm with federated learning baselines, the proposed algorithm does not utilize multi-step local training while FedAvg style algorithms use it, it is important to also compare the amount of communication when achieving the same accuracy, and the number of local steps of FedAvg style algorithms should be well-tuned.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qQIKecZ0tDY", "reviewer": "Reviewer_RUBX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a domain-aware representation learning method (FedDAR) for the non-iid FL problem. The FedDAR assumes data on clients are from multiple domains and learns a classifier head for each domain. A representation module is shared for all classifier heads and updated by the vanilla FedAvg. The authors also proposed a second-order aggregation method to update domain-aware classifier heads, whose effectiveness is validated by ablation studies. Experiments on both synthetic data and real-world datasets validate the effectiveness of the proposed method.\n \n", "summary_of_the_review": "Please refer to the comments in Strengths and Weaknesses. ", "strength_and_weaknesses": "Strengths\n\n1. The target problem is meaningful in real-world applications for FL.\n\n \n Weaknesses\n\n1. This paper’s contribution is incremental.\n\n2. Too much data sharing may lead to security and privacy concern, which is not discussed in the paper. Communication efficiency is another problem;\n\n3. Eq.2 is not bounded between 0 and 1 when local stats are globally aggregated. \n\n4. One step of FL equivalent to central training will establish for almost all optimization methods, which can not prove the advantageousness of DCCO.\n\n5. The work employed the local loss function of Barlow Twins, but replaced the calculation of C_ij (Eq.2) with correlation coefficient term. How can this replacement lead to better performance? Discussion and ablation experiments are needed.\n\n6. Lack of experiments comparison with other self-supervised FL methods such as [1][2].\n\n[1] Divergence-aware Federated Self-Supervised Learning\n\n[2] Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nl5Qcg0hICb", "reviewer": "Reviewer_j562", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of training dual encoding models on decentralized datasets, which has few existing works. Moreover, the authors consider a challenging scenario that each client possesses a small and non-IID dataset, where directly utilizing the existing centralized methods decreases efficacy. Hence, the authors propose a novel method, Distributed Cross Correlation Optimization (DCCO), to train dual encoding models on decentralized datasets with federated learning. This paper provides several experimental results and proofs to show the advantages of DCCO.", "summary_of_the_review": "This paper proposes a novel method to solve the problem of training dual encoding models on decentralized datasets and proves its advantages on small and non-IID datasets over existing centralized methods. The details of the problem and the methods are clear and easy to understand for readers. However, there are still some problems that affect the fluency of the paper. The figures and the experiments should be supplemented to help readers grasp the idea and prove the generalizability of the approach. Overall, this work is meaningful in the area of dual encoding model training and can be further studied and optimized.", "strength_and_weaknesses": "**Strengths:**\n1. This paper provides a detailed description of the motivation. The authors discover the limitations of the existing methods and solve the problems under the limited environment that is closer to reality, thus making this work challenging and meaningful. \n2. The details of the proposed approach are well supported by texts and graphs, and the experiment settings are proper for proving the advantages of the approach.\n\n**Weaknesses:**\n1. Figure 2 presents the overview of the approach with separate graphs. However, the corresponding relations of the two graphs are vague. Figure 2 can be replaced by a single detailed graph of DCCO and explain the relations in the caption.\n2. For the proposed method, the authors should give the algorithm for readers to understand better.\n3. We would like to see the influence of other environmental parameters on experimental performance, such as $\\lambda$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8XIdKG8pUmf", "reviewer": "Reviewer_xR7P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an algorithm to train dual encoding models in federated learning. The key idea is sharing aggregated encoding statistical information across clients. Experiments were conducted on two benchmark datasets, including CIFAR-100 and Dermatology. The Dermatology dataset consists of de-identified images of skin conditions captured using consumer-grade digital cameras. The results show that the proposed algorithm achieves better utility than some baselines.", "summary_of_the_review": "This paper has limited contribution. The motivation of the work is unclear; the proposed approach is simple without offering a deep understanding of how it works. Experimental results are unconvincing, and a thorough evaluation and analysis are needed.", "strength_and_weaknesses": "The problem studied in this paper is interesting. However, the paper has serval major weaknesses as follows:\n\n- The paper leverage privacy concerns to motivate the work using federated learning (no sharing data). However, the proposed approach shares statistical information from encodings, significantly increasing the privacy risk of the client's local data. Note that federated learning does not automatically offer privacy protection since the server can extract clients' local data from the shared information. Therefore, the motivation for the work is unclear.\n\n- The proposed approach is a trivial adaptation of the existing CCO loss used for training dual encoding models in Zbontar et al. (2021). The technical contribution is not significant. What are fundamental questions or challenging problems the paper aims to address?\n\n- Experimental results are unconvincing. The datasets used are small, and the model utility is shallow (<50\\% in most cases). The practicability of the proposed approach and setting is unconvincing. Who is going to use models with such utility? In addition, the comparison is unfair. The proposed approach used significantly larger data to train the model than a minimal dataset to train the centralized model. That does not show the advantage of the proposed approach. Why don't we gather all the data to train a usable centralized model instead of sticking with a poorly federated learning model? Again, the privacy concerns are invalid in this setting since the proposed approach does not offer any privacy protection. What is the cost of the proposed method in terms of communication and computation? How do the encodings improve after and before using the proposed approach? The paper does not shed light on understanding the core contribution of the proposed approach. Why FedAvg? There are better aggregations to use. That highlights a poor treatment for critical components, such as privacy, model/data utility, practicability, and usability, of the proposed approach given the learning setting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject