FedTMOS: Efficient One-Shot Federated Learning with Tsetlin Machine
Authors: Shannon How Shi Qi
Jagmohan Chauhan
Geoff V. Merrett
Jonathon Hare
Venue: ICLR 2025 Poster (2025)
Keywords: Efficient Federated Learning
One Shot Federated Learning
Tsetlin Machine
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose using Tsetlin Machine for efficient data-free one shot FL without the need for server-side training

Abstract: One-Shot Federated Learning (OFL) is a promising approach that reduce communication to a single round, minimizing latency and resource consumption. However, existing OFL methods often rely on Knowledge Distillation, which introduce server-side training, increasing latency. While neuron matching and model fusion techniques bypass server-side training, they struggle with alignment when heterogeneous data is present. To address these challenges, we proposed One-Shot Federated Learning with Tsetlin Machine (FedTMOS), a novel data-free OFL framework built upon the low-complexity and class-adaptive properties of the Tsetlin Machine. FedTMOS first clusters then reassigns class-specific weights to form models using an inter-class maximization approach, efficiently generating balanced server models without requiring additional training. Our extensive experiments demonstrate that FedTMOS significantly outperforms its ensemble counterpart by an average of $6.16$%, and the leading state-of-the-art OFL baselines by $7.22$% across various OFL settings. Moreover, FedTMOS achieves at least a $2.3\times$ reduction in upload communication costs and a $75\times$ reduction in server latency compared to methods requiring server-side training. These results establish FedTMOS as a highly efficient and practical solution for OFL scenarios.

Published as a conference paper at ICLR 2025
FEDTMOS: EFFICIENT ONE-SHOT LEARNING WITH TSETLIN MACHINE
FEDERATED
Shannon How Shi Qi, Jagmohan Chauhan, Geoff' V Merrett, Jonathan Hare University of Southampton, UK {s how, j. chauhar soton. ac. uk, (gvm, jah @ecs soton. ac.uk
ABSTRACT
One-Shot Federated Learning (OFL) is promising approach that reduce commu- nication to single round, minimizing latency and resource consumption. How- ever, existing OFL methods often rely on Knowledge Distillation, which intro- duce server-side training, increasing latency. While neuron matching and model fusion techniques bypass server-side training, they struggle with alignment when heterogeneous data is present. To address these challenges, Shot Federated Learning with Tsetlin Machine (FedTMOS), a novel data-free OFL framework built upon the cwwcooplexity and class adaativ properties of the Tsetlin Machine. FedTMOS first clusters then reassigns class-specific weights form models using inter-class maximization approach, efficiently generat- ing balanced server models without requiring additional training. Our extensive experiments demonstrate that FedTMOS significantly outperforms its ensemble counterpart by average of 6. 6.16%, and the leading state-of-the-art OFL. base- lines by 7.22% across various OFL settings. Moreover, FedTMOS achieves least 2.3x reduction in upload communication costs and a 75x reduction in server latency compared to methods requiring server-side training. These results establish FedTMOS as highly efficient and practical solution for OFL scenarios.
proposed One-
INTRODUCTION
Federated Learning (FI.) enables collaborative training across decentralized data sources while maintaining privacy. Unlike traditional machine learning approaches, where data is transferred to a central server, FL. keeps the training process on local devices, with only model parameters be- ing uploaded to a central server for aggregation, ensuring that sensitive information is not shared (McMahan et al. 2016). However, the standard FL training process can lead to significant commu- nication costs due to the multiple communication rounds between the server and devices to achieve convergence. This iterative process is time consuming and resource-ing interssveee especially for edge devices with limited computational power and bandwidth (Imteaj et al.) (022). The risk to secu- rity and the potential of failures in communication links also leads further challenges (Mothukuri
ct al. 2021 Li et al. 2020).
To address these limitations, One-Shot Federated Learning (OFL) has emerged as promising al- ternative. OFL restricts communication to a single round, thus minimizing communication errors and reducing the risk of interference caused by iterative updates (Guha et al. 2019). This approach is particularly well-suited for scenarios where continuous communication is impractical, such as in model marketplaces, where models are sold after reaching convergence (Li et al., 2020). OFL is ideal for edge devices with limited resources, as it eliminates the need for multiple rounds of trans- mitting large volumes of model parameters between clients and the server, as well as the need for repetitive on-device training (Khan al. 2021).
Current OFL methods mainly rely on Knowledge Distillation (KD) and ensemble learning, which aggregate local models into an ensemble before distilling it into a global model. A key challenge with these methods is their dependence on public datasets, which may be inaccessible or unsuitable for certain tasks (Li et al.] 1 Guha et al.] 2019). Data-free methods using generative models have been explored (Zhu et al., 2021] Zhang et al.] 2022a Dai et al.) 2024; Zhou et al., 2020), but they often suffer from performance limitations and introduce significant computational overhead.

---

Published as a conference paper at ICLR 2025
Additionally, neuron matching and model fusion techniques, which eliminate the need for server- side training, have been proposed, but they too struggle with performance when models are trained on heterogeneous data distributions,
aligning the models be challenging (Wang et al.) 2020) Singh & Jaggi 2020 Jhunjhunwala et al.). On the client side, most existing methods heavily rely on Deep Neural Networks (DNNs), which resource-intensive and often impractical for clients with limited computational capabilities, such as edge devices.
Firstly, address the computational limitations of training DNNs on the edge, we employ the Tsetlin Machine (TM), which features a low-complexity architecture based on finite-state automata and game-theoree theoretie principles (Granmo 202T). This makes TMs a highly efficient alternative to tradi- tional DNNs, as they have demonstrated the capability to significantly reduce communication costs per round by at least 1.4x and improve storage efficiency by at least 6.6x compared to Binary Neural Networks (BNNs), all without sacrificing performance (How et al. 2023).
Furthermore, to enhance the efficiency of the OFL process, we eliminate the need for server-side training for KD and the reliance leverages the class-adaptive nature of TMs by generating server models that maximize inter-class model separation, all without requiring server-side training. Our experiments on various datasets demonstrate that our proposed method outperforms statc-of-the-an baselines by an average of 7 7.22% across all settings. Additionally, it significantly reduces upload communication costs by at least 2.3x. By incorporating TMs into OFI,, we aim to provide a computationally efficient, scalable, and practical solution for FL scenarios where computational resources are limited.
auxiliary dataset. We proposed a novel data-free solution that
Our contributions are as follows:
We introduce FedTMOS, a novel OFL approach that capitalizes on the low-complexity de- sign of the TM. By leveraging TMs, FedTMOS operates data- free method that achieves a reduction in upload costs by at least 2.3x, while delivering an average accuracy improve- ment of 7.22% relative the top-performing baselines and 12.79% improvement over state-of-the-art data-free OFL. methods.
FedTMOS employs a unique inter-class weight separation technique that effectively create server models that enhance class distinction. This approach consistently outperforms its ensemble counterpart by an average of 6.16% across the evaluated datasets, while simulta- neously reducing both model size and performance instability by 1.63%.
Our approach significantly reduces server-side computation, achieving least a 75x re- duction in server latency during model aggregation compared to existing methods that rely on server-side training. This efficiency makes FedTMOS highly practical for OFL or situ- ations demanding quick model deployment.
RELATED WORK
2.1
ONE SHOT FL
OFL improves efficiency in FL by limiting communication to single round. This is typically achieved by building ensemble models and applying KD using public datasets (Guha et al. 2019
Zhou et al. 2020). However, its effectiveness depends on the quality of both the dataset and model ensemble. To enhance performance, Li et al. (2021) applied hierarchical KD, and Diao et al. (2023) introduced open-set voting to generate 'unknown' samples without predefined classes. These meth- ods rely on public data, limiting their use in data-scarce settings. In response, data-free OFL tech- niques have emerged, including DENSE, which utilizes a generator trained with an ensemble of client models (Zhang et al.) 2022a). Dai et al. (2024) further improves the performance of the global model by optimizing through KD derived from both the synthetic data and the ensemble model. Other approaches include local clustering techniques, such as those by Dennis et al. (202T), which involve uploading cluster means instead of full models. While this reduces communication over- head, it struggles with increased data complexity. (Heinbaugh et al. (2023) introduced Conditional Variational Autoencoders (CVAEs) to learn conditional data distributions from clients, with the ag- gregated decoders forming an ensemble model. Several techniques aim to improve server efficiency by avoiding server-side training altogether. Neuron matching aligns model weights (Singh & Jaggi 2020 Wang al. 2020), while model fusion combines multiple models into single robust one
2

---

Published as conference paper at ICLR 2025
Jin al. 2023). These methods uses optimal transport (Singh & Jaggi 2020 regularization (Jin
et al. 2023 and Fisher Information @Jhunjhunwala et al.), facilitate model aggregation without the overhead of KD, enhancing scalability in OFL. However, these methods often struggle with hetero- geneous data, limiting their effectiveness.
2.2 TMS IN FL
FedTM is the first FL framework to utilize TM. Unlike traditional FL frameworks that rely on DNNs, where weight aggregation typically involves a simple weighted average of integer weights, FedTM uses a distinct two-step aggregation process (How al. 2023). This process enabled by the unique architecture of the TM, depicted in Figure 2 While FedTM offers significant reductions both memory utilization and communication costs, its aggregation process still requires multiple communication rounds to achieve convergence. To our best knowledge, TMs have not been explored the context of OFL.
PRELIMINARIES
3.1
THE TSETLIN MACHINE
The Tsetlin Machine (TM) is machine learning method grounded in propositional logic and bit- based representation, leveraging Tsetlin Automata (TA) and game theory principles to derive logical propositions for classification Granmo 2021
3.1.1 THE TSETLIN AUTOMATON
The Tsetlin Automaton (TA) is an efficient and simple learning mechanism. With 2A states, and two actions, Include and Exclude, the automaton adjusts its state based on feedback by either incre- menting or decrementing the states based on received feedback (Penalty or Reward). From Figure when reward is given, the TA advances further along the path of the current action. If it receives penalty, it moves closer to the center, which may lead to a change in action.
Penalty Reward
A+1
A+2
Include
Exclude
Figure 1: two-action TA with 2A states
3.1.2 THE TSETLIN MACHINE STRUCTURE AND INFERENCE
The first step in training TM to convert the input features into Boolean format. The input feature vector = .., J. (0. 1) is transformed into set of literals, L (I,..., 12. = = ,, Tos - ........ }. which includes both the original features and their negations. Each clause Cj in the TM selects subset of these literals, denoted as L5 C L. Here, indexes the clauses, and Ly refers to the set of literals chosen by clause C, to form conjunctive expression.
The TM organizes its clauses into two groups: positive and negative clauses. For the given input x = (Z1...., }. and with binary class labels, y E {0, 1}, the TM computes a unit step function, u, to determine the final classification output. If the signed sum s(x) is negative, the TM classifies the output as = 0, if not classifies as 9 1:
N/2 ý u((s))) w E
N Σ
CF(xx)
j=N/2+1
In the Multi-Cis Class case, each class has its own set of clauses and the final classification is the class with the highest sums: ÿ = arg maax,=1.... Sm (x). As shown in Figure 2 the TM utilizes bitwise operations to compute its output, resulting in design that is both intuitive and low in complexity.
3

---

Published as a conference paper at ICLR 2025
Multi-Class Weighted TM
Class 0
Class
Class
CT
Gayz
Cj
GT
Gayz
Cayzas
Clux
Gayas
Gy
0
Class Sums: Six (x) Enr 8,07
21-N/201 B,G
Classification: y gmaxxx--.. 1., (m))
x1
Boolean Literals:
1010 10
Clause Computation
Clause Output: A-x2
Exdide
Includo
Booleanization
x2 94
Row Input Features
Figure 2: Overview of the Weighted Multi-Class Tsetlin Machine Inference Process: Raw input features are first booleanized to generate boolean literals. Clause outputs are produced through logical operations with these literals and trained TAs. The inference routine sums up the N clause outputs, weighted by their respective weights, to generate class sums for M classes, with the output determined to be the class with the highest sum (Maheshwari et al. 2023
3.1.3
THE TSETLIN MACHINE LEARNING MECHANISM
The TM operates in an online fashion, learning one training example (x, y) at a time. For each training example, the TM adjusts the states of its TAs by applying rewards and penalties, which involve incrementing or decrementing their states. Feedback to the TAs comes in two types:
Type which helps identify frequent patterns is applied randomly to clauses based on their polarity and the true label. Positive clauses receive Type feedback when y and the negative clauses receive when = O.
Type II is also given stochastically to clauses to enhance pattern discrimination. Positive clauses receive Type Feedback when y = 0, and negative clauses receive it when 1.
The feedback is applied randomly to introduce ensemble effect, guided by hyperparameter T, which represents the target sum. The TM aims to adjust the signed sum such that it reaches -T' for inputs with class y = and T for class y
facilitate this, s(x) is constrained within [-T,T] with: c(x) = clamp(s(xx) -7, 77.7) T) and the probability that each clause receives feedback is proportional the difference between the clamped sum and T:
T+c(x) 2T
if y= 0
(x) =
(1)
T T-((x)
if yf
Randomly selecting clauses helps distribute feedback across a variety significant sub-patterns instead of concentrating few. Feedback decreases the clamped sum, c(x) approaches the target +7,, ensuring that only some clauses are used to identify each sub-pattern.
3.1.4 WEIGHTED TSETLIN MACHINE
During learning, similar clauses tend to appear multiple times in the final model. Introducing weights allows each clause to be represented once with an associated weight, rather than repeat- ing it. Hence, the impact of individual clauses be quantified, resulting in real-valued quantity.
Initially, all weights are set to and they are updated based on the feedback type. In summary, Type feedback increases weights for correct pattems, while Type II feedback decreases weights to reduce false positives. This enhances the efficiency of the TM by optimizing weight assignment to clauses, leading to compact models without compromising accuracy (Phoulady al. 2020
4

---

Published as a conference paper at ICLR 2025
The resulting overall sum now, denoted s(x), becomes:
N/2 5(x) - Σ (x)
N Σ 0f j-N/2+1
(2)
3.1.5 CONVOLUTIONAL. TM
Convolutional TM (CTM) uses filters with spatial dimensions W x W and binary layers. An image with dimensions X x Y and Z binary layers is represented in TMs using an input vector x = {æx & E {0, XXXXXXXXX x2 The input vector represents a specific patch of the image, with the entire image being divided into B such patches. Each clause receives B literal inputs and unlike regular TMs, where a single output is produced per clause, a clause in a CTM outputs B outputs one for each patch. To aggregate these multiple outputs from clause j, cp..,, of into a single output, Cj. logical OR operation is applied: Cj = -V
Training the CTM extends the TM's learning process by using both Type and Type II feedback mechanisms. To update a clause during training. the CTM randomly picks patch from the set of patches where the clause evaluates to I, where [X = 1, S S B). The clause is then updated according to the selected patch, allowing the learning process to focus the most relevant regions that contribute to the clause's outcome (Granmo et al. 2019).
3.1.6 TM COMPOSITES
To enable collaboration among multiple independently trained TM models, Granmo (0023) pro- posed TM composites. Given TMs, this involve computing the normalized class sums for each TM, t by dividing it with the difference between the maximum and minimum class sums in the input set: a' maxm (s' (x)) minm (x))
The final classification result is obtained by selecting the class with the highest value from the sum of all TMs computed below:
1 arg Σ E a s (x)
TM composites enhance classification accuracy and convergence by reducing over-fitting and in- creasing robustness among individual models, thereby improving overall performance through ef- fective model combination (Granmo) 2023).
4 METHODOLOGY
Given J clients, each having local datasets D1, D2. Dy. The objective aggregate the local TM models, T (Ti T,...., Ty}. into & server models (ø < JJ) that generalizes well over D = User D; one communication round.
4.1 MOTIVATION
Similar to ensemble methods for OFI, we first introduce straightforward approach for aggregating local TM models, FedTMOS (ensemble), by applying the principles behind TM Composites. In this approach, the final classification is then determined by Equation
However, this method encounters significant limitations. Firstly, the number of clients, J in- creases, the number of local models grows proportionately, leading to issues related to redundancy and overlapping knowledge. This results in diminishing classification performance due conflict- ing information among models. Additionally, the increased number of models adds computational overhead, making the OFL process less efficient.
5

---

Published as a conference paper at ICLR 2025
4.2 FEDTMOS: ONE-SHOT FEDTM
To address these limitations, we proposed a novel approach based on the class-adaptive nature of TM as illustrated Figure Clauses, as simple logical expressions, collectively represent complex patterns, with their associated weights quantifying these patterns. Our method aims to reduce the total number of models user-defined value @ where the final classification is determined by:
Σ
1 mx()x)
ÿ = aug MaxX ar
T7
where ④ < J. Our proposed approach consists of two stages: first the weights, represented by [0,, ..,.... 0, where each 0j - {0) -f. 0F....,,'' 0M)) is Misize vector of class-specific weights for client j. and M denotes the total number classes, are scaled based on the average normalized Gini index of all clients (r)), followed by k-means clustering (Lloyd T982). Then, perform greedy reassignment of model weights to ø number of models to minimize overlap.
Our approach is intuitive: by maximizing inter-class separation within models, model's ability to distinguish between classes. The Gini index is employed to quantify the distri- bution of data and ensure balanced client participation. dynamically adjusts client weights based on data inequality, promoting fairness and engagement across clients (Li et al. 2023). This, in turn, enhances both model performance and convergence.
enhance the
Server:
Clients:
Client
Class
Class
Class
Qlea M SF
Class
Class
Class
C?
C
0
C
Client
Model Ø
Class
Class
Class
Class
Class
Class
c
Classh
Dlus
C
Ai
G
e
CY
Figure 3: Overview FedTMOS: ! Clients upload their scaled clause weights, state parameters, and normalized Gini Index to the server. 2 The server then rescales the weights using the mean normalized Gini Index and performs k-means clustering on the weights. This clustering is essential for grouping similar weights, helping prevent large disparities in class weights across models during the reassignment process while also reducing complexity. 3 Finally, the number of models, denoted by user-defined parameter ø, is initialized, and class weights from each cluster greedily reassigned to maximize inter-class separation within each model.
4.2.1 PRE-PROCESSING MODEL WEIGHTS
In the initial step, clients upload their scaled clause weights, which are adjusted based on the pro- portion of samples per class relative to the client's 's total sample size. This scaling expressed as:
D D
03
0
where 03 represents the clause weight for class in client j. |D]] is the number of samples for class i in client j. and DDy is the total number of samples client j. This ensures that the contributions of different classes are appropriately balanced, preventing over-repreeentatiio of any particular class.
Alongside these weights, clients also upload their individual normalized Gini index, which quantifies the inequality in their local data distributions (Tangirala, 2020). The Gini Index for client j is

---

Published as a conference paper at ICLR 2025
1200
120
Weightt Magnitude
1900
1533
800
400
10
10
Class
Class
(a)
(b)
Figure 4: Distribution of clause weights for each class stays within the range of T with similar means for (a) data with low normalized Gini scores compared to (b) data with high normalized Gini scores for simple TM model with T 1000 on the MNIST dataset.
computed 9ý >i P. where p; is the proportion of data belonging class To account for imbalances in the server model, we scale the clause weights of all clients, O using the normalized Gini Index, x E 9j. This scaling adjustment is applied only when exceeds predefined threshold, signaling significant inequality. The adjusted weights are updated as:
0j x
This adjustment prevents snder-representation of minority classes, which could otherwise be over- shadowed by dominant classes due to imbalanced clause weights as seen in Figure 4 Without correction. these class imbalances, amplified by the signed sum clamping in TM within [-T,T] can skew the model by decreasing the contribution of the other classes in the server model.
Next, we apply k-means clustering to group these scaled weights. This crucial as clustering similar weights ensures smoother and more balanced reassignment process, preventing large disparities in class weights across models. The objective k-means to iteratively minimize the within-cluster sum of squares (WCSS), defined as:
k
arg min
Σ BEV
00 14/2
(4)
where is the number of clusters, represents data point (i.e., vector of scaled weights from any client j), V, is the set of points assigned to cluster and μu; is the centroid (mean) of cluster i. Performing k-means on the scaled weights reduces the complexity of the subsequent reassignment process, while ensuring that the class weights within models are not largely skewed.
4.2.2 REASSIGNING WEIGHTS
In the initial step. we use k-means clustering to group scaled weights into k clusters, minimizing the WCSS. This approach clusters similar weights, enabling balanced reassignment and reducing the risk of over-fitting or under-representtng certain classes.
Building on this clustering foundation, we reassign the weights models. During this step, aim to maximize the average squared distances between the cluster centroids that represent different classes within each model. This step is critical because, our goal is to select class weights from the most distinct clusters to ensure that the models have clear class boundaries and better generalization. By doing so, we encourage greater separation between class representations each model, which helps to improve the model's 's class differentiation ability.
The objective function for maximizing the inter-cluster distance as follows:
maximize
(5)
where in and uei are the centroid means for classes and within model m that correspond to the clusters which the class weights belong.

---

Published as a conference paper at ICLR 2025
This emphasis inter-class separation ensures that each model maintains distinct class boundaries, thereby enhancing the model's 's classification performance.
Starting with the clusters containing the fewest classes, assign class weights to models based on the assignment that yields the greatest increase average squared distance (Equation3). If no model satisfies this criterion, allocate the weight the model with the fewest distinct class weights. In the event of ties, we select the model with the least overlapping clusters.
To form the final models, compute the mean of the clause weights for each class. Since clause weights do not affect the TA state updates (Granmo et al. 2019), the aggregation of TA states can occur independently. To prevent the states, represented by bits, from including all literals due to excessive aggregation, we apply the TopK procedure from FedTM (How et al. (023) with K = 2. This involves selecting only the two states from the clients with the highest number of samples class, indicating greater confidence, and combining them using the bitwise OR operator. This approach prevents problem of overly inclusive and redundant state representations.
Our method maximizes inter-class separation and ensures adequate class coverage, resulting in sell-separated models with distinct class representations. This enhances the models' effectiveness in handling data distribution variations and improves class differentiation, yielding scalable, high- performing models for classification tasks. The overall algorithm is summarized in Algorithm T
Algorithm FedTMOS
Input:
0,} and states Clients' scaled local TM model clause weights {01. .......0.)
Cs..C...... C,}, E ... Ty their normalized Gini scores, (91 92.....J11 number of clusters l for k-means clustering and number of final models Φ, scaling threshold, a Initialize: all_ allghtigh,, all_states = []. []
Compute mean normalized Gini scores: T = -'= G
Set G if G >
if S
for each client j = 1.......JJJ for each class m 1,2....,
do ......,M M do Add 07 -x: to all wweights and C.Mh all_states cluster info, cluster pluster_means- _means- = ammaans(all_weighee k)
pordereed_mddet reordered.meant final models average models(reordered ) models) return final models
weights(cluster.info cluster.means
4.3 EXPERIMENTS
4.3.1
EXPERIMENTAL DETAILS
Datasets. We evaluated our approach on four image datasets widely utilized in FL literature: MNIST
(Deng 2012), F-MNIST (iiao et al. 2017). SVHN (Netzer et al. (011) and CIFAR-10 10 (Krihheskyy 2009 To simulate heterogeneity. we applied two different methods: sampling class priors from a Dirichlet distribution, Dir(a), as described in Hsu al.) (2019), where controls the degree of heterogeneity in data splits. We also distributed data such that each client possesses samples from only 3 classes, S(3). In our experiments, we simulated non-IID settings by using a 0.05, 0.1, 0.3 and B = 2,3, 3, 4.
Baseline Methods. We compared our method with several FL algorithms representing different approaches for model aggregation: FedAvg (McMahan et al. 2016). the standard iterative FL al- gorithm evaluated after one communication round; One-hone which ensembles client classifier predictions (Guha et al. 2019); and Distilled-FedOv. which enhances performance using open-set voting and a public dataset for KD (Diao et al.) 2023). DENSE and Co-Boosting, both Rata-free methods, generate data for KD: DENSE by leveraging similarity, stability, and transferabilityy (Zhang) et al. 2022a). and Co- no-Boostiig by creating hard samples and re- ne-weighting client models (Dai et al., 2024 Additionally, we compared with three data-fres methods that do not involve server-side train- ing OT-Fusion, which employs layer-wise model merging method using optimal transport to align neurons and weights (Singh & Jaggi 2020); RegMean, which minimizes the 12 distance to individ-
8

---

Published as a conference paper at ICLR 2025
Table I: Performance of the different algorithms across various data partitions
Dusset
Panttico
Fedlog
FSd OueSttu
DENSE
Co- Boosting
Duniler
Feddy
CT.Pasoo
RegMeeu
FedFasher
(Eascatiie
PearsSOs
FeSTMOS
12.57613.00
27.80
58.79214.19
89:14:0067
30.2664774
7666424557
27.144865
$7.11411.31
93,00-381
#1 18Th
DO D010ETTT 71
XT
NM
023
94.1=2.99
978720115
M841:8:10
1697
$7.30
14.4
35
05.25:
S0
S1.S3ER000
12:0-11:55
REASE731
7002:100
R5122504
96.34
31A15331
3502299
18.11106
15:45:335
FMNNSST
11.98611.44
11:101 100
80.24-2200
13.55-265
789461669
63.24-0.58
76.4044112
25.166
75
122511005
84.92-1.30
1.30
4570-533
60
46072001
25.08
35.00:00
15.055 1000
15.0
25
06 25
00.92333
SVHN
D2703
52,905.77
21.99533
12557:645
55A52805
76005600
71532230
71375621
2L
76002104
75361331
01A3:111
7
31.314-1.10
47.48-526
439645447
SUAPHAIT
7458-075
296767-82
54,621673
57.254396
72834022
78.9:-925
MD
36.3
75-24
96.9 5.6
156
50.35:1.71
CIFAR
25
21918631
24
56981035
813
254450075
3583-405
291323326
3321134000
45642214
24.032277
310661336
923632415
51.330042
52,1K:873
ual model predictions using inner product matrices of layer inputs (Jin al. 2023); and FedFisher, which approximates Bayesian inference using Fisher information by using local posterior estimates and Fisher matrices from clients (Jhunjhunwala et al.). These baselines were selected to evaluate the performance with and without reliance on external datasets or excessive computational overhead.
Configurations. Our experiments default to 10 clients with Dir(0.1), 1), as Zhang et al. (2022a); Dai et al. (2024), unless stated otherwise. We reported the average test accuracy and standard devi- ation across different dataset splits for each setting. Following Diao et al. (2023), half of the test dataset served as public dataset for Distilled FedOV, and we evaluated all algorithms subset, ensuring consistency across evaluations. For CNN-based al algorithms, we used a 5-layer CNN architecture with batch size of 128, per Dai et al. (2024) and Jhunjhunwala et al.] For FedT- MOS, we fixed = 30 for the Dir partitions to account for class distribution variability among clients and A = 10 for the S partition. We implemented different CTM models for each dataset, ensuring the size of & server models was equal to or smaller than the distilled CNN server model for fair comparison. All algorithms were trained for 30 local epochs, as outlined in Jhunjhunwala al., See Appendix
the same
A. for more details.
From Table I FedTMOS outperforms all baseline methods without the need for synthetic or gener- ated data and server-side training in all settings. On average, FedTMOS performs better than the top performing baseline, Distilled-FedOY by 13.3%, 3.67%, 3.13% 8. 8.82% on the MNIST, F-MNIST, SVHN and CIFAR-10 10 dataset respectively. We note that Distilled FedOV leverages KD at the server using a subset of the test data, closely matching the actual samples. On average, FedTMOS outper- forms the best data-free OFI. method by 12.79%, reinforcing its effectiveness in OFI.
We note that DENSE and Co- Co-Boosting exhibit high variance, particularly with increasingly non- IID data. This is likely because classifiers trained on non-lID data generate sub-optimal knowledge samples, which are crucial for the distillation process. This hinders the student model's learning. reducing performance and increasing variability (Gou al. 2021).
We also evaluated FedTMOS against the FedTMOS ensemble. On average, FedTMOS demonstrates 6.16% better performance across all settings, while reducing variation by average of 1.63% across all settings. Furthermore, FedTMOS exhibits lower memory costs as it reduces the ensemble model size to é number of server models, making it an efficient solution. Overall, these results highlight FedTMOS as a reliable approach for data-free OFL.
4.3.2 EFFICIENCY ANALYSIS
In terms of communication costs, our results demonstrate that the bit-based architecture of the TM significantly reduces communication overhead while producing a model that generalizes well across all settings, evident in the results in Table 2 By ensuring that the server model size of the TM is equal to or smaller than the CNN models, we show that FedTMOS effectively reduces upload costs by at least 2.3x x without sacrificing performance.

---

Published as a conference paper at ICLR 2025
Table 2: CC: Upload costs for each client/storage and potential download costs of the server model
(MB); SL: Server aggregation latency (s).
Distilled- FedOV
Dataset
Metric
DENSE
Co-Boosting
OT-Fusion
RegMean
FedFisher
FedTMOS
CC
0.25/0.25
0.18/0.18
0.04/0.16
MNIST
SL.
647+3.42
738+5.76
14.86
0.71+0.05
2.49+-00.34
1..06
+.48
CC
0.25/0.25
0.18/0.18
0.05/0.14
F-MNIST
SL
624=2.75
835=3,38
317=7.53
0.7840.03
2.96=0.19
11.2=1106
0.52==0.14
CC
1,28/1
3,76/3.76
0,31/1,23
SVHN
SL
69413.40
1238:14.52
96217.18
2.2911.11
8,19211.62
13.23:1142
1.21::0:25
CC
1.28/1.28
3,76/3.76
0.57/1.24
CIFAR-10
SL
93123351
124717.45
246:13.95
2.04.1.14
6.93
9.98.0.93
3,2410017
We evaluated the average latency for model aggregation at the server using standard compute node equipped with single GPU core. FedTMOS consistently outperforms other algorithms by achiev- ing lower server aggregation latency. However, for the CIFAR-10 dataset, OT-Fusion performs slightly better, with a latency that 1.2 seconds faster than FedTMOS. Despite this, FedTMOS still achieves, on average, 20% better performance than OT-Fusion on CIFAR- 10, suggesting that FedTMOS is more efficient overall, even if its latency marginally higher this specific case.
Overall, these results indicate that FedTMOS has significant potential for efficiency, attributed to its low upload costs, which is essential for edge devices with limited bandwidth. Furthermore, reduced aggregation latency ensures that the OFL process remains efficient, regardless of the computational resources available on the aggregator server. This is particularly important in edge-based FL, where an edge server acts
aggregator managing nearby edge devices. The absence of server-side training contributes to quicker deployment in time- sensitive scenarios and allows for the extension to iterative FL while maintaining efficiency (Zhang al. 2022b).
4.3.3 SCALABILITY
We evaluated the performance of all methods by varying the number of clients. For FedTMOS, to constrain the server model size, we reduced the number of clauses in the local CTM model, scaling down the local model size by 2.9x for 20 and 50 clients, and 4x for 80 clients. From Table 3 we see that FedTMOS outperforms all other methods, although its performance declines with an increasing number of clients. This suggests that while reducing model size is necessary, it can limit performance. Future work will explore dynamic scaling techniques for the number of server models, o, and adapting the number of clusters, k, to better balance complexity and class coverage. Additionally, advanced sampling methods can be used to manage the increasing number weight vectors for reassignment, potentially enhancing performance with increasing clients.
Table 3: Performance of the different algorithms with an increasing number of clients on CIFAR-10
Clicats
FedAvg
Fed. OneShot
DENSE
Co- Co-Boosting
Distiled. RedOv
OT-Fusšon
RegMean
FedFisher
FoTMMS
26.42:73
185544992 31.441994 1.85
50
23.34
34.611360
27.4711777
24.31-208
30.4511.56
27.171136
37.5210:37
1.39010.94
23.04+5..11
25.04 1306
23.85+1170
32.47+236
25.65+1.044
20.86+2225
33.66+00.55
49.15+ 1,08
5 CONCLUSIONS
We introduced FedTMOS, a novel framework for OFL that eliminates the need for server-side train- ing for KD and the use of synthetic or generated datasets. By scaling weights with normalized Gini scores, clustering parameters, and maximizing inter-class model separation, FedTMMS reduces computational complexity, constrains server model size, and minimizes variance in performance. Notably, it surpasses all SOTA baselines by at least an average of 7.22% and the best data-free method by 12.79% across all dataset settings. Furthermore, it achieves a reduction in upload com- munication costs by least 2.3x, making FedTMOS well-suited for FL with edge devices and providing a strong foundation for further exploration into enhancing the efficiency and performance of OFI.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the UK Research and Innovation (UKRI) Centre for Doctoral Training in Machine Intelligence for Nano-electronic Devices and Systems [EP/S024298/1] and the Engi- neering and Physical Sciences Research Council (EPSRC) International Centre for Spatial Compu- tational Learning [EP/S030069/I]
REFERENCES
Hong-Yoo You Chen, Cheng-Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the Impor- tance and Applicabilityy of Pre- Trainiinn for Federated Learning. In International Conference on Learning Representations, 2023.
Rong Dai, Yonggang Zhang, Ang Li, Tongliang Liu, Xun Yang, and Bo Han. Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting. In International Conference on Learning Representations, 2024.
Li Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research (Best of the Web]. IEEE SPM, 29(6): 141-142, 2012. doi: 10.100//MS:201222214777
Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogemeity for the Win: One-Shot Federated Clustering, 2021. URL https: http:p////aaail..cccc//////....0
Yiqun Diao, Qinbin Li, and Bingsheng He. Towards Addressing Label Skews in One-Shot Federated Learning. International Conference on Learning Representations, 2023.
Jaaniing Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge Distillation: A Survey. International Journal of Computer Vision, 1573-1405, doi: 110.110//111-00000101011111 s11263- 02123-021--1453--
March 2021. ISSN
URL httt:://ax..ci..rr/11....
del. 10077
Ole-Christoffer Granmo. The Tsetlin Machine A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic, 2021.
MheCChristoffer Granmo. TMComposites: Plug-and-Play, Collaboration Between Specialized Tsetlin Machines, 2023.
Ole-Christoffer Granmo, Sondre Glimsdal, Lei Jiao, Morten Goodwin, Christian W. Omlin, and Geir Thore Berge. The Convolutional Tsetlin Machine, 2019.
Neel Guha, Ameet Talwalkar, and Virginia Smith. One-Shot Federated Learning, 2019. URL
11175
Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao. Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity. In The Eleventh International Conference on Learning Representations 2023.
Shannon Shi Qi How, Jagmohan Chauhan, Geoff Merrett, and Jonathan Hare. FedTM: Mem- ory and Communication Efficient Federated Learning with Tsetlin Machine. In 2023 Interna- tional Symposium on the Tsetlin Machine (ISTM), pp. 1-8, 2023. doi: 10.1100/ISMM588889 2023. 10454904
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification, 2019.
Ahmed Imteaj. Urmish Thakker, Shiqiang Wang, Jian Li, and M. Hadi Amini. A Survey Feder- ated Learning for Resource-Constrrinne loT Devices. IEEE ToT-J, 9()):1224, 2022.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. FedFisher: Leveraging Fisher Informa- tion for One-Shot Federated Learning.
Xisen Jin, Xiang Ren, Daniel Pecotiuc-Piettee and Pengxiang Cheng, Dataless Knowledge Fusion by Merging Weights of Language Models, In International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at ICLR 2025
Latif U. Khan, Walid Saad, Zhu Han, Ekram Hossain, and Choong Seon Hong. Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges. IEEE Communica- tions Surveys & Tutorials, 23(3):1759-1799 2021. doi: 10. (0.119//CMSS.22211390003
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009. URL https:
//api. www.eentttccoooo.ooorooooo/ooo/e
182668744
Ya Le and Xuan S. Yang. Tiny ImageNet Visual Recognition Challenge. 2015.
Jie Lei, Tousif Rahman, Rishad Shafik, Adrian Wheeldon, Alex Yakovlev, Ole-Christoffer Granmo, Fahim Kawsar, and Akhil Mathur. Low-Power Audio Keyword Spotting Using Tsetlin Ma- chines. Journal of Low Power Electronics and Applications, (2). ISSN 2079-9268. doi:
10.3390/jlpeal 1020018.
Cross-Silo Setting. In International Joint Conference on Artificial Intelligence, 2021. Qinbin Li, Bingsheng He, and Dawn Xiaodong Song. Practical One-Shot Federated Learning for
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020. doi: 10.1 10.10099/
Xiaoli Li, Siran Zhao, Chuan Chen, and Zibin Zheng. Helerogeneity-aware fair federated learning. Information Sciences, 619:968-9888 2023.
S. Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2): 129-137, 1982. doi: 1.111999TT.1192.108000
Sidharth Maheshwari, Tousif Rahman, Rishad Shafik, Alex Yakovlev, Ashur Rafiev, Lei Jiao, and Ole-Christoffer Granmo. REDRESS: Generating Compressed Models for Edge Inference Us- ing Tsetlin Machines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 11152-11168, 2023.
doi: 10.1109/PA....223... 2266.13.
H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-Efiiei Efficient Learning of Deep Networks from Decentralized Data. In A/STATS, 2016.
Viraaji Mothukuri, Reza M. Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. survey on security and privacy federated learning. Future Genera- fion Computer Systems, 115:619-640, 2021. ISSN 0167-739X. doi: https://doi.org/10..r 1016/
j.future.2020.1000 10.007 URL https: /tttpps:/www.... wwt.:.acieneeiiecc.o wttp:rrrtttcccccceeeeeeeeemeeeee www.///ceeen///aarieeee... 041/S0167739X200
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeuRIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael G. Rabbat. Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning, In International Conference on Learning Representations, 2023.
Adrian Phoulady, Ole-Christoffer Granmo, Saeed Rahimi Gorji, and Hady Ahmady Phoulady. The Weighted Tsetlin Machine: Compressed Representations with Weighted Clauses, 2020. URL
hntpss:/aaze //axiv/ http:///aaavvv..///abs//////////
12607
Tousif Rahman, Adrian Wheeldon, Rishad Shafik, Alex Yakovlev, Jie Lei, Ole-Christoffer Granmo, and Shidhartha Das. Data Booleanization for Energy Efficient On-Chip Learning using Logic Driven AI. In 2022 International Symposium on the Tsetlin Machine (ISTM), pp. 29-36, 2022.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Infor- mation Processing Systems, 33, 2020.
Chong Tang, Neelam Singh, and Jagmohan Chauhan. AdaTM: logic inspired adaptive Tsetlin ma- chines for efficient and effective continual learning on the edge. In 21 International Conference Embedded Wireless Systems and Networks, 2024.
12

---

Published as a conference paper at ICLR 2025
Suryakanthi Tangirala. Evaluating the Impact of GINI Index and Information Gain Classification using Decision Tree Classifier Algorithm". International Journal of Advanced Computer Science and Applications, 11, 2020.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated Learning with Matched Averaging. In International Conference on Learning Repre- sentations, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Mishion-MNIST:: Novel Image Dataset for Bench- marking Machine Learning Algorithms, 2017.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: Data-Free One-Shot Federated Learning. In Advances in Neural Information Processing Systems, volume 35, pp. 21414 21428, 2022a.
Zhenxiao Zhang. Zhidong Gao, Yuanxiong Guo, and Yanmin Gong. Scalable and Low-Latency Fed- crated Learning with Cooperative Mobile Edge Networking, 2022b. URL https://arxiv.
org/abs/2205 13054
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Oliver Wu. Distilled One-Shot Feder- ated Learning. ArXiv, abs/2009.07999 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 12878-12889. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL EXPERIMENTAL DETAILS
A.1.1 BASELINES MODEL CONFIGURATION
For the baseline models, we adhered to the same hyperparameters specified in the original papers Jhunjhunwala et al.) Zhang al. 2022a Dai et al. 2024 Diao et al. 2023). However, to ensure fair comparison, we fixed the number of server epochs to 100 epochs for the bascline algorithms Distilled FedOV, DENSE and Co- Co-Boosting.
A.1.2 FEDTMOS MODEL CONFIGURATION
To align with the booleanized format for training TMs, we applied variouos pre- pre-prcessiing steps for each dataset. For the MNIST dataset, we encoded the data by setting pixel values larger than 75 to I, and values below or equal to 75 to O. For the F-MNIST and SVHN dataset, we followed the original implementation by Granmo et al. (2019), to binarize the data using an adaptive Gaussian thresholding procedure with window size of 11 and threshold value of 2. For CIFAR-10, as described in Granmo (2023), we processed one copy of the data using the same adaptive Gaussian thresholding procedure and another copy with 8-level color thermometer encoding. Additionally, employed three TM composite models for CIFAR- 10: adaptive thresholding, 3x3 color thermometer, and 4x4 color thermometer.
For FedTMOS, we used the following model configurations of CTM for each dataset:
Table 4: FedTMOS Model Configuration
MNIST
F-MNIST
SVHN
CIFAR-I0
Adaptive Threshold
3x3 CT
4x4 CT
Number of Clauses
100
200
1100
200
200
200
Feedback Threshold
1000
1000
2000
400
300
300
Learning Sensitivity
5
5
5
5
5
5
Patch Dimensions
(10,10)
(5.5)
(5,5)
(10,10)
(3,3)
(4,4)
A.1.3 ADDITIONAL SETTINGS
For FedTMOS, as we wanted to constrain our models such that the final server model aligns with smaller or equal sizes to the CNN counterparts, set various values constrain our final model. For the MNIST, F-MNIST, SVHN and CIFAR-10 dataset we used: á S (4, 3, 3, (3, 3, 1)} respec- tively. Additionally, we employed threshold values based on the average normalized Gini Index (G) to scale the clause weights. When G exceeds threshold, Ø, it indicates significant class imbalance. In response, clause weights are scaled by x to prevent dominant classes from overpowering minority ones, ensuring balanced class representation in the final model and mitigating the effects of non-lll) data. The code is available at: https://github. wwt.:/coo.oooooeeeeeee homm//hhammnhs//FeTTMMMSS
G 1
if iG5o
x
We used threshold (0.5, 0.5, 0.3, 0.6) respectively for the MNIST, F-MNIST, SVHN and CIFAR- 10 dataset. Note that we used 0.3 for SVHN due to the unbalanced nature of the data. In the future, we will explore a data-driven approach to determine the threshold and & values.
A.2 ADDITIONAL RESULTS
A.2.1 PERFORMANCE WITH 200 LOCAL EPOCHS
We evaluated the baseline algorithms with 200 local epochs, as recommended in (Dai et al. 2024). ensure that local models reached convergence before server-side training. However, excluded
14

---

Published as a conference paper at ICLR 2025
Table 5: Performance of the different algorithms across various data partitions
Fed- OncShot
Distiled- FedOv
FedTMOS
Dataset
Partition
FedAvg
DENSE
Co- Co-Boosting
(Ensemble)
FeJTMOS
Dirl0.05)
18.65217.31
41.78211.34
78.362866
88,9543899
89,2523.92
13.78:111.24
93.50=3.72
Dar(U.1)
12.77
16.12113.34
85.13
94.810449
918313002
93.86
91.28
MNIST
1xr(0.3)
8(2)
21.3517.39
10.18
24.2313221
53.721
16-19-1.0
15.88
19:2219356
97.56+0.32
69.16
48.88 19.01
00.30
83.07 3351
8(3)
35.2448.86
40.45-8.99
70.73+
74.1844.20
18.75-33.46
8(4)
40,2849974
48 66.00
72.27=5.06
87.1645.35
84.08+8.11
88.68=0061
97.16=0.488
Dar(0.05)
39,8946.48
41.9745.78
62.52=5.28
64.58:12.48
72.00=3.16
70.33:6.399
72.73=3322
Dir(0.1)
50.9188277
50.56 16.66
70.07 1.007122992 2.92
13.94
78.15 3.155
75.38 3.76
78.39
3.75
F-MNIST
DAr(U.))
71.26+41.03
73.1442.55
80.75-0.59
80.1411 1.49
83.31+00611
+3.37-0.33
84.30
0.69
812)
24.7417.32
22.7813.60
40.72 6.54
48.1613. 3.24
54.12 4.06
39.801173 7.92
60,08
7.67
8(3)
44.28
54.6214.30
37.08+4.55
32.4443811
67.81+3.57
63.60-1..0 1.90
72.22
1.61
8(4)
34.00:22.01
34.73:6.02
63.14:7.00
61.8542.33
68.6847.00
76.02-46334
80.76=6.19
Dir(0.05)
28.04=18.44
24.61+2888
46. 15
13.81213.83
70.45=3.00
41,45 18.69
66.77=1.41
(ir(0.1)
72.9013.5
D)
38,0010822
76,29:24.13
67.00 3.
76.5224.79
81.352101
76.111100
80.73
SVHN
8(2)
189511.78
24.1616.06
48.4214933
8(3)
37.01
58.0317255
45.17411.12
62951 88
12.3144.57
66444
75.62 3.02
8(4)
10.06
6..6616.22
57.57177
16.7613.04
Dir10.05)
20.9641.57
26.2526.18
37 17.80=520
39.4625 07
44.19=
66
47.12=4.95
Dir(0.1)
27.1418.51
36.05:14.60
48.5723.74
10.15110.56
48.8914.23
46.8814.4
52.06-2.77
CIFAR-10
Dir(u.))
43.9916.10 10
42.01=3354
30:12:3336
G0.721383
36.342188
58.4320.08
36.69=2.07
8(2)
14.7116.36
18.47:2.02
29.262:5225
32.9415566
31.5726.20
32.1615.06
36.01
8(3)
23.01 16.68
24,68 14666
40.52 3.50
42.151334
43,06 1.94
50,66 12.82
53.21 2.09
S(4)
34.6613.31
34.9313.01
41.312312
46.911738
47.31222
37.7811377
37.851..1 1.13
OT-Fusion, RegMean, and FedFisher from this setup, as their hyperparameters are specifically opti- mized for 30 epochs according to Jhunjhunwala et al. and re-tuning for 200 epochs could inadver- tently introduce inconsistencies.
From Table 5 FedTMOS outperforms the other baseline methods without the need for synthetic or generated data in most settings. On average. FedTMOS performs better than the top performing baseline. Distilled FedOV by 7.36%, 4.07%, 0.26%. 5.14% on the MNIST, F-MNIST, SVHN and CIFAR-10 dataset respectively. We note that Distilled FedOV leverages KD at the server using subset of the test data, closely matching the actual samples. Furthermore, when compared to the best-performing data-free method for each setting, FedTMOS surpasses them in all except for least non-llD setting (Dir(0.3)) for CIFAR-I0. On average, FedTMOS outperforms the best data- free method for each setting by 8.46% reinforcing its effectiveness in tata-free OFL.
In addition, Table [ highlights that even with just 30 epochs, FedTMOS maintains its ability to outperform all basclines. This result emphasizes the effectiveness of FedTMOS, showcasing its strong performance even without requiring full convergence.
We note that DENSE and Co- To-Boosting exhibit high variance, particularly with increasingly non- IID data. This is likely because classifiers trained non-lID data generate sub-optimal knowledge samples, which are crucial for the distillation process. This hinders the student model's 's learning, reducing performance and increasing variability (Gou al. 2021)
A.2.2 ADDITIONAL EFFICIENCY ANALYSIS
Table 6: Training Latency (s)
Distilled- FedOV
Dataset
DENSE
Co-Boosting
OT-Fusion
RegMean
FedFisher
FedTMOS
MNIST
61.29
30.16+2.33
22.21
F-MNIST
61.96110.41
48.84114.15
34.20.13.49
SVIIN
271.46.14.16
25.86.11.94
320.5419.42
CIFAR-10
235.10066.86
22.54+2.33
219.58+6.85
In terms communication costs, our results demonstrate that the bit-based architecture the TM significantly reduces communication overhead while producing model that generalizes well across
15

---

Published as a conference paper at ICLR 2025
all settings, evident in the results in Table 2 By ensuring that the server model size of the TM is equal to or smaller than the CNN models, we show that FedTMOS effectively reduces upload costs by at least 2.3x without sacrificing performance. Furthermore, FedTMOS reduces storage costs by an average of 3.5x compared to FedTMOS (ensemble), while providing an average of 6.16% improved performance and 1.63% decrease in variability across all settings.
We computed the average training latency of the models on each client with a compute node with 32 CPU cores. From Table [6] even though the training latency of FedTMOS is higher compared to OT-Fusion, RegMean, and FedFisher, the results in Table [ demonstrate that FedTMOS outper- forms them by at least an average of 14.43% across all settings. This substantial improvement in performance highlights the value of FedTMOS in achieving higher accuracy with comparable server efficiency. Moreover, despite the slight trade-off training efficiency, FedTMOS remains highly suitable for practical deployment due to its low upload costs as shown in Table 2 which are crucial for edge devices with limited bandwidth (Zhang et al., 2022b), and the computational simplicity of TMs, which avoid backpropagation and make them ideal for low-power training on edge devices
(Rahman et al. 2022 Tang et al.) 2024 Lei et al.). This balance between performance and effi- eiency highlights Fed MOS as promising solution for OFL and facilitates its extension to iterative
FL while maintaining efficiency (Zhang et al. 2022b).
Furthermore, we observed that the training latency of OT-Fusion, RegMean, and FedFisher is lower than that of DENSE, Co-Boosting, and Distilled- This difference arises because the former methods utilize CNNs with three layers (Dai et al. 2024). whereas the latter methods employ CNNs with only two layers (Jhunjbunwala et al.).
A.2.3 PERFORMANCE EVALUATION IN A CENTRALIZED SETTING
Table 7: Performance of CTM and CNN in the centralized setting, where CNN1 is the CNN in FedAvg, Fed- OneSholDENSE, Co- Co-Boosting, Distilled-FedOV and CNN2 the CNN in OT-Fusion, Reg.Maan FedFisher
CTM
CNNT
CNN2
MNIST
98.23-0..1
99.15-0..01
99.51 : 0.09
F-MNIST
87.3010.56
90.0411.33
89.66=0.29
SVHN
80.07=0.20
89.5110.37
90.1910.59
CIFARTO
54.93+0.45
82.17+00.43
79.42+0.55
In centralized setting, CTM does not outperform CNNs in terms of accuracy as shown in Table.
This is expected, as CNNs are highly optimized for tasks like image classification and generally achieve higher performance in such settings (Granmo 2023
Note that the performance of CTM in our experiments lower than that reported in Granmo (0233) This discrepancy arises because, to ensure model size and efficiency constraints, we used only 200 clauses per client compared to 2000 clauses in the original paper. Additionally, we used models per client, whereas the paper utilized 4 models.
However, this highlights that the robustness of our proposed approach, FedTMOS, enables CTM to perform competitively compared to the CNN-based OFL baselines, even though CTM may not match the accuracy of CNNs in centralized settings. The performance improvements observed in FL settings stem from the strength of our proposed OFL methodology. FedTMOS, which is both robust and scalable in FI..
A.2.4 EXTENSION COMPLEX DATASETS
We compared FedTMOS with ResNet-18 on CIFAR- 100 Krizhevsky (2009) and with a pre-trained ResNet- 18 model on Liny-ImageNer(Le & Yang 22015). While TMs have yet to be directly explored for the use of pre-trained weights, the potential for applying them to tasks that share classes with the pre-training dataset is promising, demonstrated by the comparable performance on the Tiny- ImageNet dataset between a pre-trained ResNet- 18 and TM. This will be a key focus of our future work, where aim investigate how incorporating pre-trained weights can further enhance TM
16

---

Published as a conference paper at ICLR 2025
adaptability and performance. Regardless, the performance of our current model, even without large- scale pre- training, highlights the potential of TMs, suggesting that there is still room for significant improvement in their capabilities.
Table 8: Performance of the evaluated baselines on complex datasets
FedAvg
DENSE
Co-Boosting
FedTMOS
Dir(0.05)
6.93
20.12:42.40
20.16=2.75
27.16:0.557
CIFAR-100
Dir(0.1)
10.52:10.44
25.22:12.02
2..3
28.19:10.86
Dir(0.3)
13.32:10.76
30.97110.89
30.26:10.51
30.51
Dir(0.05)
7.44
8.32-F0.32
8.38+
1..64:0066
Tmngeem
Dir(0.1)
9.41+1.12
10.61+0.57
00.49+0.19
11.81+0.41
Dir(0.3)
12.2910.39
13.85+0.75
14.35=0.35
13.03#0.66
Table 9: Upload costs for each client/storage and potential download costs of the server model (MB)
Dataset
FedAvg
DENSE Co-Boosting
FeS
CIFAR- 100
45.12/45.12 12 45.12/45.12
14.2/43.4
Tiny-Image ImageNet
11.36/40.8
A.3 FURTHER DISCUSSION ON THE LIMITATIONS AND FUTURE WORK
Firstly, we acknowledge that TMs in general, are robust as DNNs for complex datasets, particularly those involving multi-channel images. This limitation stems from the boolcanization process and bit-based representation of input data, which restrict TMs' performance. Efforts to address this, such as creating composite TMs trained on extracted features like 3x3 thermometer encoding and Histograms of Gradients, have shown promise improving their capability (Granmo 2023
However, DNNs face significant challenges in FL scenarios with high data heterogeneity. When aggregated at the server, the parameter spaces of clients with heterogeneous data often fail to provide an accurate estimation of the global parameter space. Unlike DNNs, TMs leverage class-specific weights, enabling more effective contributions from individual clients in heterogeneous settings. While class-wise weights could thcoretically be implemented in DNNs, TMs are inherently more suited for this purpose due their simpler structure and efficient aggregation process.
For example, in the MNIST dataset, TMs require just 100 weights per class, compared to a simple CNN, which might involve 61,706 weights (Dai al. 2024). translating at least 6,179 weights per class. This simplicity reduces computational overhead. Furthermore, while small DNNs trained for individual classes often risk over-fitting due limited generalization capacity, TMs are simultane- ously trained across all classes with separable class weights. This design helps TMs generalize better across classes, reducing over-fitting and making them more adept handling heterogeneous data in FL settings. While DNNs might struggle in similar scenarios due to their complexity, exploring class-specific weights DNNs for FL settings remains an interesting avenue for future research.
Next, TMs remain an emerging field, and the use of pre- pre-trained models has yet to be fully explored. Unlike DNNs, which often benefit from pre-training on large datasets like ImageNet (Chen al.) Nguyen et al. 2023), TMs have not been adapted to incorporate pre-trained weights. For instance, TM pre-trained on a large dataset could potentially reuse class-specifie weights for tasks involving those same classes. This unexplored area presents exciting direction for future work. Pre-training TM on a dataset like ImageNet and fine-tur tuning it on smaller datasets, such as Tiny ImageNet CIFAR-100, could significantly enhance their applicability and effectiveness in these specific tasks.
2023
Lastly, in our current experiments, also observed that the number of resulting weight vectors from local clients can influence recombination performance, especially when mapping to a fixed number of server models. As discussed in the main text, dynamic scaling of the number of server
17

---

Published as a conference paper at ICLR 2025
models, Φ, and the number of clusters, k, remains key direction for future work. In particular, aim to explore how & can be dynamically adapted based on the number of available weight vectors, to better balance performance and model complexity.
In summary, while TMs are still an emerging field research, they offer distinct advantages, par- ticularly in FL. scenarios. Their class-specifee weight structure and computational simplicity make them well-suited for low-power, on-device training (Tagg et al. 2024 Lei et al.] Rahman et al.) 2022). However, recognize their current limitations, including challenges in handling complex datasets and the lack of pre-trained models, which can affect training efficiency.
18

---

Published as a conference paper at ICLR 2025
A.4 FULL ALGORITHM
Algorithm reassign.weightsicustter..... inno, cluster _means, cluster_means,
Initialize: reordered _models rm for rm in range(ó), reordered means track_clusters {[] for in range(ő))
{ used_clusters = [],
Sort cluster i based on number of classes in each cluster
while l < cum.ceusters do for cidx in cluster info do if cidx used clusters then
continue
for class m cidx do best model find best(cidx.m.reordeeed models, reordered means) model == False then
best_modes
find find_eestt.ce find_eeattcccreeorceeeeee wewsssccrreordeeedmmdllessseese
Add 8Th to best mosde
Add CmM to best mest_
Update the mean for class m in reordree_.necns/btttmoddlllll
Add cidx to used_clusters
Add cidx to track clusters(best model]
return reordered models
calculate avg fink(reordered.men)
2_dist== 0
num painspai-s-- =
for mi in range(len(reordered means)) do
for mj in range(mi+1 (.nnne..drerddeeree_mmamen do
ss_2_dist + recodeeeddmmaans/ rewrdereddmmaaneeeeeeeeeeeeeeeee
num pairs +=
return ss 2 diwt/nummmmmmm pairs
find bestrcidx,,,,rorddreed rooddeede menaerrddasdrr).....e)))
best_model = False, max_dist = for model in reordered modell do if model has class then continue
temp model means copy(reordered means[model]] temp_model wtwp_oode__eeannnneg meaas(modell]]]] me wupddaaateeeeordeeedeeeeeeeeeee
distance calculate avg liik((temp_oool)
if distance>> max max_dist then wax.dist=distance best_model = model
return best model
find seasttcc(rorrdered_meoe modells cidx) best_model = False, min_class = for rm reordered models do distinct.classes len(set(classes in rm))
distinct classes < min_class then
min min_class dilistint_cclsseed best_model = rm
else if distinet_elasses min _class then
cluster.class_com = sum(1 I for с in track.cluster((mm) if cluster class count < min class then best model rm
return best_model
19

---

Published as a conference paper at ICLR 2025
Algorithm 3 average models(final models) for each model fim in models do for each class m in fim do sereeesndices argsort(O"" for all j fm, in descending order)
f've + V Esorted indiccs[0:2] J fim - mean (0))), Vj E fm
return final models
20


Reviews:
[{"review_id": "bLnl6NEAts", "reviewer": "Reviewer_GkyH", "summary": "This paper presents FedTMOS, a compute efficient one-shot Federated Learning (FL) algorithm that leverages Tsetlin Machines. Tsetlin Machines present an alternative to DNNs, known for their low complexity, compute and storage efficiency along with good performance. FedTMOS learns client-specific TMs and derives an aggregated server side TM that enhances class distinction. The aggregation procedure is significantly cheaper than traditional KD based methods while being data-free. The authors show comprehensive empirical results on standard OFL benchmarks under non-IID data.", "questions": "1) The authors mention using a standard compute node for evaluating server side latency. Does this mean that the node was GPU equipped? It would be unfair to measure the latency of DNN based approaches without using a GPU equipped node.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The application of Tsetlin Machines to OFL is novel and offers an interesting alternative to standard KD based methods which are compute intensive\n- The method is data-free\n- The authors provide comprehensive evaluations on communication and compute efficiency alongside accuracy which showcase the strength of the approach", "weaknesses": "The paper can be improved on several fronts as listed below:\n1) The paper offers no discussion on the limitations of Tsetlin Machines and its broader applicability. While TMs are an evolving research area, DNNs are the norm today. Thus, an elaborate discussion of its current limitations will strengthen the paper by well informing the community on its wider applicability. For instance, can TMs be applied to NLP based tasks such as those based on transformer models as of today? \n2) A significant portion of the proposed algorithm in Section 4 is explained in sentences, making it difficult to follow without using mathematical references to the quantities being discussed. For instance, equation (4) describes general k-means clustering without reference to actual scaled weights which are being clustered.  Section 4.2.2 uses no mathematical expressions to describe the proposed algorithm. The paper can be greatly improved by defining appropriate notation for quantities being referred to at the beginning of Section 4 and then using this notation throughout while explaining the proposed approach. \n3) The paper misses an important baseline, FedFischer [1] which is more compute efficient on the server side as compared to the KD based methods and offers strong accuracy. In general, the paper misses related work involving averaging based schemes such as OT-Fusion [2] and RegMean [3] which offer low server side latency. \n4) Lack of theory to justify the performance improvements as compared to the evaluated baselines. Can the authors provide more insights into the accuracy improvements achieved?\n5) With the increasing availability of large pre-trained models, conducting OFL starting from a pre-trained initialization is shown to significantly improve performance [1]. How can a TM incorporate pre-trained weights from other TMs trained on large datasets?\n\n[1] Jhunjhunwala, Divyansh, Shiqiang Wang, and Gauri Joshi. \"FedFisher: Leveraging Fisher Information for One-Shot Federated Learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[2] Singh, Sidak Pal, and Martin Jaggi. \"Model fusion via optimal transport.\" Advances in Neural Information Processing Systems 33 (2020): 22045-22055.\n\n[3] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rXwmbLon9S", "reviewer": "Reviewer_Fgwm", "summary": "The authors leveraged Testlin Machine to resolve the bottleneck in one-shot federated learning, saving the communication cost and reducing the necessity of using a public dataset. The proposed solution views the one-shot federated learning in a different prospective, in the form of automation machines.", "questions": "1. Why do authors typically introduce Testlin Machine, which is an automation machine rather than leveraging a general reinforcement learning scheme where penalty, reward, and stage changing are involved? What is the motivation for doing so? How different is the solution with general reinforcement learning based one-shot FL? e.g. \n\n2 It is not common to use Gini index to measure data distribution. There are more common solutions. For example, the simplest way is Gaussian Model. But it is possible that clients data are non-i.i.d. In that case, a simple solution is to do some sampling. In some semi-supervised federated learning, uploading hard or soft labels is also fine.  Choosing Gini index is neither a straightforward nor a trivial option. How did you come up with that? And why can authors benefit from that?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of introducing Testlin Machine into one-shot federated learning is innovative, aiming to solve the bottleneck of using public datasets.\n\n2. The authors clearly described the background, laying emphasis on Testlin Machine, making the paper self-contained.\n\n3. The authors evaluated the solutions over client numbers of a certain scale, e.g. 20, 50, 80, which is a critical factor in one-shot federated learning.", "weaknesses": "1. The reviewer acknowledges the innovation of introducing Testlin Machine, however the motivation for doing so is not well explained. The authors spent certain paragraphs describing  Automation Machine and the mechanisms in machine learning. Nevertheless, how such a mechanism can benefit machine learning and federated learning is not illustrated. Moreover, why the key bottleneck in one-shot federated learning can be resolved is not explained. In other words, the current solution looks like converting a conventional question into a mechanism of a automation machine. For example, it likes a task converting a coding task into Moore Machine in algorithm lectures.\n\n2 Many choices of approaches are not well justified. See more details in the reviewer's questions.\n\n3. The empirical evaluation can be improved. The authors claimed that they used various datasets. However, these are very basic datasets like MNIST,SVHN, and CIFAR10. The reviewer suggested using more complex datasets such as Tiny-ImageNet. For datasets like MNIST, even if we are not doing one-shot federated learning, few epochs and communication rounds are needed to achieve convergence. The effectiveness, particularly in terms of convergence and accuracy, can be correctly justified by using a more complex dataset. \n\nOther minor writing issues:\n1. The acronym in the paper is not of common use. OFL is not a common usage for one-shot federated learning. Directly saying one-shot FL is fine. TM is usually referred to Turing Machine.\n2. Table 1 and Table 4 are out of bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1UoIBWVSS4", "reviewer": "Reviewer_Voa4", "summary": "This paper propose FedTMOS for efficient one-shot federated learning (FL). FedTMOS employs Tsetlin Machine instead of DNNs to reduce upload costs and presents a novel data-free solution to generate server model. Experimental results show that FedTMOS outperforms existing one-shot FL methods.", "questions": "- Since FedTMOS uses a non-DNN model, is its scalability being limited by Tsetin Machine? Can it achieve comparable performance when other baseline methods employ stronger networks (e.g., ResNet) on challenging datasets (e.g., Tiny-ImageNet)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Employing Tsetlin Machine in one-shot federated learning is interesting.\n- The proposed FedTMOS significantly reduce the communication costs.", "weaknesses": "- It is unclear whether the performance improvement in Table 1 comes from the performance gap between the CNNs and CTM. It is suggestted to report the performance of CNNs and CTM in a centralized(non-federated learning) setting.\n- My main concern with this work is its applicability, as it is limited to a specific machine learning model. In my view, machine learning models and tasks should primarily serve as a testbed for evaluating federated learning algorithms. They should not be restricted to particular models, unless exploring new applications of federated learning in emerging areas, such as diffusion models or large language models. However, this paper addresses a well-established image classification task and is effective only for the Tsetlin Machine, which limits its practical application.\n- The readability of this paper can be further improved. For instance, in line 146, what does the $j$ of $L_j$ stand for, and how to get the definition of the $L_j$ from the definition of $L$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)