lo-fi: distributed fine-tuning without communication
Authors: Mitchell Wortsman
Suchin Gururangan
Shen Li
Ali Farhadi
Ludwig Schmidt
Michael Rabbat
Ari S. Morcos
Venue: Accepted by TMLR ()

Abstract: When fine-tuning large neural networks, it is common to use multiple nodes and to communicate gradients at each optimization step. By contrast, we investigate completely local fine-tuning, which we refer to as lo-fi. During lo-fi, each node fine-tunes independently without any communication. Then, the weights are averaged across nodes at the conclusion of fine-tuning. When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the baseline, which observes the same amount of data but communicates gradients at each step. We also observe that lo-fi matches the baseline's performance when fine-tuning OPT language models (up to 1.3B parameters) on Common Crawl. By removing the communication requirement, lo-fi reduces resource barriers for fine-tuning large models and enables fine-tuning in settings with prohibitive communication cost.

Published in Transactions on Machine Learning Research (01/2023)
lo-fi: distributed fine-tuning without communication
Mitchell Wortsman" University of Washington
milchosuf@es...usshhngognn
Suchin Gururangan
sg01 @cs. www...asinggooonoommmm edu
University of Washington
Shen Li
shenli@mella.c com
Meta Al Research, FAIR Team
Ali Farhadi
ali@Ecs wwt.icssssssinngoo.ccmoo edu
University of Washington
Ludwig Schmidt
sdbmidd@@os..unshhiuigon.o eduu
University of Washington
Michael Rabbat
mikerubbat@@metaa.com
Meta Al Research, FAIR Team
Ari S. Morcos
urimorvoss@mella.coom
Meta AI Research, FAIR Team
Reviewed on OpenReview: https: // openreview. net/ fort/frmm id= hewta////0///edddee/e//eeee/e//d
Abstract
When fine-tuning large neural networks, it is common to use multiple nodes and to com- municate gradients at each optimization step. By contrast, we investigate completely local fine-tuning, which we refer to as lo-fi. During lo-fi, each node fine-tunes independently without any communication. Then, the weights are averaged across nodes at the conclusion of fine-tuning. When fine-tuning DeiT-base and DeiT-large on ImageNet, this procedure matches accuracy in-distribution and improves accuracy under distribution shift compared to the bascline, which observes the same amount of data. but commuicattss gradicnts at cach step. We also observe that lo-fi matches the baseline's performance when fine-tuning OPT language models (up to 1.3B parameters) on Common Crawl. By removing the communi- cation requirement, lo-fi reduces resource barriers for fine-tuning large models and enables fine-tuning in settings with prohibitive commmication cost.
1 Introduction
Many of the best performing machine learning models today come from two step procedure: First, pre-train on a large, heterogeneous dataset to learn à good representation. Next, fine-tune to adapt the model to a task of interest (Girshick et al. 2014 Yosinski et al. 2014 Kornblith et al.) 2019 Kolesnikov et al. 2020) This paper operates within the second step of this procedure fine-tuning which is increasingly important with drastic improvements in pre-trained models, e.g., CLIP (Radford al. 2021 GPT-3 (Brown et al.
2020), OPT (Zhang et al. 2022), and PaLM (Chowdhery et. al.
2022) Indeed, recent advances such
Minerva [Lewkowyez et al. 2022 InstructGPT (Ouyang et al.
2022 have come from fine-tuning rather
than training from scratch.
Work done while MW and SG FAIR.

---

Published in Transactions on Machine Learning Research (01/2023)
85.8
ImageNet accuracy
85.6
IN
IN-V2
IN-R
Sketch
TN-A
85.2
baseline (Deil-b)
85.96
76.65
62.66
46.86
57.15
lo-fi (DeiT-b)
86.00
76.84
63.25
18.37
58.43
84.8
individual node lo-fi
baseline (DeiT-1)
87.12
78.18
69.87
54.41
68.97
84.6
lo-fi (DeiT-I)
87.10
78.25
70.14
54.95
69.53
baseline
84.4
15
25
Comparing lo-fi (no communication dur- ing fine-tuning) to the baseline which communicates at each step when fine- tne-tuning the ImageNet-222 pre- trained DeiT-base and DeiT-larre model from DeiT- III [Touvron et al. 2022) on ImageNet (Deng et al, 2009). Both lo-fi and the baseline use the same num- ber of iterations, which have been tuned for the base- line. Underlined numbers indicate significantly bet- ter accuracy according to McNemar's test with siguif- icance level 0.05. Lo-fi matches performance on Ima- geNet (TN), but can outperform the baseline on some distribution shifts. The shifts we consider are IN-V2 RRecht et al.] 2019). IN-R (Hendrycks el. al. 2021a)) Sketch (Wang et al. 2019), and IN-A (Hendrycks ct al. 2021b).
Table 1:
Epochs
Figure 1: In standard multi-node distributed data- parallel fine-tuning, there is synchronization be- tween nodes at each step of fine-tuning. With lo- fi (local fine-tuning), there is no communication between nodes throughout fine-tuning. As a re- sult, each node & independently produces their own model 0²_ Then, lo-fi averages these models once for the final solution Q-- = 1 Ex_1 0h. In this four-node fine-tuning run, show (i) the average accuracy of the individual models 0h, (ii) the accu- racy of On- at the end of each fine-tuning epoch, and (iii) the accuracy of the baseline which commu- nicates among nodes every step. In particular, we fine-tune the ImageNet-21k pre-trained DeiT-base model from DeiT--1l1 (Touvron et al. (2022) on Im- ageNet (Deng al. 2009) using their code, which uses four nodes.
Most work developing learning methods still operates in the paradigm of training from scratch. Accordingly, both use similar algorithmic techniques despite important diffcrences in the pre-training and fino-tuning regimes. In particular, one notable difference between pre-training and fine-tuning is that fine-tuned models appear to lie in a single low-error region (Neyshabur al. 2020). Indeed, linearly interpolating the weights of fine-tuned models can have similar advantages as ensembling their predictions but without the added cost during inference (Wortsman et al. 2022). By contrast, linearly interpolating the weights of two models trained from scratch will encounter high error barrier (Frankle et al. 2020: Garipov et al. 2018).
Recently, the model soups approach WWorsmaan et al. 2022 leveraged this similarity between ensembling outputs and averaging weights. Given a hyperparameter sweep over fine-tuned models, they average the weights of multiple models instead of the conventional procedure of selecting one model and discarding the remainder. However, the model soups approach does not modify the fino-tuning procedure itself.
In this paper, we leverage the observation that fine-tuned models appear to lie in single low error region to remove communication between nodes during distributed fine-tuning. In standard data-parallel multi- node finc-tuning, gradients between nodes are commuicctted at each step. This synchronization of updates keeps the models at each node identical to each other during fine-tuning. However, in cerlain seltings communication costs during fine-tuning may be prohibitive, and we therefore ask whether they are necessary at all. With our method of local fino-tuning, which we refer as lo-fi, we remove all communication between nodes during fine-tuning. The models on each node therefore drift apart throughout fine-tuning. Then, arrive at the final solution at the end, we average the weights of the models produced by each node.
2

---

Published in Transactions on Machine Learning Research (01/2023)
IN
IN-V2
IN-R
Sketch
IN-A
epochs
stoch. depth no extra cost no comms
DeiTT-base
Touvron et al.
2022
85.72
76.53
61.83
47.44
57.29
50
0.15
baseline
85.96
76.65
62.66
16.86
57.15
21
0.15
individual node
85.66
76.22
62.09
46.75
56.00
0.10
lo-fi
86.00
76.84
63.25
48.37
58.43
24
0.10
lo-fi ensemble
86.08
76.91
63.05
47.67
57.80
24
0.10
DeiT-
Touvron et al. 2022
86.97
78.47
69.70
54.35
68.57
50
0.40
baseline
87.12
78.18
69.87
54.41
68.97
12
0.30
individual node
86.76
78.00
69.41
54.57
67.59
3
0.25
lo-fi
87.10
78.25
70.14
54.95
69.53
12
0.25
lo-fi ensemble
87.14
78.35
70.00
54.62
69.20
12
0.25
Table 2: Expanding the comparison between lo-fi and the baseline (Table 1 when fine-tuning the ImageNet- 21k pre-trained models from DeiT-111 (Touvron et al. 2022) on ImageNet (Deng et al. 2009). In this four node fine-tuning run, lo-fi removes communication between nodes so that each node produces an independent model. The weights of the models are then averaged at the end to produce the final solution. In this table we bold the highest number and evaluate the following models: i) paper, the fine-tuned models from the DeIT-III paper (Touvron al. 2022), , ii) baseline, which is our improved fine-tuning baseline after hyperparameter turning which requires less epochs of training but achieves slightly higher accuracy than reported in the DeiT-Illl paper [Touvron et al. 2022), iif) individual node, which is one of the individual node models that is produced by lo-fi, iv) lo-fi, which fine-tunes individual models on cach node then averages their weights once at the end, and v) lo-fi ensemble which averages the outputs of the models produced by each node during lo-fi, and therefore requires more cost during inference. In addition to evaluating on ImageNet (LN), the task used for fine-tuning, we also evaluate on the distribution shifts ImageNett- V2 (IN-V2, (Recht et al. 2019))
ImageNet-R (IN-R, (Hendrycks al. (021a)) ) ImageNet-Sketch (Wang et al. 2019), and ImageNet-A. (IN-
A (Hendrycks et al. 2 While more information is provided in Section 3.1 this table also displays some
hyperparameter changes made from the default DeiT--111 fine-tuning script. Unlike (Touvron et al. 2022
we fine-tune with LP-FT (Kumar et al. 2022). and observe it is better for the bascline to use fewer tine-tuning epochs. Lo-fi observes the same amount of data as the tuned baseline and uses the same hyperparameters with the exception of slightly decreased regularization by lowering stoch. depth (Huang et al.) 2016 drop probability by 0.05 (making the same change to the baseline decreased accuracy). Additional columns track whether the model incurs no additional cost during inference compared to single model (denoted extra cost), and also if there is communication between nodes during fine-tuning (denoted comms). Overall, lo-fi matches or outperforms the baseline without communication between nodes during fine-tuning.
We note that these techniques are a natural extension of previous work: lo-fi is just a model soup (Wortsman) ct al. 2022) formed by splitting up large fine-tuning job into multiple smaller jobs, cach isolated to node. Analogously, lo-fi is embarrassingly parallel training from muncch-rriin-mege (Li et al. 2022 applied in the setting where no domain specialization info is provided and so each expert is trained on ITD data. However, we believe that the application of these techniques in this setting is of practical interest, especially if models continue to grow.
In computer vision we use the DeiT-III codebase Touvron et al., 2022 to fine-tune the ImageNet-21k pre-trained DeiT-base and DeiT-large models, which are four-node fine-tuning jobs by default. We observe (Figure 1 Table D that lo-fi matches the accuracy of DeiT-base and DeiT-large on ImageNet, the task used for fine-tuning, while outperforming the baseline come after hyperparameter tuning the bascline to slightly exceed that in the DeiT-III paper while requiring fewer fine-tuning epochs. Moreover, lo-fi and the baseline observe the same amount of data. While overall similar results are observed when fine-tuning CLIP ViT-L (Radford et al.) 2021 on ImageNet or tasks from WILDS Koh et al.) 2021), lo-fi often requires more itcrations in this sotting. Finally, we test lo-fi beyond computer vision by fine- finettnnin OPT-125 1PT-125M and OPT-1.3B ÇZhang al.) 2022 on Common Crawl, observing that lo-fi match the baseline which communicates between nodes.
some distribution shifts. These improvements
3

---

Published in Transactions on Machine Learning Research (01/2023)
Overall, our work is test of whether communication between nodes is required during fine- fino-tuning, However, we also wanted to understand the advantages of removing this communication. Therefore, we benchmark the wall-clock overhead of communication on an AWS cluster with EFA. We use the models from the Dei repository (Touvron et al. 2022) in the context of image classification. In this setting and on the system used for this study, the advantages are overall less substantial than we initially expected, especially for large batch sizes. Notably, we observe that the trick of overlapping the communication and computation in the backwards pass (Li et al.) 2020a), which is the default in PyTTrch (Paszke et al. 2019) as of v1.5, reduces the overhead of using multiple nodes from roughly 50% slow-down to under 10% for the large DeiT model. Finally, we discuss how lo-fi can help with faster job scheduling and addresses the straggler and jitter problem in distributed training, where different nodes might experience random slowdowns.
2 Methods
This section details the methods used in our experiments. We begin with the baseline standard data- parallel training, and next outline our straightforward modification which i) removes communication wetween nodes then ii) averages the final models produced by each node.
Consider neural network f(x, () where is the input data and E Rd are the network parameters. Since we are fine-tuning, 0 is initialized as the weights of pre-trained model. Moreover, as is standard in neural network training, the input data æ is a batch rather than single data point. Finally, let n denote the number of devices, b denote total batch size, and E(G. y) denote loss for the vector of predicted labels á = f(z, 0) and a vector of mroudd-ruut labels y.
With communication. The most straightforward and common approach for training with n devices is data-parallel. In this sctting, each device has their own copy of the parameters 0. During fine-tuning, cach batch of size b split into n disjoint sub-batches of size b/n. Each device i loads the sub-batch (1),3) and computes gradients Ii
(x)(f=x))) Then the gradients node computing averaged gradient + >i gi. After synchronizing gradients, each device uses ğ to update 0. Since every device updates 0 using an identical gradient the parameters 0 remain identical across devices.
synchronized across nodes with each
lo-fi. With foeal-inetuuing (lo-fi), we partition the devices into K disjoint groups. In the majority of our experiments, each group is a single node containing 8 GPU devices. During fine-tuning we allow communication within each group, but not across groups. Each group & begins with parameters O which are initially identical across devices, but drift apart throughout fine-tuning. Then, at the end of fine-tuning there is single communication and the parameters from each group are averaged to produce final solution 1x 6*
0 0=
There are two possible implementations for lo-fi which we refer to as implementation A and B. Implementation A proceeds as before each device loads the sub-batch (æ,, 9i) and computes gradients Jr- There is then gradient synchronization only among devices belonging to the same group while devices from different groups apply different gradients. Data partitioning accomplished without communication by coordinating random seeds, so long as each device knows its rank and the total number of devices. Our experiments primarily use Implementation A.
In Implementation B, each group is a completely independent run-no knowledge of total number of devices is required. Accordingly, within each group the global batch size is scaled by 1/K so that the per-device batch size is matched. Our image classification results use Implementation while our language modelling results use Implementation B.
Our motivation for having one group per node and still allowing communication among the devices on the node is that communication within a node is faster than communication across nodes.
4

---

Published in Transactions on Machine Learning Research (01/2023)
ImageNet
ImageNet-V2
ImageNet-R
ImageNet-Sken Skketch
ImageNet-A
77.0
Accuracy
86.0
63.0
48
58
85.8
76.5
62.5
47
56
10 15
5
10
15
5
10
15
10
15
10 15
Number of nodes
Number of nodes
Number of nodes
Number of nodes
Number of nodes
We test whether the performance of lo-fi continues to improve when adding more nodes. On the contrary, this experiment suggests diminishing even negative returns after 4 nodes. This experiment is for fine-tuning DeiT-base as in Table 1 Recall that when using four nodes, lo-fi and the baseline observe the same number of images, but lo-fi does not require communication between nodes. When moving beyond nodes as we do in this experiment, lo-fi observes more images then the baseline.
Figure 2:
3 Experiments
This section presents our experiments which test whether communication is required during fine-tuning, First we use the DeiT-III codebase (Touvron et al. 2022) to fine-tune their pre-trained ImageNet-21k models on ImageNet, where we observe that lo-fi matches the baseline but without communication between nodes (Section 3.1 Next, we fine-tune CL.IP (Radford et al. 2021 on ImageNet, WILDS-FMoo (Koh et al. 2021; Christie et al. 2 and WILDS-iWildCam (Beery et al. 2021 (Section 3.2p. Finally, we show preliminary experiments applying lo-fi outside of computer vision (Section 3.3 and benchmark the associated speed-ups by removing communication (Section 3.4
3.1 Fine-tuning DeiT-IlI on ImageNet
The aim of these experiments to test whether communication between nodes is required when fine-tuning high accuracy models for image classification. To test this we begin by fino-tuning the DeiT-base and DeiT- large models from the DeiT-IlI paper (Touvron et al. 2022 using their code. In particular, we fine-tune their ImageNet-21k models on ImageNet- (Deng et al. 2009 with and without lo-fi.
We chose the models from DoiT-III for a few reasons: (i) DefT-III is represcntative of statc-of-thh-art settings as uses many advanced techniques such as stochastic depth (Huang al. 2016), CutMix (Yun et al. 2019). and the LAMB optimizer (You et al., 2019). (ii) DeiT-III provides hyperparameter configurations which they used in their fine-tuning experiments. (iii) DeiT-IIII uses nodes with 8 GPUs each when fine-tuning their pre-trained ImageNet-2lkk models on ImageNet. This provides opportunity to test lo-fi in an equivalent setting where there is normally communication between nodes.
Main results. Our overall finding is that communication between nodes is not necessary in this setting lo-fi matches the accuracy of the baseline while observing the same amount of data. These results are presented in Figure I and Tables and 2 In these experiments, lo-fi uses 4 groups each group corresponds to one node.
Figure ① illustrates accuracy throughout training when fine-tuning DeiT-base with and without lo-fi. We also report the average accuracy of the models produced by the individual nodes. To make this plot we display the accuracy of the averaged lo-fi model at the end of each epoch, though usually we would only average the models once at the end. A question emerges when looking at this plot: why does the accuracy of the individual node first dip before coming back up? The answer is due to the interaction of learning rate and batch size, which we discuss further in Appendix
Table evaluates the final models from Figure on ImageNet as well as under distribution shift (on ImageNet- V2 (Recht et al.) (2019), ImageNet-R (Hendrycks et al.) (021a), ImageNet Sketch (Wang ct al. 2019 and ImageNet-A (Hendrycks et al. (021b]). .. In addition, Table I Teppetts the experiment from Figure with the DeiT-large model. We underline any result that is significantly better (using McNemar's test with significance
5

---

Published in Transactions on Machine Learning Research (01/2023)
0.05). Overall we observe that lo-fi matches the accuracy of the baseline which uses communication, and outperforms the baseline under distribution shift.
Table 2 supplements Table ! with additional details. In particular, we consider the accuracy of the model produced by an individual node during lo-fi, before the averaging. We also evaluate the output-space ensemble of the models produced by each node during lo-fi, which more expensive during inference as pass through each model is required. Finally, we display the accuracy of the models fine-tuned in the DeiT-IIII paper [Touvron et al. 2022). We improved our bascline over that in the paper with the following hyperparemter changes: (i) Instead of removing the classification layer of the pre-trained model, we implement version of LP-FT (Kumar et al., 2022) to finc-tune- we preserved the ImageNet-21k classifier then use a class mapping from ImageNet-2lkk to ImageNet classes. (ii) We remove the grayscale, solarization, and Gaussian blur augmentations, since we found this improves accuracy. This aligns with previous research where fine-tuning requires less augmentation (Wortsman et al. 2022p. (iii) We fine-tuned for fewer epochs, which also required a switch to a cosine scheduler that updates every iteration instead of every epoch so the schedule could complete. We also considered different values for the learning rate and stochastic depth, but found the default values to be best (Touvron et al. (022). This is with the exception of Deige for which we found stochastic depth 0.3 to be better for the bascline, which is what we used.
Lo-fi was run using identical hyperparameters ex- cept we decreased the stochastic depth drop rate by 0.05 for both DeiT-base and DeiT-large since each node is effectively tunint on less data and may therefore require less regularization. The most sub- stantive change from the DeiT-IlI code was to use LP-FT [Kmar et al.) 2022), which we accomplished by preserving the classification layer from the pre- trained model and using a mapping from ImageNet- 21k to ImageNe While this change results in a minor improvement for the baseline, we found was necessary for achieving matching performance with lo-fi. Overall, despite the extensive hyperparameter tuning we performed for the baseline, lo-fi was still able to match or exceed the accuracy.
61.8
0.90999
0.9999
Avg. OOD accuracy
61.6
61.2
baseline
61.0
baseline + EMA lo-fi
A 0.99
0.999
60.8
85.825 85.850 85.875 900 85.925 85 95.950 85.975 86,000 ImageNet accuracy
Figure 3: For the experiment Table 1 lo-fi out- performs the baseline under distribution shift. We wanted to test whether this OOD performance (y- axis) could be improved by applying weight averaging techniques to the baseline. We observe that the an- swer is yes with EMA (Szegedy et al.) 2016). although this can come at slight cost in-distribution accuracy (æ-axis). In this plot we try different values of EMA decay B. Applying EMA lo-fi had minimal benefit, did applying WiSE-FT (Wortsman et al., 2021 to the baseline. The mnageNet-2lk--Imeeee
Ablations. We ran three ablation studies to bet- ter understand the performance of lo-fi in this set- ting. First, we wanted to lest whether adding more nodes was helpful. In the initial 4 node experi- ment with lo-fi, we matched the bascline in terms of total amount of data observed, allowing fair compute-matched comparison. However, there are practical settings such as privacy-preserve ML in which the benefits of reduced communication may outweigh the importance of matched compute. In Figure we observed that adding more nodes did not improve in-distribution accuracy. Interestingly, how- ever, adding additional nodes marginally improved out-of-distribution performance, most notably on ImageNet-Skotde and ImagcNet-A.
transfer setting is not characteristic of those studied in the WiSE-FT paper.
Next, we wanted to understand if four groups, one per node, was optimal in this setting. What happens if we instead use groups per node, or 2 groups each group consisting of 2 nodes? In this experiments the amount of data observed remains constant; all that changes is the amount of communication. As presented
The only class in ImageNet. but ImageNet-21k is teddy bear-we we initialize this row with bear instead.
6

---

Published in Transactions on Machine Learning Research (01/2023)
Groups
16
ImageNet Accuracy 85.95 86.00
85.85
85.73
Table 3: For our four-node fine-tuning jobs, we usually partition the 32 GPUs into 4 communication groups, one per-node. This lable shows the effect of partitioning the GPUs inlo groups of different sizes, finding slightly worse performance when the number of groups large.
ImageNet
ImageNet-V2
ImageNet-R
imageNettSketch
ImageNet-A
87.5
Accuracy
79
85
62
72.5
87.0
70.0
80
60
86.5
78
67.5
6
12
24
12
24
12
24
12 Fine-tuning epochs Fine-tuning epochs
12
Fine-tuning epochs
Fine-tuning epochs Fine-tuning epochs
lo-fi
baseline
Figure 4: We fine-tune CLIP VIT-L (Radford et al. 2021 Dosovitskiy et al. 2021 ImageNet. In contrast to the DeiT fine-tuning experiments, the models were not pre-trained with stochastic depth and we found better accuracy when fine-tuning without stochastic depth. Instead, we fine-tune for 6, 12, and 24 epochs. lo-fi shows good performance under distribution shift, but on ImageNet. requires more epochs to exceed the baseline accuracy unlike in the DeiT experiments.
in Table 3 accuracy drops slightly when using larger number of groups This result demonstrates that the best configuration is one group per node.
Finally, we found it interesting the lo-fi outperformed the baseline under distribution shift. Accordingly, we wanted to test whether we could recover these out-of-distribution (OOD) improvements by applying other weight averaging techniques to the baseline. We observe in Figure 3 that the answer yes, although at
slight cost in-distribution performance for the methods we tried. The best performing technique we tried debiased exponential moving average (EMA) (Szegedy et al.) 2016 Kingma & Ba 2014), for which tried decay values 0.99, 0.999, 0.9999, and 0.99999. We also tried applying EMA and WiSE-FT (Wortsman)
et al. 2021 to lo-fi, but did not observe out-of-distribution improvements
3.2 Fine-tuning CLIP VIT-L on ImageNet and WILDS
In the previous section we observed that lo-fi matches the baseline for DeiT-IIII on ImageNet, but how does lo-fi perform for models pre-trained larger datasets? In this section, further test lo-fi for the CLIP
ViT-I (Radford et al. 2021 Dosovitskiy et al.] 2021 when fine-tuning on ImageNet (Figure 4 as well as
two datasets from WILDS KKoh et al. 2021) (Figure 5
Unlike the DeiT models, CLIP was not pre-trained with stochastic depth and we find better accuracy when we fine-tune without stochastic depth. This unlike the DeiT-IIII models, which we found performed best when we used some stochastic depth. Indeed, this allowed did for the baseline by decreasing stochastic depth drop rate by 0.05. As this is no longer the case, instead show experiments when fine-tuning for different numbers of epochs. Other than this omission of stochastic depth and varying the training epochs, the hyperparameter configuration is identical to that discussed in the previous section and follows the ImageNet-2 2 fine-tuning set-up from Del (Touvron et al. 2022
use slightly less regularization for lo-fi then we
When using 2, 8, and 16 groups changed the stochastic depth drop rate by 0.05, -0.05, and -0.10, respectively, from the four group setting. Thee intuition from WiSE-FT Wortsman ct al. 2021 is that of combining generalist and specialist. Our intuition for why WISE-FT does not show substantial Improvements the 2ingeNet221k--mmgaee transfer setting because both models ImageNet specialists.

---

Published in Transactions on Machine Learning Research (01/2023)
WILDS-FMoW: ID
WILDS-FMoW: OOD
WILDS-WWildCam: ID
WILDS-WWidCCm: OOD
79
78.50
Accuracy
72.5
66
78
78.25
70.0
64
78.00
67.5
62
6
12
24
48
12
24
48
3
6
12
24
3
24
Fine-tuning epochs
Fine-tuning epochs
Fine-tuning epochs
Fine-tuning epochs
lo-fi
baseline
Figure 5: We repeat the CLIP VYT-L fine-tuning experiment from Figure one two other image classification
tasks: WILDS-FMoW (Koh et al. 2021 Christic et al., 2018), satelite recognition task with geographic
and temporal distribution shift and WILDS-WWidCam (Koh et al. 2021 Beery et al. 2021 camera trap dataset with a geographic distribution shift. Overall, we find similar results as in Figure
Results when fine-tuning CLIP VIT-L on ImageNet are presented in Figure 1 For this experiment, we inítialize the classification head the zero-shot model using the zero-shot classifier output by the CLIP text tower (as in Wortsman et al. 2021)). We observe that more fine-tuning epochs are required for lo-fi to outperform the baseline on ImageNet. Under distribution shift, lo-fi roughly matches or exceeds the baseline for each of the fine-tuning epochs we tried. While this result indicates that lo-fi is a promising alternative the baseline in this sctting, a key limitation is that additional fine-tuning epochs were required to enable this improvement. The accuracy improvements beyond the best baseline model are consistent with the results reported in model soups (Wortsman et al. 2022
We also test CLIP ViT-I on two further datasots, WILDS-FMoW (Koh et al., 2021 Christic ot al. 2018 a satellite image recognition datasel with temporal distribution shift and WILDS-iWildes
(Koh el al. 2021
Beery et al. 2021). a classífication dataset with camera traps the wild with a geographic distribution shift. Our motivation is test lo-fi on natural images beyond the ImageNet universe. The results are presented in Figure 5 observing very similar results to the aforementioned experiment of fine-tuning CLIP ViT-L ImageNet. However, there is an important difference in the experimental set-up. For these experiments, we first tried using the zero-shot initialization for the last layer of the model, as we did with ImageNet. However, this resulted in worse accuracy for lo-fi. Accordingly, these experiments are completed using the LP-FT method of fine-tuning (Kumar et al.) 2022). First, we train a linear probe using one node, This linear probe then used as the initialization when end-to-end fine-tuning individually on each node. We also apply this change to the baseline, but the benefit is much less substantial for the baseline than for lo-fi. Finally, for this experiment used learning rate 7e-4 which we found resulted in higher accuracy for lo-fi and the baseline.
3.3 Language model fine-tuning
We also test lo-fi outside of image classification by finc-tuning OPT-125M and OPT-1.3B (Zhang et al. 2022
Experimental Setup We report i)individual node, which is the average performance of the models pro- duced by each node when using lo-fi, ii) lo-fi, which averages models produced by each node, and iii) baseline, which uses communication between nodes. For the 125M parameter model, we set the learning rate to 6e- 5, with 1024-length sequence blocks, and 500K tokens per batch. For the 1.3B parameter model, we set the learning rale to le-5, with 512-length sequence blocks, and 1M tokens per batch. We use fp16 mixed precision (Micikevicius et al.] 2017) for all experiments. We fine-tune the 125M parameter model with 4 nodes, and we fine-tune the 1.3B parameter model with nodes. When using lo-fi there no communication between nodes, so the experiments produce and 8 models, respectively. Each node consists of 8 Volta 32GB GPUs connected with 400GBps interconnect.
8

---

Published in Transactions on Machine Learning Research (01/2023)
OPT 125M
OPT 1.3B
Perplexity (Comman Crawi)
individual node
Perplexity (Comman Crawl)
16.0
individual node
lo-fi
lo-fi
15.8
baseline
15.8
baseline
15.4
15.4
15.2
15.2
15.0
15.0
200
400
600
800
1000
1200
1400
200
400
600
800
1000
1200
1400
Iterations
Iterations
Figure 6: Fine-tuning language model (left: OPT-125M, right: OPT-1.3B) on Common Crawl with lo-fi closely approaches the performance of the baseline of multi-node fine-tuning with communication. Here, we train four lo-fi workers independently, one per node. The baseline consists of standard data-parallel fine- tuning using four nodes, where there communication between nodes at every iteration. The r-axis shows iterations, which does not take into account that lo-fi may be faster.
Results We fine-tune on the Pile's Common Crawl subset (Gao et al. 2021 using the Huggingface Trans- formers library (Wolf et al.) 2020). Results presented in Figure 6 We observe that for both model scales, when comparing by step count, lo-fi roughly matches the performance of the baseline, providing large performance improvements over the individual node sctting. These results suggest that lo-fi is an effective alternative to standard multi-node fine-tuning with communication.
3.4
How much is the speed-up, really?
We have shown that lo-fi produces high accuracy models without communication during fine-tuning. This leads an important practical question: what is the wall-clock advantage of eliminating communication between nodes during fine-tuning? We examine the wall-clock training time advantage once nodes are allocated and also the time it (akes for node allocation
slurm cluster. Note that these experiments are for the DeiT-III (Touvron et al. 2022 models in the image classification setting.
Wall-clock advantage. To examine the wall-clock advantage of lo-fi compared to the bascline A100 GPUs on AWS with fast interconnect of 400 GBps (EFA). This is representative of fast and modern large scale neural network training set-up. In particular,
want to understand the effect of using modern distributed training tools, and also varying batch size. We note that our results depend critically on the quality of the interconnect between nodes. In a setting with a slower interconnect such as standard ethernet, we would expect the training speed-ups to be more substantial. In a setting with a faster interconnect such as TPUs, the training speed-ups should be more minor.
A recent innovation in distributed training tooling is to overlap the backwards pass computation and gradient communication-tlee the gradients for layer - can be computed at the same time as communicating the gradients for layer é LLi et al.) 2020a Paszke et al. 2019 4 We experiment with turning on and off this overlapping communication www.riiiattooooooptaaaatiiio ccomputation feature, finding substantial reductions in communication overhead when overlapping communication and computation. We also experiment with changing the batch size. general, we observed that when using smaller batch size, communication will account for a larger portion of training time. This is because the size of the gradient does not depend on the batch size, absolute communication cost does not depend batch size. However, using smaller batch size will lower the total computation time and therefore communication cost will account for larger fraction of the total training time.
+Overlapping communication//coppaaaai on by default PyTorch 21.5 Passike al. 2019
al. 2020a

---

Published in Transactions on Machine Learning Research (01/2023)
Communicatioon overhead: (%)
50
Wait time for job allocation (hours)
40
30
2.5
added time
2.0
1.0
base
large
huge
giant
(1 params)
(1e9 params)
Model size
4
Number of nodes
overlapping communication/compute (batch 16)
overlapping communication/compute (batch
Overlapping communication/computee (batch
Overlapping communicationtoompue (batch 32)
Figure 7: (Left) On an AWS cluster we show on the y-axis the wall-clock overhead observed when switching from to 4 nodes using models from the DeiT-III repository (Touvron et al. 2022) and constant, per-GPU batch size, 100% indicates that the job becomes twice as slow while 0% indicates no difference switching from 4 nodes. With the method of overlapping the communication and computation in the backward pass (Li al. 2020a), the slow-down is less substantial than initially expected, especially for larger per-GPU batch sizes. The huge and giant models deeper and there is more opportunity to overlap communication and computation. (Right) Jobs requiring only one node schedule faster than jobs requiring four nodes the slurm cluster that we use for these experiments. This plot shows the median per-day wait time averaged over three months of job data on this cluster.
Our experiments with varying batch size and turning and off overlapping conmunication/comptttiii are presented in Figure (left). These experiments are for the vision transformer models DeiT-base, DeiT-large, DeiT-huge and DeiT --gaant, ranging from roughly 10 to 109 parameters. On the a-axis we show the different model sizes, while the y-axis shows the additional wall-clock time required to go from a node to node job (i.e., 0% indicates that the node job the same speed as the node job, while 100% indicates that the node job is twice as slow). In this experiment, the number of iterations and batch size per device is fixed We found that without overlapping communication/ cttpuuiiiaatioooopptttt.. compute, shifting substantial increase in training time of 25-55% (purple lines). However, overlapping communication and compute has proven surprisingly effective, reducing the communication cost to <10%.
multi-node settings results in
A potential issuc with these experiments is that they currently reflects a "statc-of-the-art" cluster sctting, and actual workflows may be slower. We also believe that both GPU memory size and network bandwidth will improve the future. Higher GPU memory capacity will allow users to train larger models, resulting in higher communication overhead, while higher network bandwidth will help reduce the communication delay.
Finally, we note that lo-fi can help the straggler and jitter problem in distributed training, where different nodes might experience random slowdowns due to various reasons. In standard data-parallel, synchronization will take place multiple times per iteration, such that any random slowdown on any node will slow down the entire run. Since lo-fi needs only one communication at the end (which can even be asynchronous), the straggler/jittor problem no longer an issuc.
We note that scaling with fixed hatch size may unrealistic for problems large batch accuracy drop, which would he reason use lo-fi.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Scheduling advantage. For modern cluster workloads, both on private and public clusters, the wait time to schedule a job can increase the total training time, especially during periods of heavy cluster usage. Since single-node jobs require fewer simultaneous resources to run, they should schedule faster, reducing the total training time. To mcasure this, we analyzed the time required to schodule a 1-node job vs. a multi-node job on a large slurm-based cluster and present the results in Figure 7 (right). These wait times are averaged over all jobs run on this cluster over three month period. We found that schoduling a single node job was notably faster than multi-node jobs, taking ~45 minutes for node, ~2 hours for 2-4 nodes, and ~-3 hours for 8 nodes.
We note that these results are specific to the cluster used in these experiments and may or may not be representative of other clusters depending on their scheduling algorithm and workload distribution, amongst other factors. We also note that the scheduling benefit will only apply when using implementation B in which each group is trained independently (as described in Section P) Regardless, we thought that it may be useful to collect and present this empirical data, providing quantitative support for the observation from (Li ct al. 2022) that jobs requiring fewer nodes schedule faster.
3.5 Does jointly training to increase diversity across groups improve lo-fi performance?
Previous work from [Gontijo-Lopes al. 2021 Wortsman et al. 2022 has shown that more diverse models trained with different hyperparameters produce larger benefits when ensembles or weight averaged and also Li et al., 2022) which showed that ensembling or weight averaging specialists trained on different domains incurs the largest benefit. We therefore asked whether encouraging diversity automatically through regularization during training might improve the performance of the final lo-fi model.
While this strategy did indeed produce models with a larger averaging benefit (avg. model - best individual model), it also decreased the accuracy of the individual models such that overall performance was the same or worse than simply training the lo-fi components independently. We also tried pulling togother the predictions of the models, which is also known as co-distillation (Anil et al., 2018 Sodhani et al., 2020). This improved the accuracy of the individual models, but model diversity decreased, the benefit from weight- was reduced, also leading to overall lower accuracy. We explored a number of variations of these approaches which we discuss in more detail in Appendix A
4 Related work
Averaging and linearly interpolating models. Averaging or interpolating the weights of neural net- works is a common technique for improving accuracy.
Weight-averaging techniques for optimization date back to carly work in convex optimization (Ruppert 1988 Polyak 1990). In deep learning, an exponential ial moving average (EMA) of weights can be used to improve accuracy (Szegedy et al. (2016). Another popular approach is Stochastic Weight Averaging (SWA) (Izmailov ct al. 2018 which uses uniform average of weights saved at each epoch while training with constant cyelic learning rate. Indeed, the SWA method was motivated in part by the analogy between weiggtt-vveaaging and ensembling.
While SWA and EMA average weights along the training trajectory, there has also been substantial interest in averaging weights across independent trajectories. In particular, Nagarajan & Kolter (Nagarajan & Kolter 2019) observe that the weights of two models that are fine-tuned independently on MNIST (LeCun et al. 2010) from a shared initialization can be interpolated without increasing loss. For more difficult problems such ImageNet, this naive linear interpolation encounters high error barrier (Frankle et al.] 2020 Fort ett al. 2020). However, Frankle et al. (2020) observe that when the first part of the optimization trajectory is shared and the remainder of training is independent, models can once again be interpolated without reducing accuracy. They refer to this phenomena interpolating weights without accuracy loss as linear mode connectivity. Neyshabur et al. (2020) observed a similar phenomenon when interpolating between model pairs that are fine-tuned from a shared initialization on a new task. This obscrvation was extended to interpolation between zero-shot model and fine-tuned model with the WiSE-FT approach (Wortsman et al. 2021 to many models fine-tuned with different hyperparameters with model soups (Wortsman et al.
11

---

Published in Transactions on Machine Learning Research (01/2023)
2022), to models fine-tuned on different datasets with Ilharco et al. (0022 and for creating better pre-trained models by Choshen et al. (2022). The overall observation that the objective landscape can appears roughly convex was also made by Li et al. (2018). While all of the aforementioned weight-averaging employ simple linear interpolation, more advanced weight-averaging techniques have also been developed with promising
results (Matena & Raffel
2021
Recently, Li et al. (0022) introduced branch-train--ee merge which is at the intersection of model combination and distributed training. They consider the case where the training data is partitioned into diffcrent textual domains, then train an individual expert model for each domain. As they are training from scratch, they first roquire an initial seed phase. They then combinc all of these experts via weight averaging or ensembling to outperform the dense baseline of training one large model on all of the data. The main difference are that our work is for fine-tuning, and we do not assume the data is partitioned into different domains.
Other rescarch in the area includes Garipov et al. (2018) and Draxler et al. (2018) who concurrently found that two neural network solutions trained independently can be connected by a simple curve along which loss remains low. These findings were generalized by Benton et al. (2021) who learn high dimensional low- loss connectors between individual solutions. Concurrent work with Benton et al. (2021). Wortsman et al. learned these high dimensional low-loss subspaces from scratch. Then, Entezari et al. (2021) conjectured that all solutions could be made to be linearly connected by applying a permutation to the weights which does not change the function. Ainsworth et al. (0022) recently made progress towards confirming this conjecture. However, unlike the model interpolations we observe here, and have previously been observed (Wortsman et al. 2022, Li et al., 2022), the interpolations in Ainsworth et al.) (2022) so far do not improve models in terms of accuracy. Regardless, they are interesting from a scientific perspective, and suggest the possibility of applying methods such as lo-fi for training from scratch in the future, although there is currently no evidence towards this.
Distributed training (Li et al. 2020a Yuan et al. 2022) and fine-tuning (Borzunov et al. 2022 Wang et al. 2022) are increasingly important in deep learning as models become larger.
Distributed training and fine-tuning.
An overview of the many standard approaches is detailed by Weng & Brockman (Weng & Brockman 2022 including i) data- parallaaallllimm, where data is split among devices, ii) pipeline parallelism, where different layers are split among devices, and fif) tensor parallelism, where individual layers are split among devices. We note that these approaches are not mutually exclusive. Indeed, one can use pipeline parallelism to distribute model across node, then use data-parallelism across nodes. lo-fi is proposing an alternative to data- parallelism across modes-iinteell of synchronizing the updates between nodes during fine-tuning, each node independently produces a model which is averaged at the end of fine-tuning, We emphasize that lo-fi can still be used if there is pipeline parallelism across the node.
There have previously been many alternatives proposed to synchronizing gradients each step. The idea of training several models in parallel and averaging their weights once at the end of training has been investigated at least since (MeDonald et al. 2009 Zinkevich et al.] 2010). The focus in those works is on convex models and training from scratch, rather than fine-tuning. While lo-fi is simply borrowing these techniques from convex optimization and applying them to tine-tuning for deep learning, we believe that these findings are interesting and useful from a practical standpoint.
Another alternative, HogWild (Recht et al., 2011) proposes asynchronous communication. The difference between HogWild and lo-fi is that lo-fi never communicates during fino-tuning, so it's as the hogs cach have their own individual farm. As another alternative, local-sgd (Stich) 2018 Ortiz al. 2021] communicates updates every l steps instead of every step. lo-fi is equivalent to local-sgd applied to fine-tuning where k is the number of fine-tuning epochs.
There have also been compelling recent methods for more efficient and accessible pipeline or lensor parallelism to enable learning or inference with extremely large models. For instance, with Petals (Borzunov et al. 2022) certain layers of very large models are computed and communicated among coordinated users. Also, rescarchers have been using decentralized training for very large models (Huang et al.] 2019 Yuan et al.
2022 which is made possible by, e.g., compressing communication [Wang et al. 2022
Kairoux et al. 2021
12

---

Published in Transactions on Machine Learning Research (01/2023)
Xic et al. 2020 Faghri al. 2020 Li et al. 2020bj. Indeed, even inference with very many-billion parameter
models can pose interesting challenges (Dettmers et al. 2022). Our approach is orthogonal to the work in compressing communication, as we are instead removing communication, but may prove useful for large-scale decentralized training.
There is also the active research area of federated learning (e-g., Kairouz et al. 2021 0: Pillutla et al. (2022)
which has recently been explored in transfer settings (Nguyen al.) 2022). In federated learning, the data each client is different and updates
usually communicated every & steps. While lo-fi only considers the easier setting of IID data, it is possible that similar approaches based weight averaging to reduce communication may prove beneficial for privacy-preservngg machine learning.
5 Limitations and conclusion
Limitations. There are many limitations discussed throughout this text. For instance, we found that when fine-tuning CT.IP VrT-L ImageNet and WILDS, lo-fi needs observe more data to exceed the baseline. This is similarly true during language model fine-tuning (Section 3.3 Therefore, when communication costs are prohibitive. A final limitation is that lo-fi can only achieve matching accuracy when no new parameters are introduced, which we accomplish with a "zero-shot"" initialization, or via LP-FT (this does not come up during language model fine-tuning).
most recommend lo-fi
Conclusion. Overall we have observed that communication between nodes is not required during fine- tuning in certain settings. These findings may prove beneficial to number settings including large-scale decentralized fine-tuning and privacy-proseriing ML and represent promising step in the overall dircction of developing models like open-source software (Raffel 2021) in which many institutions can collaborativel fine-tune a large model if none has the resource to do so individually. As more workloads shift to fine-tuning of pre-trained models and models grow increasingly larger, we hope that our results will help to reduce barriers to large-scale models.
Acknowledgements
We thank Beidi Chen, Suraa Ganguli, Caleb Ho, Gabriel Ilharco, Teng Li, Mansheej Paul, Alex G, Andrew Saxe, David Schwab, Shubho Sengupta, and Hugo Touvron for useful discussions.
References
Samuel Ainsworth, Jonathan Hayaso, and Siddhartha Srinivasa. Git ro-basin: Merging models modulo permutation symmetries. arXio preprinl wrXiw:2209048866 2022.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. ar preprint arXiv:1804.0888 033.55 2018.
Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. Conference on Computer Vision and Pallern Recognilion (CVPR) FGVC8 Workshop, 2021. https:
V/arxiv.org/a wws/////////...o///..///// 03494
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference Machine Learning, pp. 779. PMLR, 2021.
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chu- machenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and finc-tuning of large models. arXio preprint ar XrXi:2220900 188, 2022.
Benjamin Brazowski and Elad Schneidman. Collective learning by ensembles of altruistic diversifying neural networks. Xixio preprint ar Xrx::2006. 1 11671, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020. httpps://axxiv. wwtpa//.rrrgabb///o//////// 141655
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finctuned models for better pre- training. arXiu preprint arXiv:224.003044 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways, 2022.
Gordon Christic, Neil Fendley, James Wilson, and Ryan Mukherjoe. Functional map of the world. In Conference Compuler Vision and Pallern Recognilion (CVPR), 2018. https://arxiv httaaaawwi.rraaaaaaaaaaeae abs/ abs/1711. 1711. 07846
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009. https://ieeexplore.
ieee. Leee.org/ee httg/ddcummnn/522664
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale, 2022. http::///arxii..og/bbb//////////
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine- tuning pretrained language models: Weight initializations, data orders, and early stopping, arXiv preprint arXiv:2002.08 06305, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words; Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021. http:s//aardvv.org/aa////0 11929
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309-130 1318. PMLR, 2018
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in lincar mode connectivity of neural networks. arXiv preprint ar arXiv:2110.0020 06296, 2021.
Fartash Faghri, Iman Tabrizian, Ila Markov, Dan Alistarh, Daniel M Roy, and Ali Ramezani- KebrzyannnnKKKaa... Adaptive gradient quantization for data-parallel sgd. Advances in neural information processing systems, 14:3174-3185, 2020.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural langent. kernel. In Advances in Neural Information Processing Systems (NeurIPS), 2020. /ttp::///axxxvoro//// httgg/abs/2000...0
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), 2020. https:
//arxiv.org hwrrg/bbs/1912 05671
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https://arxxw.org/eeeeeee
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Conference on computer vision and paltern recogni- tion (CVPR), 2014. https://openaccees .app__aapr..pppe
peaecff.com/coonetttcccooo hrt.aett..sppppll///ppppeerrrrar _iich__
Feature Saabure_Hierarchiee.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk. No one representation to rule them all: Over- lapping features of training methods. Xiv preprinl (t:::2110.12999 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributiio generalization. International Conference Computer Vision (ICCV), 2021a. httpp://arxxtv.orrgabb///// 16241
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. Conference on Computer Vision and Pattern Recognition (CVPR), 2021b. /tttps///arxv..org
abs/1907.C .07144
Gao Huang, Yu Sun, Zhmang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 146-661. Springer, 2016.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Tee,, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in newral information processing systems, 32, 2019.
Gabriel Ilharco, Mitchell Wortsma, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Korn- blith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights, 2022.
attps://arxivv.rg//aae a20/aaa////...///////////// 05592
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averag- ing weights loads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018. http:://aaxx...orrrr/////133 05407
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412 12. 6980, 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDDS benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. https: wrg//ba/////ooooooooooooraeere//
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Ncil Houlsby. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision (ECCV), 2020. httpps://arrtv.org/aab////
11370.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. https://arxi:: wwaexiv.org///e org/ a0gg/85bs/88 08974
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine-tuning can distort pretrained features and underperform out-of-distribation.. In International Conference on Learning Representations, 2022. URL httts::/ htt.pp/eweeeeeee.eeeo/ooooo/etna
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. 2010.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models, 2022. https://arxil http//aaivvv.g.g/aee// prggabs/2206..oo. 14858
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Margaret Lí, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettle- moyer. Branch-train-merge merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.0300 2022.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. Xiv preprint arXiv:2006. 15704, 2020a.
Zhize Li, Dmitry Kovalev, Xun Qian, and Peter Richtárik. Acceleration for compressed gradient descent in distributed and federated optimization. arXiv preprint vrXxx:2002.11364 2020b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrv: 1 11692, 2019.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2021. Ittpp://aaxiiv.
org/abs/2111.09822
Ryan McDonald, Mehryar Mohri, Nathan Silberman, Dan Walker, and Gideon Mann. Efficient large-scale distributed training of conditional maximum entropy models. In Advonces in Neural Information Process- ing Systems, volume 22, 2009.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. anXiv preprint xrXit:1710.0555 08740, 2017.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain gen- eralization in deep learning. In Н. Wallach, Н. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceedding http:///gg....oooooccc/////////d
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Tn Advances in Newral Information Processing Systems (NeurIPS), 2020.
11687
John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? exploring the impact of pre-training and initialization in federated learning. arXiv preprint XrXt::2006.15387 2022.
Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat, Ari Morcos, and Nicolas Ballas. Trade-offs of local sgd at scale: An empirical study. arXiv preprint arXiv:21 10. 08100813333 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. arXiv preprint Xrx:22203 02155, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024 8035, 2019.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Foderated learning with partial model personalization. In International Conference on Machine Learning. pp. 17716 17758. PMLR, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Boris Teodorovich Polyak. New method of stochastic approximation type. Automation and remote control, 1990.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and lyya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. httpp::/aaraiv.orggaaaasa
Colin Raffel. A call to build models like we build open-source software, 2021. https://coltarafeel.com
html.
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to paral- lelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize ImageNet? In International Conference on Machine Learning (ICML), 2019. https://arxiv.orr org/
abs/1902 10811
David Ruppert. Efficient estimations from a slowly convergent robbins-monro process, 1988.
https://
ecommons. cornell......oooo wdw.haaddee/111//66
Shagun Sodhani, Olivier Delalleau, Mahmoud Assran, Koustuv Sinha, Nicolas Ballas, and Michael Rabbat.
A closer look at codistillation for distributed training. arXiv preprint arXiv:2010.022 2020.
Sebastian Stich. Local sgd converges fast and communicates little. or Xix preprint or 1xx:1100.00000 2018.
Christian Szegedy, Vincent. Vanhoucke, Sergey Toffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2818-2826, 2016.
Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. arv preprint arXivv2204.07114 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 2018.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NewrIPS), 2019.
https: wtwgg/////////reo 13549
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Re, and Ce Zhang, Fine-tuning language models over slow networks using activation compression with guarantees. arXiv preprint arXiv:2206.0133 2022.
Lilian Weng and Greg Brockman. Tochniques for training large neural networks, 2022. https://openai.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-tho-al natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrutions, pp. 38 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.16653/v1/200 emnlp-demos.6. URL htttp:://cclantholll org///00 emnlp-denee demos. 6
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastogari. Learn- ing neural network subspaces. In International Conference on Machine Learning (ICML). https: 7/proceedings. mlr. wwwps//ssssv/eseeeee https//////////ttmaaaaaaaaa html
17

---

Published in Transactions on Machine Learning Research (01/2023)
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. htpp:://axxiv.org/bbbb. /0000/0/999109033
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022. https://arxiv: www.:/o.ggooaoo.. abs aa/////a///////// 05482
Cong Xie, Shuai Zheng, Sanmi Koyejo, Indranil Gupta, Mu Li, and Haibin Lin. Cser: Communication- efficient sgd with error reset. Advances in Neural Information Processing Systems, 33:12593 12603, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NeurIPS), 2014. https://arxiv. org/abs/1411 11000///111..9999
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXic:1904 190400062 2019.
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christo- pher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. arXiv preprinl ar X2v: 01288, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022.
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. In Advances in Newral Information Processing Systems, volume 23, 2010.
18

---

Published in Transactions on Machine Learning Research (01/2023)
A Negative results when using regularization to promote model diversity
We also experiment with explicitly regularizing the models to have diverse predictions. We are motivated by previous work which shows that having diverse models can lead better ensembles or weight-averages (Bra- zowski & Schneidman 2020 Gontijo-Lopes et al. 2021 Wortsman al., 2022 Li et al. 2022). In particular, wanted to push apart model predictions from different nodes. However, in the standard set-up, different nodes observe different data. As an alternative to this, at cach itcration we had pairs of nodes observe the same data. We did this without reducing batch size by using cutmix Yun et al. 2019) with the data from one node to another. However, without any further changes, this modification reduced accuray by roughly 0.15pp on ImageNet. We suspect this is because the models on different nodes became less diverse by sharing data.
Let y1 and y/ be the predictions from two nodes who observed the same data. We experimented with adding the term A DxI((y to the loss where DRL. is the KL divergence. The motivation was that with À < 0 the models would become more diverse as in [Brazowski & Schneidman 2020p. When À > 0, this procedure would become co-distillation (Anil et al. 2018) and hopefully accelerate training. However, while < improved the absolute benefit from averaging models, reduced the performance of the individual models. Moreover, while of averaging. These experiments are presented Figure g finding that any change to À lowered the accuracy of lo-fi. We also experimented with number of other approaches to improve diversity, including using other distance metries, only pushing apart incorrect examples, pushing apart model weights, and also taking the PCA of the predictions and pushing apart in the unimportant PCs. However, all of these approaches exhibited the same qualitative effect: improving diversity reduced individual model performance and increased the benefit of averaging, but not by enough to offset the individual model reduction. Interestingly, while our search was not exhaustive, this result may suggest that simply fine-tuning models with random seeds might produce the optimal amount of diversity for ensembling and model averaging.
improved the performance of the individual models, it decreased the benefit
85.8
ImageNet accuracy
85.4
85.2
85.0
84.8
84.6
84.4
5
20
25
Epochs
coefficient 0, parameter average
KL coefficient 0, individual model
KL coefficient 1, parameter average
coefficient individual model
KL coefficient -0.1, parameter average
KL coefficient 10.1, individual model
Figure 8: We tried "pushing apart" "pulling together" the models at each of the nodes by adding the KL di- vergence between their predictions to the loss. Using positive coefficient is equivalent to co-distillation (Anil et al. 2018 which improved the individual models but decreased the accuracy of the average. Using neg- ative coefficient in (Brazowski & Schneidman 2020) overall increased the improvement from averaging but decreased the accuracy of the individual models. Overall, these approaches did not improve lo-fi.
19

---

Published in Transactions on Machine Learning Research (01/2023)
ImageNet
ImageNet-V2
ImageNet-R
ImageNet-Skecch
ImageNet-A
77.0
Accuracy
86.0
48
58
85.8
76.5
47
1all
10 15
10 15
5
10
15
10
15
5
10 15
Number of nodes
Number of nodes
Number of nodes
Number of nodes
Number of nodes
Figure 9: We supplement Figure] with the additional baseline of standard multi-node fine-tuning with communication. We do not replace Figure 2 as the intention of Figure 2 is to visualize how the curve changes as the r-axiss changes, and our intention is not for readers to view the figure as a comparison between lo-fi and the baseline.
B A comment on learning rate in Figure 1
A question emerges when examining Figure 1 why does the accuracy of the individual node first dip before coming back up? The answer is due to learning rate. While both lo-fi and the baseline use the same learning rate, which is the learning rate used by DeiT-III paper of 3c-4, the individual lo-fi nodes have a smaller global batch. Therefore, this same learning rate acts larger. We tried to increase the learning rate for the bascline so that it also had this down-then-up trend but it resulted in worse accuracy. We also tried changing the learning rate for lo-fi but this also reduced accuracy. This is similar to an observation made the context of EMA model soups (Wortsman al. 2022), which that the best model for averaging weights is not necessarily the best model overall. We believe the learning helps individual nodes produce models which are different, and there is therefore more benefit from their combination.
C Additional experimental results
This section presents additional experimental results. First, supplement Figure with the additional baseline of multi-node training (Section C.1). Next, explore the application of lo-fi for fine-tuning tasks from GLUF Wang et al. (0088)
C.1 Adding the additional baseline of multi-node training to Figure 2
In Figure supplement Figure 2 with the additional baseline of multi-node training. Due to compu- tational constraints for simultaneous nodes, we do not exceed 12 nodes. Overall we observe similar curve shapes for both the baseline and lo-fi.
C.2 Additional natural language processing results
In Section 3-3 we explored fine-tuning with the language modeling objective. This was because standard fine-tuning tasks in natural language processing are relatively small and don't require multiple nodes. Tn this section, we expand our results to also consider fine-tuning a RoBERTa-base (Liu et al. 2019 model on tasks from the GLUE benchmark (Wang et al.) 2018). Instead of fine-tuning on multiple nodes without communication, we instead fine-tuning on multiple GPUs without communication. Since we are introducing new parameters (i.c., the classification layer), we use LP-FT similarly the experiments in Section 3.1 We accomplish this via common first epoch among lo-fi workers. The baseline is then 40 epochs with communication among 4 GPUs while each of the 4 lo-fi workers is 10 epochs on GPU before final averaging step. In contrast to vision experiments, NLP fine-tuning can sometimes have low accuracy depending on the seed (odge et al.] 2020). To account for this we use the greedy averaging technique from Wortsman et al. 2002 to combine models instead of the uniform averaging used throughout the rest of the paper.
20

---

Published in Transactions on Machine Learning Research (01/2023)
SST2
COLA
QQP
MRPC
RTE
MultiNLI
baseline
94.30
61.34
89.29
92.10
77.98
87.51
lo-fi
95.30
63.33
89.12
92.20
77.98
87.83
Table 4: The performance of lo-fi compared the bascline when finc-tuning on GLUE tasks (Wang et al. 2018).
Details in Section C.2
21


Reviews:
[{"review_id": "F8izzRPOkQ", "reviewer": "Reviewer_xXhU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper has a lot of substance, experiments and ablations.\nThe results show a clear trend, there is little doubt the method works and achieves the claims in the paper.\n\nMinor:\nThe robustness to more epochs (e.g. Fig 4,5) is interesting and a good property. Have any guess as to why is it happening? It is also a small weakness that it sometimes requires more epochs to surpass the baseline, is that a cost of distributing or related to the drop rate?\n\nWeakness:\nWhile the framing discusses distributed finetuning, this fits the results only with some caveats. Finetuning can be done without multiple GPUs and reach reasonable results and this method could be seen as a way of ensembling weak models. This is little discussed, unless I missed it, but we see that training on just one node works not that bad, so essentially what the paper proposes is training without distribution, stopping before convergence (right?) and getting a lower result, and then performing a model-soup-like ensemble that, as a good ensemble, improves results over the individual models. So in essence we could just put more computing effort, calculate anything we want more times (for example the full fine-tuning or a partial one) and make an ensemble to get better results. That is far from new or surprising. So the paper does not really say deep things about the low communication, but instead could be understood as saying things on ensembling of undertrained models, which we can expect. This alternate view should be better discussed. For example, why train with a small batch size (1/K) and not regularly (accumulate to account for missing nodes) but stop in the middle of training, is it better? worse? faster? Ensembling weak models also goes a lot back, connecting this paper to that branch may allow others to more easily improve it.\n\nIt should be emphasized more how lo-fi is in comparison to regular non-distributed training. Overall data is the same, overall GPU number is the same (although don't have to be), what about logical batch size (after accumulation), per node data, overall computation time? Except for the communication what changes with it? There are 3 aspects (data, compute and accuracy) and each of them is separated (device\\node\\overall + maybe accumulated batches). It takes time to understand, what of those require further costs, what are the possible gains and what is held constant. The accuracy part is quite elaborated, but the other aspects are spread across the paper despite being the real reason for someone to prefer this method.\n\nAs far as I can tell (and the fact that I am unsure is discussed above), the difference between lo-fi one node and regular training is that lo-fi batches are 1/K size, but same learning rate. In that case, the authors show that training was worse when choosing the wrong learning rate, there is even Appendix showing it is likely the case. This raises suspicions, could the one node perform ok with an ok learning rate? (with gradient accumulation, surely it can because logically it is equivalent, the paper should mention it and explain why this is possible but not advisable as it is slow) If it can, then no problem, more probable is that we will see that the method can just boost some lower-performing models by fusing them (like model soups etc.). But then, the framing is different, we want to just train faster, not to get better results, because better results are achievable by training several times and merging (as we said model soups). This method proposes to train inefficiently without communication and not lose (much?), but gain in speed. So more emphasis should be put on times. Specifically, they can train each node better (fit the parameters, you made so much effort fitting the baselines well you can fit your own too), and stop the training given the same computation as given to the other method but not communication, or even stop before, and show at which point you can stop. Note, running until convergence will tell us nothing, the framing here is about no communication, but without communication and with more compute\\time anything could be done, without Lo-Fi, just train on one node K times longer and accumulate gradients, this will be performing the computation of the K nodes on 1, and is not interesting.\n\n\nMinor:\n\nThe introduction mentions nodes, but the method defines devices. Are those the same? Do those do anything other than processing? would they be called processing units (GPU TPU CPU?)? Device for me hints of phones, tablets, laptops, PCs etc. but, it may be just me, and I'm not even a native speaker. Only when discussing some devices talking to each other and others that do not did I understand you are referring to slurm-like definitions. You should define those somewhere. A node is a group of devices which can better communicate within themselves than with devices from different nodes. Also, make sure to define things in a general manner, if hardware would change tomorrow and call a group for example \"pod\" then the methodology of your paper, that in no way relies on the hardware should still be clear. Figure 1 is actually clearer in describing the setting.\n\nIt is not overflowing the paper, so it is not critical, but still, because no experimental setup is provided, various details are written in places where they give no benefit for the reader. For example, when trying to understand the method, \"In the majority of our experiments, each group is a single node containing 8 GPU devices.\" is of no interest.  Or which code is used for DeiT which is mentioned at least 5 times.\n \nThe introduction describes regions in the low space in quite a detail, which to me seems to miss the point. The motivation for the work is not clear. This is only the background to why it is possible or what came before this work. For example, what is a node (first mentioned when the method is introduced), or why would costs of communication between nodes (GPUs) would ever be \"prohibitive\". This is the motivation of this paper, communication during fine-tuning, not low regions in space, which are related but do not make this paper exciting, just possible. There is another possibility, and that is that the paper does make regions in space during finetuning its main motivation, but this would require reframing if not a bit different focus on the experiments. While changing this will mean rewriting the introduction, I don't see it as necessary for the paper to be published, just for it to be more exciting or convincing.\n\nWhy did you use different implementations in text and vision experiments? Did you test how they affect results or times? I also could not follow the explanation of the implementations, you state this is implementation after the one-time communication, why batches and gradients are still needed at this stage?\n\nFigure 1 which the paper heavily relies on does it some disservice. The \"lo-fo(individual node)\" has little to do with lo-fi. It is not that the paper proposes to use this method. This looks like there are two things the paper suggests, and one of them might be tolerable despite lower performance.\n\nIf every node can already train the model by itself, and you allow only one communication, why not have a small finishing touch just to improve over the noise of averaging\\merging\\fusing the model? For example, only learn the last layer to make sure it predicts based on the inputs. Or perform some final batches after the fuse (we have found it to help in other scenarios).\n\nThe tables would benefit from a split to in domain and out of domain, as this is the main claim and the names of individual test sets are necessary but not the main message. This would highlight where lo-fi \"wins\".\n\nWhen you write \"drop\" in Table 2 you mean stochastic depth drop, right? worth writing it in the caption, as drop with little context may be dropout or other things.\n\nWhy is appendix B the first one referenced in the text? Makes you think you skipped a note somewhere before. (By the way, the addition of this mystery is very nice)\n\n\"we found it was necessary for achievingmatching performance with lo-fi.\" it is quite ok not to achieve the same performance with a more restricted method. how big was the drop?\n\n\"in which this the benefits\" typo", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work presents a distributed way of finetuning by finetuning separately and ensembling\\merging models, with little communication. This allows for different nodes (groups of processing units that are interconnected well) not to communicate during training and still perform as well or even better. A crucial matter here is that, per node, the amount of computation is held constant. So it produces a protocol where training is more efficient when node communication is slow or problematic.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Mainly, things in the framing bother me: the relations to ensembling and individual training and the over discussion of the theoretical reasons to believe it would work (which are great but do not replace talking about this work too...), but under discussion of motivation and terminology.\n\nThe explanation of the motivation is unclear to me. Also some of the nuances of the experimental setup, specifically those related to amount of compute, the answer to the question what really differs between each baseline (except the hardware speed and other factors that do not affect the reported results), especially between the one node lo-fi and the regular training. I believe I understand them now, but it was not explicit in any one place.\n\nMinor adjustments are just the minor weaknesses,", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nsp7RG8ZNR", "reviewer": "Reviewer_iWNq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of lo-fi is easy to follow.\n- Improving the communication efficiency of distributed fine-tuning is a promising research direction.\n\nWeaknesses:\nMy major complaint about this paper is that the idea is simply too straightforward. Conducting distributed computation with only final gradient/model aggregation (i.e., after the convergence of all locally trained) is a simple and old idea (please see PSGD proposed in [1], which appeared at NeurIPS 2010). Lo-fi, in my view, is a simple extension of PSGD to fine-tuning tasks, which has limited novelty. \n\nMore concerns are summarized below:\n- Many methods have been developed to make fine-tuning very memory and/or computation-efficient, e.g., the ones in [2-3]. In fact, those methods also manage to save communication costs under a distributed scenario (as their number of active tunable parameters is so few). I wonder how lo-fi compares against LoRA and Adapter in [2-3].\n- The proposed lo-fi method seems to only apply to data parallelism. For the model that can not fit into a single GPU memory, how does lo-fi work? In another word, can lo-fi be modified to support model and pipeline parallelism strategies?\n- The draft seems to be written in rush, there are a few issues in formatting/writing, e.g., there is an abrupt line break at the end of the second paragraph of the Introduction.\n\n[1] https://papers.nips.cc/paper/2010/file/abea47ba24142ed16b7d8fbf2c740e0d-Paper.pdf  \n[2] https://arxiv.org/abs/2106.09685  \n[3] https://arxiv.org/pdf/1902.00751.pdf  \n[4] https://cs.stanford.edu/people/matei/papers/2021/sc_megatron_lm.pdf  \n[5] https://arxiv.org/abs/2201.12023  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper proposes a method lo-fi, which aims at achieving distributed fine-tuning without extra communication. In its vanilla version, lo-fi simply conducts local fine-tuning on each computing device before a final averaging step. \n- Motivations and intuitions are provided for the proposed lo-fi method.\n- Empirical and ablation studies were conducted to justify the effectiveness and communication efficiency of lo-fi.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- One interesting direction/study that can be added to improve the current draft is: [1] is guaranteed to converge for convex functions. We all know that the loss landscape of a deep neural network is not convex at all, however why PSGD works particularly well for fine-tuned deep neural networks? Does the pre-training make the loss landscape more convex? I believe that some studies in [2] can be borrowed.\n- The proposed method lacks novelty. The authors have to convince potential readers very well why their proposed method is important.\n- The authors are encouraged to compare lo-fi against the other efficient fine-tuning methods, e.g., LoRA and Adapter.\n- A discussion on the compatibility of lo-fi to model and pipeline parallelism can improve the quality of the paper.\n- The writing quality of the current draft can be improved.\n\n[1] https://papers.nips.cc/paper/2010/file/abea47ba24142ed16b7d8fbf2c740e0d-Paper.pdf  \n[2] https://arxiv.org/abs/1712.09913  ", "broader_impact_concerns": "I do not find any concerns about the ethical implications of the work. Thus I would not urge the authors to add a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yUZF3YVi8B", "reviewer": "Reviewer_Q9VY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- Strong experiments across different modalities that verify their claim \n- Additional experiment on wall-clock time demonstrates the actual use-case of their method \n- Good ablations on OOD data and effect of number of nodes \n\nWeaknesses\n- The NLP experiments fine-tune OPT on CommonCrawl, which is normally used for pre-training, not fine-tuning. This difference matters since pre-training is done on new parameters and fine-tuning is done on non-new parameters and the behavior of lo-fi is different in these two setups. “lo-fi can only achieve matching accuracy when no new parameters are introduced” - limitations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper demonstrates that fine-tuning a model on individual nodes and then averaging their weights afterwards can match (or even outperform) training one model on all the nodes using data parallelism. They demonstrate that the results hold when fine-tuning vision and language models. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Simply strengthen work: NLP experiment which fine-tune a LM on some standard downstream task for fine-tuning.  \n- Simply strengthen work: A baseline in figure 2 which uses gradient communication across various number of nodes. This would indicate if using gradient communication experiences diminishing returns after 4 nodes. \n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a straightforward application of existing methods (e.g. federated learning) to the setting of (distributed) transfer learning. One reviewer complained about lack of novelty, but I think that highlighting that this simple approach can work in the ubiquitous transfer learning setting is valuable. Other reviewers had some initial suggestions primarily around clarity and framing which were addressed. I think this paper makes a meaningful contribution and is worth publishing, since it could potentially open up distributed training to resource-constrained individuals in an incredibly common ML pipeline.
Decision: Accept (poster)