Title: A Unified Analysis of Stochastic Gradient Descent with Arbitrary Data Permutations and Beyond
Venue: NeurIPS 2025
Accepted: yes
Keywords: stochastic gradient descent

random reshuffling

federated learning

convergence analysis
Abstract: We aim to provide a unified convergence analysis for permutation-based Stochastic Gradient Descent (SGD), where data examples are permuted before each epoch. By examining the relations among permutations, we categorize existing permutation-based SGD algorithms into three categories: Arbitrary Permutations, Independent Permutations (including Random Reshuffling and FlipFlop Rajput et al., 2022), Dependent Permutations (including GraBs Lu et al., 2022a; Cooper et al., 2023). Existing unified analyses failed to encompass the Dependent Permutations category due to the inter-epoch permutation dependency. In this work, we propose a generalized assumption that explicitly characterizes the dependence of permutations across epochs. Building upon this assumption, we develop a unified framework for permutation-based SGD with arbitrary permutations of examples, incorporating all the existing permutation-based SGD algorithms. Furthermore, we adapt our framework for Federated Learning (FL), developing a unified framework for regularized client participation FL with arbitrary permutations of clients.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new general framework for analyzing the stochastic gradient descent algorithm with various kinds of permutations in the smooth setting. A more general assumption was introduced to handle the potential dependency of the permutations across different rounds. The framework was further extended to the federated setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n> 1. The author proposed a novel assumption which is more general that handles potential dependencies of permutations effectively. \n> 2. The framework proposed was further developed and extended into the federated learning setting.\n> 3. Convergence guarantees were provided together with numerical experiments further supporting its validity.\n\nWeaknesses:\n> 1. The newly proposed assumption brings additional difficulty into practical implementations, in Theorem 1, the step size $\\gamma$ seems to be dependent on the parameters such as $A_i,B_i$. Given the complexity in practice, I am not sure whether the theoretical indication actually can be helpful at all.\n\n> 2. Do the authors have an example, where all of the assumptions in Theorem 1 holds? The sets of assumptions seem quite restrictive to me. In order to apply the theorem, do we need to determine all of $A_i, B_i$ and $D$? If it is the case, how can we actually determine them in a practical setting? \n\n> 3. I am curious about the original motivation of the authors. If dependent permutation is better, why bother developing a theory that includes the three cases. Why not focusing alone on the dependent case.\n\n> 4. Given the complication that the theories involve, I would suggest the authors test the algorithm on a broader scope of machine learning and deep learning tasks.\n\nQUESTIONS:\n> 1. In practical federated learning setting, the data is often heterogeneous across different clients. How would this affects the algorithm, would it be reflected by the convergence guarantee?\n\n> 2. Would the same kind of technique be applicable to other optimization algorithms besides SGD? Is there a specific reason of picking SGD as the base optimizer in this case?\n\n> 3. How does the algorithm scales with large systems. In fact, is there a reason of choosing dependent permutation over the other algorithms?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified framework for analyzing permutation-based Stochastic Gradient Descent (SGD), including arbitrary, independent, and dependent permutations. By introducing a more general assumption (Assumption 2) that captures inter-epoch permutation dependencies, the authors generalize existing analyses and derive upper bounds for prior algorithms (e.g., GraB). The framework is further extended to Federated Learning (FL) with regularized client participation. The authors propose FL-GraB, a new algorithm that improves FL training performance under permutation constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper presents a new upper bound for dependent permutation-based SGD algorithms.\n2. It offers a clean and comprehensive theoretical analysis, with results that are consistent with prior work.\n3. The findings are interesting: FlipFlop and PairGraB are shown to achieve upper bounds comparable to those of RR and the original GraB, respectively.\n4. The proposed unified framework is extended to federated learning FL. FL-GraB empirically outperforms FL-RR in the experiments provided.\n\nWeaknesses\n1. Only covers regularised participation in FL; the much more common independent sampling setting (e.g., Wang & Ji 2022) seems outside the framework.\n2. The advantage of FL-GraB or dependent client sampling is not clear. There is no comparison with FL baselines such as SCAFFOLD (Karimireddy et al., 2020) under heterogeneous data distributions.\n3. The implementation AP / FL-AP are not clearly described, and the corresponding results are not discussed.\n4. Figure 1 lacks an x-axis label (likely \"epochs\").\n5. \"Initilize\" in Algorithm 2 line 4 should be \"Initialize\".\n\nReferences\n1. Wang, S., & Ji, M. A unified analysis of federated learning with arbitrary client participation. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.\n2. Karimireddy, et al. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning (ICML), 2020.\n\nQUESTIONS:\n1. Can the unified framework be extended to cover independent client sampling, as in Wang & Ji (NeurIPS 2022)?\n2. Can you provide a comparison or discussion of how FL-GraB performs against prior client sampling strategies, such as SCAFFOLD (Karimireddy et al., 2020)\n3. Can you clarify the implementation details of the AP and FL-AP algorithms used in the experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives a unified framework for studying different permutation-based algorithms. The framework is general enough to include arbitrary parameter orders, random reshuffling, and some adaptive shuffling algorithms such as GraB. The main novelty is in Assumption 2-- It connects the gradient error (difference between shuffled gradients and the true gradients) over an epoch *between* epochs. It vaguely reminds me of some of the unified assumptions made in prior work in other contexts like federated learning [1] where the gradient error at epoch $k$ is related to the error at epoch $k-1$. Theorem 1 gives a convergence rate under this assumption in the nonconvex and smooth setting, and is specialized to various algorithms in Propositions 1-7. Not all of these results are restatements of existing results, as e.g. for FlipFlop and PairGraB. Finally, the results are directly applicable to the case when the different \"data points\" represent clients in a federated setting, and we can control how we sample them.\n\n[1] Gorbunov, Eduard, Filip Hanzely, and Peter Richtárik. \"Local SGD: Unified theory and new efficient methods.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\nSTRENGTHS AND WEAKNESSES:\n1. (Strength) The framework is quite elegant and unifies many existing results. It provides a direct point-of-view as to *why* methods like GraB work. I actually like that no new methods are introduced here, it is relatively rare to see papers that do a good job at taking a field with a diverse number of methods and find a unifying framework to look at them.\n2. (Strength) The paper provides a new analysis for PairGraB, a practical variant of GraB that was not analyzed prior.\n3. (Weakness) The paper's analysis framework is to treat all of these methods as equivalent to doing one large step of gradient descent plus deviation error. Unfortunately, this means that all the convergence results obtained here outperform SGD only in the same regime that GD outperforms SGD, and underperform SGD in the regime when SGD is better than GD. This can be seen from e.g. the last column in Table 1, where the paper's leading term scales as $1/Q$ rather than $1/(NQ)$ as in the analysis of [2]. This is unfortunately common to a lot of the literature on RR.\n4. (Weakness) The relevance of shuffling-based algorithms to modern training is a bit in doubt, since now we've looped back to the case where datasets are far too large to actually shuffle through and we instead rely on sampling. This isn't really this paper's fault though but is a more generic concern. I guess I would ask: how might we use some of the insights here to develop methods where shuffle only a subset of the dataset?\n\n[2] Koloskova, Anastasia, et al. \"A unified theory of decentralized SGD with changing topology and local updates.\" International conference on machine learning. PMLR, 2020.\n\nQUESTIONS:\n1. Can your analysis extend to a sort-of hybrid algorithm where we choose a number (say $K$ that is smaller than $N$), sample $K$ differrent data points and then shuffle them? This might be more relevant to modern large-scale language model training.\n2. Can your analysis be used with the shuffling variance technique of [3] to recover the $1/(NQ)$ dependence at least in the strongly convex setting?\n\n[3] Mishchenko, Konstantin, Ahmed Khaled, and Peter Richtárik. \"Random reshuffling: Simple analysis with vast improvements.\" Advances in Neural Information Processing Systems 33 (2020): 17309-17320.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This work focuses on the analysis of permutation-based SGD. The authors propose a framework that enables a unified analysis for the three categories: arbitrary permutations, independent permutations, and dependent permutations. They address the dependency of permutations by introducing a more general assumption. They extend the analysis to Federated Learning scenarios where client permutations are considered. The framework establishes new convergence rate results and also results that are consistent with existing works.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper is well-structured and clearly written.
2. The provide extensive comparisons with previous work.
3. The authors propose a more general assumption that leads to a unified theorem for all three classes of SGD with data permutations. Mainly, the assumption accounts for dependency of the permutations between epochs.
4. The authors define a new order error for Federated Learning and extend the techniques for analysis of the convergence of FL with regularized participation.
5. The authors apply their main theorems to analyze multiple existing algorithms. They show that the results match the existing bounds. They also establish new results for methods that previously lack convergence guarantees (namely, FL-GraB), which indicates the effect of client ordering in Federated Learning.

Weakness:
The probabilistic bounds becomes less and less effective with increasing $Q$ due to $Q \delta$.

QUESTIONS:
Corresponding to the weakness, is there any hidden factors that prevent one from choosing, e.g., $\delta = o(1/Q^2)$ so that the bounds will hold with higher probability with increasing $Q$?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05