Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper derives the convergence guarantee of SL for non-convex objectives on non-IID data.  It compares also SL against FL theoretically and empirically,
strength_and_weaknesses: See Below
clarity,_quality,_novelty_and_reproducibility: See Below
summary_of_the_review: -In Algo 1. I have a problem understanding exactly what is the communication round here, because in step 8 and 10 there is communication between clients and server. You don't count that as a communication round?

To have the local update in step 12 (local model update) according to algo 1, communication is needed between the server and the client, so what is local here since there is a communication between server and client?...

It seems that there are "local" communication rounds which are denoted by k=0.1,...K and there are "global" communication rounds which are r=0.1...R can you clarify what this is, because for me the communication happens for both iterations k and r? am i missing something?

-In step 12 you put, client-side and server-side updates, again I'm confused here, how the server can access the local gradients to do this local update.

- In step 3, since in SL things work in parallel, why do you need to sample a subset of clients only?

-Table 1: in local updates for SL, is it a typo of indices there x_{i}^{r,k+1} = x_{i-1}^{r,K}- ... or am I missing something again?

-if my understanding is correct, x^r of FL is the same as x^{r,K} of SL. Then from Table 1, SL and FL and producing the same iterations if they use the same learning rates and batches? if my understanding is wrong, can you tell what prevents SL from using x^r in local updates.

-The reported complexity bounds between SL and FL are the same up to the multiplicative constant N.
The reason is, I think, the choice of learning rates. 
If you change the lr of SL by the lr of FL over N you get the same bounds for FL and SL!

-Does K depend on i? you have K=En_i/b.


------after rebuttal----

After discussions with authors and rebuttal with other reviewers. The authors addressed some of my concerns. I still think that the complexity results given in the first version of the paper, SL and FL have exactly the same complexity bounds by choosing the right learning rates. See my comments and discussions on this. I decided to increase my rating of the paper to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the convergence rate of split learning theoretically. With constant step sizes in both inner and outer loops, the outer loop complexity is $T^{-1/2}$ with additional terms depending on noise and dissimilarity. Then it compares the rate between SL and FL and presents discussions/explanations.
strength_and_weaknesses: The topic is of great importance for theoretical analysis of optimization in distributed machine learning. I believe the proof is correct and solid. The experiments are concrete.

Some questions:
- I suggest a table for notations, and highlight the super/sub-scripts of $x$. 
- Eq (2) is correct but a bit misleading for readers. Although there is a sum over $k$, one cannot get all $x_i^{r,k}$ over $k$ at the same time because the $k$ stands for iterations, so the $x_i^{r,k}$ comes sequentially like in Algo 1 box. When I saw a sum, I felt the terms can somehow be parallelized but actually not, so it would be great to clarify in a footnote, or always write as in Algo 1. 
- Assumption 3: Could you explain why “In the IID case, $B = 1, G = 0$”? I suspect (e.g. in stochastic case) $E\|\nabla f_i\|^2 = \|\nabla f\|^2 + Cov(\nabla f)$, was it wrong? Could you use a few simple examples in appendix calculating $B,G$, like linear functions, $f(x) = ax$ and $f_i(x) = a_i(x)$, and so on?
- Step size: you choose $T=NKR$, where is $B$? In Thm.1, the third term in $\min$ depends on $B$. 
- Rate: Is the rate outer loop complexity, and the total gradient calculation $K$ times that number? Could you compare it with SGD, and GD if you simply calculate all stochastic gradients together and average them? I didn’t mean SL has to beat them, but it would be helpful and interesting to compare. If the rate is not as good, it would be great to explain the reason. Is it due to the fact that you cannot shuffle or randomize the individual/local gradients? Are you aware of any lower bound of RHS of Cor. 1 and how suboptimal the result is from that optimal rate?
- Constant step size: if the step size depends on $T$ and error depends on $T$, would it be better to choose a larger error and a small $T$ at the beginning, i.e. use a large step size at the beginning, then refine the error, increase $T$ and decrease the step size? In SGD one can use the decreasing step size.

clarity,_quality,_novelty_and_reproducibility: I think the work is novel and important. I like the presentation and clarity of the paper. In Sec.3, the paper gives one sentence summary of techniques of each paper. Later, following each theorem, the paper presents plenty of discussions. 
I suggest adding a table of notations, and a table of main techniques (the iteration expression) and rates of the prior works in appendix. The SplitFedv2/3/SFLG can be explained more clearly. 
summary_of_the_review: I think the paper is important and solid, and the clarity is great. With a few more revision regarding my questions above, I believe it's ready for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work establishes the convergence bound of split learning (SL) for non-convex objectives with non-iid data. The convergence results provide some insight on the parameter tuning and potential benefits/limitations of split learning, compared with standard federated learning (FL, local SGD). The authors also conduct empirical comparison between spilt learning and federated learning with respect to data heterogeneity, which provides a deeper understanding between them.
strength_and_weaknesses: Strengths:
- The analysis seems solid and correct, though I didn't go through all the details in the appendix.
- To my knowledge, the convergence result is new, which complements the theoretical study of split learning.

Weaknesses:
- The main takeaway of this work is that SL is faster than FL and generalizes similar to FL on mildly non-IID data, while FL prevails on highly non-IID data. This conclusion does not seem surprising to me since previous works (e.g., Gao et al. (2020; 2021) cited in the submission) already made such observations.
- For the theoretical results, since split learning runs sequentially for its local updates, it is basically minibatch SGD with biased updates. Thus, as long as the accumulated error (client drift) can be bounded, establishing its convergence seems straightforward to me. I hope the authors can discuss the technical difficulties on deriving the convergence bound.  
- The discussion (or guidance for hyperparameter selection) for the theorem is inadequate. For example, in section 4.1, "one needs to meticulously design the value of K for the best performance", "large value of K forces SL prone to converge to local optima", "to reduce the communication overhead without hurting the convergence rate" all require further discussion, such as to analyze the communicated bits of split learning. The Effect of $\eta_g$ seems interesting, but it is not thoroughly studied. It seems that we can choose an arbitrarily large $\eta_g$?
- The presentation also needs to be improved (see below). 

clarity,_quality,_novelty_and_reproducibility: The clarity can be improved, for example,
- The notations in Eq. 2 are not well defined until Algorithm 1, it is better to reorder the contents to ensure a clearer flow of reading. 
- In section 4.2, the paragraph "Client drift in SL is upper bounded, as shown...." does not have a clear flow of ideas. 
- $\eta_{SL}$ and $\eta_{FL}$ are not clearly defined. 
summary_of_the_review: This work provides some new analysis of SL on non-iid data and compare it with FL, which is only marginally interesting from my perspective. It is not clear what are the technical difficulties of proving the theorems, and the discussion of the theorems is inadequate. The overall presentation also needs to be improved.

---------------------------------------------------
After rebuttal: Thank the authors for the detailed responses and revision, I really appreciate the effort. I have also checked other reviews and responses. The revision greatly improved the clarity of this paper. However, I still think the novelty of this paper is below the bar as there seems to be no strong or really exciting result. I keep my score but will not be upset if this work gets accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies convergence guarantees of split learning (SL) for non-convex objectives on the realistic case of heterogeneous data. This question is of great interest to the community. The paper compares standard federated (FL) with SL and observed SL can beat FL when data homogeneity is high, but will suffer for more heterogeneous data.

Unfortunately after discussions the impression remained that the paper in its current form remains narrowly below the bar. It could have gone either way. The main remaining concerns are due to the level of novelty (wrt existing results on stepsizes as special cases) and improvements to the presentation of the paper. We are also concerned that the bounds are sometimes loose and greatly affected by stepsizes (i.e., $\eta_l, \eta_g$, which is pointed out by reviewer SLb6, and the authors also admit that the theoretical study of $\eta_g$ is still insufficient.

We hope the detailed feedback helps to strengthen the paper for a future occasion, and are positive for that as the paper definitely has merit.
justification_for_why_not_higher_score: This could go either way, depending if the two negative reviewers acknowledge the author response. The paper does have some value, placing it narrowly around the bar for ICLR
Decision: Reject