Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a novel personalized method for FCO via the forward-backward envelope. Personalized models are firstly updated, then the local models are updated with Second-order optimization. After the local update, the global update is done via FedAVG. Convergence results of the proposed algorithm are shown under similar assumptions in pFedMe. Numerical experiments on the federated lasso, federated matrix completion, and nonsmooth deep neural network were done.
strength_and_weaknesses: Strength:
the topic is important for distributed learning and federated learning. This paper proposed an interesting method to obtain explicit forms of gradients. A convergence proof is performed to justify the algorithm's efficacy. The paper is not hard to follow.
Weaknesses:
My major concern is the novelty. It seems that the proposed method uses the framework of pFedMe but estimates the local gradient using an explicit form, plus additional computation on the Hessian matrix. Although the experimental result of FedFBE shows faster convergence over other methods, it is not fully convincing to say that FedFBE has higher efficiency. Estimating the Hessian matrix definitely helps with convergence, but it also leads to the consumption of much more computation. pFedMe doesn't use the explicit form partly because it can save computational power by avoiding that Hessian matrix. Therefore, I am not sure that the problem will be solved without creating another bigger problem. Thus, the novelty in terms of the solution technique is thus limited.
I wonder about the influence of hessian components of the proposed method on the final results. Some ablation studies can be done to justify the efficacy of FBE.
clarity,_quality,_novelty_and_reproducibility: 
The paper is written clearly and easy to follow. The authors can highlight their contributions better by stressing the differences between FedFBE and pFedMe in the algorithm derivation and convergence proof.
summary_of_the_review: Generally speaking, the topic of the paper is of importance and interest. However, the authors can improve the work by addressing the above-mentioned concerns, especially about the computational efficiency and the ablation studies in the second-order info in the optimization.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes federated composite optimization, i.e., the nonconvex minimization of sum_i f_i + h in a federated setting. Here, f_i is smooth, h is not smooth but proximable. Note that the f_i are not assumed to be equal to each other (heterogeneous data).

The Forward Backward Envelope (FBE) of f_i + h is a function F_i s.t. one gradient step on F_i is roughly equivalent to one Forward Backward step for f_i + h (it involves the Hessian though). Similarly to the Moreau envelope, the FBE is smooth and depends on a parameter lambda. The paper proposes to minimize F = sum F_i. Tuning lambda allows for more federation or more personalization.

To minimize F, one can use basically any Federated Learning algorithm since the F_i are smooth. 
The main theorem upper bounds the squared norm of the gradient of F evaluated at the federated iterate, along the algorithm.
strength_and_weaknesses: Weaknesses

- The gradient of the FBE is actually not a FB step, because it requires the Hessian of the smooth function. So the methods based on the FBE require the Hessian.

-I think that the assumptions A2 and A3 are too strong. They basically say that the first second and third derivatives are bounded. Usually, the smooth term is gradient Lipschitz and we do not assume Lipschitzness of the nonsmooth term. The third derivatives are needed because the Hessian appears in the algo.

- The approach based on the FBE is not justified. Is it a good model to minimize F instead of the true loss f? Why won't we apply a federated version of the Forward Backward algorithm for example? (and then take one additional FB step for personalisation?)


Strength

- Numerical exp show the superiority of the approach over concurrent methods in several contexts.

- The F_i is smooth, so the model can be solved by any FL algorithm which gives a lot of flexibility. 
clarity,_quality,_novelty_and_reproducibility: Clarity and quality: 

Overall the paper is well written. However, the paragraph after Eq 8 could be clarified. 

Besides, it is not clear that if lambda = 0 then there is more personalisation. Indeed, the objective becomes constant in this case. 

The presentation of personalisation vs federation is a bit confusing and is not discusses theoretically. The main theorem deals with the federated iterate. More generally, one would need more justifications for the approach. 

Novelty:

The paper provides one way to deal with composite functionals in a Federated setting, a problem which seems to be new. Otherwise, the proof technique seems rather standard given the assumptions. I mean that, once the model F = sum F_i is formed, the reminder seems to follow from existing analyses in 1. the FBE 2. the FL algorithm used to minimize F. However, the numerical experiments show that at least in practice the approach performs well.

MINOR

"Bregman distrance"

 "unbaised"

"Caculate"

" standrad"
summary_of_the_review: This paper proposes a meta algorithm to solve composite problems in a federated setting. This meta algorithm relies on minimizing the sum of the FBEs with any FL algorithm, because the FBEs are smooth. Once the approximated problem with FBEs is formed, the paper forgets about the original problem (except in the numerical experiment section).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a personalization federated learning method (i.e., pFedFBE) by using forward-backward envelope as clients’ loss functions. It provided the convergence analysis  of the proposed method, which shows the same convergence complexity results as FedAvg for FL with unconstrained smooth objectives. Numerical experimental results demonstrate the effectiveness of the proposed pFedFBE method.
strength_and_weaknesses: Strength:

This paper proposed a personalization federated learning method (i.e., pFedFBE) by using forward-backward envelope as clients’ loss functions. It provided the convergence analysis of the proposed method, which shows the same convergence complexity results as FedAvg for FL with unconstrained smooth objectives.

Weakness:

The novelty of this paper is limited. This paper basically follows the existing methods and  theoretical analysis such as in (https://arxiv.org/pdf/2011.08474.pdf).
clarity,_quality,_novelty_and_reproducibility: The writing can significantly be improved. 
The novelty of this paper is limited.  This paper basically follows the existing methods and 
theoretical analysis such as in (https://arxiv.org/pdf/2011.08474.pdf).
summary_of_the_review: This paper proposed a personalization federated learning method (i.e., pFedFBE)  by using forward-backward envelope as clients’ loss functions. It provided the convergence analysis  of the proposed method, which shows the same convergence complexity results as FedAvg for FL with unconstrained smooth objectives. Numerical experimental results demonstrate the effectiveness of the proposed pFedFBE method.

Some Comments:

1. Assumption 1-(A3) is stronger than the basic assumptions in the existing FL methods such as FedAvg.

2. It would be great if the authors would detail the tuning parameters in the experiments.


------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------

The authors still did not solve my main concern: the limited novelty of this paper. So I support to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The author(s) extends the Moreu-envelop based personalized FL to the composite problems. Convergence analysis under certain assumptions are developed. Experiments on synethtic data and MNIST are conducted to evaluate the proposed algorithm.
strength_and_weaknesses: Pros:
- The story line of the paper is clear and related works are well-addressed to my knowledge.

Cons:
- The presentation of this paper should be improved: (1) the author(s) should use \citep in many places, for example in the first paragraph. (ii) Figure 1,2,3,4,5,6 are not properly cropped. (3) Notations are kind of heavy, it is better to include a subsection for notations.
- From the perspective of personalized FL, the paper seems to be a direct extension of [1] to composite problems. However, generalizing [1] to the composite problem does not seem to be a challenging task.
- The author(s) spent much space for the theoretical analysis (page 5 and page 6). However, Theorem 1 does not seem to be a particularly strong results, the algorithm uses second order information and does not coverge faster.
- Using second order information (the Hessian matrix) is expensive in practice, this issue is only briefly mentioned above Section 3. It is better to discuss more on how to efficiently leverage the second order information, i.e., how to do the Hessian-vector product efficiently and what is the time complexity of the proposed algorithm.
- Experiments are also not very convincing, the datasets are mostly aftifically generated, the only real dataset is the MNIST dataset. It is more convincing the evaluate the proposed algorithm on more real-world datasets.


[1] Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394–21405, 2020.
clarity,_quality,_novelty_and_reproducibility: The writting of the paper should be improved; novelty is maringal; reproducibility is unclear;
summary_of_the_review: The presentation of the paper should be improved. On the theory side, the contribution of the paper is kind of marginal compared with the exiting Moreu-envelop based personalized FL; on the empirical side, the experiments are mostly on synthetic and tiny datasets. The author(s) should consider improving either from the theory or the empirical aspect.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work proposes a personalization approach for federated learning (FL) using the forward backward envelope (FBE) of the clients' loss functions. 

While the approach is interesting, its superiority beyond the existing approaches (such as with Moreau envelopes) is unclear. Moreover, it looks like the overall approach is a second order approach, which has scalability and storage issues without the presence of quasi-Newton type approximations. 
justification_for_why_not_higher_score: The writing of the paper should be improved significantly based on the reviewers' remarks. The scalability concerns should be addressed as well.
Decision: Reject