Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors propose a new method for efficient vertical FL. Specifically, the authors propose to find a small subset of clients whose features could be used to learn a model that yields high accuracy with significantly less amount of time using 1. Mutual Information to evaluate how informative a selected subset of features is, 2. a group testing method. Empirical results show that the proposed method is able to achieve comparable/stronger performance than regular training with significantly less time.
strengths_and_weaknesses: Strength:
- Overall this paper is well written with clear intuitions on how each component of the algorithm is designed. The novel idea of using mutual information with homomorphic encryption is interesting and sensible.
- The authors evaluate on a thorough set of datasets and the empirical results show that the proposed method is strong compared to prior baselines.

Weakness:
- The authors propose a group testing method to select the best k clients. However, this method could be potentially costly. Could the authors provide any justification how the number of tests $T$ affects the convergence of finding best top-k clients? Moreover, the experiments in this work focus on cases where the number of clients is small (<5). Could this method generalize to large scale federated network where the number of partitions is large?
- For all the datasets evaluated in this work, the authors use a logistic regression model to evaluate the proposed method. Could the authors provide evaluations of VF-PS on more sophisticated ML task, e.g. non convex loss?
questions: Questions are listed in the previous section.
limitations: Limitations are listed in weakness section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors studies the problem of participant selection in vertical federated learning, and model it with a mutual information-based view. The first technical contribution is VF-MINE, a vertically federated mutual information estimator. The second contribution is to further optimize VF-MINE and get a group testing-based participant selection framework, VF-PS. Some experimental evaluations were conducted to show the performance of VF-PS.
strengths_and_weaknesses: **Strengths:**
+ The idea of boosting the efficiency of vertical federated learning with participant selection is novel and interesting.
+ The Fagin-inspired optimization is novel.
+ The analysis of security and the experimental results are good.


**Weaknesses:**
- Some parts of the algorithms are not clear and easy to follow. Please see my questions below. 


questions: The paper seems to suggest that VS-PF is a once-and-for-all method. But what if the “good selection of participants” varies in different stages of training?
limitations: The authors have addressed the limitations that the mutual information may not be the best selection criterion. Beyond that, the authors may need to consider the social impact of inequality of contribution and reward between selected and unselected participants.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the problem of client sampling in the context of Vertical FL, i.e., FL-PS, where each client corresponds to a feature subset and thus differs in their importance to the global model. Specifically, a mutual information (MI) based approach is used for importance evaluation. The Fagin algorithm and a so-called batched group testing scheme are employed to make the selection efficient. Security analysis is provided. Results of evaluation on various classification datasets show that the proposed framework is competitve to full-participation training with very low sampling overheads.  
strengths_and_weaknesses: Strengths:
- Client/feature subsets sampling in VFL is an intriguing problem under-explored in the field of FL.
- The authors introduce multiple techniques, e.g., Fagin, group testing, batching, to make their MI-based sampling framework more practicaly feasible.
- The empirical results are relatively comprehensive, covering different scale of data settings and multiple models.
- Privacy protection is considered in the whole process.

Weaknesses:
- An identical instance space $\mathcal{D}$ is assumed, which may not be the case for many VFL scenarios.
- The complexity  and efficacy of group testing scheme is subject to how groups are generated. Random grouping over a multitude of clients can be problematic.
- Some usage of symbols and terms seem to be too casual. E.g., "# participants", "$NN_{k, y_q}$", "All-train", etc.
questions: - Is it possible that the kNN-based MI estimator is biased towards smaller local feature subsets with Euclidean distance?
- In the Security Analysis, is the framework still secure when the leader participant is curious and seeks to reconstruct non-local features?
- How are the groups for testing generated? How is the number of groups $T$ determined?
limitations: The methodology, in my understanding, is practically limited from two aspects: i) a globally aligned, complete instance space $\mathcal{D}$, as mentioned above, and that ii) the MI-based design is only applicable to classification tasks, which is a common limitation for most VFL algorithms/frameworks though.
review_title: null

Meta-review: metareview: This paper studies the problem of vertical federated learning, where each client (e.g., hospital) has access to a disjoint subset of features. The reviewers agree that the paper provides an interesting technique for client selection. The authors have adequately addressed the reviewers' questions. 

The authors should consider dedicating part of their revision to addressing concerns raised by the ethics reviewers. In particular, there are concerns of unfairness due to the selection of a small subset of participants. Even though the paper does not readily provide a solution to this problem, the authors should acknowledge this potential issue and suggest directions for future work. If there is not enough space in the paper, the authors could use part of the appendix for a detailed discussion.
confidence: Less certain
Decision: Accept