Title: Collaboratively Learning Linear Models with Structured Missing Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Collaborative Learning

Missing Data

Sensors

Linear Regression
Abstract: We study the problem of collaboratively learning least squares estimates for $m$ agents. Each agent observes a different subset of the features---e.g., containing data collected from sensors of varying resolution. Our goal is to determine how to coordinate the agents in order to produce the best estimator for each agent. We propose a distributed, semi-supervised algorithm Collab, consisting of three steps: local training, aggregation, and distribution. Our procedure does not require communicating the labeled data, making it communication efficient and useful in settings where the labeled data is inaccessible. Despite this handicap, our procedure is nearly asymptotically, local-minimax optimal---even among estimators allowed to communicate the labeled data such as imputation methods. We test our method on US Census data. We also discuss generalizations of our method to non-Gaussian feature settings, non-linear settings, and Federated Learning.

Collaboratively Learning Linear Models with Structured Missing Data

Abstract

We study the problem of collaboratively learning least squares estimates for m agents. Each agent observes a different subset of the features-e.g., containing data collected from sensors of varying resolution. Our goal is to determine how to coordinate the agents in order to produce the best estimator for each agent. We propose a distributed, semi-supervised algorithm COLLAB, consisting of three steps: local training, aggregation, and distribution. Our procedure does not require communicating the labeled data, making it communication efficient and useful in settings where the labeled data is inaccessible. Despite this handicap, our procedure is nearly asymptotically, local-minimax optimal-even among estimators allowed to communicate the labeled data such as imputation methods. We test our method on US Census data. We also discuss generalizations of our method to non-Gaussian feature settings, non-linear settings, and Federated Learning.

1 Introduction

Consider a set of agents that collect data to make predictions, where different agents may collect different features-because of different sensor availability or specialization-but wish to leverage shared structure to achieve better accuracy. Concretely, suppose we have m agents, where each agent i ∈ [ m ] observes n samples of ( x i + , y ) where x i + ∈ R d i is some subset of x ∈ R d . Weset this as a regression problem where the data ( x, y ) has the linear relationship y = 〈 x, θ 〉 + ξ for some noise variable ξ . For example, these agents could be a network of satellites, each collecting data with a distinct set of sensors of varying resolution and specialization, with the purpose of estimating quantities like crop-yields [22], biomass [18], and solar-flare intensity [12]. Or these agents could be a group of seismic sensors, using acoustic modalities or accelerometers to predict whether an earthquake will occur [2]. Other examples may include networks of hospitals or phones [13]. In these settings, the agents can share information to collaboratively train a model; however, they are limited by communication bandwidth constraints, a situation satellites and seismic sensors often face due to radio frequency spectrum scarcity and interference [6, 20]. Without being too rigorous, we will define a communication efficient algorithm as one where the per-agent communication cost does not scale with n , d , or m . No dependence on the dataset size n is suited for applications with significant data volume but limited communication resources. No dependence on the overall dimension d of the data ensures that agents are not incentivized to turn away agents with access to a richer set of features that would improve statistical performance but increase d . Finally, no dependence on m ensures that the algorithm is scalable to large collectives of agents. Can we construct a statistically optimal and communication efficient procedure to estimate θ ?

We answer in the affirmative and introduce our estimator COLLAB. COLLAB consists of three steps: local training on all agents, aggregation on a coordinating server, and distribution back to all agents. Our algorithm is communication-efficient: each agent i ∈ [ m ] syncs twice with a coordinating server

and incurs communication cost scaling like Θ ( d 2 i ) . We prove local minimax lower bounds which prove that COLLAB is (nearly) instance-optimal. We choose to study this problem in a stylized linear setting so that we can provide stronger guarantees for the algorithms we make. Indeed, our results which pair the exact asymptotic covariance of our estimator COLLAB with matching asymptotic local minimax lower bounds heavily rely on the linearity of our problem and would not be possible without strong structural assumptions. Having said this, the theory we develop for linear models does hint at potential methods for non-linear settings, which we discuss in Section 7. We also acknowledge privacy considerations are important for real world systems such as hospitals. We choose to focus instead on sensor settings where privacy is less of a concern. We leave adapting our results to privacy-sensitive settings to future work.

We compare our methods to single-imputation methods theoretically and empirically. We choose to baseline against imputation methods for three reasons. First, if we ignore communication constraints, our problem is a missing data problem, where formally the data is 'missing at random' (MAR) [16]. MAR problems are well studied, so we know that imputation methods work well theoretically and in practice [25, 15]. Second, because we have instance-optimal lower bounds, we know that imputation methods are also optimal for our problem. Finally, because imputation methods use more information than the method we propose, imputation will serve as a 'oracle' baseline of sorts.

Contributions. We briefly summarize our contributions.

We design a communication-efficient, distributed learning algorithm COLLAB which performs a weighted de-biasing procedure on the ordinary least squares estimator of each agent's data.

We show COLLAB is asymptotically locally minimax optimal among estimators which have access to the ordinary least squares estimator of each agent's data. We also show that with some additional assumptions, COLLAB is also asymptotically locally minimax optimal among estimators that have access to all of the training data of all agents.

We propose and develop theory for various baseline methods based on imputation. We compare the statistical error and communication cost of COLLAB against these baseline methods both theoretically and empirically on real and synthetic data.

We discuss generalizations of COLLAB for non-Gaussian feature settings and non-linear settings. We highlight open problems and identify possible directions for future work.

1.1 Related Work

Missing data. If we ignore the communication and computational aspects of our problem, the problem we study reduces to one of estimation with missing data. There has been a lot of work on this topic; please see [16] for an overview. The data in our problem is missing at random (MAR)-the missing pattern does not depend on the value of the data and is known given agent i . There are many approaches to handling missing data such as weighting and model-based methods [24]. Most related to our work are methods on single imputation. Schafer and Schenker [25] shows imputation with conditional mean is nearly optimal with special corrections applied. More recently, Chandrasekher et al. [3] show that single imputation is minimax optimal in the high dimensional setting. Another closely related popular approach is multiple imputation [23, 1]. Previous work [27, 29] has shown that multiple imputation in low dimensional settings produces correct confidence intervals under a more general set of assumptions compared to single imputation settings. However, we choose to focus on single imputation methods for two reasons. First, we are interested in estimation error and not confidence intervals, and our lower bounds show that single imputation has optimal estimation error for our setting. Second, in our problem context, multiple imputation would require more rounds of communication and consequently higher communication cost. Other methods for missing data include weighting and model-based methods.

Distributed learning. Learning with communication constraints is a well studied practical problem. We provide a couple of examples. Suresh et al. [26] study how to perform mean estimation with communication constraints. Duchi et al. [8] develop communication-constrained minimax lower bounds. Distributed convex optimization methods like Hogwild [21] have also been well studied. However, the works mentioned all concern the no-missing-data regime. A more relevant subfield of distributed learning is federated learning. In federated learning, a central server coordinates a

collection of client devices to train a machine learning model. Training data is stored on client devices, and due to communication and privacy constraints, clients are not allowed to share their training data with each other or the central server [13]. In the no-missing-features regime, optimization algorithms for federated optimization are well studied. There is also more theoretical work, which focus on characterizing communication, statistical, and privacy tradeoffs, albeit for a more narrow set of problems such as mean and frequency estimation [4]. More related to the missing data regime we consider is cross-silo federated learning [13] or vertical federated learning [30]. In this paradigm, the datasets on client machines are not only partitioned by samples but also by features. Researchers have studied this problem in the context of trees [5], calculating covariance matrices [14], k-means clustering [28], support vector machines [31], and neural nets [17]. Most related to our work is Gascón et al. [9], Hardy et al. [11]; they study how to privately perform linear regression in a distributed manner. However, unlike our work, these works focus more on developing algorithms with privacy guarantees rather than statistical ones.

2 Mathematical model

We assume we have m agents that observes a subset of the dimensions of the input data x ∈ R d . Each agent i has a 'view' permutation matrix Π /latticetop i := [ Π /latticetop i + Π /latticetop i -] ∈ R d × d . Π i + ∈ R d i × d describes which feature dimensions the agent sees, and Π i -∈ R ( d -d i ) × d describes the dimensions the agent does not see. For a feature vector x ∈ R d and corresponding label y ∈ R , the i -th agent observes ( x i + , y ) where x i + := Π i + x ∈ R d i . Each agent has n fresh, independent observations (independent across agents) denoted as a matrix X i + ∈ R n × d i and vector y i ∈ R n . We let X i -∈ R n × ( d -d i ) denote the unobserved dimensions of the input data x drawn for the i -th agent, and we let X i ∈ R n × d denote the matrix of input data x drawn for the i -th agent, including the dimensions of x unobserved by the i -th agent. To simplify discussions in the following sections, for any vector v ∈ R d we use the shorthand v i + = Π i + v and v i -= Π i -v . Similarly for any matrix A ∈ R d × d we denote by

For a p.s.d. matrix A , we let ‖ x ‖ A = 〈 x, Ax 〉 .

Weassume the data from the m agents follow the same linear model. The feature vectors x comprising the data matrices X 1 , . . . , X m are i.i.d. with zero mean and covariance Σ /follows 0 . We will assume that each agent has knowledge of Σ i + -e.g., they have a lot of unlabeled data to use to estimate this quantity. We also assume for each index tuple pair ( j, k ) for 1 ≤ j, k ≤ d , there exists some agent that observes dimensions j and k . This ensures the coordinating server can construct the complete covariance matrix Σ from individual Σ i + . The labels generated follow the linear model

Throughout this work we consider a fixed ground truth parameter θ .

Objectives. We are interested in proposing a method of using the data of the agents to form an estimate ˆ θ which minimizes the full-feature prediction error on a fresh sample x ∈ R d

We are also interested in forming an estimate ˆ θ i which minimizes the missing-feature prediction error of a fresh sample x i + ∈ R d i for agent i -i.e., x i + = Π i + x where x ∈ R d is fresh. Define T i := [ I d i Σ -1 i + Σ i ± ] Π i and the Schur complement Γ i -:= Σ \ Σ i + := Σ i --Σ i ∓ Σ -1 i + Σ i ± . The local test error is then

We see that ‖ θ i -‖ 2 Γ i -is irreducible error. The role of the operator T i is significant, as (2) shows that T i θ is the best possible linear predictor for the i th agent 2 . Through this paper, we will also highlight the communication costs of the methods we consider. Recall that we would like our methods to have communication cost that do not scale with n or d .

3 Method

We begin with some intuition of why collaboration should help. Then we outline an approach of solving this problem for general feature distributions. The general approach is not immediately usable because it requires some knowledge of θ , so we need to do some massaging. In Section 3.3, we show how to circumvent this issue in the Gaussian feature setting and introduce our method COLLAB. Adapting the general approach to other non-Gaussian settings is an open problem, but we discuss some potential approaches in Section 7.

3.1 Intuition

In order to build some intuition for how our approach leverages collaboration, we begin with some basic analysis of the no-collaboration solution: where each agent i only performs ordinary least squares on their local data

here A † denotes the Moore-Penrose inverse for a general matrix A , and (i) holds whenever rank ( X i + ) ≥ d i . Because we focus on the large sample asymptotics regime ( n /greatermuch d i ), (i) will hold with probability 1 . Expanding out the y i gives equality (ii). Ignoring the zero-mean noise term, we see that θ must satisfy θ i + + ̂ Σ -1 i + ̂ Σ i ± θ i --ˆ θ i ≈ 0 for all i ∈ [ m ] . Our approach essentially tries to find such a θ .

3.2 Our General approach

In particular, after each agent i performing ordinary least squares on their local data to produce { ˆ θ i } i ∈ [ m ] , we recover an estimate of θ by performing a form of weighted empirical risk minimization parameterized by the positive definite matrices W i ∈ R d i × d i

We know by first order stationarity that ˆ θ = (∑ m i =1 T /latticetop i W i T i ) -1 (∑ m i =1 T /latticetop i W i ˆ θ i ) . We can show that ˆ θ is a consistent estimate of θ regardless the choice of weighting matrices W i . In fact, if the features X i , ξ i are Gaussian, ˆ θ is also unbiased. See Lemma B.1 in the Appendix for this result. While Lemma B.1 shows that the choice of weighting matrices W i does not affect consistency, the choice of weighting matrices W i does affect the asymptotic convergence rate of the estimator. In the next theorem, we show what the best performing choice of weighting matrices are. The proof is in Appendix B.2.

Theorem 3.1. For any weighting matrices W i , the aggregated estimator ˆ θ = ˆ θ ( W 1 , · · · , W m ) is asymptotically normal

with some covariance matrix C ( W 1 , · · · , W m ) . The optimal choice of weighting matrices is

The main challenge of using Theorem 3.1 is in constructing the optimal weights W /star i , as at face value, they depend on knowledge of θ . While we will discuss high level strategies of bypassing this issue in non-Gaussian data settings in Section 7, we will currently focus our attention on how we can make use of Gaussianity to construct our estimator COLLAB.

Algorithm 1: COLLAB algorithm

3.3 COLLAB Estimator - Gaussian feature setting

If X i are distributed as N (0 , Σ ) , W /star i has an explicit closed form as

where Γ i -= Σ i --Σ i ∓ Σ -1 i + Σ i ± is the Schur complement. Recall we assume that each agent has enough unlabeled data to estimate Σ i + . The denominator E x,y [( 〈 x i + , ˆ θ i 〉 -y ) 2 ] cannot be computed but we can use 1 n ‖ X i + ˆ θ i -y ‖ 2 2 , which is a consistent estimator of E x,y [( 〈 x i + , ˆ θ i 〉 -y ) 2 ] , in its place. Thus, each agent is able to construct estimates of W g i by computing

Now we construct our global and local COLLAB estimators defined respectively as

We summarize the COLLAB algorithm in Algorithm 1. At a high level, ˆ θ clb is an estimate of θ which also minimizes the full-feature prediction error (1) and ˆ θ clb i minimizes the missing-feature prediction error for agent i (2). Now we show that using the collective 'biased wisdom' of local estimates ˆ θ i , our collaborative learning approach returns an improved local estimator. The proof is in Appendix B.3.

Corollary 3.2. Let X i ∼ N (0 , Σ ) and define C g := ( ∑ m i =1 T /latticetop i W g i T i ) -1 . The global COLLAB estimator ˆ θ clb and the local ˆ θ clb i on agent i are asymptotically normal

The following are true

(i) W g i are the optimal choice of weighting matrices i.e.,particular, C ( W 1 , · · · , W m ) /followsequal C ( W g 1 , · · · , W g m ) = C g .

(ii) On agent i , we have √ n ( ˆ θ i -T i θ ) d → N (0 , ( W g i ) -1 ) . The asymptotic variance of ˆ θ i is larger than that of the COLLAB estimator ˆ θ clb i -i.e., ( W g i ) -1 /followsequal T i C g T /latticetop i .

Remark 3.3 . In Corollary 3.2, we can replace Σ i + in ˆ W g i with local sample covariances ˆ Σ i + for the plug-in estimator in Eq. (4). We prove this stronger statement in the proof of Corollary 3.2. However, as the coordinating server still need to use exact operators T i to compute the COLLAB estimator, relaxing this condition does not bring practical benefit.

This result characterizes the error distribution of the COLLAB estimator; guarantees about prediction error, defined in (1) and (2), directly follow. In particular, letting z ∈ R d be distributed as N (0 , C g ) , by the continuous mapping theorem we have

For large dimension d , ‖ z ‖ 2 Σ concentrates around its expectation E [ ‖ z ‖ 2 Σ ] = Tr( C g ) . The same arguments can be applied to make a statement about the local test error (2).

Communication Cost In the first round of communication, agent i sends d 2 i + d i +1 real numbers to the coordinating server. In the second round of communication, the server sends back the updated local model which is d i real numbers. In total, this amounts to d 2 i +2 d i +1 communication cost per agent. The communication cost of COLLAB does not scale with n , d , or m , satisfying our desiderata.

4 Comparison with other methods

In this section, we compare our collaborative learning procedure with other popular least squares techniques based on imputation and comment on the statistical efficacy and communication cost differences. We summarize our analysis in Table 1. The proofs of the theorems are in Appendix C. For the sake of brevity, our results here will center around estimating θ . Analogous results centered around estimating T i θ follow directly. These results can be connected to the full-feature and local test error by the same argument we made earlier to derive (5).

Local imputation w/ collaboration. Suppose a coordinating server collected covariance information Σ i from each agent and then distribute Σ back to each of them. Then one intuitive strategy is to use this information to impute each agent's local data by replacing X i + with E [ X i | X i + ] = X i + T i , before performing local linear regression. In other words, instead of computing ˆ θ i , compute

to send back to the coordinating server. Note that we use Moore-Penrose inverse here as T /latticetop i X /latticetop i + X i + T i is in general of rank d i , and ˆ θ imp i is then the min-norm interpolant of agent i 's data. Similar to COLLAB, we can use weighted empirical risk minimization parameterized by W i ∈ R d × d and to aggregate ˆ θ imp via

The next theorem, in conjunction with Theorem 3.1, implies that under the WERM optimization scheme, aggregation of least squares estimators on imputed local data does not bring additional statistical benefit. In fact, the local imputation estimator is a linearly transformed on local OLS ˆ θ i .

Theorem 4.1. For ˆ θ imp i from agent i , we have ˆ θ imp i = T /latticetop i ( T i T /latticetop i ) -1 ˆ θ i . Given any weighting matrices W i ∈ R d × d , the aggregated imputation estimator ˆ θ imp is consistent and asymptotically normal

Using the same weights W /star i ∈ R d i × d i as in Theorem 3.1 for aggregated ˆ θ imp , we have under p.s.d. cone order, for weights W i , C imp ( W 1 , · · · , W m ) /followsequal C /star , where C /star = ( ∑ m i =1 T /latticetop i W /star i T i ) -1 . In addition, the equality holds when W i = T /latticetop i W /star i T i .

As we will see in Sec. 5 where we provide minimax lower bound for weak observation models, the fact that the weighted imputation does not outperform our COLLAB approach is because the WERM on local OLS without imputation is already optimal. In fact, having access to the features will not achieve better estimation rate for both the global parameter θ and local parameters T i θ .

In terms of communication cost, this local imputation method requires more communication than COLLAB, as a coordinating server needs to communicate Σ to all the agents. Each agent must first send d 2 i real numbers corresponding to Σ i to the coordinating server, and then the coordinating server will send back d 2 -d 2 i real numbers corresponding to the entries of Σ that agent i has not observed. This amounts to d 2 total communication cost per agent. The per-agent communication cost for this method scales with d , which is not desirable for the reason outlined in the introduction.

Global imputation. Finally, we analyze the setting where we allow each agent to send the coordinating server all of their data ( X i + , y i ) for i = 1 , · · · , m instead of their local estimators, ˆ θ i or ˆ θ imp i . Having all the data with structured missingness available, a natural idea is to have the coordinating server first impute the data, replacing X i + with E [ X i | X i + ] = X i + T i , and then perform weighted OLS on all of the nm data points, before sending a d i dimensional local model back to each agent. Namely for scalars α 1 , · · · , α m > 0 , we take

Surprisingly, in spite of the additional power, ˆ θ imp-glb still does not beat ˆ θ in Theorem 3.1.

Theorem 4.2. For any scalars α 1 , · · · , α m > 0 , ˆ θ imp-glb is consistent and asymptotically normal

Recall the lower bound matrix C /star := ( ∑ m i =1 T /latticetop i W /star i T i ) -1 in Theorem 3.1. If X i ∼ N (0 , Σ ) , we have under p.s.d. cone order and any α i > 0 , C imp-glb ( α 1 , · · · , α m ) /followsequal C /star . In addition, the equality holds when α i = 1 / ( ‖ θ i -‖ 2 Γ i -+ σ 2 ) .

The communication cost for this method is significantly larger than the other methods we discussed. Each agent must send all of its data to a coordinating server. Factoring in the additional communication cost of receiving a new local model from the server, this amounts to a total nd i + d i communication cost per agent. The fact that communication cost for this method scales with n is a significant disadvantage for the reason we outlined in the introduction.

5 Asymptotic Local Minimax Lower Bounds

In this section, we prove asymptotic local minimax lower bounds that show COLLAB is (nearly) optimal. We work in the partially-fixed-design regime. For every sample x ∈ R d , x i + ∈ R d i is a fixed vector. We draw x i -from N ( µ i -, Γ i -) where µ i -and Γ i -is the conditional mean and variance of x i -given x i + . Here Γ i -is also the Schur complement. We draw x i -from N ( µ i -, Γ i -) . The samples x i + ∈ R d i comprise the matrices X i + ∈ R n × d i . For all i ∈ [ m ] , we will assume we have an infinite sequence (w.r.t. n ) of matrices X i + . This partially-fixed-design scheme gives the estimators knowledge of the observed features and the distribution of the unobserved features, which is consistent with knowledge that COLLAB has access to. In this section we fix θ ∈ R d . The corresponding label y = x i + θ i + + x i -θ i -+ ξ , where ξ ∈ R is drawn from i.i.d. N (0 , σ 2 ) . We use y j ∈ R n to denote its vector form for the agent j . To model the estimator's knowledge about the labels, we will have two observation models-one weaker and one stronger-which we will specify later when we present our results.

For each observation model, we will have two types of results. The first type of result is a minimax lower bound for full-featured data; i.e., how well can estimator perform on a fresh sample without

missing features. This type of result will concern the full-feature asymptotic local minimax risk

We will show that there exists a B ∈ R d × d such that the local minimax risk in the previous display is lower bounded by u T Bu for all u ∈ R d . In other words, we have lower bounded the asymptotic covariance of our estimator with B (with respect to the p.s.d. cone order). The second type of result is an agent specific minimax lower bound; i.e., what is the best prediction error an estimator (for the given observation model) can possibly have on a fresh sample for a given agent. This type of result will deal with the missing-feature asymptotic local minimax risk, defined as

Similar to the first minimax error definition, we will again show that there exists a B i ∈ R d i × d i such that the local minimax risk we just defined is lower bounded by u T B i u for all u ∈ R d i . Recall (2) for discussion surrounding why T i θ is the right object to compare against.

5.1 Weak Observation Model: Access only to local models and features

Recall the local least squares estimator ˆ θ i = ( X /latticetop i + X i + ) -1 X /latticetop i + y i . Let P ˆ θ θ be a distribution over ˆ θ 1 , . . . , ˆ θ m induced by θ and ( ξ 1 , . . . , ξ m ) iid ∼ N (0 , σ 2 I n ) . We define the following family of distributions P ˆ θ n,c := { P ˆ θ θ ′ : ‖ θ ′ -θ ‖ 2 ≤ cn -1 / 2 } which defines our observation model. Intuitively, in this observation model, we are constructing a lower bound for estimators which have access to the features X 1+ , . . . , X m + , the population covariance Σ , and access to ˆ θ 1 , . . . , ˆ θ m . In comparison, our estimator COLLAB only uses Σ and ˆ θ 1 , . . . ˆ θ m . We present our first asymptotic local minimax lower bound result here. The proof of this result can be found in Appendix D.1.

Theorem 5.1. Recall that C g := ( ∑ m i =1 T /latticetop i W g i T i ) -1 . For all ∈ [ m ] and n let the rows of X i + be drawn i.i.d. from N (0 , Σ i + ) . Then for all u ∈ R d , with probability 1, the full-feature asymptotic local minimax risk for P ˆ θ n,c is bounded below as,

For all u ∈ R d i , with probability 1, the missing-feature asymptotic local minimax risk for P ˆ θ n,c is bounded below as

This exactly matches the upper bound for COLLAB we presented in Corollary 3.2.

5.2 Strong Observation Model: Access to features and labels

Define the family of distributions P y n,c := { P y θ ′ : ‖ θ ′ -θ ‖ 2 ≤ cn -1 / 2 } as the observation model. Intuitively, in this model, we are constructing a lower bound for estimators having access to all of the features X 1+ , . . . , X m + and access to y 1 , . . . y m . This observation model is stronger than the previous observation model because estimators now have access to the labels y . We note again that our estimator COLLAB only uses Σ and ˆ θ 1 , . . . ˆ θ m . The quantities our estimator rely on do not scale with n , making our estimator much weaker than other potential estimators in this observation model, as estimators are allowed to depend on y i , which grows in size with n . We present our second asymptotic local minimax lower bound result here, starting with defining the strong local lower bound matrix C s := ( ∑ m i =1 2 Σ / ( ‖ θ i -‖ 2 Γ i -+ σ 2 )) -1 . The proof of this result is in Appendix D.2.

Theorem 5.2. For all i ∈ [ m ] and n let the rows of X i + be drawn i.i.d. from N (0 , Σ i + ) . Then for all u ∈ R d , with probability 1, the full-feature asymptotic local minimax risk for P y n,c is bounded below as

For all u ∈ R d i , with probability 1, the missing-feature asymptotic local minimax risk for P y n,c is bounded below as

(a) Prediction Error for large

n

(b) Prediction Error for small

n

(c) Small

In view of the lower bound in the strong observation model and that of the weak observation model in Theorem 5.1, it is clear that the lower bound in the strong observation setting is in general smaller as

We argue that the two lower bounds are comparable in the missing completely at random [16]. Consider for every agent i , each coordinate is missing independently with probability p . In this case, ( d i , Σ i + , T i ) are i.i.d. random triplets parameterized by p .

which further implies C g /followsequal ( ∑ m i =1 Σ / ( ‖ θ i -‖ 2 Γ i -+ σ 2 )) -1 /followsequal C s .

Corollary 5.3. Under the random missingness setup with missing probability p , let the eigenvalue of Σ be λ 1 ( Σ ) ≥ · · · ≥ λ d ( Σ ) > 0 and define its condition number κ = λ 1 ( Σ ) / λ d ( Σ ) . Suppose p ≤ 1 2 κ -1 (1 + ‖ θ ‖ 2 Σ / σ 2 ) -1 , we have the limits lim m →∞ mC g and lim m →∞ mC s exist and

6 US Census Experiment

We experiment on real US census data modified from the ACSTravelTime dataset from the folktables package [7]. The code can be found at https://github.com/garyxcheng/collab . We (artificially) construct m = 5 datacenters (agents), each containing data from only one of California, New York, Texas, Florida, and Illinois. The goal is to collaboratively learn a model for each datacenter in a communication efficient way. This setup models potentially real settings where state governments are interested in similar prediction tasks but may not be allowed to directly transfer data about their constituents directly to one another due to privacy or communication constraints. In this setup, there is covariate shift across agents, and the feature and error distributions are not Gaussians-presenting significant deviations from our theoretical assumptions which serve as a proving ground to test how well COLLAB works in practice. For an experimental setup that more closely matches the assumptions we made in our theory, see Appendix A.2.

After dataset preprocessing, described in the Appendix A.1, we have d = 37 features. We compute the feature covariance matrix from training data across all the datacenters, which we plot in Appendix A.1. We assume we are able to do this because this computation can be done in a distributed manner, without communicating training data points or labels. The California datacenter will have access to 37 features, New York to 36 , Texas to 35 , Florida to 30 , and Illinois to 27 . This models the feature heterogeneity which varies across geography. Each datacenter will have n datapoints, which we vary in this experiment. The objective is to predict people from Illinois's travel time to work given all 37 features. This task models the setting where the datacenter of interest does not have access to labeled full-featured, data to use to predict on full-featured test data.

We compare our method COLLAB against methods we call Naive-Local, Naive-Collab, OptimizedNaive-Colllab, Imputation, and RW-Imputation. We briefly describe each method here; Appendix A.1 contains a more detailed description of each method. Naive-Local refers to each agent locally perform OLS to construct ˆ θ i . Naive-Collab does an equal-weighted average of the agent OLS models-∑ i ∈ [ m ] Π /latticetop i + ˆ θ i /m . Optimized-Naive-Collab uses gradient descent to optimize the choices of weights used in Naive-Collab. Optimized-Naive-Collab uses fresh labeled samples without any missing

n

omitting Impute algs.

features during gradient descent, so in this sense, Optimized-Naive-Collab is more powerful than our method. Imputation refers to the global imputation estimator ˆ θ imp-glb with α i = 1 /m . RW-Imputation is Imputation but with the optimal choice of weights α i . We also compare against Naive-Local trained with 5 n datapoints. We choose 5 n to model the hypothetical scenario setting where all of the other datacenters available contain data (albeit with missing features) from Illinois. For each method that we test, we run 80 trials to form 95% confidence intervals.

We see that for n ≤ 800 in Figures 1(b) and 1(c), COLLAB performs the best; the imputation methods do the worst, and have much higher variance. In this small n regime, even the Naive-Local method with 5 times the data does worse than COLLAB. For n ≥ 2000 in Figure 1(a), the aggregation methods do worse than the imputation methods, and Naive-Local method with 5 times the data is the best performing method. However, COLLAB remains better than Optimized-Naive-Collab and Naive-Collab. The performance of Naive-Collab and Optimized-Naive-Collab being much closer to the performance of COLLAB here than in the synthetic data experiment in Appendix A.2 is not surprising, as the covariance of the features here is much more isotropic, meaning that the naive aggregation methods will not incur nearly as much bias.

7 Discussion and Future Work

Optimal weights beyond Gaussianity. E [ x i + θ /latticetop i -z i + z /latticetop i + θ i -x /latticetop i + ] has a nice closed form in Gaussian setting because z i + and x i + are independent-which is in general not true without Gaussianity. If we can directly sample from the feature distribution P (e.g., unlabeled data), then we can empirically estimate E [ x i + θ /latticetop i -z i + z /latticetop i + θ i -x /latticetop i + ] by sampling from P and using any consistent plug-in estimate ˆ θ (e.g., run COLLAB with weights W i = I d i ). This will return a good estimate of the optimal weights. An interesting future direction is to prove lower bounds without the Gaussianity assumption.

Generalization to non-linear models. Recall in the Gaussian setting, the optimal weights in COLLAB are W g i = Σ i + / ( E x,y [( 〈 x i + , ˆ θ i 〉-y ) 2 ]) . Then, the optimal loss function in Eq. (3) becomes

This hints at a generalization to non-linear models. Suppose the local agents train on models f i ( x i + ; θ i ) , R d i × R d i ↦→ Y and the global model f ( x ; θ ) , R d × R d ↦→ Y satisfies for some mapping T i : R d → R d i , f ( x ; T i θ ) = f i ( x i + ; θ i ) . Consider a loss function /lscript ( · , · ) : Y × Y → [0 , ∞ ) . Then we can consider the following way of aggregation inspired by COLLAB for linear models

We can consistently estimate the denominators (weights) using training time loss. An interesting future direction is to investigate the performance of this general approach for non-linear problems.

Application to Federated Learning. Federated Learning algorithms generally consist of repeating the following steps until termination: Step 1. server sends ˆ θ to all agent, Step 2. each agent i does some local training initialized at ˆ θ to form ˆ θ i , and Step 3. server collects ˆ θ i and aggregates them to form a new ˆ θ . Most federated learning algorithms assume the agent model sizes are the same across agents because aggregating models (Step 3) of different sizes is not obvious [19]. However, now (6) suggests a solution to this issue: in step 3, we can solve (6), albeit with population expectations replaced with empirical averages, to aggregate models of different sizes. Note that because data is assumed to stay with each agent in Federated Learning, this optimization problem may have to be solved via a federated learning algorithm.

References

Melissa J. Azur, Elizabeth A. Stuart, Constantine Frangakis, and Philip J. Leaf. Multiple imputation by chained equations: what is it and how does it work? International Journal of Methods in Psychiatric Research , 20, 2011.

David Chas Bolton, Parisa Shokouhi, Bertrand Philippe Gerard Rouet-Leduc, Claudia Hulbert, Jacques Rivière, Chris Marone, and Paul Allan Johnson. Characterizing acoustic signals and searching for precursors during the laboratory seismic cycle using unsupervised machine learning. Seismological Research Letters , 2019.

Kabir Aladin Chandrasekher, Ahmed El Alaoui, and Andrea Montanari. Imputation for highdimensional linear regression. arXiv preprint arXiv:2001.09180 , 2020.

Wei-Ning Chen, Peter Kairouz, and Ayfer Özgür. Breaking the communication-privacy-accuracy trilemma. IEEE Transactions on Information Theory , 69:1261-1281, 2020.

Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, and Qiang Yang. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems , 36:87-98, 2019.

Giacomo Curzi, Dario Modenini, and Paolo Tortora. Large constellations of small satellites: A survey of near future challenges and missions. Aerospace , 2020.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Neural Information Processing Systems , 2021.

John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Yuchen Zhang. Optimality guarantees for distributed statistical estimation. arXiv: Information Theory , 2014.

Adrià Gascón, Phillipp Schoppmann, Borja Balle, Mariana Raykova, Jack Doerner, Samee Zahur, and David Evans. Privacy-preserving distributed linear regression on high-dimensional data. Proceedings on Privacy Enhancing Technologies , 2017:345 - 364, 2017.

Elisabeth Gassiat. Revisiting the van Trees inequality in the spirit of Hájek and Le Cam. 2014.

Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. ArXiv , abs/1711.10677, 2017.

Zhenbang Jiao, Hu Sun, Xiantong Wang, Ward B. Manchester, Tamas I. I. Gombosi, Alfred O. Hero, and Yang Chen. Solar flare intensity prediction with machine learning models. Space Weather , 18, 2019.

Peter Kairouz, H. B. McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary B. Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Salim Y. El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecn ý , Aleksandra Korolova, Farinaz Koushanfar, Oluwasanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, R. Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Xiaodong Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Found. Trends Mach. Learn. , 14:1-210, 2019.

Alan F. Karr, Xiaodong Lin, Ashish P. Sanil, and Jerome P. Reiter. Privacy-preserving analysis of vertically partitioned data using secure matrix products. Journal of Official Statistics , 25: 125-138, 2009.

Gayaneh Kyureghian, Oral Capps, and Rodolfo M. Nayga. A missing variable imputation methodology with an empirical application. 2011.

Roderick J. A. Little and Donald B. Rubin. Statistical analysis with missing data, third edition. Wiley Series in Probability and Statistics , 2019.

Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang. A secure federated transfer learning framework. IEEE Intelligent Systems , 35:70-82, 2020.

Lamin R. Mansaray, Adam Sheka Kanu, Lingbo Yang, Jingfeng Huang, and Fumin Wang. Evaluation of machine learning models for rice dry biomass estimation and mapping using quad-source optical imagery. GIScience & Remote Sensing , 57:785 - 796, 2020.

H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics , 2016. URL https://api. semanticscholar.org/CorpusID:14955348 .

Board on Physics. A strategy for active remote sensing amid increased demand for radio spectrum. 2015.

Benjamin Recht, Christopher Ré, Stephen J. Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS , 2011.

Ritvik Sahajpal, Lucas Fontana, Pedro Lafluf, Guillermo Leale, Estefania Puricelli, Dan O'Neill, Mehdi Hosseini, Mauricio Varela, and Inbal Becker-Reshef. Using machine-learning models for field-scale crop yield and condition modeling in argentina. 2020.

Joseph L Schafer. Multiple imputation: a primer. Statistical methods in medical research , 8(1): 3-15, 1999.

Joseph L. Schafer and John W. Graham. Missing data: our view of the state of the art. Psychological methods , 7 2:147-77, 2002.

Joseph L Schafer and Nathaniel Schenker. Inference with imputed conditional means. Journal of the American Statistical Association , 95(449):144-154, 2000.

Ananda Theertha Suresh, Ziteng Sun, Jae Hun Ro, and Felix X. Yu. Correlated quantization for distributed mean estimation and optimization. ArXiv , abs/2203.04925, 2022.

Anastasios A. Tsiatis. Semiparametric theory and missing data. 2006.

Jaideep Vaidya and Chris Clifton. Privacy-preserving k-means clustering over vertically partitioned data. In Knowledge Discovery and Data Mining , 2003.

Naisyin Wang and James M. Robins. Large-sample theory for parametric multiple imputation procedures. Biometrika , 85:935-948, 1998.

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. arXiv: Artificial Intelligence , 2019.

Hwanjo Yu, Jaideep Vaidya, and Xiaoqian Jiang. Privacy-preserving svm classification on vertically partitioned data. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , 2006.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the idea of estimating least squares collaboratively when each agent has access to a different set of features of the same data. The aim is to design an effective and efficient algorithm in terms of communication cost (various agents transferring/communicating information/data). The paper introduces a new algorithm *COLLAB* that is efficient and also applicable in security fields where features can not be transferred between agents/sensors/machines/systems. The authors theoretically prove that the proposed algorithm *COLLAB* is minimax optimal and perform a set of experiments to showcase the capability of the proposed algorithm *COLLAB*.\n\nSTRENGTHS:\nThe paper discusses the setup typical in many fields (especially security applications) where data can not be transferred between agents due to security reasons or input-output constraints (network bottleneck). In the setup discussed, when the agent has a linear model, ordinary least squares (OLS) is a logical way to solve and get the parameter value. The idea behind *COLLAB* is logical and intuitive however needs more clarity in the presentation. The comparison against the imputation methods is also logical, as these are go-to models for such setups. The authors derive local minimax lower bounds to showcase that the proposed algorithm *COLLAB* is very close to the optimal, which is interesting. For the correctness of this section, I would rely on other reviewers as I was not able to follow the derivations clearly. The experiments are also a good mixture of real-world and synthetic data sets where the capability of the proposed algorithm is shown.\n\nWEAKNESSES:\n* A nice idea, but some assumptions are very hard/limited, and the evaluations are limited.\n* Some details (especially Sections 3.1 and 3.2) are presented convolutedly and are hard to grab. Some minor clarifications:\n  - L110, is the `x` written same as L99? It probably should be `X`?\n  - L132, L133: I am still not sure what `(i)` refers to here.\n  - In general, I believe it is convenient to stick to the notation where bold **`X`** is a matrix, bold **`x`** is a vector and `x` is a scalar. It makes it easy to follow the equations.\n* Evaluations are limited. More empirical evaluations are to be performed to showcase the results of the proposed algorithm against other methods. A limitation here is the linear model in the agents, which limits the modelling capability of the model.\n* Considering non-linear cases as well would make the paper more solid. Then the agents are more flexible, and the authors can experiment with more complex data sets.\n* Computation benefits should be shown empirically as well. Currently, it is only shown theoretically.\n\nQUESTIONS:\n* The paper assumes that all the agents have the same linear model. How restrictive is it? One of the agents has a $R^{16}$ dimension feature vector, whereas the other might have a $R^2$ dimension feature vector. So, the same complex model does not seem logical here.\n* Second assumption is that the agent has enough data to estimate $\\Sigma$. How will it impact the algorithm if there is little data and $\\Sigma$ is unreliable/biased?\n* A toy example to plot various parameter $\\theta_i$ of the agents, how they combine to give $\\theta_{global}$, and how they compare with the oracle value of $\\theta$.\n* A setup where the computational benefits can be shown empirically as well?\n\nLIMITATIONS:\nThe authors do not discuss them, and I do not see a limitation directly of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate collaborative learning of least squares estimates for multiple agents with varying feature subsets. The goal is to coordinate the agents efficiently to achieve optimal estimators without exchanging labeled data. To address this, the authors propose the distributed algorithm Collab, consisting of local training, aggregation, and distribution steps. Despite not sharing labeled data, Collab approaches near-asymptotic local minimax optimality, outperforming methods that do utilize labeled data. They validate our approach through experiments on real and synthetic datasets.\n\nSTRENGTHS:\nDistributed learning with heterogeneous data sources is a problem of broad interest. In this study, the authors tackle this problem in a simplified setting and provide robust theoretical guarantees. Their theoretical results are strong, demonstrating the solid foundation of their approach. In addition, the point of minimizing communication resources is an interesting angle. Furthermore, the experimental results are compelling, further supporting the effectiveness of their method.\n\nWEAKNESSES:\n- Settings need more justification: the authors discuss a setting where a linear regression problem is running on satellites with completely different features. This seems restrictive and can the authors elaborate more on the motivation of their study?\n- Random X with zero mean. If X is not zero-mean, then the regression with partial information is not consistent anymore (different features may have correlations). I can imagine this is a very common scenario and can the authors provide some discussions on this?\n\nQUESTIONS:\nIn Section 7, is that the numerator in *generalizing to non-linear models* requires sharing global information for $x_{i+}$? What are the possible ways to alleviate this constraint?\n\nLIMITATIONS:\nI do not foresee any potential societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies statistical inference in learning a linear regression model in the cross-silo or vertical federated learning setting. The paper formulates the problem as a missing data problem and chooses single imputation methods to deal with the associated inference of the common parameter. The paper shows the theoretical properties of the proposed estimator under some assumptions. Finally the paper compares their results with other existing methods.\n\nSTRENGTHS:\nThe paper studies a very important problem and relatively easy to follow.\n\nWEAKNESSES:\n1. The paper is not carefully written with confusing notations and problem formulation:\n\n1(a). In Section 2, it says that “the $i^{th}$ agent has data $(x_{i+}, y)$” and then later uses $y_i$ to denote the $i^{th}$ agent labeled data. It is unclear whether these labels are the same or not. If they are the same, this is an unrealistic assumption in practice and obviously contrary to the common assumption in the literature that only one active party has access to the label data. \n\n1(b). Equation (1) defines a weighted MSE, with expectation assessed with respect to the feature $x$, which is problematic. Why not consider the typical unweighted MSE with expectation evaluated with respect to $y$ in regression setting? \n\n2. It seems that the contribution of the paper is to apply existing single imputation based missing data method to vertical federated learning setting, with the exception that here they assume each agent has access to its own label data (which again is problematic).\n\n3. The paper does compare their methods with several other methods in the experiment. However, there are not enough discussions of these methods.\n\nQUESTIONS:\nWhat is the motivation to consider minimizing a weighted empirical loss in Equation (3)?\n\nLIMITATIONS:\nThe paper considers single imputation, claiming that this is okay given that the goal is estimation error instead of confidence intervals. However, many theoretical results given in the paper are about the asymptotical distributions of the proposed estimators. Do these asymptotical distributions properly consider the uncertainty associated with the missing data imputation?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary\n-------\n\nThe paper studies collaborative linear regression when m agents attempt to collaboratively\nestimate a linear model, under communication constraints. Each agent i only observes di\nof the d features. A central server designs a protocol to elicit sufficient information\nfrom each agent and compute the parameter theta of the linear model so as to minimize\ncommunication costs and the estimation error.\nThe authors present, what appears to be a near-complete solution, for the case where the\ncovariates are distributed as a Gaussian and when the covariance matrix is known. This\nincludes asymptotic normality results for the proposed estimator and lower bounds which\nmatch as n goes to infinity. The authors also compare their method against other baselines\nbased on imputing data and show that their estimation errors are no worse but at\nsignificantly lower communication cost.\n\nDecision: While the paper is well-presented and pleasant to read, I am not an expert in\nthis topic and did not have the time to go through the proofs in detail. As such,\nI am unable to evalute the technical merit of the paper (challenges of the problem,\nnovelty of proof techniques). I have given a positive score with low confidence to reflect\nthis but will heed to more expert reviewers during the discussion.\n\n\nDetailed comments\n-----------------\n\nThe authors have used local and global imputation\nbaselines in Table 1 to show the communication benefits, and have gone on to show that\ntheir method does no worse theoretically than these methods.\nHowever, I am not sure if these are particularly strong baselines to compete with; for\ninstance, I would not have expected communicating all data points to be necessary.\nIn fact, at the outset, my intuition suggested a solution which computes local\ncoefficients by each agent which are then aggregated by the server in an appropriate\nfashion.  It would have been helpful if the authors had better illustrated the challenges\nin doing so. For instance, is the method in 3.1 the most natural way to solve this\nproblem, or are there other naive ways to aggregate the coefficients that yield\nsub-optimal solutions?\n\nThe same applies to the non-Gaussian case and the setting when the co-variance matrices\nare unknown. The authors could have done a better job of illustrating the challenges.\nIn general, I did not get a sense for how challenging this problem setting was to\nappreciate the contributions by the authors.\n\nDo the results in Theorems 3.1-3.2 implicitly capture the difficulty of the problem in\nterms of the number of covariates each agent has access to?\n- For instance, in the worst case there could be only n samples (when there is a perfect\n    partition of the covariates among the agents, and they have the same points),\n    but in the best case there could be mn\n    samples (when all agents have all the covariates and have distinct points)\nIf so, is it possible to make this more explicit in the results.\n\nThe paper is largely well-written. The problem is well-motivated, the setting is described\nclearly, and the method/results are organized well. I would have however liked to see a\nsketch of the main proof ideas and how they differ from similar results in the linear\nregression literature. A breakdown of the key proof challenges that the authors had to\novercome would have also been useful.\n\nWhat was the reason for the discussion around o(n) communication complexity? It appears\nthat local imputation methods and the method of the authors are able to achieve consistent\nestimation with communication cost that does not depend on n.\n\nIn the experiments, the imputation-based methods outperform the method of the authors when\nthere are more samples. Intuitively, I would have expected this since they are more\ncommunication-heavy. However, this is not the case in the low-sample regime. Can you\nexplain why this is the case?\n\nSTRENGTHS:\nSee above.\n\nWEAKNESSES:\nSee above.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

comments: SUMMARY:
This paper studied the problem of collaboratively learning least squares estimates with multiple agents, each of which only observes a different subset of the features. The authors proposed a distributed, semi-supervised algorithm called Collab consisting of three steps: 1) local training, 2) aggregation, and 3) distribution. The authors showed that the proposed Collab algorithm is nearly asymptotically local minimax optimal. The authors conducted experiments to verify their algorithm on real and synthetic data.

STRENGTHS:
1. This paper provides deep theoretical insights into their proposed Collab algorithms. The results in Theorems 4.1 and 4.2 that the performance of Collab is no worse than local imputation with collaboration and global imputation are novel and surprising.

2. The asymptotic local minimax lower bound is interesting and could be of independent interest.

WEAKNESSES:
Although the results in this paper are interesting, as the authors themselves admitted, they are only limited to the linear model and Gaussian features. But the authors did provide some interesting discussions in Section 7 on future directions.

QUESTIONS:
1. In Step 2 of the main loop in the Collab algorithm, what happens if the estimate $\hat{\Sigma}_i$ is inaccurate? Could the authors analyze the impact of such errors?

2. Although the paper is well written in general, there are some minor typos. For example, in Line 162, it appears that the global estimator should be $\hat{\theta}^{clb}$.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-23