# ZenoPS: A Distributed Learning System Integrating Communication Efficiency and Security
## 4 Methodology
The servers simply relay all updates sent from the workers to the validators and respond to any pulling request from the workers and the validators. In the synchronous mode, the servers will wait until the approved updates are received from all validators. In the asynchronous mode, the servers will update the global model parameters whenever an approved update is received from any validator. Note that the validators can send vectors of all 0 values to the server, as a notification that an update fails the validation. The servers will not move on to the next iteration until a non-zero approved update is received. The validator uses the criterion defined in Definition 6 to validate the candidate updates sent from the servers. If a candidate update passes the validation procedure, it will be sent to the servers; otherwise the validator will send a vector of all 0 values to the server. Periodically, the validators will pull the model parameters from the servers and update the validator vector. Note that, in the synchronous mode, the validators always pull the latest version of the global model; in the asynchronous mode, the pulled model can have some delay. ```
1:Server
2:Initialize \(x_{0}\in\mathbb{R}^{d}\)
3:for all server step \(t\in[T]\)do
4:\(u_{t}\gets 0\)
5:while\(u_{t}=0\)do
6: Send \(x_{t-1}\) to workers on request
7: Receive updates \(\{\tilde{u}_{i}:i\in[n+q]\}\) from workers, and relay to validators
8:if Synchronous then
9:\(u_{t}\leftarrow\frac{1}{n_{v}}\sum_{i\in[n_{v}]}u_{i}\), after all validators respond
10:else
11:\(u_{t}\gets u_{i}\), after receiving \(u_{i}:i\in[n_{v}]\) from a validator
12:endif
13: Update the parameters \(x_{t}\gets x_{t-1}+\alpha u_{t}\)
14:endwhile
15:endfor
1:Worker (honest) \(i=1,\ldots,n\)
2:Initialize \(e_{i}\gets 0\)
3:while until terminated do
4: Receive \(x_{t^{\prime}}\) from the server, and initialize \(x_{i,t^{\prime},0}\gets x_{t^{\prime}}+e_{i}\)
5:for all local iteration \(h\in[H]\)do
6:\(x_{i,t^{\prime},h}\gets x_{i,t^{\prime},h-1}-\eta\
abla f(x_{i,t^{\prime },h-1};z_{i}\sim\mathcal{D}_{i})\)
7:endfor
8: Compute the accumulated update \(u_{i}\gets x_{i,t^{\prime},H}-x_{t^{\prime}}\)
9: Compress \(u^{\prime}_{i}\leftarrow\mathcal{C}(u_{i})\), and update the local residual error \(e_{i}\gets u_{i}-u^{\prime}_{i}\)
10: Replace \(u^{\prime}_{i}\) by arbitrary value with probability \(p_{byz}\) (optional)
11: Send \(u^{\prime}_{i}\) to the server
12:endwhile
1:Validator\(j=1,\ldots,n_{v}\)
2:Initialize \(counter=1\)
3:while until terminated do
4:if\(\mod(counter,n/n_{v})=1\)then
5: Pull \(x_{t^{\prime}}\) from server, update for \(H\) SGD steps, and obtain \(x_{t^{\prime},H}\)
6: Update the validation vector \(v_{t}=x_{t^{\prime},H}-x_{t^{\prime}}\)
7:endif
8:\(counter\gets counter+1\)
9: Wait until any update \(\tilde{u}\) arrives
10:\(u\leftarrow\tilde{u}\) if the validation defined in Definition 6 is passed, else \(u\gets 0\)
11: Send \(u\) to the server
12:endwhile
```  
**Algorithm 3** ZenoPS with score-based validation. To make the system design and the relationship between the three groups of nodes clearer, we illustrate the ZenoPS architecture in Figure 2.  
In the remainder of this section, we present more details of the optional features provided by ZenoPS.