SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
U \leftarrow \left(\left(e ^ {\varepsilon} + | \mathcal {Y} | - 1\right) \cdot \max  (\mathcal {Y}) - \sum_ {y \in \mathcal {Y}} y\right) / \left(e ^ {\varepsilon} - 1\right)
$$
$$
\Delta \leftarrow (U - L) / (n - 1)
$$
$$
\text {r e t u r n} \hat {\mathcal {Y}} = (L, L + \Delta , L + 2 \Delta , \dots , U - \Delta , U)
$$
We defer the proof to Appendix F; the main idea is that these endpoints create an interval that contains the interval one would obtain from the debiased randomized response on the two label set of the minimum label and maximum label. Because the  $\hat{\mathcal{V}}$  we choose has labels less than this minimum and larger than this maximum, all labels in between can be generated by interpolation.
Additionally, we show that using the randomizer optimized on the grid  $\hat{\mathcal{V}}$  gives bounded excess noisy label loss compared to optimizing over a continuous set of labels for Lipschitz loss functions.
Lemma 8. Let  $\mathcal{M}$  be the optimal unbiased randomizer with  $\hat{\mathcal{Y}} = [L,U]$ , and let  $\mathcal{M}_{\Delta}$  be the optimal unbiased randomizer with  $\hat{\mathcal{Y}} = \{L,L + \Delta ,\dots ,U - \Delta ,U\}$ . If  $g(\hat{y},y)$  is  $K$ -Lipschitz in  $\hat{y}$ , then  $\mathcal{G}(\mathcal{M}_{\Delta};\mathcal{P})\leq \mathcal{G}(\mathcal{M};\mathcal{P}) + K\Delta$
In the case of  $g(\hat{y}, y) = \frac{1}{2} (\hat{y} - y)^2$ , we have  $K = (U - L)$ . Therefore, as the grid gets finer, the excess noisy label loss of  $\mathcal{M}_{\Delta}$  obtained from 1 over the noisy label loss of  $\mathcal{M}$  scales linearly in the discretization parameter  $\Delta$ . We present the proof in Appendix E.
For large  $\varepsilon$  values, we experimentally observe that the optimal unbiased randomizer appears to approach  $\varepsilon$ -dbRR. For small  $\varepsilon$  values, we also experimentally observe that the optimal unbiased randomizer appears to be supported on the labels of the debiased randomized response on the two label set of the minimum label and the maximum label. This justifies our choice of endpoints for grid.
Splitting budget between prior and label.
So far, we have assumed a known prior  $P$  over input labels  $\mathcal{V}$ . However, this is typically not the case, and so we use the standard Laplace mechanism to privately estimate the prior. Given  $n$  samples drawn from  $P$ ,  $\mathcal{M}_{\varepsilon}^{\mathrm{Lap}}$  constructs a histogram over  $\mathcal{V}$  and adds Laplace noise with scale  $2 / \varepsilon$  to each entry, followed by clipping (to ensure that entries are non-negative) and normalization. For completeness, we include a formal description of  $\mathcal{M}_{\varepsilon}^{\mathrm{Lap}}$  in Appendix C.
Our randomizer for the unknown prior case, described in Algorithm 3, thus operates by splitting
the privacy budget into  $\varepsilon_1, \varepsilon_2$ , using  $\mathcal{M}_{\varepsilon_1}^{\mathrm{Lap}}$  to get an approximate prior distribution  $P'$ , and using the randomizer returned by ComputeOptUnbiasedRand $_{\varepsilon_2}$ ( $\mathcal{P}', \hat{\mathcal{V}}$ ) to randomize the labels. It follows that the entire algorithm is  $(\varepsilon_1 + \varepsilon_2)$ -DP.
Algorithm 3 LabelRandomizer  $\varepsilon_{1},\varepsilon_{2}$
Parameters: Privacy parameters  $\varepsilon_1, \varepsilon_2 \geq 0$ .
Input: Labels  $y_{1},\ldots ,y_{n}\in \mathcal{V}$
Output:  $\hat{y}_1, \dots, \hat{y}_n \in \hat{\mathcal{V}}$ .
$$
P ^ {\prime} \leftarrow \mathcal {M} _ {\varepsilon_ {1}} ^ {\text {L a p}} \left(y _ {1}, \dots , y _ {n}\right)
$$
$\hat{\mathcal{V}} \gets$  FeasibleOutputSet $_{\varepsilon_1}(\mathcal{V})$
$\mathcal{M}\gets$  ComputeOptUnbiasedRand  $\varepsilon_{2}(\mathcal{P}^{\prime},\hat{\mathcal{Y}}^{\prime})$
for  $i\in [n]$  do
$$
\hat {y} _ {i} \leftarrow \mathcal {M} (y _ {i})
$$
return  $(\hat{y}_1,\dots ,\hat{y}_n)$
Handling continuous  $\mathcal{Y}$ . While we focused on the case of finite sets  $\mathcal{V}$ , the approach can be extended to the case where  $\mathcal{V}$  is a continuous set, say  $\mathcal{V} = [L,U]$ . In this case, we can first choose a finite discrete subset  $\widetilde{\mathcal{V}}\subseteq \mathcal{V}$  that contains both  $L$  and  $U$ , e.g.,  $\widetilde{\mathcal{V}} = \{L,L + \Delta ,\ldots ,U\}$ , and then apply unbiased randomized rounding  $\mathrm{URR}_{\widetilde{\mathcal{V}}}(\cdot)$  to all labels before applying any mechanism  $\mathcal{M}$ .
Namely, for any label  $y \in [L, U]$  such that  $y_1 \leq y \leq y_2$  for  $y_1, y_2$  being consecutive elements of  $\widetilde{\mathcal{V}}$ ,  $\mathrm{URR}_{\widetilde{\mathcal{V}}}(\mathcal{Y})$  returns  $\widetilde{\mathcal{Y}}$  drawn as  $y_1$  with probability  $\frac{y_2 - y}{y_2 - y_1}$  and  $y_2$  with probability  $\frac{y - y_1}{y_2 - y_1}$ . This ensures that  $\mathbb{E}[\widetilde{y}] = y$  and hence, for any unbiased mechanism  $\mathcal{M}$  that acts on inputs in  $\widetilde{\mathcal{V}}$ , it holds for all
Figure 3: Illustration of various  $\varepsilon$ -DP label randomizers for  $\varepsilon = 4$ . The 2D density plot contours are generated in log scale using Gaussian kernel density estimates. The legends show the MSE between the original labels and the  $\varepsilon$ -DP randomized labels.
$y \in \mathcal{V}$  that  $\mathbb{E}[\mathcal{M}(\mathrm{URR}_{\widetilde{\mathcal{Y}}}(\mathcal{y}))] = \mathbb{E}[\mathrm{URR}_{\widetilde{\mathcal{Y}}}(\mathcal{y})] = y$ . Furthermore, as the gap between consecutive points of  $\widetilde{\mathcal{Y}}$  decreases, the variance introduced by  $\mathrm{URR}_{\widetilde{\mathcal{Y}}}$  also decreases.
We evaluate our randomizer on three datasets, and compare with the Laplace mechanism [DMNS06], the additive staircase mechanism [GV14], and the RR-on-Bins method  $\mathrm{[GKK^{+}23]}$ . The Laplace mechanism and the additive staircase mechanism both have a discrete and a continuous variant. Following  $\mathrm{[GKK^{+}23]}$ , we use the continuous variants for real-valued labels (the Criteo Sponsored Search Conversion dataset), and the discrete variants for integer-valued labels (the US Census dataset and the App Ads Conversion Count dataset). Note that in the experiments from  $\mathrm{[GKK^{+}23]}$ , the noised labels from both the Laplace mechanism and the additive staircase mechanism were clipped to be in the valid label value range, as for small  $\varepsilon$ 's, the magnitude of the noised labels could be orders of magnitudes larger than the original label values, potentially causing numerical instability in model training. However, in our study, we found that with more careful hyperparameter tuning and early stopping (see Appendix G), the learning could be stabilized for sufficiently small values (e.g.,  $\varepsilon \geq 0.3$ ) that is practically useful in ML, and in this case, the unclipped (therefore also unbiased) version of both the Laplace and the additive staircase mechanisms outperform their clipped counterpart. For reference, we present the results for both clipped and unclipped variants for those two mechanisms. We note that all of these mechanisms can be implemented in the feature-oblivious label DP setting (Figure 1). More details on model and training configurations are presented in Appendix G.
4.1 Criteo Sponsored Search Conversion
The Criteo Sponsored Search Conversion Log Dataset [TY18] is a collection of 15,995,634 data points derived from a sample of 90-day logs of live traffic from Criteo Predictive Search (CPS). Each example contains information of an user action (e.g., a click on an advertisement) and a subsequent conversion (purchase of the related product) within a 30-day attribution window. We use the setup of feature-oblivious label DP to predict the revenue in € (i.e., the SalesAmountInEuro field in the dataset) of a conversion. We apply the following preprocessing steps: filtering out examples with SalesAmountInEuro being  $-1$ , and clipping the labels at the 95th percentile of the value distribution  $(400\mathbb{E})$
In Figure 3, we visualize an example of the various label randomizers. For  $\varepsilon = 4$ , the optimal RR-on-Bins randomizer maps the input values to one of four distinct values. On the other hand, for the optimal unbiased randomizer, the joint distribution of the sensitive labels and randomized labels maintains an overall concentration along the diagonal. The joint distribution for the Laplace mechanism is quite spread out.
In Figure 4, we compare the noisy label loss on the training set and the mean squared error on the test set for all the randomizers considered. For each randomizer, the label randomization and training was run with 10 independent random seeds, and the plots show the average and standard deviation bars. For RR-on-Bins and the optimal unbiased randomizer, we use  $\varepsilon_{1} = 0.017$  for privately estimating the prior using  $\mathcal{M}_{\varepsilon_1}^{\mathrm{Lap}}$ , and  $\varepsilon_{2} = \varepsilon -\varepsilon_{1}$  for randomizing the label, following the guidance in Appendix C.
(a) Noisy Label Loss
(b) Test Loss
Figure 4: Comparison of different label DP randomizers on the Criteo Search dataset: (a) shows the mean squared error between the original labels and the privatized labels on the training set from each randomizer; (b) shows the mean squared error between the model predictions and the groundtruth labels on the test set. The solid line indicates the non-private baseline.
(a) US Census
(b) App Ads
Figure 5: Comparison of different label DP randomizers: (a) shows the mean squared error of the prediction on the US Census test set. The solid line indicates the non-private baseline. (b) shows the relative Poisson loss (i.e.,  $(L - L^{\star}) / L^{\star}$ ) with respect to the non-private baseline Poisson loss  $L^{\star}$ .
We find that the optimal unbiased randomizer (Algorithm 3) achieves the smallest test mean squared error across a wide range of  $\varepsilon$  values. It is interesting to note that the unbiased randomizers (Laplace and additive staircase mechanisms and the optimal unbiased randomizer) vastly outperform the biased randomizers (clipped versions of Laplace and additive staircase mechanisms as well as the RR-on-Bins) in terms of test loss, even though the noisy label loss is an order of magnitude larger for the unbiased randomizers.
4.2 US Census
The 1940 US Census dataset<sup>3</sup> is widely used in the evaluation of data analysis with DP [WDZ+19, CJG21, GKM+21]. This digitally released dataset in 2012 contains 131,903,909 examples. We follow the task studied in [GKK+23] and evaluate label DP algorithms by learning to predict the number of weeks each respondent worked during the previous year (the WKSWORK1 field).
In Figure 5a we compare the mean squared error on the test set for all the randomizers considered. For each randomizer, the label randomization and training was run with 10 independent random seeds, and the plots show the average and standard deviation bars. For RR-on-Bins and the optimal unbiased randomizer, we use  $\varepsilon_{1} = 0.002$  for privately estimating the prior using  $\mathcal{M}_{\varepsilon_1}^{\mathrm{Lap}}$ , and  $\varepsilon_{2} = \varepsilon -\varepsilon_{1}$  for randomizing the label, following the guidance in Appendix C. We find that the optimal unbiased randomizer achieves the smallest test mean squared error across a wide range of  $\varepsilon$  values.
<sup>3</sup>https://www.archives.gov/research/census/1940
4.3 App Ads Conversion Count
We also evaluate on a conversion count prediction dataset collected from a commercial mobile app store. Each example in this dataset corresponds to an ad click and the task is to predict the number of post-click conversion events in the app after a user installs the app within a certain time window.
In Figure 5b we compare the relative Poisson loss on the test set for all the randomizers considered. For each randomizer, the label randomization and training was run with 6 independent random seeds. For the optimal unbiased randomizer, Laplace and additive staircase mechanisms, at low  $\varepsilon$ 's we experience blowup in the training loss during training. The plot shows the average and standard deviation bars, at the time of lowest training loss, right before blowup. We see that for all  $\varepsilon$ 's, all of the unbiased randomizers vastly outperform the others, including RR-on-Bins. Among the unbiased randomizers (optimal unbiased randomizer, Laplace and additive staircase mechanisms), the optimal unbiased randomizer performs the best at all  $\varepsilon$ 's. One caveat is that for the smaller  $\varepsilon$ 's, one can see the standard error increase. We hypothesize this is not an inherent feature of the randomizers but due to the blow up of the model training, reducing the training length and adding additional variance to the error, a behavior that seems specific to the AppAds dataset.
DP learning has been the subject of considerable research spanning different settings that include empirical risk minimization [CMS11], PAC learning [BLM20], training neural networks  $\mathrm{[ACG^{+}16]}$ , online learning [JKT12], and regression [Wan18]. For the label DP setting, [CH11, BNS16] studied the sample complexity of classification, while [WX19] studied sparse linear regression in the local DP model [EGS03,  $\mathrm{KLN^{+}11}$ ]. For training deep neural networks with label DP,  $\mathrm{[GGK^{+}21]}$  studied the classification setting and gave a multi-stage training procedure where the priors on the labels in the previous stage are used to define an LP that is optimized to find the randomization mechanism for the next stage. For the classification loss, they characterized the optimal solution as the so-called RRTop- $k$ . By contrast, the work of  $\mathrm{[GKK^{+}23]}$  showed that for regression objectives the optimal solution to the LP is an RR-on-Bins solution. A crucial insight in our work is that the addition of an unbiasedness constraint to the LP leads to solutions that (i) are not RR-on-Bins solutions, (ii) can have substantially higher variance than RR-on-Bins, and (iii) nevertheless have a much lower train and test error due to the reduction in bias.
Kairouz et al. [KOV16] defined a family of staircase mechanisms and showed they are optimal among local DP algorithms for minimizing an objective function given a prior. We extend the notion of staircase mechanisms to include real-valued labels that affect the optimization function. The structural properties that we prove (Theorem 6) utilize the ordering of the labels and the unbiasedness condition, which we show is critical for training neural networks for regression with label DP and high accuracy. Moreover, their work shows the presence of staircase mechanisms in the context of optimizing mutual information and KL-divergence objectives, whereas we show their presence in the context of regression.
A two-party learning setting with one features party and one labels party was recently studied in the work of  $\mathrm{[LSY^{+}21]}$ . They focus on the interactive setting, which is arguably less practical than the (non-interactive) feature-oblivious setting studied in  $\mathrm{[GKK^{+}23]}$  that we also consider.
We also note that the label DP mechanism of  $\mathrm{[MEMP^{+}21]}$ , which builds on the PATE framework  $\mathrm{[PAE^{+}17, PSM^{+}18]}$ , and the works of [EMSV22, TNM $^{+}22$ ], which leverage unsupervised and semi-supervised learning algorithms, cannot be implemented in the feature-oblivious setting. The DP-SGD algorithm of  $\mathrm{[ACG^{+}16]}$ , which protects both the features and the label of each training example and which has been applied to different domains including computer vision  $\mathrm{[DBH^{+}22]}$  and language models  $\mathrm{[YNB^{+}22]}$  also cannot be implemented in the feature-oblivious label DP setting. The same is also true for label DP algorithms for logistic regression [GR21] that are based on linear queries (where the features are assumed to be known and non-sensitive, and the linearity is with respect to the labels).
Over the past decade, there has been a significant body of research on DP ML (e.g., [CMS11, ZZX+12, SCS13]). In particular, DP regression has been the focus of several prior papers including [ZZX+12, KST12, Wan18, SÁZL18, AMS+22].
The label DP setting has also been studied in several papers including [CH11, BNS16, WX19,  $\mathrm{GGK}^{+}21$ ,  $\mathrm{MEMP}^{+}21$ , YSMN21, EMSV22, BMS22,  $\mathrm{GKK}^{+}23$ ]. The Randomized Response (RR) mechanism [War65], a basic form of label DP, was introduced several decades ago and is widely studied/used.
In this work, we show that using unbiased  $\varepsilon$ -DP label randomizers lead to better trained models, and in particular, choosing the label randomizer that minimizes the noisy label squared loss seems to perform the best in terms of test performance, by empirically demonstrating this on three datasets. We also provide theoretical results shedding light on the structure of these randomizers, as well as why they might lead to better trained models.
Discussion. While we focused entirely on  $\varepsilon$ -DP ("pure-DP"), for our specific approach, it does not seem that relaxing to  $(\varepsilon, \delta)$ -DP ("approximate-DP") will be beneficial. For the first stage of estimating the histogram privately, one could potentially use an approximate-DP mechanism, but we believe that is unlikely to change the prior significantly. For the second stage of randomizing labels, it is known that approximate-DP may not be helpful in the local model [BNS18].
In this work, we use hyperparameter tuning to choose the best architecture, which in general has additional privacy costs, and how to tune hyperparameters privately and efficiently is an active research topic [PS22, SSS23]. Consequently, it is common in the private ML literature to separate the question of private hyperparameter tuning and private training, and focus on comparing the privacy-utility trade-off under optimal hyperparameters, e.g.,  $\mathrm{[MEMP^{+}21, HLY^{+}23, KCS^{+}22, DBH^{+}22]}$ . We also follow this convention.
Limitations and future work. While LPs are somewhat efficient, it would be desirable to have a more efficient algorithm for computing the optimal unbiased randomizer. The RR-on-Bins randomizer introduced in  $\mathrm{[GKK^{+}23]}$  has an important advantage that one does not need to construct the set of potential output labels  $\hat{\mathcal{V}}$  before hand, and in fact, there is a simple dynamic programming algorithm to compute the optimal randomizer along with the output labels. Moreover, the RR-on-Bins family of randomizers has a clean structure. Understanding more structural properties of the optimal unbiased randomizers, and designing better algorithms for computing them, would be an interesting future direction to pursue.
To further improve the benefit of this optimal prior-based unbiased label randomizer, it might be possible to first partition the input examples into groups of "related examples", purely using the input features and compute the optimal unbiased randomizer for each group, by privately estimating the prior over labels for each group. This could lead to better test performance, as the randomizer can adapt to the different priors across these groups, unlike the "static" label randomizers such as Laplace and additive staircase mechanisms. For example, such an approach was used in the setting of image classification using self-supervised learning in  $\mathrm{[GGK^{+}21]}$ .
As shown in (2), there is a bias-variance trade-off when it comes to choosing a label randomizer. While RR-on-Bins minimizes the noisy label loss (corresponding to the variance), the method in our work minimizes the bias (in fact, making it zero), at the cost of greatly increasing the variance. However, it might be possible in some settings that allowing a small amount of bias might greatly reduce the variance, thereby improving performance. This might be especially relevant when  $\varepsilon$  is very small. This could be done for example by adding a regularizer to reduce the bias, without enforcing hard unbiased constraints, as currently done in Algorithm 1. We leave this investigation to future work.
[YNB+22] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In ICLR, 2022.
[YSMN21] Sen Yuan, Milan Shen, Ilya Mironov, and Anderson Nascimento. Label private deep learning training based on secure multiparty computation and differential privacy. In NeurIPS Workshop Privacy in Machine Learning, 2021.
$\left[\mathrm{ZZX}^{+}12\right]$  Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Yin Yang, and Marianne Winslett. Functional mechanism: regression analysis under differential privacy. VLDB, 5(11):1364-1375, 2012.
A Bayes Optimal Predictor Using Noisy Labels
We prove Theorem 5, restated below for convenience.
Proof. If  $\mathcal{M}$  is unbiased, then  $\mathbb{E}_{\hat{y} \sim \mathcal{M}(y)}[\hat{y}] = \sum_{\hat{y} \in \hat{\mathcal{Y}}} \hat{y} \cdot \operatorname*{Pr}[\mathcal{M}(y) = \hat{y}] = y$  for all input labels  $y \in \mathcal{V}$ . We have the following equalities (we restrict ourselves to the case of finite  $\mathcal{V}$  and  $\hat{\mathcal{V}}$ , but the proof generalizes easily to measurable sets  $\mathcal{V}$  and  $\hat{\mathcal{V}}$ ):
$$
\begin{array}{l} \underset { \begin{array}{c} \hat {y} \sim \mathcal {M} (y) \\ (x, y) \sim \mathcal {D} \end{array} } {\mathbb {E}} [ \hat {y} \mid x = x _ {0} ] = \sum_ {y \in \mathcal {Y}, \hat {y} \in \hat {\mathcal {Y}}} \hat {y} \cdot \Pr [ \hat {y} \mid y, x = x _ {0} ] \cdot \Pr [ y \mid x = x _ {0} ] \\ = \sum_ {y \in \mathcal {Y}, \hat {y} \in \hat {\mathcal {Y}}} \hat {y} \cdot \Pr [ \hat {y} \mid y ] \cdot \Pr [ y \mid x = x _ {0} ] (\because \text {f e t u r e o b l i v i o u s n e s s}) \\ = \sum_ {y \in \mathcal {Y}} \Pr [ y \mid x = x _ {0} ] \cdot \left(\sum_ {\hat {y} \in \hat {\mathcal {Y}}} \hat {y} \cdot \Pr [ \hat {y} \mid y ]\right) \\ = \sum_ {y \in \mathcal {Y}} \Pr [ y \mid x = x _ {0} ] \cdot y (\because \text {u n b i a s e d n e s s}) \\ = \underset {y \sim D (x _ {0})} {\mathbb {E}} [ y \mid x = x _ {0} ], \\ \end{array}
$$
showing that the Bayes optimal predictors are equal at all values  $x_0$ .
Conversely, suppose we have a  $y^\prime \in \mathcal{V}$  such that  $\mathbb{E}_{\hat{y}\sim \mathcal{M}(y^{\prime})}[\hat{y} ]\neq y^{\prime}$ . Then take any distribution  $D$  for which there is some  $x_0$  with  $\operatorname *{Pr}[y = y^{\prime}\mid x = x_{0}] = 1$ . Then clearly  $\mathbb{E}_{y\sim D(x_0)}[y\mid x = x_0] = y'$ . However,  $\mathbb{E}_{\hat{y}\sim \mathcal{M}(y),y\sim D(x_0)}[\hat{y}\mid x = x_0] = \mathbb{E}_{\hat{y}\sim \mathcal{M}(y')}[\hat{y} ]\neq y'$ . Hence the Bayes optimal predictors differ at  $x_0$ .
B Structural Properties of Optimal Unbiased Randomizers
We prove Theorem 6, restated below for convenience. We use the following assumption about the label loss  $g$ , which in particular, is satisfied by  $\ell_{\mathrm{sq}}$  that we primarily use.
Assumption 9. The loss  $g: \mathbb{R} \times \mathbb{R} \to \mathbb{R}$  is such that for all  $y$ , it holds that  $g(\hat{y}, y)$  is convex in  $\hat{y}$ , and minimized at  $\hat{y} = y$  for all  $y$ .
We recall the definition of a signature matrix from  $\mathrm{[GKK^{+}23]}$ . Consider any feasible solution  $M\coloneqq (M_{y\rightarrow i})_{y\in \mathcal{Y},i\in \mathcal{I}}$  of the LP in Algorithm 1. Note that for reasons that will be clear shortly, we do not require the  $\hat{y}_i$  values to be distinct, and we will assume without loss of generality that  $\hat{y}_i\leq \hat{y}_j$  for all  $i < j\in \mathcal{I}$ . Let  $p_i^{\mathrm{min}} = \min_y M_{y\rightarrow i}$  and let  $p_i^{\mathrm{max}} = \max_y M_{y\rightarrow i}$ . Note that if  $M$  encodes an  $\varepsilon$ -DP randomizer, it holds that  $p_i^{\mathrm{max}}\leq e^\varepsilon \cdot p_i^{\mathrm{min}}$ .
Definition 10 (Signature matrix). For any  $\varepsilon$ -DP randomizer encoded by  $M$ , the corresponding signature matrix entry  $S_M(y, i)$  for all  $y \in \mathcal{V}$  and  $i \in \mathcal{I}$  is defined as