SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingtai Xiao
Penn State University yxx5224@psu.edu
Guanlin He
Penn State University  
gbh5146@psu.edu
Danfeng Zhang
Penn State University dbz5017@psu.edu
Daniel Kifer
Penn State University
duk17@psu.edu
Noisy marginals are a common form of confidentiality-protecting data release and are useful for many downstream tasks such as contingency table analysis, construction of Bayesian networks, and even synthetic data generation. Privacy mechanisms that provide unbiased noisy answers to linear queries (such as marginals) are known as matrix mechanisms.
We propose ResidualPlanner, a matrix mechanism for marginals with Gaussian noise that is both optimal and scalable. ResidualPlanner can optimize for many loss functions that can be written as a convex function of marginal variances (prior work was restricted to just one predefined objective function). ResidualPlanner can optimize the accuracy of marginals in large scale settings in seconds, even when the previous state of the art (HDMM) runs out of memory. It even runs on datasets with 100 attributes in a couple of minutes. Furthermore ResidualPlanner can efficiently compute variance/covariance values for each marginal (prior methods quickly run out of memory, even for relatively small datasets).
Margins are tables of counts on a set of attributes (e.g., how many people there are for each combination of race and gender). They are one of the most common formats for the dissemination of statistical data [8, 2], studying correlations between attributes, and are sufficient statistics for loglinear models, including Bayesian networks and Markov random fields. For this reason, a lot of work in the differential privacy literature has considered how to produce a set of noisy marginals that is both privacy-preserving and accurate.
One line of work, called the matrix mechanism [32, 52, 30, 53, 37, 51, 46, 18, 42] designs algorithms for answering linear queries (such as marginals) so that the privacy-preserving noisy answers are accurate, unbiased, and have a simple distribution (e.g., multivariate normal). These crucial properties allow statisticians to work with the data, model error due to data collection (sampling error) and error due to privacy protections. It enables valid confidence intervals and hypothesis tests and other methods for quantifying the uncertainty of a statistical analysis (e.g., [20, 29, 50, 25, 26]). Incidentally, sets of noisy marginals are also used to generate differentially private synthetic data (e.g., [54, 4, 41, 10]).
For the case of marginals, significant effort has been spent in designing optimal or nearly optimal matrix mechanisms for just a single objective function (total variance of all the desired marginals) [32, 49, 13, 52, 53, 31, 51] and each new objective function requires significant additional effort [6, 18, 42, 46]. However, existing optimal solutions do not scale and additional effort is needed to design scalable, but suboptimal, matrix mechanisms for marginals [37, 38]. This is because prior work used mathematical properties specific to their chosen objective function in order to improve runtime. Furthermore, computing the individual variances of the desired noisy marginals is a slow process and more difficult is computing the covariance between cells in the same marginal.
Contributions. Our paper addresses these problems with a novel matrix mechanisms called ResidualPlanner. It can optimize for a wide variety of convex objective functions and return solutions that are guaranteed to be optimal under Gaussian noise. It is highly scalable – running in seconds even when other scalable algorithms run out of memory. It also efficiently returns the variance and covariances of each cell of the desired marginals. It leverages the following insights. Since a dataset can be represented as a vector  $\mathbf{x}$  of counts, and since a marginal query on a set  $\mathbf{A}$  of attributes can be represented as a matrix  $\mathbf{Q}_{\mathbf{A}}$  (with  $\mathbf{Q}_{\mathbf{A}} \mathbf{x}$  being the true answer to the marginal query), we find a new linearly independent basis that can parsimoniously represent both a marginal  $\mathbf{Q}_{\mathbf{A}}$  and the "difference" between two marginals  $\mathbf{Q}_{\mathbf{A}}$  and  $\mathbf{Q}_{\mathbf{A}'}$  (subspace spanned by the rows of  $\mathbf{Q}_{\mathbf{A}}$  that is orthogonal to the rows of  $\mathbf{Q}_{\mathbf{A}'}$ ). Using parsimonious linear bases, instead of overparametrized mechanisms, accounts for the scalability. Optimality results from a deep analysis of the symmetry that marginals impose on the optimal solution – the same linear basis is optimal for a wide variety of loss functions. Our code is available at https://github.com/dkifer/ResidualPlanner.
The Kronecker product between a  $k_{1} \times k_{2}$  matrix  $\mathbf{V} = \left[ \begin{array}{ccc} v_{1,1} & \dots & v_{1,k_2} \\ \vdots & \dots & \vdots \\ v_{k_1,1} & \dots & v_{k_1,k_2} \end{array} \right]$  and an  $\ell_{1} \times \ell_{2}$  matrix  $\mathbf{W}$ , denoted by  $\mathbf{V} \otimes \mathbf{W}$ , is the  $k_{1}\ell_{1} \times k_{2}\ell_{2}$  matrix that can be represented in block matrix form as:  $\left[ \begin{array}{ccc} v_{1,1}\mathbf{W} & \dots & v_{1,k_2}\mathbf{W} \\ \vdots & \dots & \vdots \\ v_{k_1,1}\mathbf{W} & \dots & v_{k_1,k_2}\mathbf{W} \end{array} \right]$ . A dataset  $\mathcal{D} = \{r_1,\ldots ,r_n\}$  is a collection of records (there is exactly one record for each individual in the dataset). Every record  $r_i$  contains  $n_a$  attributes  $Att_1,\ldots ,Att_{n_a}$  and each attribute  $Att_j$  can take values  $a_1^{(j)},\ldots ,a_{|Att_j|}^{(j)}$ . An attribute value  $a_i^{(j)}$  for attribute  $Att_j$  can be represented as a vector using one-hot encoding. Specifically, let  $e_i^{(j)}$  be a row vector of size  $|Att_j|$  with a 1 in component  $i$  and 0 everywhere else. In this way  $e_i^{(j)}$  represents the attribute value  $a_i^{(j)}$ . A record  $r$  with attributes  $Att_1 = a_{i_1}^{(1)}$ ,  $Att_2 = a_{i_2}^{(2)}$ , ...,  $Att_{n_a} = a_{i_{n_a}}^{(n_a)}$  can thus be represented as the Kronecker product  $e_{i_1}^{(1)}\otimes e_{i_2}^{(2)}\otimes \dots \otimes e_{i_{n_a}}^{(n_a)}$ . This vector has a 1 in exactly one position and 0s everywhere else. The position of the 1 is the index of record  $r$ . With this notation, a dataset  $\mathcal{D}$  can be represented as a vector  $\mathbf{x}$  of integers. The value at index  $i$  is the number of times the record associated with index  $i$  appears in  $\mathcal{D}$ . The number of components in this vector is denoted as  $d = \prod_{i=1}^{n_a}|Att_i|$ . Given a subset  $\mathbf{A}$  of attributes, a marginal query on  $\mathbf{A}$  is a table of counts: for each combination of values for the attributes in  $\mathbf{A}$ , it provides the number of records in  $\mathcal{D}$  having those attribute value combinations. The marginal query can be represented as a Kronecker product  $\mathbf{Q}_{\mathbf{A}} = \mathbf{V}_1\otimes \dots \otimes \mathbf{V}_{n_a}$  where  $\mathbf{V}_i$  is the row vector of all ones (i.e.,  $\mathbf{1}_{[Att_i]}^T$ ) if  $Att_i \notin \mathbf{A}$  and  $\mathbf{V}_i$  is the identity matrix  $\mathcal{I}_{|Att_i|}$  if  $Att_i \in \mathbf{A}$ . The answer to the marginal query is obtained by evaluating the matrix-vector product  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$ . For convenience, the notation introduced in this paper is summarized as a table in Section A in the supplementary material.
EXAMPLE 2.1. As a running example, consider a dataset in which there are two attributes: Att  ${}_{1}$  with values "yes" and "no",and Att  ${}_{2}$  with values "low", "med", "high". The record (no, med) is represented by the kron product  $\left\lbrack  \begin{array}{lll} 0 & 1 &  \end{array}\right\rbrack   \otimes  \left\lbrack  \begin{array}{lll} 0 & 1 & 0 \end{array}\right\rbrack$  and the marginal query on the set  $\mathbf{A} = \left\{  {\text{Att}}_{1}\right\}$  is represented as  ${\mathbf{Q}}_{\left\{  {\text{Att}}_{1}\right\}  } = \left\lbrack  \begin{array}{lll} 1 & 0 &  \\  0 & 1 &  \end{array}\right\rbrack   \otimes  \left\lbrack  \begin{array}{lll} 1 & 1 & 1 \end{array}\right\rbrack$  . Similarly,the marginal on attribute Att  ${}_{2}$  is represented as  ${\mathbf{Q}}_{\left\{  {\text{Att}}_{2}\right\}  } = \left\lbrack  {11}\right\rbrack   \otimes  \left\lbrack  \begin{array}{lll} 1 & 0 & 0 \\  0 & 1 & 0 \\  0 & 0 & 1 \end{array}\right\rbrack$  . The query representing all one-way marginals is obtained by stacking them:  ${\mathbf{Q}}^{I - {way}} = \left\lbrack  \begin{matrix} {\mathbf{Q}}_{\left\{  {\text{Att}}_{1}\right\}  } \\  {\mathbf{Q}}_{\left\{  {\text{Att}}_{2}\right\}  } \end{matrix}\right\rbrack$  and  ${\mathbf{Q}}^{I - {way}}\mathbf{x}$  consists of the five query answers (number of records with Att  ${}_{1} =$  yes,number with Att  ${}_{1} =$  no,number with Att  ${}_{2} =$  low,etc.).
2.1 Differential Privacy
A mechanism  $\mathcal{M}$  is an algorithm whose input is a dataset and whose output provides privacy protections. Differential privacy is a family of privacy definitions that guide the behavior of mechanisms so that they can inject enough noise to mask the effects of any individual. There are many versions of differential privacy that support Gaussian noise, including approximate DP, zCDP, and Gaussian DP.
DEFINITION 2.2 (Differential Privacy). Let  $\mathcal{M}$  be a mechanism. For every pair of datasets  $\mathcal{D}_1, \mathcal{D}_2$  that differ on the presence/absence of a single record and for all (measurable) sets  $S \subseteq \text{range}(\mathcal{M})$ ,
- If  $P(\mathcal{M}(\mathcal{D}_1) \in S) \leq e^\epsilon P(\mathcal{M}(\mathcal{D}_2) \in S) + \delta$  then  $\mathcal{M}$  satisfies  $(\epsilon, \delta)$ -approximate differential privacy [17];
- If  $\Phi^{-1}(P(\mathcal{M}(\mathcal{D}_1) \in S)) \leq \Phi^{-1}(P(\mathcal{M}(\mathcal{D}_2) \in S)) + \mu$ , where  $\Phi$  is the cdf of the standard Gaussian distribution, then  $\mathcal{M}$  satisfies  $\mu$ -Gaussian DP [15].
- If the Rényi divergence  $D_{\alpha}(\mathcal{M}(\mathcal{D}_1)||\mathcal{M}(\mathcal{D}_2))$  between the output distributions of  $\mathcal{M}(\mathcal{D}_1)$  and  $\mathcal{M}(\mathcal{D}_2)$  satisfies  $D_{\alpha}(\mathcal{M}(\mathcal{D}_1)||\mathcal{M}(\mathcal{D}_2)) \leq \rho \alpha$  for all  $\alpha > 1$ , then  $\mathcal{M}$  satisfies  $\rho$ -zCDP [7].
Queries that are linear functions of the data vector  $\mathbf{x}$  can be answered privately using the linear Gaussian mechanism, which adds correlated Gaussian noise to a linear function of  $\mathbf{x}$ , as follows.
DEFINITION 2.3 (Linear Gaussian Mechanism [46]). Given a  $m \times d$  matrix  $\mathbf{B}$  and  $m \times m$  covariance matrix  $\boldsymbol{\Sigma}$ , the (correlated) linear Gaussian mechanism  $\mathcal{M}$  is defined as  $\mathcal{M}(\mathbf{x}) = \mathbf{B}\mathbf{x} + N(\mathbf{0},\boldsymbol{\Sigma})$ . The privacy cost matrix of  $\mathcal{M}$  is defined as  $\mathbf{B}^T\mathbf{\Sigma}^{-1}\mathbf{B}$ . The privacy cost of  $\mathcal{M}$ , denoted by  $p\mathrm{cost}(\mathcal{M})$ , is the largest diagonal of the privacy cost matrix and is used to compute the privacy parameters:  $\mathcal{M}$  satisfies  $\rho$ -zCDP with  $\rho = p\mathrm{cost}(\mathcal{M}) / 2$  [46], satisfies  $(\epsilon, \delta)$ -approximate DP with  $\delta = \Phi(\sqrt{p\mathrm{cost}(\mathcal{M})} / 2 - \epsilon / \sqrt{p\mathrm{cost}(\mathcal{M})}) - e^{\epsilon}\Phi(-\sqrt{p\mathrm{cost}(\mathcal{M})} / 2 - \epsilon / \sqrt{p\mathrm{cost}(\mathcal{M})})$  (this is an increasing function of  $p\mathrm{cost}(\mathcal{M})$  [5]), and satisfies  $\mu$ -Gaussian DP with  $\mu = \sqrt{p\mathrm{cost}(\mathcal{M})}$  [15, 46].
The use of a non-diagonal covariance matrix is crucial because it will help simplify the description of the optimal choices of  $\mathbf{B}$  and  $\boldsymbol{\Sigma}$ . In particular, using non-diagonal covariance allows us to provide explicit formulas for the entries of the  $\mathbf{B}$  matrices. We note that an algorithm  $\mathcal{M}^*$  that releases the outputs of multiple linear Gaussian mechanisms  $\mathcal{M}_1, \ldots, \mathcal{M}_k$  (with  $\mathcal{M}_i(\mathbf{x}) = \mathbf{B}_i\mathbf{x} + N(\mathbf{0}, \mathbf{\Sigma}_i)$ ) is again a linear Gaussian mechanism. It is represented as  $\mathcal{M}^*(\mathbf{x}) = \mathbf{B}^*\mathbf{x} + N(\mathbf{0}, \mathbf{\Sigma}^*)$  with the matrix  $\mathbf{B}^*$  obtained by vertically stacking the  $\mathbf{B}_i$  and with covariance  $\boldsymbol{\Sigma}^*$  being a block-diagonal matrix where the blocks are the  $\boldsymbol{\Sigma}_i$ . Its privacy cost  $pcost(\mathcal{M}^*) = pcost(\mathcal{M}_1, \ldots, \mathcal{M}_k)$  is the largest diagonal entry of  $\sum_{i=1}^{k} \mathbf{B}_i^T \boldsymbol{\Sigma}_i^{-1} \mathbf{B}_i$ .
2.2 Matrix Mechanism
The Matrix Mechanism [32, 52, 30, 53, 37, 38, 51, 46, 18, 42] is a framework for providing unbiased privacy-preserving answers to a workload of linear queries, represented by a matrix  $\mathbf{W}$  (so that the true non-private answer to the workload queries is  $\mathbf{Wx}$ ). The matrix mechanism framework consists of 3 steps: select, measure, and reconstruct. The purpose of the select phase is to determine what we add noise to and how much noise to use. More formally, when a user's preferred noise distribution is Gaussian, the select phase chooses a Gaussian linear mechanism  $\mathcal{M}(\mathbf{x}) \equiv \mathbf{Bx} + N(\mathbf{0}, \boldsymbol{\Sigma})$  whose noisy output can be used to estimate the true query answer  $\mathbf{Wx}$ . Ideally,  $\mathcal{M}$  uses the least amount of noise subject to privacy constraints (specified by a privacy definition and settings of its privacy parameters). The measure phase runs the mechanism on the data to produce (noisy) privacy-preserving outputs  $\omega = \mathcal{M}(\mathbf{x})$ . The reconstruct step uses  $\omega$  to compute an unbiased estimate of  $\mathbf{Wx}$ . The unbiased estimate is typically  $\mathbf{W}(\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B})^\dagger \mathbf{B}^T \boldsymbol{\Sigma}^{-1} \omega$ , where  $\dagger$  represents the Moore-Penrose pseudo-inverse. This is the best linear unbiased estimate of  $\mathbf{Wx}$  that can be obtained from  $\omega$  [32]. This means that the goal of the select step is to optimize the choice of  $\mathbf{B}$  and  $\boldsymbol{\Sigma}$  so that the reconstructed answer is as accurate as possible, subject to privacy constraints. Ideally, a user would specify their accuracy requirements using a loss function, but existing matrix mechanisms do not allow this flexibility - they hard-code the loss function. In fact, adding support for new loss function used to require significant research and new optimization algorithms [53, 46, 18] because each new algorithm was customized to specific properties of a chosen loss function. On top of this, existing optimal matrix mechanism algorithms do not scale, while scalable matrix mechanisms are not guaranteed to produce optimal solutions [37]. Additionally, the reconstruction phase should also compute the variance of each workload answer. The variances are the diagonals of  $\mathbf{W}(\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B})^\dagger \mathbf{W}^T$  and making this computation scale is also challenging.
3 Additional Related Work
The marginal release mechanism by Barak et al. [6] predates the matrix mechanism [32, 52, 30, 53, 13, 43, 37, 51, 46, 18, 42, 38] and adds noise to the Fourier decomposition of marginals. We add noise to a different basis, resulting in the scalability and optimality properties. The SVD bound [31] is a lower bound on total matrix mechanism error when the loss function is the sum of variances. This lower bound is tight for marginals and we use it as a sanity check for our results and implementation (note ResidualPlanner provides optimal solutions even when the SVD bound is infeasible to compute).
Alternative approaches to the matrix mechanism can produce privacy preserving marginal query answers that reduce variance by adding bias. This is often done by generating differentially private synthetic data or other such data synopses from which marginals can be computed. State-of-the-art approaches iteratively ask queries and fit synthetic data to the resulting answers [22, 34, 4, 19, 39, 35, 44, 56]. For such mechanisms, it is difficult to estimate error of a query answer but recently AIM [39] has made progress in upper bounding the error. PGM [41] provides a connection between the matrix mechanism and this line of work, as it can postprocess noisy marginals into synthetic data. It is a better alternative to sampling a synthetic dataset from models fit to carefully chosen marginals [54, 11, 55, 10]. Synthetic data for answering marginal queries can also be created from random projections [48], copulas [33, 3], and deep generative models [23, 1, 35].
With respect to the matrix mechanism, the reconstruction step is often one of the bottlenecks to scalability. While PGM [41] provides one solution, another proposal by McKenna et al. [40] is to further improve scalability by sacrificing some consistency (the answers to two different marginals may provide conflicting answers to submarginals they have in common). Work on differential privacy marginals has also seen extensions to hierarchical datasets, in which records form meaningful groups that need to be queried. That is, in addition to marginals on characteristics of people, marginals can be computed in different hierarchies such as geographic level (state, county, etc.) and marginals on household composition (or other groupings of people) [2, 28, 36].
4 ResidualPlanner
ResidualPlanner is our proposed matrix mechanism for optimizing the accuracy of marginal queries with Gaussian noise. It is optimal and more scalable than existing approaches. It supports optimizing the accuracy of marginals under a wide variety of loss functions and provides exact variances/covariances of the noisy marginals in closed-form. In this section, we first explain the loss functions it supports. We then describe the base mechanisms it uses to answer marginal queries. We next show how to reconstruct the marginal queries from the outputs of the base mechanisms and how to compute their variances in closed form. We then explain how to optimize these base mechanisms for different loss functions. The reason this selection step is presented last is because it depends on the closed form variance calculations. Then we analyze computational complexity. To aid in understanding, we added a complete numerical run-through of the steps in Section B of the supplementary material. All of our proofs also appear in the supplementary material.
4.1 Loss Functions Supported by ResidualPlanner
The loss functions we consider are generalizations of the sum of variances and max of variances used in prior work. Our more general class of loss functions is based on the following principle: different marginals can have different relative importance but within a marginal, its cells are equally important. That is, a loss function can express that the two-way marginal on the attribute set  $\{\text{Race}, \text{Marital Status}\}$  is more important (i.e., requires more accuracy) than the 1-way marginal on  $\{\text{EducationLevel}\}$ , but all cells within the  $\{\text{Race}, \text{MaritalStatus}\}$  marginal are equally important. This is a commonly accepted principle for answering differentially private marginal queries (e.g., [32, 52, 30, 53, 37, 51, 46, 18, 42, 39, 4, 34]) and is certainly true for the 2020 Census redistricting data [2].
Let  $Wkload = \{\mathbf{A}_1, \dots, \mathbf{A}_k\}$  be a workload of marginals, where each  $\mathbf{A}_i$  is a subset of attributes and represents a marginal. E.g.,  $Wkload = \{\{\text{Race}, \text{MaritalStatus}\}, \{\text{EducationLevel}\}\}$  consists of 2 marginals, a two-way marginal on Race/MaritalStatus, and a one-way marginal on Education. Let  $\mathcal{M}$  be a Gaussian linear mechanism whose output can be used to reconstruct unbiased answers to the marginals in  $Wkload$ . For each  $\mathbf{A}_i \in Wkload$ , let  $Var(\mathbf{A}_i; \mathcal{M})$  be the function that returns the
variances of the reconstructed answers to the marginal on  $\mathbf{A}_i$ ; the output of  $Var(\mathbf{A}_i; \mathcal{M})$  is a vector  $v_i$  with one component for each cell of the marginal on  $\mathbf{A}_i$ . A loss function  $\mathcal{L}$  aggregates all of these vectors together:  $\mathcal{L}(v_1, \ldots, v_k)$ . We have the following regularity conditions on the loss function.
DEFINITION 4.1 (Regular Loss Function). We say the loss function  $\mathcal{L}$  is regular if: (1)  $\mathcal{L}$  is convex and continuous; (2)  $\tilde{\mathcal{L}}(v_1, \ldots, v_k)$  is minimized when all the  $v_i$  are the 0 vectors; and (3) for any  $i$ , permuting just the components of  $v_i$  does not affect the value of  $\mathcal{L}(v_1, \ldots, v_k)$ . This latter condition just says that cells within the same marginal are equally important.
Loss functions used on prior work are all regular. For example, weighted sum of variances [32, 52, 30, 53, 37, 51] can be expressed as  $\mathcal{L}(v_1,\ldots ,v_k) = \sum_i c_i\mathbf{1}^T v_i$ , where the  $c_{i}$  are the nonnegative weights that indicate the relative importance of the different marginals. Another popular loss function is maximum (weighted) variance [46, 18, 42], expressed as  $\mathcal{L}(v_1,\dots ,v_k) = \max \left\{\frac{\max(v_1)}{c_1},\dots,\frac{\max(v_k)}{c_k}\right\}$ . Thus, the optimization problem that the selection step needs to solve is either privacy constrained: minimize loss while keeping privacy cost (defined at the end of Section 2.1) below a threshold  $\gamma$ ; or utility constrained: minimize privacy cost such that the loss is at most  $\gamma$ .
Privacy constrained:  $\arg \min_{\mathcal{M}}\mathcal{L}(Var(A_1;\mathcal{M}),\ldots ,Var(A_k;\mathcal{M}))$  s.t.  $pcost(\mathcal{M})\leq \gamma$  (1)
Utility constrained: arg min  $pcost(\mathcal{M})$  s.t.  $\mathcal{L}(Var(A_1;\mathcal{M}),\ldots ,Var(A_k;\mathcal{M}))\leq \gamma$  (2)
We note that regular loss functions cover other interesting cases. For instance, suppose Alicia, Bob, and Carol wish to minimize the sum of variances on their own separate workloads. Taking the max over these three sum-of-variances as the loss function allows the data curator to minimize the largest unhappiness among the three data stakeholders.
4.2 Base Mechanisms used by ResidualPlanner
In the most common application setting, the user picks a privacy budget, a workload of marginals and a loss function  $\mathcal{L}$ . Based on these choices, a matrix mechanism must decide what linear queries to add noise to and how much noise to add to them. Then it uses those noisy measurements to reconstruct answers to the workload queries. In the case of ResidualPlanner, the linear queries that need noise are represented by base mechanisms that are described in this section. Each base mechanism has a scalar noise parameter that determines how much noise it uses (so optimizing the loss function  $\mathcal{L}$  is equivalent to finding a good value for the noise parameter of each base mechanism). As long as the loss function  $\mathcal{L}$  is regular, we prove that an optimal mechanism can be constructed from the set of base mechanisms that we describe here.
To begin, we define a subtraction matrix  $\mathbf{Sub}_m$  to be an  $(m - 1) \times m$  matrix where the first column is filled with 1, entries of the form  $(i, i + 1)$  are -1, and all other entries are 0. For example,  $\mathbf{Sub}_3 = \left[ \begin{array}{ccc}1 & -1 & 0 \\ 1 & 0 & -1 \end{array} \right]$  and  $\mathbf{Sub}_2 = \left[ \begin{array}{c}1 & -1 \end{array} \right]$ . We use these subtraction matrices to define special matrices called residual matrices that are important for our algorithm.