SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The query matrix of  $\overline{\mathcal{M}}$  is  $\overline{\mathbf{B}} = \left[ \begin{array}{c}\mathbf{B}_{opt}\mathbf{W}_{\pi_1}\\ \vdots \\ \mathbf{B}_{opt}\dot{\mathbf{W}}_{\pi_{|\Pi |}} \end{array} \right]$  and the covariance matrix  $\overline{\Sigma}$  is a block diagonal matrix with the scaled matrix  $|\Pi |\Sigma_{opt}$  in each block. Clearly, by Lemma D.2, it also provides unbiased answers to the marginals in Wkload.
First, we claim that the  $pcost(\overline{\mathcal{M}}) \leq pcost(\mathcal{M}_{opt})$  so that the privacy parameters are at least as good. Recall  $pcost(\overline{\mathcal{M}})$  is the largest diagonal entry of:
$$
\overline {{\mathbf {B}}} ^ {T} \overline {{\boldsymbol {\Sigma}}} ^ {- 1} \overline {{\mathbf {B}}} = \frac {1}{| \Pi |} \sum_ {\pi \in \Pi} \mathbf {W} _ {\pi} ^ {T} \mathbf {B} _ {o p t} ^ {T} \boldsymbol {\Sigma} _ {o p t} ^ {- 1} \mathbf {B} _ {o p t} \mathbf {W} _ {\pi}, \tag {4}
$$
Since the privacy cost  $pcost(\mathcal{M}_{\pi,1})$  is the largest diagonal of  $\mathbf{W}_{\pi}^{T}\mathbf{B}_{opt}^{T}\pmb{\Sigma}_{opt}^{-1}\mathbf{B}_{opt}\mathbf{W}_{\pi}$  and equals  $pcost(\mathcal{M}_{opt})$ , Equation 4 (and convexity of the max function) shows that the  $pcost(\overline{\mathcal{M}}) \leq pcost(\mathcal{M}_{opt})$ .
Next we consider the loss function. Let  $\mathbf{A}_i \in Wkload$  be a set of attributes and let  $\mathbf{Q}_{\mathbf{A}_i}$  be the corresponding query matrix for the marginal on  $\mathbf{A}_i$ . Then the reconstructed variances of the answers to this marginal, based on the output of  $\overline{\mathcal{M}}$  is:
$$
\begin{array}{l} V a r (\mathbf {A} _ {i}; \overline {{\mathcal {M}}}) = d i a g \left(\mathbf {Q} _ {\mathbf {A} _ {i}} (\overline {{\mathbf {B}}} ^ {T} \overline {{\boldsymbol {\Sigma}}} ^ {- 1} \overline {{\mathbf {B}}}) ^ {\dagger} \mathbf {Q} _ {\mathbf {A} _ {i}} ^ {T}\right) \\ = \operatorname {d i a g} \left(\frac {1}{| \Pi |} \sum_ {\pi \in \Pi} \mathbf {Q} _ {\mathbf {A} _ {i}} \left(\mathbf {W} _ {\pi} ^ {T} \mathbf {B} _ {o p t} ^ {T} \boldsymbol {\Sigma} ^ {- 1} \mathbf {B} _ {o p t} \mathbf {W} _ {\pi}\right) ^ {\dagger} \mathbf {Q} _ {\mathbf {A} _ {i}} ^ {T}\right) \\ = \frac {1}{| \Pi |} \sum_ {\pi \in \Pi} V a r (\mathbf {A} _ {i}; \mathcal {M} _ {\pi , 1}) \\ \end{array}
$$
For any  $\pi \in \Pi$ , Lemma D.2 tells us that  $\mathcal{L}(Var(\mathbf{A}_1; \mathcal{M}_{opt}), \ldots, Var(\mathbf{A}_k; \mathcal{M}_{opt})) = \mathcal{L}(Var(\mathbf{A}_1; \mathcal{M}_{\pi,1}), \ldots, Var(\mathbf{A}_k; \mathcal{M}_{\pi,1}))$  and so regularity of  $\mathcal{L}$  (which includes convexity), means that  $\mathcal{L}(Var(\mathbf{A}_1; \overline{\mathcal{M}}), \ldots, Var(\mathbf{A}_k; \overline{\mathcal{M}})) \leq \mathcal{L}(Var(\mathbf{A}_1; \mathcal{M}_{opt}), \ldots, Var(\mathbf{A}_k; \mathcal{M}_{opt})).$
Thus  $\overline{\mathcal{M}}$  is no worse in privacy or utility than  $\mathcal{M}_{opt}$  and hence is optimal.
Thus we consider the symmetries of the privacy cost matrix of  $\overline{\mathcal{M}}$ , which is given in Equation 4. Clearly it has the desired symmetry property that  $\Gamma = \mathbf{W}_{\pi}^{T}\Gamma \mathbf{W}_{\pi}$  for any  $\pi \in \Pi$  as the permutation space  $\Pi$  is an algebraic group.
D.3 From permutations to interpretations
Let  $\mathcal{M}_{opt}(\mathbf{x})\equiv \mathbf{B}_{opt}\mathbf{x} + N(\mathbf{0},\boldsymbol{\Sigma}_{opt})$  be an optimal mechanism that has the symmetries guaranteed by Lemma D.3. Our goal is to use the symmetries in the privacy cost matrix  $\Gamma_{opt}\equiv \mathbf{B}_{opt}^{T}\pmb{\Sigma}_{opt}^{-1}\mathbf{B}_{opt}$  to examine the structure of  $\Gamma_{opt}$ .
If  $\gamma_{i,j}$  is the  $(i,j)^{\mathrm{th}}$  entry of  $\Gamma_{opt}$  and if there is a renaming permutation that maps  $r_i$  (the record associated with index  $i$ ) to some  $r_{i'}$  (at index  $i'$ ) and maps  $r_j$  to some  $r_{j'}$  then  $\gamma_{i,j} = \gamma_{i',j'}$ . Note that if  $r_i$  and  $r_j$  have the same values for attributes  $Att_1$  and  $Att_2$  then  $r_{i'}$  and  $r_{j'}$  must match on the same attributes because renaming permutations just change the names of values within each attribute. Thus we introduce notation for the set of attributes on which two records match:
DEFINITION D.4 (Common Attributes). Define  $\zeta$  to be the function that takes two records and outputs the set of attributes on which they match. We emphasize that  $\zeta(r_i, r_j)$  is a set of attributes, not attribute values.
This discussion leads to the following result which characterizes the privacy cost matrix of an optimal mechanism.
LEMMA D.5. Under the same conditions as Lemma D.3, there exists an optimal mechanism with a privacy cost matrix  $\Gamma_{opt}$  for which the following holds. In addition to the symmetry guaranteed by Lemma D.3, for every subset of attributes  $S \subseteq \{Att_1, \ldots, Att_{n_a}\}$ , there exists a number  $c_S$  such that  $\gamma_{i,j}$ , the  $(i,j)^{th}$  entry of  $\Gamma_{opt}$ , is equal to  $c_{\zeta(r_i, r_j)}$ . In other words, the  $(i,j)^{th}$  entry is completely determined by the set  $\zeta(r_i, r_j)$  (recall  $r_i$  the record value associated with index  $i$  and  $r_j$  the record value associated with index  $j$ ).
Proof. By Lemma D.3, there exists an optimal mechanism with privacy cost matrix  $\Gamma_{opt}$  that is invariant under renaming permutations of its rows as long as the columns are permuted in the same way. Thus if  $r_i$  is the record value corresponding to position  $i$  and  $r_j$  is the record value corresponding to position  $j$ , there exists a renaming permutation that maps  $r_i$  to some  $r_{i'}$  and  $r_j$  to some  $r_j'$  if and only if the attributes on which  $r_i$  and  $r_j$  match are the same as the attributes on which  $r_{i'}$  and  $r_{j'}$  match each other (in symbols:  $\zeta(r_i, r_j) = \zeta(r_{i'}, r_{j'})$ ). When there exists such a renaming permutation then  $\gamma_{i,j} = \gamma_{i',j'}$ . Thus the value of  $\gamma_{i,j}$  is completely determined by  $\zeta(r_i, r_j)$  and the result follows.
From Theorem 4.2, we know that the rows of the matrices of  $\mathbf{R}_{\mathbf{A}}$ , for all  $\mathbf{A} \subseteq \{Att_1, \ldots, Att_{n_a}\}$  are a linearly independent basis for  $\mathbb{R}^d$ , where  $d = \prod_{i=1}^{n_a} |Att_i|$ . Thus we call the rows a residual basis. DEFINITION D.6. A row vector  $\mathbf{v}$  is a residual basis vector if it is a row in  $\mathbf{R}_{\mathbf{A}}$  for some  $\mathbf{A} \subseteq \{Att_1, \ldots, Att_{n_a}\}$ .
We now provide an interpretation of the residual bases. First, for an attribute  $Att_{\ell}$ , define the vector  $e_{i,j}^{(\ell)}$  to be a vector of length  $|Att_{\ell}|$  such that the element at position  $i$  is 1, the element at position  $j$  is -1 and everywhere else is 0. In other words,  $e_{i,j}^{(\ell)} = e_i^{(\ell)} - e_j^{(\ell)}$  (recall  $e_i^{(\ell)}$  is 1 in position  $i$  and 0 everywhere else and is a one-hot encoding of the attribute  $a_i^{(\ell)}$ ). Now, each element of the residual basis has the form  $\mathbf{v}^{(1)} \otimes \dots \otimes \mathbf{v}^{(n_a)}$  where, for each  $\ell$ ,  $\mathbf{v}^{(\ell)}$  is either the vector  $\mathbf{1}_{|Att_{\ell}|}^T$  or a vector  $e_{1,i_\ell}^{(\ell)}$ . When the vector for attribute  $Att_{\ell}$  is the vector  $\mathbf{1}_{|Att_{\ell}|}^T$ , we say that all attribute values of  $Att_{\ell}$  are selected. When the vector for  $Att_{\ell}$  is  $e_{1,i_\ell}^{(\ell)}$ , then we say attribute value  $a_1^{(\ell)}$  is positively selected and  $a_{i_\ell}^{(\ell)}$  is negatively selected (the other attribute values of  $Att_{\ell}$  are not selected at all). The attributes for which the kron term is not  $\mathbf{1}_{|Att_{\ell}|}^T$  are called the discriminative attributes.
As an example of this notation and terminology, consider Table 8. Suppose we have three attributes:  $Att_1$  takes values 'a' or 'b';  $Att_2$  takes values 'y' or 'n';  $Att_3$  takes values 1 or 2 or 3.
bn1: [0, 1]⊗[0, 1]⊗[1, 0, 0]
[1, 1]⊗[1, -1]⊗[1, -1, 0]
[1, -1]⊗[1, 1]⊗[1, 0, -1]
ay1	ay2	ay3	an1	an2	an3	by1	by2	by3	bn1	bn2	bn3
0	0	0	0	0	0	0	0	0	1	0	0
1	-1	0	-1	1	0	1	-1	0	-1	1	0
1	0	-1	1	0	-1	-1	0	1	-1	0	1
Table 8: Kron product representations.
In this case, the data vector  $\mathbf{x}$  would have 12 components. The first component corresponds to the number of appearances of record "a,y,1" in the dataset, the second component corresponds to record "a,y,2" and so on. The records corresponding to each index of  $\mathbf{x}$  are listed in order as the column headings in Table 8. The first row shows the representation of record "b,n,1" which is composed of the second value (b) for  $Att_{1}$ , the second value (n) for  $Att_{2}$  and the first value (1) for  $Att_{3}$ . Hence its kron representation is  $[0,1] \otimes [0,1] \otimes [1,0,0]$  and when the kron product is evaluated, the resulting vector has a 1 in the index corresponding to "bn1" (10th column) and 0 everywhere else.
The second and third rows show the expansions of two residual basis vectors  $[1,1]\otimes [1, - 1]\otimes [1, - 1,0]$  (its discriminative attributes are  $Att_{2}$  and  $Att_{3}$ ) and  $[1, - 1]\otimes [1,1]\otimes [1,0, - 1]$  (its discriminative attributes are  $Att_{1}$  and  $Att_{3}$ ). Consider again the kron product  $[1,1]\otimes [1, - 1]\otimes [1, - 1,0]$ . Note that the first part of the kron product,  $[1,1]$  refers to the first attribute and selects both of its values (sets them to 1). The second part of the kron product  $[1, - 1]$  refers to the  $Att_{2}$  and positively selects the first attribute value 'y' (sets it to 1) and negatively selected the second attribute value 'n' (sets it to -1). The third part is  $[1, - 1,0]$  and it positively selects the first attribute value, negatively selects the second, but the third attribute value is not selected at all (i.e., the 3rd position is 0). These attribute selections can help us determine what the kron product looks like when it is expanded as follows. For the residual basis vector  $\mathbf{v}^{(1)}\otimes \dots \otimes \mathbf{v}^{(n_a)}$  the value at the index associated with a record  $r$  is
- 0 if  $r$  has an attribute whose value is not selected by the residual basis vector's kron product. In this case we say the residual basis vector assigns a 0 to record  $r$ . For example, in the residual basis vector corresponding to kron product  $[1,1] \otimes [1, -1] \otimes [1, -1,0]$ , the third value of the third attribute is not selected. For any record that assigns the attribute value 3 to  $Att_3$ , this residual basis vector assigns a 0 to such a record.
- 1 if for every attribute, the value assigned to it by  $r$  is selected (positively or negatively), and the number of negatively selected attribute values is even. In this case we say the residual basis vector assigns a 1 to record  $r$ .
- -1 if the attribute value for each attribute is selected, and the number of negatively selected attribute values is odd. In this case we say the residual basis vector assigns a  $-1$  to record  $r$ .
For example, for the residual basis vector  $[1, 1] \otimes [1, -1] \otimes [1, -1, 0]$ , the attribute value 3 for  $Att_3$  is not selected. Hence the value at indices corresponding to records an3,bn3,ay3,by3 are all 0 (see Table 8). Next, consider the record an2. The value "a" is positively selected, "n" is negatively selected, and "2" is negatively selected. Hence all attributes are selected and an even number of attributes are negatively selected. Therefore the value at the index associated with an2 is 1. Now for the record by2. The "b" is positively selected, "y" is positively selected, and "2" is negatively selected. Hence there are an odd number of negative selections and so the value at the index associated with by2 is -1.
With this discussion and associated notation, we can now show that each residual basis vector is an eigenvector of the optimal privacy cost matrix, and the eigenvalue only depends on which attributes are discriminative.
THEOREM D.7. Under the same conditions as Lemma D.3, there exists an optimal mechanism such that the eigenvectors of its privacy cost matrix  $\Gamma$  are the residual basis vectors (Definition D.6). Furthermore, if two residual basis vectors  $\mathbf{v}^{(1)}\otimes \dots \otimes \mathbf{v}^{(n_a)}$  and  $\mathbf{w}^{(1)}\otimes \dots \otimes \mathbf{w}^{(n_a)}$  have the same discriminative attributes (i.e., for all  $i$ ,  $\mathbf{w}^{(i)}\neq \mathbf{1}_{|Att_i|}^T$  if and only  $\mathbf{v}^{(i)}\neq \mathbf{1}_{|Att_i|}^T$ ) then the two residual basis vectors have the same eigenvalues (in other words, all rows of the same residual matrix have the same eigenvalues).
Proof. Recall from Definition D.4 that  $\zeta(r_i, r_j)$  is the set of attributes on which  $r_i$  and  $r_j$  are equal.
Let  $\Gamma$  be the privacy cost matrix guaranteed by Lemma D.5 with the properties guaranteed by Lemma D.5, namely that for every subset of attributes  $S \subseteq \{Att_1, \ldots, Att_{n_a}\}$ , there exists a number  $c_S$  such that  $\gamma_{i,j}$ , the  $(i,j)^{\mathrm{th}}$  entry of  $\Gamma$ , is equal to  $c_{\zeta(r_i, r_j)}$  - the constant associated with the set  $\zeta(r_i, r_j)$  where  $r_i$  the record value associated with index  $i$  and  $r_j$  is the record value associated with index  $j$ .
Let  $r_{\ell}$  be a record associated with index  $\ell$ . We consider the dot product between a residual basis vector  $\mathbf{v} = \mathbf{v}^{(1)} \otimes \dots \otimes \mathbf{v}^{(n_a)}$  and the  $\ell^{\mathrm{th}}$  row of  $\Gamma$ . Since the entries of the  $\ell^{th}$  row are  $c_{\zeta(r_\ell, r_1)}, \ldots, c_{\zeta(r_\ell, r_d)}$  and the entries of  $\mathbf{v}$  are  $0, 1, -1$ , this dot product can be expressed as:
$$
\sum_ {\substack {r \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (r _ {\ell}, r)} - \sum_ {\substack {r \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (r _ {\ell}, r)} \tag{5}
$$
We analyze this in three cases.
Case 1:  $\mathbf{v}$  assigns a 0 to  $r_\ell$ . In this case, there is an attribute for which  $r_\ell$  has a value that is not selected. Without loss of generality, we may assume this is the first attribute  $Att_1$  so that  $\mathbf{v}^{(1)} = e_{1,i}$  (the vector with a 1 at the first index and -1 at the  $i^{\text{th}}$  index for some  $i$  and 0 everywhere else) and the value of  $Att_1$  for  $r_\ell$  is therefore not  $a_1^{(1)}$  or  $a_i^{(1)}$  (because  $r_\ell$  got assigned 0 by  $\mathbf{v}$  due to attribute  $Att_1$ ). Now, if a record  $r$  appears in the left summation of Equation 5 then its value for  $Att_1$  is either  $a_1^{(1)}$  or  $a_i^{(1)}$  and it does not match  $r_\ell$  on the first attribute. But this means that we can transform  $r$  into a record  $r'$  by replacing  $a_1^{(1)}$  and  $a_i^{(1)}$  with each other. This  $r'$  would be on the right hand side of the summation (because we are flipping the sign of the selection by  $\mathbf{v}$  of attribute  $Att_1$  in  $r'$ ). Furthermore  $r'$  also does not match  $r_\ell$  on  $Att_1$  and therefore  $r$  matches  $r_\ell$  on exactly the same attributes as  $r'$  matches  $r_\ell$ . Thus  $\zeta(r_\ell, r) = \zeta(r_\ell, r')$ . Thus the summation term from record  $r$  is cancelled out by  $r'$  in Equation 5. Using the same argument, we see that every term in the left summation is canceled out by a unique term in the right summation, and vice versa. Hence, if  $\mathbf{v}$  assigns a 0 to record  $r_\ell$  (i.e., has a 0 in index  $\ell$  when its kron product representation is expanded) then the dot product between  $\mathbf{v}$  and the  $\ell^{\text{th}}$  row of  $\Gamma$  is 0.
Case 2:  $\mathbf{v}$  assigns a 1 to  $r_{\ell}$ . In this case, every attribute of  $r_{\ell}$  has a value that is (either positively or negatively) selected by  $\mathbf{v}$  and an even number are negatively selected. Our goal is to show that if
some other record  $r_t$  is also assigned a 1 by  $\mathbf{v}$ , then the dot product between  $\mathbf{v}$  and  $\ell^{\mathrm{th}}$  row of  $\Gamma$  is the same as the dot product between  $\mathbf{v}$  and the  $t^{\mathrm{th}}$  row of  $\Gamma$ . That is, we want to show:
$$
\sum_ {\substack {r \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (r _ {\ell}, r)} - \sum_ {\substack {r \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (r _ {\ell}, r)} = \sum_ {\substack {r \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (r _ {t}, r)} - \sum_ {\substack {r \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (r _ {t}, r)} \tag{6}
$$
Let  $S$  be the set of attributes on which  $r_{\ell}$  and  $r_t$  disagree. Now define a mapping  $\phi$  between records such that  $\phi$  only modifies attributes in  $S$ . For each attribute  $Att$  in  $S$ , it maps the value that record  $r_{\ell}$  has into the value that  $r_t$  has an vice versa. (For example, suppose  $S = \{Att_1, Att_2\}$  and  $r_{\ell}$  has values  $a_2^{(1)}$  and  $a_3^{(2)}$  for those attributes, respectively, and suppose that  $r_t$  has values  $a_4^{(1)}$  and  $a_5^{(2)}$  for those attributes. Then  $\phi$  changes  $a_2^{(1)}$  in  $Att_1$  to  $a_4^{(1)}$  and changes  $a_4^{(1)}$  into  $a_2^{(1)}$ ; for  $Att_2$  it changes  $a_3^{(2)}$  into  $a_5^{(2)}$  and changes  $a_5^{(2)}$  into  $a_3^{(2)}$ . Thus  $\phi(r_{\ell}) = r_t$  and  $\phi(r_t) = r_{\ell}$  and  $\phi$  is its own inverse. Furthermore, for any record  $r$ ,  $\zeta(r_{\ell}, r) = \zeta(\phi(r_{\ell}), \phi(r)) = \zeta(r_t, \phi(r))$  since renaming attribute values the same way in two records does not affect the set of attributes on which they match (and the last equality is because  $\phi(r_{\ell}) = r_t$ ).
We next note that since  $r_t$  and  $r_\ell$  are both assigned 1 by  $\mathbf{v}$ , then they must differ on an even number of discriminative attributes of  $\mathbf{v}$  (if they differ on a discriminative attribute, one must have a value that is positively selected and the other must have a value that is negatively selected – there cannot be a 0 because  $r_\ell$  and  $r_t$  are not assigned a 0 by  $\mathbf{v}$ ). Therefore, due to its definition,  $\phi$  modifies an even number of discriminative attributes and therefore for any record  $r$ , both  $r$  and  $\phi(r)$  get assigned the same value by  $\mathbf{v}$ .
Putting these facts together, we get:
$$
\begin{array}{l} \sum_{\substack{r\text{assigned}\\ \text{value I by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} - \sum_{\substack{r\text{assigned}\\ \text{value - I by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} \\ = \sum_{\substack{\phi (r)\text{assigned}\\ \text{value 1 by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} - \sum_{\substack{\phi (r)\text{assigned}\\ \text{value -1 by}\mathbf{v}}}c_{\zeta (r_{\ell},r)}\quad \text{since}\phi \text{doesn't change the summation set} \\ = \sum_ {\substack {\phi (r) \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (\phi (r _ {\ell}), \phi (r))} - \sum_ {\substack {\phi (r) \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (\phi (r _ {\ell}), \phi (r))} \quad \text {since $\phi$ preserves the outcome of $\zeta$} \\ = \sum_{\substack{\phi (r)\text{assigned}\\ \text{value 1 by}\mathbf{v}}}c_{\zeta (r_{t},\phi (r))} - \sum_{\substack{\phi (r)\text{assigned}\\ \text{value -1 by}\mathbf{v}}}c_{\zeta (r_{t},\phi (r))}\quad \text{since}\phi (r_{\ell}) = r_{t} \\ = \sum_{\substack{r^{\prime}\text{assigned}\\ \text{value} 1\text{by}\mathbf{v}}}c_{\zeta (r_{t},r^{\prime})} - \sum_{\substack{r^{\prime}\text{assigned}\\ \text{value} -1\text{by}\mathbf{v}}}c_{\zeta (r_{t},r^{\prime})}\quad \text{renaming the summation variable from}\phi (r)\text{to} r^{\prime} \\ \end{array}
$$
and that proves Equation 6
Case 3:  $\mathbf{v}$  assigns a  $-1$  to  $r_{\ell}$ . In this case, every attribute of  $r_{\ell}$  has a value that is (either positively or negatively) selected by  $\mathbf{v}$  and an odd number are negatively selected. Our goal is to show that if some other record  $r_t$  is assigned a 1 by  $\mathbf{v}$ , then the dot product between  $\mathbf{v}$  and  $\ell^{\mathrm{th}}$  row of  $\Gamma$  is the negative of the dot product between  $\mathbf{v}$  and the  $t^{\mathrm{th}}$  row of  $\Gamma$ . That is, we want to show:
$$
\sum_ {\substack {r \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (r _ {\ell}, r)} - \sum_ {\substack {r \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (r _ {\ell}, r)} = - \sum_ {\substack {r \text {assigned} \\ \text {value 1 by v}}} c _ {\zeta (r _ {t}, r)} + \sum_ {\substack {r \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (r _ {t}, r)} \tag{7}
$$
As in the previous case, we define  $\phi$  in the same way and reasoning as before we see that for any record  $r$ ,  $\zeta(r_{\ell}, r) = \zeta(\phi(r_{\ell}), \phi(r)) = \zeta(r_{t}, \phi(r))$  and since now  $\phi$  must change an odd number of
discriminative attributes (since  $r_{\ell}$  and  $r_t$  are assigned -1 and 1 by  $\mathbf{v}$ ) then for any record  $r$ , the value assigned to  $r$  by  $\mathbf{v}$  is the negative of the value assigned to  $\phi(r)$  by  $\mathbf{v}$ . Thus we have:
$$
\begin{array}{l} \sum_{\substack{r\text{assigned}\\ \text{value} 1\text{by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} - \sum_{\substack{r\text{assigned}\\ \text{value} -1\text{by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} \\ = \sum_{\substack{\phi (r)\text{assigned}\\ \text{value - 1 by}\mathbf{v}}}c_{\zeta (r_{\ell},r)} - \sum_{\substack{\phi (r)\text{assigned}\\ \text{value + 1 by}\mathbf{v}}}c_{\zeta (r_{\ell},r)}\quad \text{since}\phi \text{flips the summation sets} \\ = \sum_ {\substack {\phi (r) \text {assigned} \\ \text {value - 1 by v}}} c _ {\zeta (\phi (r _ {\ell}), \phi (r))} - \sum_ {\substack {\phi (r) \text {assigned} \\ \text {value + 1 by v}}} c _ {\zeta (\phi (r _ {\ell}), \phi (r))} \quad \text {since} \phi \text {preserves the outcome of} \zeta \\ = \sum_{\substack{\phi (r)\text{assigned}\\ \text{value - 1 by}\mathbf{v}}}c_{\zeta (r_{t},\phi (r))} - \sum_{\substack{\phi (r)\text{assigned}\\ \text{value + 1 by}\mathbf{v}}}c_{\zeta (r_{t},\phi (r))}\quad \text{since}\phi (r_{\ell}) = r_{t} \\ = \sum_{\substack{r^{\prime}\text{assigned}\\ \text{value - 1 by}\mathbf{v}}}c_{\zeta (r_{t},r^{\prime})} - \sum_{\substack{r\text{assigned}\\ \text{value + 1 by}\mathbf{v}}}c_{\zeta (r_{t},r^{\prime})}\quad \text{renaming the summation variable from}\phi (r^{\prime})\text{to} r^{\prime} \\ \end{array}
$$
and that proves Equation 7.
Thus what these 3 cases show us are that there exists some constant  $\beta$  such that:
- If the  $i^{\text{th}}$  position of the expansion of  $\mathbf{v}$  is 0 (i.e.,  $r_i$  is assigned 0 by  $\mathbf{v}$ ), then the  $i^{\text{th}}$  position of  $\Gamma \mathbf{v}$  is also 0 (the dot product between the  $i^{\text{th}}$  row and  $\mathbf{v}$  is 0).
- If the  $i^{\text{th}}$  position of the expansion of  $\mathbf{v}$  is 1 (i.e.,  $r_i$  is assigned 1 by  $\mathbf{v}$ ), then the  $i^{\text{th}}$  position of  $\Gamma \mathbf{v}$  is  $\beta$  (the dot product between the  $i^{\text{th}}$  row and  $\mathbf{v}$  is  $\beta$ ).
- If the  $i^{\text{th}}$  position of the expansion of  $\mathbf{v}$  is -1 (i.e.,  $r_i$  is assigned -1 by  $\mathbf{v}$ ), then the  $i^{\text{th}}$  position of  $\Gamma \mathbf{v}$  is  $-\beta$  (the dot product between the  $i^{\text{th}}$  row and  $\mathbf{v}$  is  $-\beta$ ).
Thus  $\mathbf{v}$  is an eigenvector of  $\Gamma$  with eigenvalue  $\beta$ . That proves the first part of the theorem.
The next part of the theorem is to show that if two residual basis vectors have the same discriminative attributes, then they have the same eigenvalue. So let  $\mathbf{v} = \mathbf{v}^{(1)}\otimes \dots \otimes \mathbf{v}^{(n_a)}$  and  $\mathbf{w} = \mathbf{w}^{(1)}\otimes \dots \otimes \mathbf{w}^{(n_a)}$  be two residual basis vectors that have the same discriminative attributes. Define a renaming permutation  $\pi$  as follows:
- For an attribute  $Att_{\ell}$  that is not discriminative for  $\mathbf{v}$  (and hence also not for  $\mathbf{w}$ ),  $\pi$  does not rename its values (i.e., it acts as the identity for those attribute values).