SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Major notations used
Notation	Description
l	number of layers of a Transformer model
n	number of tokens of an input sequence
d	model embedding dimension
np	number of VIP-tokens
nc	number of non-VIP/remaining tokens, so np + nc = n
r	length of a compressed sequence
α(.,.,.)	multi-head attention taking three inputs for query/key/value embeddings
β(·)	two-layer feed-forward network
γ(·)	function representing all heavy computation of a Transformer layer
X	embedding matrix representing a input sequence
P	embedding matrix representing the VIP-tokens
C	embedding matrix representing the non-VIP/remaining tokens
Xnew	updated embedding matrix of a input sequence, the output of a Transformer layer
Pnew	updated embedding matrix representing the VIP-tokens
Cnew	updated embedding matrix representing the non-VIP/remaining tokens
S	compression matrix
Sc	compression submatrix for the non-VIP/remaining tokens
S†	decompression matrix
Sc†	decompression submatrix for the non-VIP/remaining tokens
s	resolution of approximation, represents the number of non-VIP token embeddings being averaged
x	location of s-length segment with the original sequence
k	increment ratio of resolution, s ∈ {1, k, k2, k3, ···, nc}
h	number of k-length segments are split into 1-length sub-segments, used for experiments
hs	number of s-length segments are split into shorter sub-segments, used in Appendix discussion
J	set of components bx that is constructed via Alg. 1, the elements are the rows of Sc
bx	components used for 1-D wavelet transform of scaling s and translation x
cx	local average of x-th s-length segment of sequence C
(cnew)s	local average of x-th s-length segment of sequence C new
T(·)	data structure for storing the input sequence C or C new
Δcs	state stored in T(C) defined as Δcs := c2s[x/2] - cs
Δ(cnew)s	state stored in T(C new) defined as Δ(cnew)s := (cnew)2s[x/2] - (cnew)s
[·]i	i-th entry/row of the input vector/matrix
[·]i,j	(i, j)-th entry of the input matrix
We provide a table 5 of notations that are used for more than once so that the readers can refer to their definition easily.
7.2 Details of Multi-Resolution Compression
We describe the omitted technical details of a modified formulation of [39] to construct  $\mathbf{S}_c\mathbf{C}$  and corresponding  $\mathbf{S}_c$  satisfying good approximation of
$$
\exp \left(\mathbf {P} \mathbf {C} ^ {\top} \mathbf {S} _ {c} ^ {\top}\right) \mathbf {S} _ {c} \approx \exp \left(\mathbf {P} \mathbf {C} ^ {\top}\right). \tag {19}
$$
Before diving into the technical details of constructing  $\mathbf{S}_c$ , we introduce some notations and tools that will be used later. We use  $[\cdot]_i$  to refer the  $i$ -th entry/row of the input vector/matrix and  $[\cdot]_{i,j}$  to refer the  $(i,j)$ -th entry of the input matrix. BOLD uppercase letters denote matrices, bold lower case letters denote vectors, and regular lower case letters denote scalars. Let  $\mathbf{v}_i$  be a vector, when we write a matrix of form
$$
\left[ \begin{array}{l l l l} \mathbf {v} _ {1} & \mathbf {v} _ {2} & \dots & \mathbf {v} _ {m} \end{array} \right], \tag {20}
$$
we treat  $\mathbf{v}_i$  as a column of the matrix. When we write a matrix of form
$$
\left[ \begin{array}{l} \mathbf {v} _ {1} \\ \mathbf {v} _ {2} \\ \dots \\ \mathbf {v} _ {m} \end{array} \right], \tag {21}
$$
we treat  $\mathbf{v}_i$  as a row of the matrix.
7.2.1 Basic Problem Setup
Let  $\mathbf{b}_x^s\in \mathbb{R}^{n_c}$  be a multi-resolution component defined as
$$
\left[ \mathbf {b} _ {x} ^ {s} \right] _ {i} := \left\{ \begin{array}{l l} \frac {1}{s} & \text {i f} s x - s <   i \leq s x \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {22}
$$
for  $s \in \{k^0, k^1, k^2, \dots, n_c\}$  assuming  $n_c$  is a power of  $k$  (we assume  $k = 2$  for simplicity). Here,  $s$  and  $x$  represent the scaling and translation (similar to the concepts in wavelet basis) of the component, respectively. Fig. 6 is the visualization of  $\mathbf{b}_x^s$ .
Then, any 1D signal  $\mathbf{f} \in \mathbb{R}^{n_c}$  can be represented as a linear combination of  $\mathbf{b}_x^s$ :
Figure 6: Visualization of  $\mathbf{b}_x^s$  for some scaling  $s$  and translation  $x$ . The y axis for different plots are not the same.
$$
\mathbf {f} = \sum_ {s, x} c _ {x} ^ {s} \mathbf {b} _ {x} ^ {s} \tag {23}
$$
where  $c_{x}^{s}$  are the coefficients for the linear combination. For a signal with multi-resolution structure (that is, signal has high frequency in some regions and has low frequency in other regions), we can find an approximation  $\hat{f}^*$  that can be expressed as a sparse linear combination where most coefficients are zeros, as shown in Fig. 7.
$$
\mathbf {f} \approx \hat {\mathbf {f}} ^ {*} := \sum_ {\mathbf {b} _ {x} ^ {s} \in \mathcal {J}} c _ {x} ^ {s} \mathbf {b} _ {x} ^ {s} \tag {24}
$$
We denote  $\mathcal{J}$  as the set of major components  $\mathbf{b}_x^s$  corresponding to the large coefficients, that is,  $\mathcal{J} := \{\mathbf{b}_x^s \mid |c_x^s|$  being large\}. Since the set of all possible  $\mathbf{b}_x^s$  is an over-complete dictionary, there are multiple possible linear combinations. To reduce the search space of the best set  $\mathcal{J}$ , we place a mild restriction on the set  $\mathcal{J}$ :
$$
\left[ \sum_ {\mathbf {b} _ {x} ^ {s} \in \mathcal {J}} \mathbf {b} _ {x} ^ {s} \right] _ {i} \neq 0 \quad \forall i \quad \langle \mathbf {b} _ {x} ^ {s}, \mathbf {b} _ {x ^ {\prime}} ^ {s ^ {\prime}} \rangle = 0 \quad \forall \mathbf {b} _ {x} ^ {s}, \mathbf {b} _ {x ^ {\prime}} ^ {s ^ {\prime}} \in \mathcal {J}, \mathbf {b} _ {x} ^ {s} \neq \mathbf {b} _ {x ^ {\prime}} ^ {s ^ {\prime}} \tag {25}
$$
The conditions state that each entry of signal  $\mathbf{f}$  is included in the support region of exactly one component in  $\mathcal{I}$ . With these tools, we will first describe the approximation when  $\mathcal{I}$  is given, then discuss how the approximation connects the set  $\mathcal{I}$  to our target  $\mathbf{S}_c$  and  $\mathbf{S}_c\mathbf{C}$ . Finally, we will discuss how to construct this  $\mathcal{I}$ .
Figure 7: An example of approximating an 1D signal using a truncated wavelet transform with components defined in (22). It uses a set  $\mathcal{J}$  of size 79 to represent a signal in  $\mathbb{R}^{512}$ .
7.2.2 Plugging Our Problem into the Setup
A recent result shows that the self-attention matrix  $\exp (\mathbf{X}\mathbf{X}^{\top})$  has the multi-resolution structure discussed above [39]. Since  $\exp (\mathbf{PC}^{\top})$  is a sub-matrix of  $\exp (\mathbf{X}\mathbf{X}^{\top})$ , we conjecture that the multi-resolution structure also holds in  $\exp (\mathbf{PC}^{\top})$ . As a result, we can find a sparse combination of  $\mathbf{b}_x^s$  to represent rows of  $\exp (\mathbf{PC}^{\top})$ .
Claim 7.1. Given the set  $\mathcal{I}$  satisfying restriction (25), we can define an approximation of the  $i$ -th row of  $\exp(\mathbf{PC}^\top)$  similar to (24) as illustrated in Fig. 7
$$
\left[ \widehat {\operatorname {e x p} \left(\mathbf {P C} ^ {\top}\right) ^ {*}} \right] _ {i} := \sum_ {\mathbf {b} _ {x} ^ {s} \in \mathcal {J}} c _ {x} ^ {s} \mathbf {b} _ {x} ^ {s} \tag {26}
$$
where  $c_x^s$  is the optimal solution that minimizes
$$
\left\| \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i} - \left[ \widehat {\exp \left(\mathbf {P C} ^ {\top}\right)} ^ {*} \right] _ {i} \right\| _ {2} ^ {2} \tag {27}
$$
Then, the approximation can be written as:
$$
\left[ \widehat {\operatorname {e x p} \left(\mathbf {P C} ^ {\top}\right) ^ {*}} \right] _ {i, j} = \langle \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i}, \mathbf {b} _ {x} ^ {s} \rangle \tag {28}
$$
where  $\mathbf{b}_x^s\in \mathcal{I}$  is the component that is supported on  $j$  (a.k.a.  $[\mathbf{b}_x^s ]_j\neq 0$  and there is exactly one  $\mathbf{b}_x^s\in \mathcal{I}$  satisfy this condition due to restriction (25)).
Proof. If  $\mathcal{J}$  is given, let  $\mathbf{B} \in \mathbb{R}^{n_c \times |\mathcal{I}|}$  be a matrix whose columns are elements  $\mathbf{b}_x^s \in \mathcal{J}$  and let  $\mathbf{c} \in \mathbb{R}^{|\mathcal{I}|}$  be a vector whose entries are the corresponding  $c_x^s$ :
$$
\mathbf {B} := \left[ \begin{array}{l l l l} \mathbf {b} _ {x _ {1}} ^ {s _ {1}} & \mathbf {b} _ {x _ {2}} ^ {s _ {2}} & \dots & \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] \tag {29}
$$
$$
\mathbf {c} := \left[ \begin{array}{l l l l} c _ {x _ {1}} ^ {s _ {1}} & c _ {x _ {2}} ^ {s _ {2}} & \dots & c _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] ^ {\prime} \tag {29}
$$
then the approximation can be expressed as
$$
\left[ \widehat {\operatorname {e x p} \left(\mathbf {P C} ^ {\top}\right)} ^ {*} \right] _ {i} = \sum_ {\mathbf {b} _ {x} ^ {s} \in \mathcal {J}} c _ {x} ^ {s} \mathbf {b} _ {x} ^ {s} = \mathbf {B} \mathbf {c} \tag {30}
$$
If we solve for
$$
\mathbf {c} := \arg \min  _ {\beta} \| \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i} - \mathbf {B} \beta \| \tag {31}
$$
$$
\mathbf {c} = \left(\mathbf {B} ^ {\top} \mathbf {B}\right) ^ {- 1} \mathbf {B} ^ {\top} \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i} \tag {32}
$$
Due to the restriction (25), the columns of  $\mathbf{B}$  are orthogonal, so  $\mathbf{B}^{\top}\mathbf{B}$  is a diagonal matrix:
$$
\mathbf {B} ^ {\top} \mathbf {B} = \left[ \begin{array}{c c c c} 1 / s _ {1} & & & \\ & 1 / s _ {2} & & \\ & & \dots & \\ & & & 1 / s _ {| \mathcal {I} |} \end{array} \right] \tag {33}
$$
We can also write down  $\mathbf{B}^{\top}\left[\exp (\mathbf{PC}^{\top})\right]_{i}$
$$
\mathbf {B} ^ {\top} \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i} = \left[ \begin{array}{c} \left\langle \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i}, \mathbf {b} _ {x _ {1}} ^ {s _ {1}} \right\rangle \\ \left\langle \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i}, \mathbf {b} _ {x _ {2}} ^ {s _ {2}} \right\rangle \\ \dots \\ \left\langle \left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i}, \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \right\rangle \end{array} \right] \tag {34}
$$
$$
\mathbf {B} \mathbf {c} = \left[ \begin{array}{l l l l} s _ {1} \mathbf {b} _ {x _ {1}} ^ {s _ {1}} & s _ {2} \mathbf {b} _ {x _ {2}} ^ {s _ {2}} & \dots & s _ {| \mathcal {I} |} \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] \left[ \begin{array}{c} \langle [ \exp (\mathbf {P C} ^ {\top}) ] _ {i}, \mathbf {b} _ {x _ {1}} ^ {s _ {1}} \rangle \\ \langle [ \exp (\mathbf {P C} ^ {\top}) ] _ {i}, \mathbf {b} _ {x _ {2}} ^ {s _ {2}} \rangle \\ \dots \\ \langle [ \exp (\mathbf {P C} ^ {\top}) ] _ {i}, \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \rangle \end{array} \right] \tag {35}
$$
We note that  $s\mathbf{b}_x^s$  simply re-scale the entry of  $\mathbf{b}_x^s$  such that any non-zero entry becomes 1. Then, let us consider  $j$ -th entry of  $\mathbf{Bc}$ . Due to the restriction (25), we have exactly one  $\mathbf{b}_x^s \in \mathcal{I}$  whose support region contains  $j$ , so the  $j$ -th row of the first matrix at the right hand side of (35) contains exactly a 1 and the remaining entries are 0. Therefore, we have
$$
\left[ \right. \widehat {\left. \mathbf {e x p} \left(\mathbf {P C} ^ {\top}\right) ^ {*} \right]} _ {i, j} = \left[ \mathbf {B c} \right] _ {j} = \langle \left[ \exp \left(\mathbf {P C} ^ {\top}\right)\right] _ {i}, \mathbf {b} _ {x} ^ {s} \rangle \tag {36}
$$
where  $\mathbf{b}_x^s\in \mathcal{I}$  is the component that is supported on  $j$  , which concludes our proof.
7.2.3 Efficient Approximation
We note that computing (28) for all  $j$  would require access to the entire  $[\exp (\mathbf{PC}^{\top})]_i$ . We exploit the same strategy as described in [39], so the exponential of inner product is used as an approximation to inner product of exponential.
$$
\left[ \widehat {\operatorname {e x p} \left(\mathbf {P C} ^ {\top}\right)} \right] _ {i, j} := \exp \left(\left\langle \left[ \mathbf {P C} ^ {\top} \right] _ {i}, \mathbf {b} _ {x} ^ {s} \right\rangle\right) \tag {37}
$$
We note that  $\langle [\mathbf{PC}^{\top}]_i, \mathbf{b}_x^s \rangle$  is the local average of the support region of  $\mathbf{b}_x^s$ , which is also the  $x$ -th  $s$ -length segment of sequence  $[\mathbf{PC}^{\top}]_i$ :
$$
\langle [ \mathbf {P C} ^ {\top} ] _ {i}, \mathbf {b} _ {x} ^ {s} \rangle = \frac {1}{s} \sum_ {[ \mathbf {b} _ {x} ^ {s} ] _ {j} \neq 0} \left[ \mathbf {P C} ^ {\top} \right] _ {i, j} \tag {38}
$$
By using some arithmetic manipulations, (37) can be efficiently computed
$$
\begin{array}{l} \exp (\langle [ \mathbf {P C} ^ {\top} ] _ {i}, \mathbf {b} _ {x} ^ {s} \rangle) = \exp (\frac {1}{s} \sum_ {[ \mathbf {b} _ {x} ^ {s} ] _ {j} \neq 0} [ \mathbf {P C} ^ {\top} ] _ {i, j}) = \exp (\frac {1}{s} \sum_ {(\mathbf {b} _ {x} ^ {s}) _ {j} \neq 0} \langle [ \mathbf {P} ] _ {i}, [ \mathbf {C} ] _ {j} \rangle) \\ = \exp \left(\left\langle [ \mathbf {P} ] _ {i}, \frac {1}{s} \sum_ {\left[ \mathbf {b} _ {x} ^ {s} \right] _ {j} \neq 0} [ \mathbf {C} ] _ {j} \right\rangle\right) = \exp \left(\left\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x} ^ {s} \right\rangle\right) \tag {39} \\ \end{array}
$$
where  $\mathbf{c}_x^s$  is defined in the main text:
$$
\mathbf {c} _ {x} ^ {s} := \frac {1}{s} \sum_ {s x - s <   i \leq s x} [ \mathbf {C} ] _ {i} = \mathbf {b} _ {x} ^ {s} \mathbf {C} \tag {40}
$$
We note that  $\mathbf{c}_x^s$  is the local average of the  $x$ -th  $s$ -length segment of sequence  $\mathbf{C}$ . The  $\mathbf{c}_x^s$  can be efficiently computed via
$$
\mathbf {c} _ {x} ^ {2 s} = \frac {1}{2} \mathbf {c} _ {2 x - 1} ^ {s} + \frac {1}{2} \mathbf {c} _ {2 x} ^ {s} \quad \mathbf {c} _ {x} ^ {1} = [ \mathbf {C} ] _ {x} \tag {41}
$$
Claim 7.2. Given the set  $\mathcal{J}$  satisfying restriction (25), let  $\mathbf{S}_c$  be a matrix whose rows are elements  $\mathbf{b}_x^s\in \mathcal{J}$
$$
\mathbf {S} _ {c} = \left[ \begin{array}{l} \mathbf {b} _ {x _ {1}} ^ {s _ {1}} \\ \mathbf {b} _ {x _ {2}} ^ {s _ {2}} \\ \dots \\ \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] \quad \mathbf {D} = \left[ \begin{array}{c c c c} s _ {1} & & & \\ & s _ {2} & & \\ & & \dots & \\ & & & s _ {| \mathcal {I} |} \end{array} \right] \tag {42}
$$
$$
\exp \left(\mathbf {P C} ^ {\top} \mathbf {S} _ {c} ^ {\top}\right) \mathbf {D S} _ {c} = \widehat {\exp \left(\mathbf {P C} ^ {\top}\right)} \tag {43}
$$
where  $\widehat{\mathrm{exp}(\mathbf{PC}^{\top})}$  is defined as (37).
Proof. Consider  $i$ -th row of  $\exp (\mathbf{PC}^{\top}\mathbf{S}_c^{\top})$
$$
\begin{array}{l} \left[ \exp (\mathbf {P} (\mathbf {S} _ {c} \mathbf {C}) ^ {\top}) \right] _ {i} = \exp ([ \mathbf {P} ] _ {i} (\mathbf {S} _ {c} \mathbf {C}) ^ {\top}) \\ = \exp \left(\left[ \mathbf {P} \right] _ {i} \left[ \begin{array}{l l l l} \mathbf {c c c c} _ {x _ {1}} ^ {s _ {1}} & \mathbf {c c c c} _ {x _ {2}} ^ {s _ {2}} & \dots & \mathbf {c c c c} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right]\right) \tag {44} \\ = \left[ \exp \left(\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x _ {1}} ^ {s _ {1}} \rangle\right) \quad \dots \quad \exp \left(\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \rangle\right) \right] \\ \end{array}
$$
$$
\left[ \exp (\mathbf {P} (\mathbf {S} _ {c} \mathbf {C}) ^ {\top}) \mathbf {D} \mathbf {S} _ {c} \right] _ {i} = \left[ \exp (\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x _ {1}} ^ {s _ {1}} \rangle) \quad \dots \quad \exp (\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \rangle) \right] \left[ \begin{array}{c} s _ {1} \mathbf {b} _ {x _ {1}} ^ {s _ {1}} \\ \dots \\ s _ {| \mathcal {I} |} \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] \tag {45}
$$
We note that  $s\mathbf{b}_x^s$  simply re-scales the entry of  $\mathbf{b}_x^s$  such that any non-zero entry becomes 1. Then, let us consider  $j$ -th entry of  $[\exp (\mathbf{PC}^\top \mathbf{S}_c^\top)\mathbf{DS}_c]_i$ . Due to the restriction (25), we have exactly one
$\mathbf{b}_x^s \in \mathcal{J}$  whose support region contains  $j$ , so the  $j$ -th column of the second matrix in the right hand side of (45) contains exactly a 1 and the remaining entries are 0. Therefore, we have
$$
\left[ \exp (\mathbf {P} (\mathbf {S} _ {c} \mathbf {C}) ^ {\top}) \mathbf {D} \mathbf {S} _ {c} \right] _ {i, j} = \exp (\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x} ^ {s} \rangle) = \exp (\langle [ \mathbf {P C} ^ {\top} ] _ {i}, \mathbf {b} _ {x} ^ {s} \rangle) = \left[ \widehat {\exp (\mathbf {P C} ^ {\top})} \right] _ {i, j} \tag {46}
$$
where  $\mathbf{b}_x^s\in \mathcal{I}$  is the component that is supported on  $j$ . The second equality is based on (39).
Claim 7.3. If  $\mathbf{S}_c$  and  $\mathbf{D}$  are defined as Claim 7.2, the pseudo inverse of  $\mathbf{S}_c$  is simply  $\mathbf{S}_c^\dagger = \mathbf{S}_c^\top \mathbf{D}$ , so each row of  $\mathbf{S}_c^\dagger$  and  $\mathbf{S}^\dagger$  contain exactly a 1 (so the number of nonzero entries of  $\mathbf{S}_c^\dagger$  and  $\mathbf{S}^\dagger$  are  $n_c$  and  $n$  respectively).
Proof. Since each row of  $\mathbf{S}_c$  is some  $\mathbf{b}_x^s\in \mathcal{I}$ , due to the restriction (25), for  $i\neq j$
$$
\left[ \mathbf {S} _ {c} \mathbf {S} _ {c} ^ {\top} \mathbf {D} \right] _ {i, i} = \langle [ \mathbf {S} _ {c} ] _ {i}, [ \mathbf {D S} _ {c} ] _ {i} \rangle = \left\langle \mathbf {b} _ {x _ {i}} ^ {s _ {i}}, s _ {i} \mathbf {b} _ {x _ {i}} ^ {s _ {i}} \right\rangle = s _ {i} \frac {1}{s _ {i}} = 1 \tag {47}
$$
$$
\left[ \mathbf {S} _ {c} \mathbf {S} _ {c} ^ {\top} \mathbf {D} \right] _ {i, j} = \left\langle \left[ \mathbf {S} _ {c} \right] _ {i}, \left[ \mathbf {D S} _ {c} \right] _ {j} \right\rangle = \left\langle \mathbf {b} _ {x _ {i}} ^ {s _ {i}}, s _ {j} \mathbf {b} _ {x _ {i}} ^ {s _ {j}} \right\rangle = s _ {j} 0 = 0
$$
As a result,  $\mathbf{S}_c\mathbf{S}_c^\top \mathbf{D} = \mathbf{I}$ . Further,  $\mathbf{S}_c^\top \mathbf{D}\mathbf{S}_c$  is a symmetric matrix. So, all Moore-Penrose conditions are verified.  $\mathbf{S}_c^\dagger = \mathbf{S}_c^\top \mathbf{D}$ .
From the restriction (25), we have every column of  $\S_c$  contains exactly a non-zero entry. Also,
$$
\mathbf {S} _ {c} ^ {\dagger} = \mathbf {S} _ {c} ^ {\top} \mathbf {D} = \left[ \begin{array}{l l l l} s _ {1} \mathbf {b} _ {x _ {1}} ^ {s _ {1}} & s _ {2} \mathbf {b} _ {x _ {2}} ^ {s _ {2}} & \dots & s _ {| \mathcal {I} |} \mathbf {b} _ {x _ {| \mathcal {I} |}} ^ {s _ {| \mathcal {I} |}} \end{array} \right] \tag {48}
$$
Since the non-zero entry of  $\mathbf{b}_x^s$  is simply  $\frac{1}{s}$  by definition,  $s\mathbf{b}_x^s$  simply re-scales the entry of  $\mathbf{b}_x^s$  such that any non-zero entry becomes 1. As a result, each row of  $\mathbf{S}_c^\dagger$  has exactly a 1. Also, by the relation between  $\mathbf{S}$  and  $\mathbf{S}_c$ :
$$
\mathbf {S} = \left[ \begin{array}{c c} \mathbf {I} _ {n _ {p} \times n _ {p}} & 0 \\ 0 & \mathbf {S} _ {c} \end{array} \right] \quad \mathbf {S} ^ {\dagger} = \left[ \begin{array}{c c} \mathbf {I} _ {n _ {p} \times n _ {p}} & 0 \\ 0 & \mathbf {S} _ {c} ^ {\dagger} \end{array} \right] \tag {49}
$$
each row of  $\mathbf{S}^{\dagger}$  has exactly a 1.
At the end, the approximation
$$
\widehat {\operatorname {e x p} \left(\mathbf {P C} ^ {\top}\right)} = \exp \left(\mathbf {P C} ^ {\top} \mathbf {S} _ {c} ^ {\top}\right) \mathbf {D S} _ {c} \approx \exp \left(\mathbf {P C} ^ {\top}\right) \tag {50}
$$
does not look exactly as (19), but we can insert a simple diagonal matrix  $\mathbf{D}$  to the formulation (19) and make the whole thing work.
7.2.4 How to Construct  $\mathcal{J}$  for  $\mathbf{S}_c$  and  $\mathbf{S}_c\mathbf{C}$ ?
The derivation so far assume access to  $\mathcal{I}$ , but in practice, we have no knowledge of  $\mathcal{I}$  and need to construct  $\mathcal{I}$  that leads to good approximation. With the approximation scheme in place, we can now analyze the approximation error, which will be leveraged later to find a reasonable set of components  $\mathcal{I}$ . The approximation error of  $i$ -th row of  $\exp(\mathbf{PC}^\top)$  can be expressed as
$$
\begin{array}{l} \mathcal {E} _ {i} := \left\| \left[ \exp (\mathbf {P C} ^ {\top}) \right] _ {i} - \left[ \widehat {\exp (\mathbf {P C} ^ {\top})} \right] _ {i} \right\| _ {F} ^ {2} \\ = \sum_ {j = 1} ^ {n _ {c}} (\left[ \exp (\mathbf {P C} ^ {\top}) \right] _ {i, j} - \left[ \widehat {\exp (\mathbf {P C} ^ {\top})} \right] _ {i, j}) ^ {2} \\ = \sum_ {\mathbf {b} _ {x} ^ {s} \in \mathcal {I}} \sum_ {[ \mathbf {b} _ {x} ^ {s} ] _ {j} \neq 0} \left(\left[ \exp \left(\mathbf {P C} ^ {\top}\right) \right] _ {i, j} - \exp \left(\left\langle [ \mathbf {P C} ^ {\top} ] _ {i}, \mathbf {b} _ {x} ^ {s} \right\rangle\right)\right) ^ {2} (51) \\ = \sum_ {B _ {x} ^ {s} \in \mathcal {J}} \exp \left(\left\langle \left[ \mathbf {P C} ^ {\top} \right] _ {i}, \mathbf {b} _ {x} ^ {s} \right\rangle\right) \sum_ {\left(B _ {x} ^ {s}\right) _ {j} \neq 0} \left(\exp \left(\left[ \mathbf {P C} ^ {\top} \right] _ {i, j} - \left\langle \left[ \mathbf {P C} ^ {\top} \right] _ {i}, \mathbf {b} _ {x} ^ {s} \right\rangle\right) - 1\right) ^ {2} (34) \\ = \sum_ {B _ {x} ^ {s} \in \mathcal {J}} \exp \left(\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x} ^ {s} \rangle\right) \sum_ {\left(B _ {x} ^ {s}\right) _ {j} \neq 0} \left(\exp \left(\langle [ \mathbf {P} ] _ {i}, [ \mathbf {C} ] _ {j} \rangle - \langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x} ^ {s} \rangle\right) - 1\right) ^ {2} \\ = \sum_ {B _ {x} ^ {s} \in \mathcal {J}} \exp \left(\left\langle [ \mathbf {P} ] _ {i}, \mathbf {c} _ {x} ^ {s} \right\rangle\right) \sum_ {\left(B _ {x} ^ {s}\right) _ {j} \neq 0} \left(\exp \left(\left\langle [ \mathbf {P} ] _ {i}, [ \mathbf {C} ] _ {j} - \mathbf {c} _ {x} ^ {s} \right\rangle\right) - 1\right) ^ {2} \\ \end{array}
$$
Algorithm 1 Constructing  $\mathcal{I}$
Input: VIP-tokens  $\mathbf{P}$  and  $\mathbf{c}_x^s$  for all  $s$  and  $x$   
Input:  $h_s$ : number of  $s$ -length segments to refine for each  $s \in \{2,4,\dots,n_c\}$   
Initialize empty  $\mathcal{J}$   
Compute  $\mu_1^{n_c}$  (53) and add  $\mathbf{b}_1^{n_c}$  (22) to  $\mathcal{J}$  (compute root node)  
for  $s \gets n_c, n_c/2, \dots, 2$  do  
    Pop  $h_s$  elements  $\mathbf{b}_x^s$  with the largest  $\mu_x^s$  (53) (select nodes with higher attention scores)  
    for each  $\mathbf{b}_x^s$  do  
        Compute  $\mu_{2x-1}^{s/2}, \mu_{2x}^{s/2}$  (53) and add  $\mathbf{b}_{2x-1}^{s/2}, \mathbf{b}_{2x}^{s/2}$  (22) to  $\mathcal{J}$  (split selected nodes)  
    end for  
end for  
Output:  $\mathcal{J}$
The second equality and fourth equality are due to (25) and (39). The approximation error is governed by two components multiply together: attention score between  $[\mathbf{P}]_i$  and the local average  $\mathbf{c}_x^s$  of the  $x$ -th  $s$ -length segment of sequence  $\mathbf{C}$  and the inner product of  $[\mathbf{P}]_i$  with the amount of deviation of  $[\mathbf{C}]_j$  from its local average  $\mathbf{c}_x^s$ .
When  $s = 1$ , the deviation is simply zero:
$$
\mathbf {c} _ {x} ^ {1} = [ \mathbf {C} ] _ {x}. \tag {52}
$$
It is reasonable to assume that the deviation  $[\mathbf{C}]_j - \mathbf{c}_x^s$  is smaller if  $s$  is smaller. Therefore, this actually suggests a simple heuristic for selecting  $\mathcal{J}$ : when  $\exp (\langle [\mathbf{P}]_i,\mathbf{c}_x^s\rangle)$  is large, we should approximate the  $x$ -th  $s$ -length segment of  $\mathbf{C}$  with higher resolution (by splitting the segment to shorter sub-segments and using finer approximation). This heuristic describes the selection criteria for one row of  $\exp (\mathbf{PC}^\top)$ , which corresponds to a single VIP-token, for multiple rows of  $\exp (\mathbf{PC}^\top)$  (for multiple VIP-tokens), we simply use
$$
\mu_ {x} ^ {s} = \sum_ {i = 1} ^ {n _ {p}} \exp \left(\left\langle \left[ \mathbf {P} \right] _ {i}, \mathbf {c} _ {x} ^ {s} \right\rangle\right) \tag {53}
$$
as selection criteria since  $\mathcal{I}$  is shared by all VIP-tokens.