SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, in Section D.1, we provide several basic definitions and notations. In Section D.2, we briefly the definition of submodular function. In Section D.3, we define the dynamic submodular framework, which gives the formal version of Definition 4.1. In Section D.4, we briefly review the static attention computation problem. In Section D.5, we formulate the attention computation in recursive fashion. In Section D.6, we briefly review our eviction policy, which gives the formal version of Definition 4.3. In Section D.7, we discuss the diminishing return for submodular. In Section D.8, we discuss the high-level ideas for submodular. In Section D.9, we analyze the robust greedy algorithm error propagation. In Section D.10, we explain how to add items into sets via approximate function. In Section D.11, we provide several definitions related to dynamic properties. In Section D.12, we prove an induction lemma for the exact function. In Section D.13, we prove an induction lemma for the approximation function. In Section D.14, we provide theoretical guarantees for both the full-knowledge version (formal version of Lemma 3.1) and the limited-cache-size version (formal version of Theorem 4.4). In Section D.15, we provide a more detailed discussion of theoretical work about attention computation and regression-related problems. In Section D.16, we provide a mathematical formulation for sparsity preserving. In Section D.17, we provide the definition of loss function which can potentially generate sparse (heavy hitter type attention sore). In Section D.18, we explain how to compute the gradient of the loss function. In Section D.19, we show how to compute the Hessian of the loss function. In Section D.20, we show that Hessian is positive definite. In Section D.21, we prove the Lipschitz property for the Hessian matrix. In Section D.22, we show that using a gradient-type algorithm is sufficient to optimize that (heavy hitter type) loss function.
D.1 Notations
For a positive integer  $n$ , let  $[n] := \{1, 2, \dots, n\}$ .
For a vector  $x \in \mathbb{R}^n$ , let  $\sqrt{x} \in \mathbb{R}^n$  denote the vector with the  $i$ -th entry being  $\sqrt{x_i}$  and  $\mathrm{diag}(x) \in \mathbb{R}^{n \times n}$  denote the diagonal matrix with the  $i$ -th digonal entry being  $x_i$ . For two matrices  $A, W \in \mathbb{R}^{n \times n}$ , let  $\| A \|_W := (\sum_{i=1}^n \sum_{j=1}^n W_{i,j} A_{i,j}^2)^{1/2}$  and  $W \circ A$  denote the matrix where  $(W \circ A)_{i,j} = W_{i,j} A_{i,j}$ . For matrix  $W \in \mathbb{R}^{n \times n}$ , let  $D_{W_i} := \mathrm{diag}(W_{i,:})$  with  $i \in [n]$ .
For two vectors  $x \in \mathbb{R}^n$  and  $w \in \mathbb{R}_{\geq 0}^n$ , let  $\|x\|_w := (\sum_{i=1}^n w_i x_i^2)^{1/2}$ . For a vector  $x$ , its  $\ell_2$  norm is defined as  $\|x\|_2 := (\sum_{i=1}^n x_i^2)^{1/2}$  and its  $\ell_p$  norm is defined as  $\|x\|_p := (\sum_{i=1}^n |x_i|^p)^{1/p}$ . For a square matrix  $A$ , we denote  $\operatorname{tr}[A]$  as the trace of matrix  $A$ .
For a matrix  $A \in \mathbb{R}^{n \times k}$  (suppose  $n \geq k$ ), we use  $\| A\|$  to denote its spectral norm, i.e.,  $\| A\| = \sup_x \|Ax\|_2 / \|x\|_2$ . We use  $\| A\|_F$  to denote its Frobenius norm  $\| A\|_F := (\sum_{i=1}^n \sum_{j=1}^k A_{i,j}^2)^{1/2}$ .
Suppose matrix  $A \in \mathbb{R}^{n \times k}$  has SVD decomposition  $U\Sigma V^{\top}$  where  $U \in \mathbb{R}^{n \times k}$  (this matrix has orthonormal columns),  $\Sigma \in \mathbb{R}^{k \times k}$  is a diagonal matrix, and  $V \in \mathbb{R}^{k \times k}$ . We call columns of  $U$  singular vectors. We use  $A^{\dagger} \in \mathbb{R}^{k \times n}$  to denote the Moore-Penrose pseudoinverse, then  $A^{\dagger} = V\Sigma^{-1}U^{\top}$ . Suppose  $\Sigma \in \mathbb{R}^{k \times k}$  is sorted diagonal matrix, let  $\sigma_1, \dots, \sigma_k$  denote the diagonal entries of  $\Sigma$ . Then we call  $\sigma_i$  the  $i$ -th singular value of the matrix, and we write it as  $\sigma_i(A)$ .
For any symmetric matrix  $B \in \mathbb{R}^{k \times k}$ , we denote its eigenvalue decomposition as  $U \Lambda U^{\top}$ , where  $\Lambda$  is a diagonal matrix. Let  $\lambda_1, \dots, \lambda_k$  denote the entries on diagonal of  $\Lambda \in \mathbb{R}^{k \times k}$ . We say  $\lambda_i$  is the  $i$ -th eigenvalue. Usually, we write it as  $\lambda_i(B)$ .
The connection between eigenvalues and singular values is
$$
\sigma_ {i} ^ {2} (A) = \lambda_ {i} (A ^ {\top} A)
$$
We use the notation  $A \succeq 0$  to denote that matrix  $A$  is positive semidefinite (psd). Mathematically,  $A \succeq 0$  means for all vectors  $x$ , we have  $x^{\top}Ax \geq 0$ .
Similarly, for two squarer matrices  $A$  and  $B$ , we use  $A \succeq B$  to denote the case where for all vectors  $x, x^{\top}Ax \geq x^{\top}Bx$ .
Let  $\operatorname{Pr}[]$  and  $\mathbb{E}[]$  denote the probability and expectation. We define the maximum between  $a$  and  $b$  as  $\max\{a, b\}$ . We denote  $\min\{a, b\}$  (resp.  $\max\{a, b\}$ ) as the minimum (reps. maximum) between  $a$  and  $b$ .
Throughout, for non-negative real numbers  $a$  and  $b$ , we use the notation  $a = (1 \pm \epsilon)b$  if  $a \in [(1 - \epsilon)b, (1 + \epsilon)b]$ .
D.2 Submodular
We provide the standard definition of the submodular function.
Definition D.1 (Submodular function [123]). For a finite set  $\Omega$ , a submodular function is defined as  $f: 2^{\Omega} \to \mathbb{R}$ , where  $2^{\Omega}$  denotes the power set of  $\Omega$ . It is characterized by the fulfillment of any of the following equivalent criteria:
Condition 1.
- For  $S, T, \subseteq \Omega$  with  $S \subseteq T$  and every  $a \in \Omega \backslash T$  we have that  $f(S \cup \{a\}) - f(S) \geq f(T \cup \{a\}) - f(T)$ .
Condition 2.
- For every  $S, T \subseteq \Omega$  we have that  $f(S) + f(T) \geq f(S \cup T) + f(S \cap T)$ .
Condition 3.
- For every  $S \subseteq \Omega$  and  $a_1, a_2 \in \Omega \backslash S$  such that  $a_1 \neq a_2$  we have that  $f(S \cup \{a_1\}) + f(S \cup \{a_2\}) \geq f(S \cup \{a_1, a_2\}) + f(S)$ .
For convenience of discussion, in this paper, we always choose  $\Omega = [n]$  when we want to discuss the submodular function.
Next, we provide some examples/types of submodular functions. One important class is called monotone,
Definition D.2 (Monotone). A set function  $f$  is monotone if for every  $T \subseteq S$  we have  $f(T) \leq f(S)$ .
Here are a number of monotone submodular functions
- Linear (Modular) functions
- A linear function can be represented as  $f(S) = \sum_{i \in S} w_i$ . If all the weights  $w_i$  are nonnegative, then the function  $f$  is considered monotone.
Budget-additive functions
- A budget-additive function has the form  $f(S) = \min B, \sum_{i \in S} w_i$  where each weight  $w_i$  and the budget  $B$  are nonnegative.
- Coverage functions
- We have a set  $\Omega = \{E_1, E_2, \dots, E_n\}$  where each  $E_i$  is a subset of a broader set  $\Omega'$ . The coverage function can be expressed as  $f(S) = |\cup_{E_i \in S} E_i|$  for any  $S \subset \Omega$ . This function can be generalized by assigning non-negative weights to the elements.
D.3 Dynamic Submodular
The standard submodular function is mapping  $2^{[n]}$  to real. Here we need a more general definition that maps  $2^{[n]} \times 2^{[n]}$  to real.
Definition D.3 (Strong submodular). We define function  $F: 2^{[n]} \times 2^{[n]} \to \mathbb{R}$ , then for any set  $Z \subset [n]$ , we assume that  $F(Z, \cdot): 2^{[n]} \to \mathbb{R}$  is a submodular function.
In fact, the above definition is stronger than we want, what we really need can be written as follows. We remark that in Definition 4.1 we provide an informal definition. Here we provide a more detailed version of the definition.
Definition D.4 (Dynamic submodular framework, formal version of Definition 4.1). Define function
$$
F: 2 ^ {[ n ]} \times [ n ] \times 2 ^ {[ n ]} \to \mathbb {R}.
$$
Then for any index  $i \in [n]$ , any set  $Z \subseteq [i - 1]$ , we assume that
$$
F (Z, i, \cdot): 2 ^ {[ n ]} \to \mathbb {R}
$$
is a submodular function w.r.t. to  $Z$ , i.e.,
$$
f _ {Z, i} (X \cup \{x \}) - f _ {Z, i} (X) \geq f _ {Z, i} (Y \cup \{x \}) - f _ {Z, i} (Y),
$$
where  $f_{Z,i}(\cdot)\coloneqq F(Z,i,\cdot)$
Remark D.5. We remark that Definition D.4 is a weaker version of Definition D.3. We also like to mention that the informal definition (see Definition 4.1) only contains two input parameters, but in fact we need three input parameters (see Definition D.4).
In the later, when we use  $f_{Z,i}(\cdot)$ , we will replace  $Z$  by  $S_i$  for convenient of analysis, for example see Definition D.23, Definition D.24, Definition D.25 and Definition D.26.
D.4 Static Attention
Before we describe the recursive attention computation, we will first describe the static version of attention computation as follows (for examples, see Definition 1.1 in [98] and others [97, 100, 101, 106, 103, 118, 108]):
Definition D.6. Given three matrices  $Q, K, V \in \mathbb{R}^{d \times d}$ , the goal is to compute
$$
\operatorname {A t t} (Q, K, V) := D ^ {- 1} A \cdot V
$$
where square matrix  $A\in \mathbb{R}^{n\times n}$  can be rewritten as follows
$$
A = \exp (Q K ^ {\top})
$$
and diagonal matrix  $D\in \mathbb{R}^{n\times n}$  can be written as follows
$$
D = \operatorname {d i a g} \left(A \mathbf {1} _ {n}\right)
$$
Here we apply  $\exp()$  to a matrix entry-wisely to a matrix.  $\mathbf{1}_n$  is a length- $n$  vector where all the entries are ones. The operator  $\mathrm{diag}()$  is turning a vector into a diagonal matrix.
D.5 Recursive Attention Definition
We first provide some definitions.
Definition D.7. Given a query matrix  $Q \in \mathbb{R}^{n \times d}$ , we use  $Q_{i,*}$  to denote a length- $d$  row vector that represents the  $i$ -th row of  $Q$  for each  $i \in [n]$ .
Definition D.8. Given a key matrix  $K \in \mathbb{R}^{n \times d}$ , we use  $K_{\leq i,*}$  to denote a  $i \times d$  matrix that selects the first  $i$  rows from  $K$ .
(b)
Figure 14: (a) Exact version of the attention computation. Here, our example is about the language modeling task. At this stage, the model predicts the word 'apple' and computes the exact attention vector  $o_4$ . (b) Approximate version of the attention computation. Let the budget of the number of tokens we can track be 3. We truncate the key matrix for the first token  $K_{1,*}$  and only keep  $K_{2,*}, K_{3,*}$  and  $K_{4,*}$  in the memory. Compared with the exact version, we don't compute  $o_{4,1}$ .
Next, we show the exact version of computing attention.
Definition D.9 (Exact Version). Given  $Q$ ,  $K \in \mathbb{R}^{n \times d}$
- For each  $i \in [n]$ , we use  $o_i \in \mathbb{R}^i$  to denote a length- $i$  vector.
- For each  $j \in [i]$ , we use  $o_{i,j}$  to denote the  $j$ -th coordinate of vector  $o_i \in \mathbb{R}^i$
For the first layer, we have
$o_{1,1} = 1$
For the second layer, we have
- length-2 vector  $o_2 \coloneqq \underbrace{D_2^{-1}}_{\text{scalar}} \cdot \exp(\underbrace{Q_{2,*}}_{1 \times d} \underbrace{(K_{\leq 2,*})^\top}_{d \times 2})$
- scalar  $D_{2}:= \underbrace{\exp(Q_{2,*}(K_{\leq 2,*})^{\top})}_{1\times 2}\cdot \underbrace{\mathbf{1}_{2}}_{2\times 1}$
For each  $i\in [n]$  ,for the  $i$  -th layer,we have
- length-i vector  $o_i := \underbrace{D_i^{-1}}_{\text{scalar}} \cdot \exp(\underbrace{Q_{i,*}}_{1 \times d} \underbrace{(K_{\leq i,*})^\top}_{d \times i})$
- scalar  $D_{i}:= \underbrace{\exp(Q_{i,*}(K_{\leq i,*})^{\top})}_{1\times i}\cdot \underbrace{\mathbf{1}_{i}}_{i\times 1}$
Now, we show the approximate version of computing attention. Instead of computing the entire attention  $o_i$ , we only compute the attention of the tokens that are being tracked.
Definition D.10 (Approximate Version). Given  $Q$ ,  $K \in \mathbb{R}^{n \times d}$
- For each  $i \in [n]$ , we use  $o_i \in \mathbb{R}^i$  to denote a length- $i$  vector.
- For each  $j \in [i]$ , we use  $o_{i,j}$  to denote the  $j$ -th coordinate of vector  $o_i \in \mathbb{R}^i$
- Let  $k \in [n]$  be the budget of the number of tokens we can track (due to the memory issue).
For each  $i\in [n]$  , for the  $i$  -th token, we have
- Let  $S_{i} \subset [n]$  denote the tokens we're tracking when we are predicting the  $i$ -th token.
-  $\left| {S}_{i}\right|  = k$
-  $|S_i \backslash S_{i-1}| \leq 1$  or equivalently  $|S_i \cap S_{i-1}| \geq k - 1$
- length-i vector  $o_i \coloneqq \underbrace{D_i^{-1}}_{\text{scalar}} \cdot \exp(\underbrace{Q_{i,*}}_{1 \times d} \underbrace{(K_{S_i,*})^\top}_{d \times i})$
- scalar  $D_{i} \coloneqq \underbrace{(\exp(Q_{i,*}(K_{S_{i},*})^{\top}) - 1_{[i]\backslash S_{i}})}_{1\times i}\cdot \underbrace{\mathbf{1}_{i}}_{i\times 1}$
In a certain sense, the above definition is related to finding heavy hitters in compressive sensing (for more background on compressive sensing, we refer readers to [124, 125, 126]).
D.6 Eviction Policy
The goal of this section is to our eviction policy under space limitation. We start by giving a process without having space limitations. We denote the attention query matrix as  $Q \in \mathbb{R}^{n \times d}$  and the key matrix as  $K \in \mathbb{R}^{n \times d}$ .  $Q_{i,*}$  represents the  $i$ -th row of  $Q$  and  $K_{\leq i,*}$  represents the first  $i$  rows of  $K$ . For simplicity,  $K_{S_i,*}$  denotes a sub-matrix of  $K$  which selects  $S_i$  rows from  $K$ .
Definition D.11 (The generative process with Full knowledge). For each  $i \in [n]$ , for the  $i$ -th token, we have
- length- $i$  vector  $o_i \coloneqq D_i^{-1} \cdot \exp(Q_{i,*}(K_{\leq i,*})^\top)$ , which denotes the attention of the  $i$ -th word.
- scalar  $D_{i} \coloneqq \exp(Q_{i,*}(K_{\leq i,*})^{\top}) \cdot \mathbf{1}_{i}$ , which denotes the normalization factor.
In the above process, since we have no space limitation, then we can keep all the scores. In the following process, we show how to handle things when there is a space limitation. In this case, we need to dynamically maintain set  $S_{i}$  such that  $|S_{i}| \leq k$  (Compared to the full knowledge case, we can think of  $|S_{i}| = i$ ).
Definition D.12 (H $_2$ O Eviction Policy, Formal version of Definition 4.3). Let  $k$  denote the budget of space and  $k < n$ . Let  $F_{\mathrm{score}}: 2^{[n]} \to \mathbb{R}$  denote certain score function. Let  $S_{i-1}$  denote the source set. Let  $S_i$  denote the target set. We defined the eviction policy  $g: S_{i-1} \to S_i$  s.t.
-  $|S_i| \leq k$  (KV cache size is not changing over the time)
-  $|S_i \backslash S_{i-1}| \leq 1$  (we can evict at most 1 KV in the KV cache)
- We construct  $S_{i} \gets (S_{i - 1} \cup \{i\}) \setminus \{u\}$  as  $u \gets \arg \max_{v \in (S_{i - 1} \cup \{i\})} F_{\mathrm{score}}(S_{i - 1} \cup \{i\} \setminus \{v\})$
Remark D.13. We remake that, in Definition 4.3, we introduce a simplified notation where we state that  $|S_i| = k$  in the first bullet point, but in general, we consider the case where  $|S_i| \leq k$  for the first  $k$  tokens. To accommodate this more general scenario, we present Definition D.12, which provides a broader definition that handles the situation where  $|S_i| \leq k$ .
Algorithm 2  $\mathsf{H}_2$  Eviction Algorithm, Query matrix  $Q\in \mathbb{R}^{n\times d}$ , key matrix  $K\in \mathbb{R}^{n\times d}$ , budget size of KV cache  $k\in \mathbb{N}$ . Formal and detailed version of Algorithm 1.
1: procedure  $\mathsf{H}_2$ -EVICTION( $Q \in \mathbb{R}^{n \times d}$ ,  $K \in \mathbb{R}^{n \times d}$ ,  $k \in \mathbb{N}$ )  
2:  $S_0 \gets \emptyset, \widetilde{o}_0 \gets 0$  ▷ Initialization the algorithm  
3: for  $i = 1 \to n$  do ▷ Linear scan each token  
4: if  $i \leq k$  then ▷ If the KV cache is not full  
5:  $S_i \gets S_{i-1} \cup \{i\}$  ▷ Expand the set directly  
6: else ▷ If the KV cache is full  
7:  $D_i \gets (\exp(Q_{i,*}(K_{S_{i-1},*})^\top) - 1_{[i] \setminus S_{i-1}}) \cdot 1_i$  ▷ Compute the normalization factor  
8:  $o_i \gets D_i^{-1} \cdot \exp(Q_{i,*}(K_{S_{i-1},*})^\top)$  ▷ Compute score vector  
9:  $\widetilde{o}_i \gets \widetilde{o}_{i-1} + o_i$  ▷ Accumulate the scores (Remark D.15)  
10: Let score function be  $F_{\text{score}}(T) := h(\sum_{s \in T} \widetilde{o}_{i,s})$  ▷ Score function (Remark D.14)  
11:  $u \gets \arg \max_{v \in (S_{i-1} \cup \{i\})} F_{\text{score}}(S_{i-1} \cup \{i\} \setminus \{v\})$  ▷ Find an entry to swap  
12:  $S_i \gets (S_{i-1} \cup \{i\}) \setminus \{u\}$  ▷ Construct  $S_i$   
13: end if  
14: end for  
15: end procedure
Remark D.14. We remark the above function  $F_{\mathrm{score}}$  can have multiple choices. In the later analysis, we provide two choices. If we use the exact function, then  $F_{\mathrm{score}}$  is  $f_{S_i,i}$  (see Definition D.23 and Definition D.25). If we use the approximate function, then  $F_{\mathrm{score}}$  if  $\widetilde{f}_{S_i,i}$  (see Definition D.24 and Definition D.26). Let  $h: \mathbb{R} \to \mathbb{R}$  (the  $h$  being used Line 10 in Algorithm 2) denote function. We hope to choose  $h$  such that it gives the submodular property for the function  $F_{\mathrm{score}}$  in the sense of our dynamic framework. For example,  $h(z)$  is usually chosen to be some non-decreasing concave function such as  $\sqrt{z + 1}$  and  $\log (z + 1)$ . For simplicity, in Algorithm 1 (which is the informal version of Algorithm 2), we choose  $h(z) = z$  for the purpose of providing readers with an intuitive understanding and due to space limitation.
Remark D.15. We remark that, in Line 10 in Algorithm 2, the  $\widetilde{o}_{i,s}$  is the accumulation of attention score for the token  $s$  in set  $T$ . In our Algorithm 1, we only use  $o_s$  in Line 10 (see Algorithm 1) to represent that accumulation of the attention score for simplicity.
In Algorithm 2, for simplicity of the presentation, we can create multiple vectors  $\widetilde{o}_0,\dots \widetilde{o}_n$  (see Line 2, Line 9, Line 10). However, every time at  $i$ -th, we only need to use the information for  $\widetilde{o}_i$  which has at most length  $n$  size. Thus, a straightforward and better implementation for only using one length- $n$  to store accumulative score can reduce  $n^2$  usage to  $n$ .
D.7 Explaining Submodular Diminishing Return Property in Attention Scheme
In the standard submodular problem [123], we are given a ground set  $[n]$  and a function  $f: 2^{[n]} \to \mathbb{R}$ . The goal is to find a set of elements  $S$  such that  $f(S)$  is maximized.
We say function  $f$  is submodular (Recall the formal definition in Definition D.1), if for every  $X, Y \subset [n]$  with  $X \subseteq Y$  and every  $x \in [n] \backslash Y$  we have
$$
f (X \cup \{x \}) - f (X) \geq f (Y \cup \{x \}) - f (Y)
$$
Submodular functions can represent the cost of items, due to their diminishing returns property [127, 128]. It suggests that the increase in information obtained from selecting a candidate object, such as a word or sentence, becomes smaller as more objects have already been chosen for the summary.
For instance, we introduce a new token, denoted as "w", into two sets,  $S$  and  $S_0$ , where the concepts covered by  $S_0$  are a subset of those covered by  $S$ . By intuition, the information added to  $S_0$  by "w" should be larger compared to adding it to  $S$ , as the new concepts carried by "w" might have already been covered by the concepts present in  $S$  but not in  $S_0$ . This property is known as the diminishing return property. Hence, we propose that the neural coverage function [129] should exhibit a desirable characteristic called submodularity.
D.8 Submodular: High Level Ideas
We formalize the submodular function maximization problem with cardinality constraint in this section. Informally, a set function is submodular if it has decreasing marginal increment.
Definition D.16 (Submodular function). We denote  $f: 2^{[n]} \to \mathbb{R}$  as a set function. The discrete derivative  $\Delta_f$  is defined as follows:
$$
\Delta_ {f} (i \mid S) := f (S \cup \{i \}) - f (S).
$$
A function  $f$  is submodular if, for any  $S \subseteq T$  and  $i \in [n] \backslash T$ , the following inequality holds:
$$
\Delta_ {f} (i \mid T) \leq \Delta_ {f} (i \mid S).
$$
For simplicity, we present the problem of maximizing a submodular function with a cardinality constraint (1). Our goal is to solve the optimization problem efficiently.
$$
\begin{array}{l} \max  _ {S \subseteq [ n ]} f (S) \\ \begin{array}{l l} \text {s . t .} & | S | \leq k \end{array} \\ \end{array}
$$
Representation of  $f(S)$  One challenge in designing the algorithm is determining the representation of input instances. Since the constraint in optimization problem (1) is straightforward, we need to decide how to represent  $f(S)$ . Suppose  $S = i_{1},i_{2},\dots ,i_{m}\subseteq [n]$ , we can decompose  $f(S)$  into a sum of increments as follows:
$$
f (S) = f \left(S _ {0}\right) + \sum_ {j = 1} ^ {m} \left(f \left(S _ {j}\right) - f \left(S _ {j - 1}\right)\right), \tag {2}
$$
where  $S_0 = \emptyset$  and  $S_{j} = S_{j - 1} + i_{j}$ . Without loss of generality, we assume  $f(\emptyset) = 0$ . By the definition of  $\Delta_f(i|S)$ , we have  $f(S_j) - f\big(S_{j - 1}\big) = \Delta_f(i_j|S_{j - 1})$ . Therefore, the decomposition (2) can be simplified as follows:
$$
f (S) = \sum_ {j = 1} ^ {m} \Delta_ {f} \left(i _ {j} \mid S _ {j - 1}\right) \tag {3}
$$
To introduce our advanced data structure later, we further represent  $\Delta_f(i|S)$  in the form of
$$
\Delta_ {f} (i | S) = u _ {i} ^ {\top} h (S) u _ {i} \tag {4}
$$
where  $u_{i}\in \mathbb{R}^{d}$  is a  $d$  -dimensional vector and  $h(S)\in \mathbb{R}^{d\times d}$  is a  $d$  -by-  $d$  matrix.
In practice, a significant subclass of submodular functions is the monotone submodular functions, i.e. functions  $f$  satisfying  $f(A) \leq f(B)$  for all  $A \subseteq B \subseteq [n]$ . When  $f$  is monotone, we could restrict all  $h(S)$  to be positive semidefinite (PSD) matrixes. When the matrix  $h(S)$  is positive semidefinite (PSD), it makes us achieve a faster acceleration.
D.9 Robust Greedy with Error Propagation
The procedure for the greedy selection algorithm initiates with empty set  $S_0 = \emptyset$ . For each iteration in the main loop, the algorithm chooses the element that maximizes the marginal increment to add to the set. When the size of the set eventually reaches  $k$ , the algorithm returns that set  $S$ . Specifically, for iteration  $t \in \{1, 2, \dots, k\}$ , we let
$$
S _ {t} \leftarrow S _ {t - 1} \cup \{j _ {t} \}
$$
where the element in the singleton is  $j_{t} = \arg \max_{j\in \overline{S}_{t - 1}}f(S_{t - 1}\cup \{j\})$ . The greedy strategy is effective in the sense that the approximation error of it is  $1 - 1 / e$
Theorem D.17 ([130]). For a monotone submodular function  $f$ , the greedy algorithm (Algorithm 3) guarantees to output a set  $S$  satisfying
$$
f (S) \geq (1 - 1 / e) \max  _ {| T | = k} \{f (T) \}.
$$
Corollary D.18 ([131]). Given
Algorithm 3 Greedy algorithm benchmark
1: procedure GreEDYALGORITHM(submodular function  $f$  2:  $S_0\gets \emptyset$  Initialize an empty set  $S_{0}$    
3: for  $t = 0\to k - 1$  do   
4:  $j\gets \arg \max_{i}\{f(S_{t}\cup \{i\})\}$  Find the element  $i\in [n]$  that maximize  $f(S_{t}\cup \{i\})$    
5:  $S_{t + 1}\leftarrow S_t\cup \{j\}$  Append element i to set  $S_{t}$  to get  $S_{t + 1}$    
6: end for   
7: return  $S_{k}$    
8: end procedure
accuracy parameter  $\epsilon >0$
integer  $k\geq 1$
- Let  $O$  denote an oracle that takes an arbitrary set  $S \subseteq [n]$  and  $i \in [n] \setminus S$ , returns a value  $O(S, i)$  with guarantee that  $\Delta(i|S) - \epsilon \leq O(S, i) \leq \Delta(i|S) + \epsilon$ .
- Let  $A$  denote an algorithm that each time step  $t = 1,2,\dots ,k$ , it selects  $j_{t} = \arg \max_{j}\{O(S_{t - 1},j)\}$  and lets  $S_{t}\gets S_{t - 1}\cup \{j_{t}\}$ .
Then this algorithm  $A$
- returns a set  $S_{k}$
- the  $S_{k}$  satisfy that
$$
f(S_{k})\geq (1 - 1 / e)\max_{|T| = k}\{f(T)\} -k(2 - 1 / e)\epsilon .
$$
D.10 Robust Submodular and Adding Items
We first propose a lemma that shows the robustness of the submodular.
Lemma D.19. Suppose that for each  $i\in [n]$  , we have