SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\mathrm {d} \log f (x)}{\mathrm {d} x _ {i}} = f (x) ^ {- 1} \circ \frac {\mathrm {d} f (x)}{\mathrm {d} x _ {i}} \\ = f (x) ^ {- 1} \circ (- \langle f (x), A _ {*, i} \rangle f (x) + f (x) \circ A _ {*, i}) \\ = - \langle f (x), A _ {*}, i \rangle f (x) ^ {- 1} \circ f (x) + f (x) ^ {- 1} \circ f (x) \circ A _ {*}, i \\ = - \langle f (x), A _ {* i} \rangle \cdot \mathbf {1} _ {n} + A _ {* i} \\ \end{array}
$$
Proof of Part 2. We have
$$
\begin{array}{l} \langle \frac {\mathrm {d} \log f (x)}{\mathrm {d} x _ {i}}, b \rangle = \langle - \langle f (x), A _ {*, i} \rangle \cdot \mathbf {1} _ {n} + A _ {*, i}, b \rangle \\ = \left\langle A _ {*, i}, b \right\rangle - \left\langle f (x), A _ {*, i} \right\rangle \cdot \left\langle b, \mathbf {1} _ {n} \right\rangle , \\ \end{array}
$$
where the first step follows from Part 1 and the second step follows from simple algebra.
Proof of Part 3. The proof directly follows from Part 2 and Definition of  $L_{\mathrm{cent}}(x)$  (See Definition C.7).
C.4 Hessian Computation for Function  $\log f(x)$
In this section, we will show how to compute the Hessian for function  $\log f(x)$ .
Lemma C.11. If the following conditions hold
- Let  $f$  be defined as Definition C.4.
$$
\frac {\mathrm {d} ^ {2} \log f (x)}{\mathrm {d} x _ {i} ^ {2}} = \left(\langle f (x), A _ {*, i} \rangle^ {2} - \langle f (x), A _ {*, i} \circ A _ {*, i} \rangle\right) \cdot \mathbf {1} _ {n}
$$
$$
\frac {\mathrm {d} ^ {2} \log f (x)}{\mathrm {d} x _ {i} \mathrm {d} x _ {j}} = (\langle f (x), A _ {*, i} \rangle \langle f (x), A _ {*, j} \rangle - \langle f (x), A _ {*, i} \circ A _ {*, j} \rangle) \cdot \mathbf {1} _ {n}
$$
$$
\begin{array}{l} \frac {\mathrm {d} ^ {2} \log f (x)}{\mathrm {d} x _ {i} ^ {2}} = \frac {\mathrm {d}}{\mathrm {d} x _ {i}} (\frac {\mathrm {d} \log f (x)}{\mathrm {d} x _ {i}}) \\ = \frac {\mathrm {d}}{\mathrm {d} x _ {i}} (- \langle f (x), A _ {*, i} \rangle \cdot \mathbf {1} _ {n} + A _ {*, i}) \\ = - \frac {\mathrm {d}}{\mathrm {d} x _ {i}} \left(\left\langle f (x), A _ {* , i} \right\rangle\right) \cdot \mathbf {1} _ {n} \\ = \left(\left\langle f (x), A _ {* i} \right\rangle^ {2} - \left\langle f (x), A _ {* i} \circ A _ {* i} \right\rangle\right) \cdot \mathbf {1} _ {n} \\ \end{array}
$$
where the 2nd step comes from Part 1 of Lemma C.10, the 3rd step follows from  $A_{*,i}$  is independent of  $x$ , and the forth step follows from Part 2 of Lemma C.9.
Proof of Part 2.
Similarly, we can provide a proof for Part 2.
C.5 Hessian Computation for Function  $\langle \log f(x), b \rangle$
The goal of this section is to prove Lemma C.12.
Lemma C.12. If the following conditions hold
$$
\langle \frac {\mathrm {d} ^ {2} \log f (x)}{\mathrm {d} x _ {i} ^ {2}}, b \rangle = (\langle f (x), A _ {*, i} \rangle^ {2} - \langle f (x), A _ {*, i} \circ A _ {*, i} \rangle) \cdot \langle \mathbf {1} _ {n}, b \rangle
$$
$$
\left. \left\langle \frac {\mathrm {d} ^ {2} \log f (x)}{\mathrm {d} x _ {i} \mathrm {d} x _ {j}}, b \right\rangle = \left(\left\langle f (x), A _ {*, i} \right\rangle \left\langle f (x), A _ {*, j} \right\rangle - \left\langle f (x), A _ {*, i} \circ A _ {*, j} \right\rangle\right) \cdot \left\langle \mathbf {1} _ {n}, b \right\rangle \right.
$$
Proof. The proof directly follows from Lemma C.11.  $\square$
C.6 Hessian Computation for Function  $L_{\mathrm{cent}}(x)$
For convenient of analyzing the  $d \times d$  Hessian matrix, we will start with defining  $n \times n$  matrix  $B$ .
Definition C.13. We define  $B(x)\in \mathbb{R}^{n\times n}$  as follows
$$
B (x) := \langle \mathbf {1} _ {n}, b \rangle \cdot (\operatorname {d i a g} (f (x)) - f (x) f (x) ^ {\top})
$$
Lemma C.14. If the following conditions hold
- Let  $f$  be defined as Definition C.4.
- Let  $L_{\mathrm{cent}}$  be defined as Definition C.7
- Let  $B$  be defined as Definition C.13
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} x _ {i} ^ {2}} L _ {\text {c e n t}} = \left(- \langle f (x), A _ {*, i} \rangle^ {2} + \langle f (x), A _ {*, i} \circ A _ {*, i} \rangle\right) \cdot \langle \mathbf {1} _ {n}, b \rangle
$$
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} x _ {i} \mathrm {d} x _ {j}} L _ {\mathrm {c e n t}} = (- \langle f (x), A _ {*, i} \rangle \langle f (x), A _ {*, j} \rangle + \langle f (x), A _ {*, i} \circ A _ {*, j} \rangle) \cdot \langle \mathbf {1} _ {n}, b \rangle
$$
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} x ^ {2}} L _ {\mathrm {c e n t}} = A ^ {\top} B (x) A
$$
Proof. The proof trivially follows from Lemma C.12 and Definition C.13.  $\square$
C.7 Hessian is Positive Definite
Previous work [20] doesn't consider cross entropy into the final loss function. Here we generalize previous lemma so that cross entropy is also being considered.
Lemma C.15 (A cross entropy generalization of Lemma 6.3 in [20]). Suppose the following conditions hold
- Let  $A \in \mathbb{R}^{n \times d}$ ,  $R \geq 4$ ,  $l > 0$ , suppose that  $R_0 = \exp(O(R^2 + \log n))$
#
$$
L (x) = \underbrace {L _ {\mathrm {r e g}} (x)} _ {\text {D e f i n i t i o n C . 8}} + \underbrace {L _ {\mathrm {c e n t}} (x)} _ {\text {D e f i n i t i o n C . 7}} + \underbrace {L _ {\mathrm {e x p}} (x)} _ {\text {D e f i n i t i o n C . 6}}.
$$
- Let  $\widetilde{B}(x) = B(x) + W^2$
- Part 1.  $\min_{i\in [n]}w_i^2\geq 10R_0 + l / \sigma_{\min}(A)^2$  , then we have
- Part 2.  $\min_{i\in [n]}w_i^2\geq 10^4\cdot R_0 + l / \sigma_{\min}(A)^2$  , then we have
$$
(1 - 0. 0 1) \cdot \widetilde {B} (x) \preceq W ^ {2} \preceq (1 - 0. 0 1) \cdot \widetilde {B} (x).
$$
Proof. Using the definition of  $B$  for  $L_{\mathrm{cent}}$  (see Definition C.13), definition/bound of  $B$  for  $L_{\mathrm{exp}}$  (see [20]), and tools developed in Section 6 in [20], we can finish the proof.  $\square$
C.8 Hessian is Lipschitz
Lemma C.16 (A cross entropy version of Lemma 7.1 in [20]). Suppose the following condition holds
- Let  $H(x) = \frac{\mathrm{d}^2L}{\mathrm{d}x^2}$  and  $R > 4$ .
- Suppose that  $\max \{\| x\| _2,\| y\| _2\} \leq R$  , and  $\max \{\| A\| ,\| b\| _2\} \leq R$
$\| A(x - y)\|_{\infty} < 0.01$
$$
\left\| H (x) - H (y) \right\| \leq n ^ {4} \exp \left(O \left(R ^ {2} + \log n\right)\right) \cdot \| x - y \| _ {2}
$$
Proof. Using the definition of  $B$  for  $L_{\mathrm{cent}}$  (see Definition C.13), definition/bound of  $B$  for  $L_{\mathrm{exp}}$  (see [20]), and tools developed in Section 7 in [20], we can finish the proof.  $\square$
Algorithm 1 Our Algorithm.
1: procedure OURALGORITHM  $(A\in \mathbb{R}^{n\times d},b\in \mathbb{R}^n,w\in \mathbb{R}^n,\epsilon ,\delta)$  ▷ Theorem D.1  
2: We choose  $x_0$   
3:  $T\gets \log (\| x_0 - x^*\| _2 / \epsilon)$  ▷  $T$  denotes the number of iterations  
4: for  $t = 0\to T$  do  
5: Implicitly formulate exact Hessian and use that to construct an approximate Hessian  $\widetilde{H}$  (similar as Section 8 in [20])  
6: Compute gradient  
7:  $\widetilde{H}\gets A^{\top}\widetilde{D} A$   
8:  $x_{t + 1}\gets x_t + \widetilde{H}^{-1}g$   
9: end for  
10:  $\widetilde{x}\gets x_{T + 1}$   
11: return  $\widetilde{x}$   
12: end procedure
D Main Theoretical Guarantees
Previous work [20] has proved the similar result without considering the cross entropy. We generalize the techniques in previous paper [20] from only considering  $\ell_2$  task loss to considering both  $\ell_2$  task loss and cross entropy loss ( $L_{\mathrm{cent}}$  see formal definition in Definition C.7). Our algorithm is a version of approximate Newton method, such methods have been widely used in many optimization tasks [18, 59, 9, 52, 87, 54, 38, 85, 49, 74, 20, 39, 53, 86]. In this work, we focus on the approximate Newton method along the line of [85, 22, 20].
Theorem D.1 (Formal version of Theorem 3.1). Let  $x^{*}$  denote an length- $d$  vector that is satisfying,
$$
\arg \min  _ {x \in \mathbb {R} ^ {d}} \underbrace {L _ {\mathrm {e x p}}} _ {\text {D e f i n i t i o n C . 6}} + \underbrace {L _ {\mathrm {c e n t}}} _ {\text {D e f i n i t i o n C . 7}} + \underbrace {L _ {\mathrm {r e g}}} _ {\text {D e f i n i t i o n C . 8}}
$$
Suppose the following conditions are holding:
-  $R \geq 4$ ,  $g(x^{*}) = \mathbf{0}_{d}$ .
-  $\| x^{*}\|_{2}\leq R,\| A\| \leq R,\| b\|_{2}\leq R.$
-  $M = \exp (O(R^2 + \log n))$ .
$\min_{i\in [n]}w_i^2\geq 100M + l / \sigma_{\mathrm{min}}(A)^2$
- Suppose that  $\epsilon \in (0, 0.1)$  is the final and  $\delta \in (0, 0.1)$  is the failure probability.
- Suppose  $x_0$  satisfy condition  $M\|x_0 - x^*\|_2 \leq 0.1l$ .
- Suppose that  $T = \log (\| x_0 - x^*\| _2 / \epsilon)$
Then there is a randomized algorithm (Algorithm 1) such that
- it runs  $T$  iterations
- in each iteration, it spends time†
$$
O \left(\left(\operatorname {n n z} (A) + d ^ {\omega}\right) \cdot \operatorname {p o l y} \left(\log (n / \delta)\right)\right).
$$
- generates a vector  $\widetilde{x} \in \mathbb{R}^d$  that is satisfying
$$
\| \widetilde {x} - x ^ {*} \| _ {2} \leq \epsilon
$$
- the succeed probability is  $1 - \delta$
Proof. The high level framework of our theorem is similar to previous work about exponential regression [63], softmax regression [20] and rescaled softmax regression [36]. Similarly as previous work [63, 20, 36, 80], we use the approximate newton algorithm (for example see Section 8 in [20]). So in the proof, we only focus on the difference about the Hessian positive definite lower bound and Hessian Lispchitz property.
Using Lemma C.15 and Lemma C.16 and approximate Newton algorithm analysis in [20], then we complete the proof.
E Comparison with Parameter-Efficient Fine-tuning Methods Not Based on Prompt Tuning
In this section, we focus on the comparison between InfoPrompt  $(N_{p} = 4)$  and some PEFT (Parameter-Efficient Fine-tuning Methods) baselines which are not based on prompt tuning:
- Adapter [45]: Similar to prompt tuning, this is also a way of parameter-efficient training for pretrained language models. Specifically, instead of adding the prompt tokens in the input, we add adapters after the feed-forward module in each transformer layer.
Here  $\omega$  denotes the exponent of matrix multiplication. Currently  $\omega \approx 2.373$ .
Table 5: Comparison with parameter-efficient fine-tuning methods which are not based on prompt tuning. The number of prompt is fixed to  $n_p = 4$  for the soft prompt tuning method.
Full datasets	CoLA	RTE	MRPC	SST2	RE	NER	SemEval	Average
LoRA	0.5880	0.6715	0.8235	0.9541	0.6636	0.8228	0.7214	0.7492
Adapter	0.5552	0.5776	0.6814	0.9472	0.5073	0.8329	0.6570	0.6798
InfoPrompt	0.6018	0.6968	0.8137	0.9599	0.7616	0.8962	0.7917	0.7888
N=64	CoLA	RTE	MRPC	SST2	RE	NER	SemEval	Average
LoRA	0.0991	0.5596	0.6985	0.5677	0.1232	0.1345	0.1711	0.3362
Adapter	0.0627	0.5487	0.5931	0.4908	0.1086	0.2345	0.1211	0.3085
InfoPrompt	0.1567	0.6137	0.7059	0.6697	0.2119	0.3331	0.2113	0.4146
N=256	CoLA	RTE	MRPC	SST2	RE	NER	SemEval	Average
LoRA	0.2854	0.5740	0.7206	0.8222	0.2291	0.1955	0.3817	0.4583
Adapter	0.2486	0.5668	0.6250	0.6640	0.1815	0.2437	0.1770	0.3866
InfoPrompt	0.1750	0.6580	0.7377	0.7305	0.2993	0.4739	0.4034	0.4968
- LoRA [47]: Another parameter-efficient training method for pretrained language models. Specifically, LoRA adds additional low-rank decomposed matrices into each Transformer layer via residual connections.
In Table 5, we can observe that LoRA is a stronger baseline than Adapter. However, our method can still outperform LoRA, especially in the downstream tasks which require more task-relevant information, e.g., NER and RE.