SlowGuess/NAACL_2022_OCR huggingface dataset:
where  $B, D, E$  and  $F \in \mathbb{R}^{d \times d}$  are all Jacobian matrices.  $\pmb{o}_c(\pmb{h}_{t-1})$  and  $\pmb{o}_h(\pmb{h}_{t-1})$  are remainder terms of the Taylor expansion.
We concatenate the memory state and hidden state and view the concatenation as an "extended hidden state". The context representation for the "extended hidden state" at time step  $t$  (assuming of zero vectors as initial states) will be written as:
$$
\left[ \begin{array}{c} \hat {\boldsymbol {c}} _ {t} \\ \hat {\boldsymbol {h}} _ {t} \end{array} \right] = \sum_ {i = 1} ^ {t} \left[ \begin{array}{l} \boldsymbol {v} _ {i: t} ^ {c} \\ \boldsymbol {v} _ {i: t} ^ {h} \end{array} \right] = \sum_ {i = 1} ^ {t} \left[ \prod_ {k = t} ^ {i + 1} A \left(x _ {k}\right) \right] \left[ \begin{array}{l} g _ {c} \left(\boldsymbol {x} _ {i}\right) \\ g _ {h} \left(\boldsymbol {x} _ {i}\right) \end{array} \right], \tag {18}
$$
where  $\hat{\pmb{c}}_t$  and  $\hat{\pmb{h}}_t$  refer to the context representations corresponding to the memory state and hidden state respectively.  $g_{c}, g_{h} \in \mathbb{R}^{d}$ , and  $A \in \mathbb{R}^{2d \times 2d}$  are all functions of inputs.  $A(x_{k})$  contains many interaction terms resulting from the gating mechanism, which may result in a strong expressive power. As the hidden state  $h_t$  is commonly used for downstream tasks, we will only consider  $\pmb{v}_{i:t}^{h}$  as the  $n$ -gram representation on our tasks, and the context representation will be  $\sum_{i=1}^{t} \pmb{v}_{i:t}^{h}$ .