SlowGuess/NAACL_2022_OCR huggingface dataset:
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In Proceedings of ICLR.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul):2121-2159.
J. Elman. 1990. Finding structure in time. Cogn. Sci., 14:179-211.
Melikasadat Emami, Mojtaba Sahraee-Ardakan, Parthe Pandit, Sundeep Rangan, and Alyson K Fletcher. 2021. Implicit bias of linear rnns. In Proceedings of ICML.
Rémi Eyraud and Stéphane Ayache. 2020. Distillation of weighted automata from recurrent neural networks using a spectral approach. https://arxiv.org/abs/2009.13101.
Gottlob Frege. 1948. Sense and reference. The philosophical review, 57(3):209-230.
Surya Ganguli, Dongsung Huh, and Haim Sompolinsky. 2008. Memory traces in dynamical systems. Proceedings of the National Academy of Sciences, 105:18970 - 18975.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2018. Improving neural language models with a continuous cache. In Proceedings of ICLR.
Pankaj Gupta and Hinrich Schütze. 2018. LISA: Explaining recurrent neural network judgments via layer-wIse semantic accumulation and example to pattern transformation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019. Modeling recurrence for transformer. In Proceedings of NAACL.
Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146-162.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Šéaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multiway classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent transformers. https://arxiv.org/abs/2203.07852.
Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, et al. 2017. Efficient softmax approximation for gpus. In Proceedings of ICML.
Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2nd Edition). Prentice-Hall, Inc., USA.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality.
Sekitoshi Kanai, Yasuhiro Fujiwara, and Sotetsu Iwamura. 2017. Preventing gradient explosions in gated recurrent units. In Proceedings of NeurIPS.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. http://arxiv.org/abs/1506.02078.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of NeurIPS.
Gérard Lallement. 1979. Semigroups and combinatorial applications. John Wiley & Sons, Inc.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL.
Tao Lei. 2021. When attention meets fast recurrence: Training language models with reduced compute. In Proceedings of EMNLP.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of NAACL.
Jiwei Li, Thang Luong, and Dan Jurafsky. 2015a. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of ACL.
Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015b. When are tree structures necessary for deep learning of representations? In Proceedings of EMNLP.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of NAACL.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of NAACL.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL.
Niru Maheswaranathan and David Sussillo. 2020. How recurrent networks implement contextual processing in sentiment analysis. In Proceedings of ICML.
Niru Maheswaranathan, Alex H. Williams, Matthew D. Golub, S. Ganguli, and David Sussillo. 2019. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Proceedings of NeurIPS.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. https://arxiv.org/abs/1609.07843.
William Merrill. 2019. Sequential neural networks as automata. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges.
William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. 2020. A formal hierarchy of RNN architectures. In Proceedings of ACL.
Jose Meseguer and Ugo Montanari. 1990. Petri nets are monoids. Information and computation, 88(2):105-155.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS.
Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In Proceedings of ICLR.
Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of ICML.
W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In Proceedings of ICLR.
W. James Murdoch and Arthur Szlam. 2017. Automatic rule extraction from long short term memory networks. In Proceedings of ICLR.
Emin Orhan and Xaq Pitkow. 2020. Improved memory in recurrent neural networks with sequential nonnormal dynamics. In Proceedings of ICLR.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP.
Guillaume Rabusseau, Tianyu Li, and Doina Precup. 2019. Connecting weighted automata and recurrent neural networks through spectral learning. In Proceedings of AISTATS.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of ACL.
Grzegorz Rozenberg and Arto Salomaa. 2012. Handbook of Formal Languages: Volume 3 Beyond Words. Springer Science & Business Media.
Sebastian Rudolph and Eugenie Giesbrecht. 2010. Compositional matrix-space models of language. In Proceedings of ACL.
Nathan Schneider, Emily Danchik, Chris Dyer, and Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. Transactions of the Association for Computational Linguistics, 2:193-206.
Chandan Singh, W James Murdoch, and Bin Yu. 2019. Hierarchical interpretations for neural network predictions. In Proceedings of ICLR.
Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of ENMLP.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Xiaobing Sun and Wei Lu. 2020. Understanding attention for text classification. In Proceedings of ACL.
Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL.
Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of EMNLP.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2605.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of ACL.
Ronald J. Williams and Jing Peng. 1990. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 2(4):490-501.
Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. 2021. Self-attention networks can process bounded hierarchical languages. In Proceedings of ACL-IJCNLP.
Ainur Yessenalina and Claire Cardie. 2011. Compositional matrix-space models for sentiment analysis. In Proceedings of EMNLP.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NeurIPS.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of ACL.
The statistics of the sentiment analysis, relation classification and NER datasets are shown in Table 6. The language modeling datasets are obtained from Einstein.ai and the statistics are shown in Table 7.
Data	Train	Dev	Test	V.size	Max.len	Class
SST-2	98,794	872	1,821	17,404	54	2
IMDB	17,212	4,304	4,363	63,311	437	2
AG-news	110,000	10,000	7,600	85,568	212	4
SST-5	318,582	41,447	82,600	18,025	54	5
SemEval	7,000	1,000	2,717	27,115	91	10
CoNLL-2003	14,987	3,466	3,684	26,873	113	20
Table 6: Statistics of the sentiment analysis, relation classification and NER datasets. "V.size" indicates the vocabulary size and "Max.len" indicates the maximum length of the instances. "SemEval" refers to the SemEval 2010 Task 8 dataset for relation classification. For CoNLL-2003, "class" refers to the tag size.
We created the binary dataset SST-2 by extracting instances (including phrases) with polarity from the constituency parse trees in the original SST dataset (Socher et al., 2013). We merged the labels extremely positive and positive as positive and the labels extremely negative and negative as negative. We also extracted all the phrases in the constituency parse trees from the original dataset and created the 5-class dataset SST-5. The labels extremely positive, positive, neutral, negative and extremely negative were mapped into  $+2$ ,  $+1$ , 0, -1, and -2 respectively.
B More Result from the SST datasets
B.1 Negation and Intensification
Figure 4 shows that the  $n$ -gram representations from the LSTM model together with its corresponding MVMA-L and MVM-L models can also capture negation on the extracted adjectives from SST-2. However, VA-EW fails to capture the negation phenomenon for the negative adjectives, which may be explained by that: the  $n$ -gram representation of VA-EW solely involves the current token, thus being less expressive compared to the one from models such as MVMA-L and MVMA-G. More
Dataset	Train	Dev	Test
PTB	Token Num	887,521	70,390	78,669
Vocab Size	10,000
Wiki2	Token Num	2,088,628	217,646	245,569
Vocab Size	33,278
Wiki103	Token Num	103,227,021	217,646	245,569
Vocab Size	267,735
Table 7: Statistics of the language modeling datasets.
(a) VA-EW
(b)  $\mathrm{LSTM}_{n\text{-gram}}$
(c) MVM-L
(d) MVMA-L
Figure 4: Distribution of polarity scores for adjectives and their negation bigrams.  $p$ -adj and  $n$ -adj refer to the positive and negative adjectives respectively. [-] refers to the negation operation (prepending the word "not"). Circles refer to outliers.
(a) GRUcontext, positive adjectives
(b) GRUcontext, negative adjectives
Figure 5: Context polarity scores for positive adjectives (a) and negative adjectives (b) along with their corresponding negation and intensification bigrams from SST-5.
over, the MVMA-G model can also capture the negation and intensification phenomena on SST-5 as shown in Figure 5. The intensification token will generally strengthen the polarity of an adjective while the negation token will generally weaken the polarity of it.
We also visualized the polarity score of each  $n$ -gram within a sentence. Two examples are shown in Figures 6a and 6b, where a warmer color indicates a higher polarity score (i.e., the  $n$ -gram is more positive). For example, "never" itself has a remarkably negative polarity score while "loses" has a remarkably positive one. Consequently, the  $n$ -grams starting from "never" (while ending with another word) generally have positive polarity scores.
(a)
(b)
Figure 6: Polarity scores for  $n$ -gram representations within two example sentences. SST-2, MVMA-G.
Such visualization results show that our identified representations defined over the linguistic units as captured by RNNs can be highly interpretable.
B.2 First-order Approximation
To examine how well the recurrence relation in Equation 7 can approximate the standard RNNs, we followed the method in the work of Maheswaranathan and Sussillo (2020) and compared the hidden state of the standard RNNs  $(\pmb{h}_t = RNN(\pmb{x}_t, \pmb{h}_{t-1}))$  at each time step to the corresponding context representations  $(\hat{\pmb{h}}_t = \pmb{g}(\pmb{x}_t) + \pmb{A}(\pmb{x}_t)\pmb{h}_{t-1})$ . The error at each time step is defined as
$$
\left| \left| \boldsymbol {h} _ {t} - \hat {\boldsymbol {h}} _ {t} \right| \right| _ {2} / \left| \left| \boldsymbol {h} _ {t} \right| \right| _ {2}. \tag {15}
$$
We used the current standard hidden state to predict the next hidden state and the context representations on the SST-2 test set.
We noticed that the weight decaying coefficient has a remarkable impact on the error. Specifically, a larger coefficient can result in smaller errors. When the coefficient is  $1e - 5$ , the average errors on the Elman, GRU, and LSTM models were  $26.2\%$ ,  $21.7\%$  and  $46.6\%$  and respectively. When the coefficient is  $3e - 4$  the average errors dropped to  $17.1\%$ ,  $15.1\%$ , and  $33.3\%$  respectively. Note that since this is the single step error, the accumulated errors across many times steps can be large, particularly for LSTM, and thus the first-order approximation cannot fully replace standard RNNs. Despite this,
(a) MVMA-G
(b) MVM-G
Figure 7: (a) and (b): T-sne visualization of the context representation for phrases (<30 tokens) from the AGnews dataset with four topics.
the resulting context and  $n$ -gram representations can help us understand how RNNs process contextual information such as  $n$ -gram features.
C T-sne Visualization
We visualized the context representations from the MVMA-G model using t-sne (van der Maaten and Hinton, 2008), which provides us with an intuitive understanding on the efficacy of our identified representations. We automatically extracted 2,188 phrases with less than 30 tokens from AG-news with 4 topics<sup>17</sup> and projected their context representations to a two-dimension space. Figures 7a and 7b show there exist four major clusters corresponding to the four topics, indicating those representations can generally learn the topic information and explain the differences. Similar to the previous analysis, the MVM-G model is able to learn the topic information with the  $n$ -gram representations.
D Results on Transformer
We have also run the Transformer model on the sentiment analysis datasets and the results are listed in Table 8.
17Although SST-5 has 5 lables, most of its phrases are neutral, we therefore did not use this dataset for visualization.
SST-2	AG-news	IMDB
dev	test	dev	test	dev	test
83.4±0.4	82.0±0.1	90.9±0.5	90.5±0.4	88.4±0.2	88.1±0.2
Table 8: Accuracy on sentiment analysis tasks. Transformer
E.1 Sentiment Analysis
Settings For the SST-2, AG-news, and IMDB datasets, we used the cross-entropy as the loss function to train the models. Embeddings were randomly initialized and trainable during training. For the SST-5 dataset, we treated the classification as a regression problem as the labels are polarity intensity. The mean-squared error was used as the loss function during training. Note that we initialized embeddings with pre-trained GloVe (Pennington et al., 2014) and fixed them during training on SST-5 for the analysis of both the negation and intensification phenomena.
Furthermore, for the MM model, each token was represented as a matrix and the matrix size was set as  $32 \times 32$ . For the other models, the embedding and hidden sizes were both set as 300.
Polarity Adjectives We automatically extracted adjectives with polarity (examples shown in Table 9) from SST-2 in two steps. In the first step, following the method of Sun and Lu (2020), we calculated a frequency ratio for each token (in the vocabulary) between the frequencies of the token seen in the positive and negative instances respectively. If a token has a frequency ratio either larger than 3 or less than  $1/3$ , it will be extracted as an positive token or an negative token. In the second step, we used the textblob package to detect positive and negative adjectives from those positive tokens and negative tokens respectively.
E.2 Relation Classification
Following the work of Gupta and Schütze (2018), we examined the RNN, baseline, MVMA and MVM models on SemEval 2010 Task 8 (Hendrickx et al., 2010) which has 9 directed relationships and an undirected other type. We used the final hidden states of the standard RNNs (or context representations of the MVMA, MVM and baseline models) as the instance representations for classification. The cross-entropy loss was employed during training.
E.3 Named Entity Recognition
At each time step, we concatenated the context representations (or hidden states) from both directions in a bidirectional model, fed them to a projection layer and then to a linear CRF layer. More details about the architecture can be referred to the biLSTM-CRF model in the work of Lample et al. (2016). We also referred to the code at https://github.com/allanj/pytorch_neural_crf for the implementation of the linear CRF layer.
CoNLL-2003 contains four types of entities: persons (PER), organizations (ORG), locations (LOC) and miscellaneous names (MISC). The original dataset was labeled with the BIO (Beginning-Inside-Outside) format. For example, "United Arab Emirates" are labeled as "B-LOC I-LOC I-LOC". We transform the tags into the IOBES format where two prefixes "E-" and "S-" are added. Specifically, "E-" is used to label the last token of an entity span. The "S-" prefix is used for a single-token span. For example, "United Arab Emirates" are labeled as "B-LOC I-LOC E-LOC" in this format. There are 20 categories of tags in total including the starting, ending and padding tags. We trained the models to predict each entity.
The embedding size and hidden size were set to 300 and 200 respectively. The SGD optimizer was used to learn parameters.
E.4 Language Modeling
The embedding size and hidden size were both 512 for PTB and Wiki2, and 256 and 512 respectively for Wiki103. The cross-entropy loss was used during training. For PTB and Wiki2, the output of the final fully-connected layer was fed to a softmax function while the Adaptive softmax (Joulin et al., 2017) was used for Wiki103 (because of its large vocabulary size). We only considered the word-level models. We trained each model for 50 epochs, chose the model which had the best performance on the development set as the final model and evaluated the final model on the test set.
F Jacobian matrix of LSTM
Unlike GRU and Elman RNN, LSTM has a memory cell apart from a hidden state. Here, we describe how to get their Jacobian matrices. An
<sup>18</sup>https://textblob.readthedocs.io/en/dev/
Type	Adjectives	Size
Pos	outstanding, ecological, inventive, comfortable, nice, authentic, spontaneous, sympathetic, lovable, unadulterated, controversial, suitable, grand, happy, enthusiastic, adventurous, successful, noble, true, detailed, sophisticated, sensational, exotic, fantastic, remarkable, impressive, charismatic, good, effective, rich, popular, unforgettable, famous, comical, energetic, ingenious, extraordinary, ...	73
Neg	bad, tedious, miserable, psychotic, didactic, inexplicable, feeble, sloppy, disastrous, stupid, amateurish, false, cynical, farcical, terrible, unhappy, horrible, atrocious, idiotic, wrong, pathetic, angry, uninspired, vicious, unfocused, unnecessary, artificial, troubled, questionable, arduous, stereotypical, ...	47
Table 9: Examples of the extracted adjectives from the SST-2 dataset. "Pos" refers to positive adjectives and "Neg" refers to negative adjectives.
LSTM cell can be written as
$$
\boldsymbol {i} _ {t} = \sigma \left(\boldsymbol {W} _ {i i} \boldsymbol {x} _ {t} + \boldsymbol {W} _ {h i} \boldsymbol {h} _ {t - 1}\right),
$$
$$
\boldsymbol {f} _ {t} = \sigma \left(\boldsymbol {W} _ {i f} \boldsymbol {x} _ {t} + \boldsymbol {W} _ {h f} \boldsymbol {h} _ {t - 1}\right),
$$
$$
\boldsymbol {o} _ {t} = \sigma \left(\boldsymbol {W} _ {i o} \boldsymbol {x} _ {t} + \boldsymbol {W} _ {h o} \boldsymbol {h} _ {t - 1}\right), \tag {16}
$$
$$
\boldsymbol {c} _ {t} ^ {m} = \tanh  (\boldsymbol {W} _ {i c} \boldsymbol {x} _ {t} + \boldsymbol {W} _ {h c} \boldsymbol {h} _ {t - 1}),
$$
$$
\boldsymbol {c} _ {t} = \boldsymbol {f} _ {t} \odot \boldsymbol {c} _ {t - 1} + \boldsymbol {i} _ {t} \odot \boldsymbol {c} _ {t} ^ {m}, \boldsymbol {h} _ {t} = \boldsymbol {o} _ {t} \odot \tanh  (\boldsymbol {c} _ {t}),
$$
where  $\pmb{i}_t, \pmb{f}_t, \pmb{o}_t \in \mathbb{R}^d$  are the input gate, forget gate and output gate respectively.  $c_t^m \in \mathbb{R}^d$  is the new memory, and  $c_t$  is the final memory.
Let us expand the memory state and hidden state at time step  $t$  as
$$
\begin{array}{l} \boldsymbol {c} _ {t} = \boldsymbol {g} _ {c} (\boldsymbol {x} _ {t}) + B (\boldsymbol {x} _ {t}) \boldsymbol {c} _ {t - 1} \\ + D \left(\boldsymbol {x} _ {t}\right) \boldsymbol {h} _ {t - 1} + \boldsymbol {o} _ {c} \left(\boldsymbol {c} _ {t - 1}, \boldsymbol {h} _ {t - 1}\right), \tag {17} \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {h} _ {t} = \boldsymbol {g} _ {h} (\boldsymbol {x} _ {t}) + E (\boldsymbol {x} _ {t}) \boldsymbol {c} _ {t - 1} \\ + F \left(\boldsymbol {x} _ {t}\right) \boldsymbol {h} _ {t - 1} + \boldsymbol {o} _ {h} \left(\boldsymbol {c} _ {t - 1}, \boldsymbol {h} _ {t - 1}\right), \\ \end{array}
$$