SlowGuess/NAACL_2021_OCR huggingface dataset:
Note that some of the results listed in Table 3 are based on models that utilize additional resources. Zhang and Yang (2018) and Ma et al. (2020) utilized Chinese lexicon features to enrich the token representations. Meng et al. (2019) combined Chinese glyph information with BERT pre-training. In contrast, the proposed MCRF approach is simple yet performant. It achieves comparable or better results without relying on additional resources.
4.1.2 Results on Slot Filling
One of the main features of the AITS and SNIPS datasets is the large number of slot labels (79 and 39 respectively) with relatively small training set (4.5k and 13k respectively). This requires the sequence labeling model learn the transition rules in a sample-efficient manner. Both ATIS and SNIPS provide an intent label for each utterance in the datasets, but in our experiments we did not use this information and rely solely on the slot labels.
The results are reported in Table 5. It can be seen that MCRF-training outperforms the baseline models and achieves competitive results compared
to previous published results.
Model	ATIS	SNIPS
(Goo et al., 2018)	95.4	89.3
(Li et al., 2018)	96.5	-
(Zhang et al., 2019)	95.2	91.8
(E et al., 2019)	95.8	92.2
(Siddhant et al., 2019)	95.6	93.9
BERT-tags-retain	95.2 (92.9)	93.2 (92.1)
BERT-tags-discard	95.6 (93.1)	93.5 (92.3)
BERT-CRF-retain	95.5 (93.5)	94.6 (93.7)
BERT-CRF-discard	95.8 (93.9)	95.1 (94.3)
BERT-MCRF-decoding	95.8 (93.9)	95.1 (94.4)
BERT-MCRF-training	95.9 (94.4)	95.3 (94.6)
4.1.3 Results on Chunking
The results on CoNLL2000 chunking task are reported in Table. 6. The proposed MCRF-training outperforms the CRF baseline by 0.4 in F1-score.
Table 5: Test F1-scores on slot filling datasets.
Model	F1
ELMo (Peters et al., 2017)	96.4
CSE (Akbik et al., 2018)	96.7
GCDT (Liu et al., 2019)	97.3
BERT-tags-retain	96.1 (95.7)
BERT-tags-discard	96.3 (96.0)
BERT-CRF-retain	96.5 (96.2)
BERT-CRF-discard	96.6 (96.3)
BERT-MCRF-decoding	96.6 (96.4)
BERT-MCRF-training	96.9 (96.5)
Table 6: Results on CoNLL2000 chunking task.
In this section, we investigate the influence of various factors that may impact the performance of MCRF. In particular, we are interested in the quantity MCRF gain, which we denote by  $\Delta$ , defined simple as the difference of F1-score of MCRF-training and that of the conventional CRF (with either "retain" or "discard" strategy).
4.2.1 Effect of Tagging Scheme
In the previous experiments we have always used the BIO scheme. It is of interest to explore the performance of MCRF under other tagging schemes such as BIOES. The BIOES scheme is considered
Figure 4: Ablation over the tagging scheme (BIO vs. BIOES). The F1-scores on the dev sets are plotted.
more expressive than BIO as it introduces more labels and more transition restrictions.
We have re-run the experiments in Section 4.1.1 using the BIOES scheme. Our results are reported in Fig. 4 and Table 7. It is clearly seen that under the BIOES scheme MCRF still always outperforms the CRF baselines. Note that compared to the case under BIO scheme, the MCRF gain is less significant against the CRF-retain baseline, but larger against CRF-discard.
BIO	BIOES
Δret.	Δdisc.	Δret.	Δdisc.
Resume	2.1	0.3	1.0	0.6
MSRA	1.6	0.4	0.8	0.6
Ontonotes	1.5	0.3	0.9	0.6
Weibo	2.0	0.8	0.9	0.8
4.2.2 Effect of Sample Size
One may hypothesize that the occurrence of illegal paths might be due to the scarcity of training data, i.e. a model should be less prone to illegal paths if the training dataset is larger. To test this hypothesis, we randomly sample  $10\%$  of the training data from MSRA and Ontonotes, creating a smaller version of the respective dataset. We compare the proportion of the illegal segments produced by BERT-
CRF trained on the original dataset with the one trained on the smaller dataset. We also report the performance gain brought by MCRF in these two scenarios. Our findings are summarized in Table 8. As can be seen from the table, the models trained with fewer data do yield slightly more illegal segments, but the MCRF gains under the two scenarios are close.
Table 7: A comparison of the average MCRF gain under BIO and BIOES schemes. The symbols  $\Delta_{\mathrm{ret}}$  and  $\Delta_{\mathrm{disc}}$  stand for the gain against BERT-retain and BERT-discard, respectively.
MSRA-full	MSRA-10%
ill.	F1	Δ	ill.	F1	Δ
retain	1.8%	94.2	1.6	2.4%	90.4	1.2
discard	-	95.4	0.5	-	90.7	0.9
MCRF	0%	95.8	-	0%	91.6	-
Ontonotes-full	Ontonotes-10%
ill.	F1	Δ	ill.	F1	Δ
retain	4.2%	79.2	1.6	4.7%	78.7	1.2
discard	-	80.4	0.4	-	79.1	0.8
MCRF	0%	80.8	-	0%	79.9	-
Table 8: Ablation over the training set size. The column named "ill." indicates the proportion of illegal segments over all predicted segments.
4.2.3 Effect of Encoder Architecture
So far we have experimented with BERT-based models. Now we explore effect of neural architecture. We trained a number of models on CoNLL2003 with varying encoder architectures. The key components are listed as follows:
- ELMo: pretrained language model<sup>4</sup> that serves as an sequence encoder.
- CNN: CNN-based character embedding layer, with weights extracted from pretrained ELMo. It is used to generate word embeddings for arbitrary input tokens.
- LSTM- $n$ :  $n$ -layer bidirectional LSTM with hidden dimension  $h = 200$ .
The results of our experiments are given in Table 9. We observe that the encoder architecture has a large impact on the occurrence of illegal paths, and the BERT-based models appear to generate much more illegal paths than ELMo-based ones. This is probably due to the fact that transformer-encoders are not sequential in nature. A further study is needed to investigate this phenomenon, but it is beyond the scope of the current work. We also notice that
4Model downloaded from https://github.com/allenai/bilm-tf
the MCRF gain seems to be positively correlated with the proportion of the illegal paths generated by the underlying model. This is expected, since the transition-blocking mechanism of MCRF will (almost) not take effect if the most probable path estimated by the underlying CRF model is already legal.
Encoder	ill.	err.	CRF	MCRF	Δ
LSTM-1	3.1%	11.7%	82.2	83.2	1.0
LSTM-2	1.4%	8.3%	84.3	85.1	0.8
CNN + LSTM-1	0.4%	4.0%	94.1	94.3	0.2
CNN + LSTM-2	0.3%	2.3%	94.0	94.5	0.5
ELMo + LSTM-1	0.4%	3.3%	95.1	95.3	0.2
ELMo + LSTM-2	0.6%	5.5%	95.0	95.3	0.3
BERT	1.3%	12.5%	94.5	95.4	0.9
BERT + LSTM-1	1.0%	13.1%	94.7	95.3	0.6
BERT + LSTM-2	0.9%	10.3%	93.9	95.0	1.1
Table 9: Ablation over the encoder models. The column named "err." indicates the proportion of erroneous predictions that are due to illegal segments.
4.3 Related Work
Some models are able to solve sequence labeling tasks without relying on BIO/BIOES type of tagging scheme to distinguish the boundary and the type of the text segments of interest, thus do not suffer from the illegal path problems. For instance, Semi-Markov CRF (Sarawagi and Cohen, 2005) uses an additional loop to search for the segment spans, and directly yields a sequence of segments along with their type. The downside of Semi-Markov CRF is that it incurs a higher time complexity compared to the conventional CRF approach. Recently, Li et al. (2020b) proposed a Machine Learning Comprehension (MRC) framework to solve NER tasks. Their model uses two separate binary classifiers to predict whether each token is the start or end of an entity. They introduced an additional module to determine which start and end tokens should be matched.
We notice that the CRF implemented in PyTorch-Struct (Rush, 2020) has a different interface than usual CRF libraries in that it takes not two tensors for emission and transition scores, but rather one score tensor of the shape (batch size, sentence length, number of tags, number of tags). This allows one to incorporate even more prior knowledge in the structured prediction by setting a constraint mask as a function of not only a pair of tags, but also words on which the tags are assigned. Such
feature may be exploited in future work.
Finally, we acknowledge that the naive version of MCRF that does constrained decoding has already been implemented in AllenNLP $^{5}$  (Gardner et al., 2018). As shown in Section 4.1, such approach is suboptimal compared to the proposed MCRF-training method.
Our major contribution is the proposal of MCRF, a constrained variant of CRF that masks illegal transitions during CRF training, eliminating illegal outcomes in a principled way.
We have justified MCRF from a theoretical perspective, and shown empirically in a number of datasets that MCRF consistently outperforms the conventional CRF. As MCRF is easy to implement and incurs zero additional overhead, we advocate always using MCRF instead of CRF when applicable.
We thank all anonymous reviewers for their valuable comments. We also thank Qin Bin and Wang Gang for their support. This work is also supported by the National Natural Science Foundation of China (NSFC No. 61701547).
https://github.com/allenai/allennlp/blob/main/allennlp/modules/conditional_random_field.py
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Leonard E. Baum and Ted Petrie. 1966. Statistical inference for probabilistic functions of finite state markov chains. The Annals of Mathematical Statistics, 37(6):1554-1563.
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The journal of machine learning research, 12(null):2493-2537.
Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5467-5471, Florence, Italy. Association for Computational Linguistics.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and YunNung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 753-757, New Orleans, Louisiana. Association for Computational Linguistics.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv:1508.01991 [cs]. 00487 arXiv:1508.01991.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, page 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sydney, Australia. Association for Computational Linguistics.
Changliang Li, Liang Li, and Ji Qi. 2018. A self-attentive model with gate mechanism for spoken language understanding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3824–3833, Brussels, Belgium. Association for Computational Linguistics.
Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020a. FLAT: Chinese NER using flat-lattice transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6836-6842, Online. Association for Computational Linguistics.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020b. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859, Online. Association for Computational Linguistics.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020c. Dice loss for data-imbalanced nlp tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465-476.
Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. GCDT: A global context enhanced deep transition architecture for sequence labeling. In Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 2431-2441, Florence, Italy. Association for Computational Linguistics.
Ruotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2020. Simplify the usage of lexicon in Chinese NER. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5951-5960, Online. Association for Computational Linguistics.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for Chinese character representations. In Advances in Neural Information Processing Systems 32, pages 2746-2757.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 548-554, Lisbon, Portugal. Association for Computational Linguistics.
Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365 [cs]. 00277 arXiv: 1802.05365.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 335–342, Online. Association for Computational Linguistics.
Tjong Kim Sang, Erik F., and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop.
Sunita Sarawagi and William W Cohen. 2005. Semimarkov conditional random fields for information extraction. In Advances in Neural Information Processing Systems, volume 17. MIT Press.
Aditya Siddhant, Anuj Goyal, and Angeliki Metallinou. 2019. Unsupervised transfer learning for spoken language understanding in intelligent agents. In AAAI.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, and Robert Belvin. 2011. Ontonotes release 4.0.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5259-5267, Florence, Italy. Association for Computational Linguistics.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1554-1564, Melbourne, Australia. Association for Computational Linguistics.
Denote by  $L$  and  $L^{\prime}$  the likelihood function of sample  $(x,y)$  for CRF and MCRF model respectively:
$$
L (W, A) = \frac {\exp s (y , x , W , A)}{\sum_ {p \in \mathcal {P}} \exp s (p , x , W , A)}, \tag {10}
$$
$$
L ^ {\prime} (W, A) = \frac {\exp s (y , x , W , A)}{\sum_ {p \in \mathcal {P} / \mathcal {I}} \exp s (p , x , W , A)}. \tag {11}
$$
To simplify the notations, we also write
$$
L (W, A) = \frac {N (W , A)}{D (W , A)}, \quad L ^ {\prime} (W, A) = \frac {N (W , A)}{D ^ {\prime} (W , A)},
$$
$$
N (W, A) = \exp s (y, x, W, A)
$$
$$
D (W, A) = \sum_ {p \in \mathcal {P}} \exp s (p, x, W, A)
$$
$$
D ^ {\prime} (W, A) = \sum_ {p \in \mathcal {P} / \mathcal {I}} \exp s (p, x, W, A)
$$
Proposition 1 is a direct corollary of the following result:
Lemma 2. Let  $(x,y)$  be a sample with  $y\in \mathcal{P} / \mathcal{I}$ . Then for arbitrary  $(W_0,A_0)$ , we have
$$
\lim  _ {c \rightarrow - \infty} L \left(W _ {0}, \bar {A} _ {0} (c)\right) = L ^ {\prime} \left(W _ {0}, A _ {0}\right) \tag {12}
$$
$$
\lim  _ {c \rightarrow - \infty} \nabla_ {W} L \left(W _ {0}, \bar {A} _ {0} (c)\right) = \nabla_ {W} L ^ {\prime} \left(W _ {0}, A _ {0}\right) \tag {13}
$$
$$
\lim  _ {c \rightarrow - \infty} \nabla_ {a _ {i j}} L \left(W _ {0}, \bar {A} _ {0} (c)\right) = \nabla_ {a _ {i j}} L ^ {\prime} \left(W _ {0}, A _ {0}\right). \tag {14}
$$
$$
s (p, x, W, A) = \sum_ {i = 1} ^ {T} l _ {i, n _ {i}} + \sum_ {i = 1} ^ {T - 1} a _ {n _ {i}, n _ {i + 1}}, \tag {15}
$$
and the masked transition matrix  $\bar{A} (c) = \bigl (\bar{a}_{ij}(c)\bigr)$  is defined as
$$
\bar {a} _ {i j} (c) = \left\{ \begin{array}{l l} c & \text {i f} (i, j) \in \Omega , \\ a _ {i j} & \text {o t h e r w i s e}, \end{array} \right. \tag {16}
$$
where  $\Omega$  is the set of illegal transitions.
Since  $\bar{A}(c)$  differs from  $A$  only on entries corresponding to illegal transitions and a legal path contains only legal transitions, it follows from (15) that  $\forall p \in \mathcal{P} / \mathcal{I}$
$$
s (p, x, W _ {0}, \bar {A} _ {0} (c)) = s (p, x, W _ {0}, A _ {0}). \tag {17}
$$
$$
N \left(W _ {0}, \bar {A} _ {0} (c)\right) = N \left(W _ {0}, A _ {0}\right). \tag {18}
$$
Next, we show
$$
D ^ {\prime} \left(W _ {0}, \bar {A} _ {0} (c)\right) \xrightarrow [ c \rightarrow - \infty ]{} D \left(W _ {0}, A _ {0}\right). \tag {19}
$$
By (10) (11) and (17), it suffices to demonstrate for any illegal path  $p \in \mathcal{I}$
$$
\lim  _ {c \rightarrow - \infty} \exp s (p, x, W _ {0}, \bar {A} _ {0} (c)) = 0. \tag {20}
$$
To achieve this, we rewrite  $s(p,x,W_0,\bar{A}_0(c))$  as a product of three terms:
$$
\begin{array}{l} \exp s (p, x, W _ {0}, \bar {A} _ {0} (c)) \\ = \prod_{i = 1}^{T}e^{l_{i,n_{i}}}\prod_{\substack{(i,j)\in \mathcal{T} / \Omega \\ (i,j)\sim p}\\ \text{legal transitions}}}\hskip -14.226378pt e^{\bar{a}_{ij}(c)}\prod_{\substack{(i,j)\in \Omega \\ (i,j)\sim p\\ \text{illegal transitions}}}e^{\bar{a}_{ij}(c)} \\ \end{array}
$$
where  $(i,j)\sim p$  means that  $(i,j)$  is a transition contained in path  $p$ . Let  $E(p)$  be the number of illegal transitions in  $p$ . If  $p$  is illegal, then  $E(p) > 0$ ; otherwise  $E(p) = 0$ . Since  $\bar{a}_{ij}(c) = c$  for  $(i,j)\in \Omega$  by definition,
$$
\begin{array}{l} \exp s (p, x, W _ {0}, \bar {A} _ {0} (c)) \\ = \Big(\prod_{i = 1}^{T}e^{l_{i,n_{i}}}\prod_{\substack{(i,j)\in \mathcal{T} / \Omega \\ (i,j)\sim p}}e^{a_{ij}}\Big)\cdot e^{cE(p)}. \\ \end{array}
$$
Now that the terms in the parenthesis do not depend on  $c$  and  $e^{cE(p)}$  vanish as  $c \to -\infty$ , we achieve (20). Then (12) of Lemma 2 is proved.
Now we turn to the proof of (13). By elementary calculus we have
$$
\begin{array}{l} \nabla_ {W} L = \left(D \cdot \nabla_ {W} N - N \cdot \nabla_ {W} D\right) \cdot D ^ {- 2} \\ \nabla_ {W} L ^ {\prime} = \left(D ^ {\prime} \cdot \nabla_ {W} N - N \cdot \nabla_ {W} D ^ {\prime}\right) \cdot D ^ {\prime - 2}. \\ \end{array}
$$
By (18) and (19), it remains to show
$$
\nabla_ {W} D ^ {\prime} \left(W _ {0}, \bar {A} _ {0} (c)\right) \xrightarrow [ c \rightarrow - \infty ]{} \nabla_ {W} D \left(W _ {0}, A _ {0}\right). \tag {21}
$$
By the same argument as in the proof of (17) and (20), it is easily seen that for  $p \in \mathcal{P} / \mathcal{I}$
$$
\begin{array}{l} \nabla_ {W} \Big (\exp s (p, x, W, A) \Big) \big | _ {W _ {0}, A _ {0}} \\ = \nabla_ {W} \Big (\exp s (p, x, W, A) \Big) \Big | _ {W _ {0}, \bar {A} _ {0} (c)} \\ \end{array}
$$
and for  $p\in \mathcal{I}$
$$
\nabla_ {W} \Big (\exp s (p, x, W, A) \Big) \Big | _ {W _ {0}, \bar {A} _ {0} (c)} \xrightarrow {c \to - \infty} 0.
$$
Thus (21) is achieved and (13) follows.
Finally, the proof of (14) is similar to that of (13).