SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Multi-domain translation. Boldface identifies best system for each domain.
Model / Domain	LAW	LAW1	LAW2
Adapter [+151m]	61.0	60.4 (-0.6)	60.2 (-0.8)
LaMGD Transformer [+0m]	60.4	60.4 (=)	60.4 (=)
Table 4: Experiments with two similar pseudo-domains
O2M-related	AZE	BEL	CES	GLG	POR	RUS	SLK	TUR	AVG
Transformer [91.6m]	4.8	7.3	20.8	21.1	39.7	19.8	22.6	15.2	18.9
Adapter [+13m]	4.3	6.8	21.1	22	39.7	20	23	15.2	19
LaMGD Transformer [+0m]	5.2	9.4	20.6	22.8	39.6	19.6	22.4	15.0	19.3
M2O-related	AZE	BEL	CES	GLG	POR	RUS	SLK	TUR	AVG
Transformer [67.8m]	11.4	16.6	28.5	27.1	43.7	24.6	30.3	25.6	26.0
Adapter [+13m]	10.1	15.8	28.4	26.8	43.7	24.5	30.2	25.6	25.6
LaMGD Transformer [+0m]	11.3	17.4	28.6	28.7	43.7	24.5	30.7	25.6	26.3
O2M-diverse	BOS	MAR	HIN	MKD	ELL	BUL	FRA	KOR	AVG
Transformer [96.9m]	10.2	4	12.7	22.2	31.8	34.0	38.3	8.3	20.2
Adapter [+13m]	10.2	4	13.3	21.9	32.2	34.1	38.5	8.3	20.3
LaMGD Transformer [+0m]	10.1	3.8	12.6	22.8	31.8	33.4	38.1	8.1	20.1
M2O-diverse	BOS	MAR	HIN	MKD	ELL	BUL	FRA	KOR	AVG
Transformer [70.4m]	22.4	9.7	20.5	31.8	37.5	38.7	39.8	19.0	27.4
Adapter [+13m]	22.5	9.4	20.0	30.6	37.2	38.2	39.3	19.0	27.0
LaMGD Transformer [+0m]	23.5	9.6	21.5	32.2	37.7	38.6	40.0	18.9	27.7
Table 5: Multilingual Translation experiments. Boldface denotes significant gains over Transformer  $(p = 0.05)$
space using Principal Component Analysis (PCA).
For multilingual (TED-related) experiments, the training data contains four language families: (1) Turkic, with Azerbaijani and Turkish(AZE,TUR); (2) Slavic, with Belarusian and Russian (BEL,RUS); (3) Romance, with Galician and
Portuguese (GLG, POR); and (4) Czech-Slovak, with Slovak and Czech (CES, SLK). We provide in Figure 2b the heatmap of the similarities between the dropout masks of our objective languages. We observe that each pair of languages in the same family correspond to brightest color except the di
(a) Multidomain
(b) Multilingual (Related)
Figure 2: Heatmap visualization of the similarities between dropout masks of domains(languages).
agonal in every column or every row.
We also plot the languages based on their dropout masks in Figure 3 using a 2d PCA projection.
We discuss here the choice of the hyper-parameters  $k$ , the number of activated nodes in each layer, and its impact on the sharing level between the tasks. Table 7 shows the variance of performance when the number of activated nodes is changed, and the sharing level between tasks decreases in consequence. In addition, we also report in this sec
k	AVG	sharing rate
8	18.1	0.63
10	19.15	0.73
12	19.33	0.78
14	19.44	0.88
tion the effect of not choosing the number of groups  $n_p$ , which is assigned to 16 in the comparison of LaMGD and the contrasting methods. We show that setting  $n_p$  to the layer's size, which means the group size is 1, has a very similar performance as
choosing  $n_p$  heuristically.
Table 6: Variation of the performance w.r.t  $k$  ,while we fix  ${n}_{p} = {16}$  (o2m-related experiment).
k/np	AVG
12 / 16	19.33
384 / 512	19.26
Table 7: Setting the size of group to 1 (o2m-related experiment). The quota of activated nodes is keep unchanged to  $75\%$
Multidomain and multilingual translation systems have received considerable attention in the recent years, and a exhaustive survey is beyond the goal of this paper. Domain adaptation for neural MT is surveyed in (Chu et al., 2017), while multidomain MT systems are notably studied in (Saunders, 2021; Pham et al., 2021); for multilingual MT, the reader is referred eg. to (Chu and Dabre, 2018; Dabre et al., 2020). We focus on the most relevant subset of this literature below.
Language similarity The methods developed by (Sen et al., 2019; Kong et al., 2021) use language proximity to design parameter sharing strategies. The authors propose a multi-decoder model sharing the same encoder among languages and routing languages in different families to different decoders. These approaches share the same interest in expressing the proximity between tasks in the selection of task-specific parameters as our approach. However, our method learn the selection from a latent commonality in data instead of using a predefined selection such as "One language family per decoder" in (Kong et al., 2021).
Language-specific sub-networks. Frankle and Carbin (2019); Liu et al. (2019) study techniques to identify the most important parameters for the current task, so that masking the less important parameters during training does not hurt performance. Lin et al. (2021) adapts this idea for multilingual NMT, trying to identify language dependent subsets of parameters by pruning a fine-tuned model. Our approach also aims to map sub-networks to tasks: we do so by masking the output of each layer, rather than masking parameters. Furthermore, Lin et al. (2021) computes the masks via a heuristic selection; while our approach learns the masks with variational techniques.
Sparse Transformer The idea of adaptive sparsity is studied in several works. For instance, Li et al. (2020) propose to use a variable depth for dif
(a) TED-Related
(b) TED-Diverse
Figure 3: Visualization of languages according to their dropout masks (a large vector concatenating the dropping masks of all the layers of the model) constructed by PCA.
ferent tasks. The authors aimed to match the depth of the sub-network to the complexity of the task. Gong et al. (2021b,a) also take an interest in the adaptive sparse Transformers, in which differ each task triggers the selection of specific heads in multihead attention, layers, and blocks in feedforward matrices. Mixture-of-experts (MoE) constitute another effective approach to achieve sparsity. Using the Transformer architecture, the GShard model replaces a single feedforward (FFN) sub-layer with an MoE module consisting of multiple FFN sublayers (Lepikhin et al., 2021; Fedus et al., 2021).
Adapter modules Adapters have proven to be very efficient for multi-task NLP (Houlsby et al., 2019; Bapna and First, 2019; Pham et al., 2020; Pfeiffer et al., 2020). In a nutshell, this technique consists in plugging several so-called adapter modules to the intermediate layers of a pretrained Transformer and finetuning these adapters on the downstream tasks. Adapters can also be trained without supervision for multilingual translation (Philip et al., 2020). However, the hard-coded separation between the domains of different tasks may lead to a catastrophic forgetting effect (Pfeiffer et al., 2021), which is a common problem in multi-task modeling using neural networks (McCloskey and Cohen, 1989). In multidomain translation, Pham et al. (2021) recently demonstrated the brittleness of adapters against fuzzy domain separations, out-of-domain distributions, and erroneous domain tags. Several subsequent studies have aimed to mitigate this weakness through a mixture of expert mechanism (e.g. (Pfeiffer et al., 2021)).
Zhang et al. (2021) propose to learn to route between shared and language-specific representations with a conditional language-specific routing while training the parameters of the underlying Transformer. This method is related to the Fusion-Adapters of Pfeiffer et al. (2021). Both approaches aim to select between shared and task-specific representations. The proximity between tasks is not taken into account in the routing mechanism. We propose a different approach to the problem of multi-task routing in the underlying network.
6 Conclusions and outlook
In this work, we have presented a novel method for multidomain and multilingual translation. It allows us to jointly search for an optimal assignment of sub-networks to tasks and to learn the parameters of the underlying network. Our method relies on a sound mathematical framework and an end-to-end optimization procedure; it only adds a small number of extra parameters. The additional training cost is also reasonable, amounting to 100k iterations in the multidomain setting, given the observed gains in performance. Experimentally, we achieve a large improvement over a Transformer baseline; our performance are also comparable to that of a strong a multi-task baseline using residual adapter modules which rely on a large number of extra parameters. For multilingual translation, our model outperforms multilingual Transformer and Language Adapters in 3 our of 4 settings. LAMGD seems specially beneficial for training languages with little parallel data, which can take advantage
of the resources that are available for related languages. Besides, we provided an thorough analysis of the similarities between learned sub-networks and demonstrate a strong correlation between the learned similarities and the proximity of the corresponding tasks (domain or language).
There are several ways in which our methodology can be improved. In future work, we would first like to provide an complete variational framework to model both the number of groups,  $k$  and the selection of the dropout masks. Second, we also intend to dispense with the domain information during inference: this would mean replacing the dependency on  $d$  in the variational distribution by a dependency on the input  $x$ . Another interesting direction will be to consider adapting the size and capacity allocated to each domain / language, depending on the difficulty of the associated translation task. Addressing these questions will allow us to replace heuristic choices in the architecture design with an increased dependency on the training data.
MT technologies are generally intended to facilitate cross-lingual as well as cross-cultural communications. The methods presented here are notably interesting in the view to improve MT from and into English for low-resource languages, subject to the availability of data for a related language. We acknowledge that (a) our results should ultimately be backed-up large scale experiments involving much more languages – even though this goes against the idea of limiting the computing cost our experiments; (b) better architectures and training regimes can improve the translation quality for low-resource languages, yet will not solve the problem entirely. This means that additional work focusing specifically on developing resources for these languages should remain an important objective for future work.
This work was granted access to the Jean Zay HPC resources of [TGCC/CINES/IDRIS] under the allocation 2020- [AD011011270] made by GENCI (Grand Equipement National de Calcul Intensif).
Brandon Amos, Vladlen Koltun, and J. Zico Kolter. 2019. The limited multi-label projection layer. CoRR, abs/1906.08707.
Brandon Amos and Denis Yarats. 2020. The differentiable cross-entropy method. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 291-302. PMLR.
Ankur Bapna and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538-1548, Hong Kong, China. Association for Computational Linguistics.
Chenhui Chu and Raj Dabre. 2018. Multilingual and multi-domain adaptation for neural machine translation. In Proceedings of the 24st Annual Meeting of the Association for Natural Language Processing (NLP 2018), pages 909-912, Okayama, Japan.
Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385-391, Vancouver, Canada. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine translation. ACM Comput. Surv., 53(5).
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Hongyu Gong, Xian Li, and Dmitriy Genzel. 2021a. Adaptive sparse transformer for multilingual translation. ArXiv, abs/2104.07358.
Hongyu Gong, Yun Tang, J. Pino, and Xian Li. 2021b. Pay better attention to attention: Head selection in
multilingual and multi-domain sequence modeling. ArXiv, abs/2106.10840.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. volume 97 of Proceedings of Machine Learning Research, pages 2790-2799, Long Beach, California, USA. PMLR.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72. Association for Computational Linguistics.
Xiang Kong, Adithya Renduchintala, James Cross, Yuqing Tang, Jiatao Gu, and Xian Li. 2021. Multilingual neural machine translation with deep encoder and multiple shallow decoders. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1613-1624, Online. Association for Computational Linguistics.
Wouter Kool, Herke Van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3499-3508. PMLR.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.
Xian Li, Asa Cooper Stickland, Yuqing Tang, and Xiang Kong. 2020. Deep transformers with latent depth. In Advances in Neural Information Processing Systems, volume 33, pages 1736-1746. Curran Associates, Inc.
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 293-305, Online. Association for Computational Linguistics.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019. Rethinking the value of network pruning. In International Conference on Learning Representations.
Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation - Advances in Research and Theory, 24(C):109-165.
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 35-41, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Minh Quang Pham, Josep Crego, and François Yvon. 2021. Revisiting multi-domain machine translation. Transactions of the Association for Computational Linguistics, 9(0):17-35.
Minh Quang Pham, Josep Maria Crego, François Yvon, and Jean Senellart. 2020. A study of residual adapters for multi-domain neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 617-628, Online. Association for Computational Linguistics.
Jerin Philip, Alexandre Berard, Matthias Galle, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods
in Natural Language Processing (EMNLP), pages 4465-4470, Online. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 529-535, New Orleans, Louisiana. Association for Computational Linguistics.
Danielle Saunders. 2021. Domain adaptation and multi-domain adaptation for neural machine translation: A survey. CoRR, abs/2104.06951.
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multilingual unsupervised NMT using shared encoder and language-specific decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083-3089, Florence, Italy. Association for Computational Linguistics.
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 963-973, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020a. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8526-8537, Online. Association for Computational Linguistics.
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020b. On negative interference in multilingual models: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, Online. Association for Computational Linguistics.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan First. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
This section explains how to compute  $\hat{m}_l^d (\tau)$  by solving the optimization problem (4) and then how to compute the gradients  $\frac{\partial\hat{m}_l^d(\tau)}{\partial\Phi_l^d}$ .
First, to solve (4) we follow the same approach as in (Amos et al., 2019; Amos and Yarats, 2020) by applying the Karush-Kuhn-Tucker (KKT) conditions to (4). The solution of (4) will have the following form:
$$
\hat {m} _ {l} ^ {d} (\tau) = \sigma \left(\frac {g _ {l} ^ {d} + \Phi_ {l} ^ {d} + \bar {\nu}}{\tau}\right) \tag {5}
$$
in which  $\sigma(.)$  is the sigmoid function and  $\bar{\nu}$  is the solution of the following equation:
$$
\sum_ {i = 1} ^ {n _ {p}} \sigma \left(\frac {g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i) + v}{\tau}\right) = k \tag {6}
$$
Because sigmoid is monotonically increasing, equation (6) has a unique solution. Furthermore, because of the smoothness of  $g(\nu, \Phi_l^d) = \sum_{i=1}^{n_p} \sigma\left(\frac{g_l^d(i) + \Phi_l^d(i) + \nu}{\tau}\right)$  w.r.t  $\nu$  and  $\Phi_l^d$ , we can perform the implicit differentiation of its solution  $\bar{\nu}$  w.r.t  $\Phi_l^d$  as below, even though the solution of (6) does not have an explicit form.
$$
\begin{array}{l} \frac {\partial g}{\partial \bar {v}} \times \frac {\partial \bar {v}}{\partial \Phi_ {l} ^ {d}} + \frac {\partial g}{\partial \Phi_ {l} ^ {d}} = 0 \\ \Rightarrow \frac {\partial \bar {v}}{\partial \Phi_ {l} ^ {d}} = - \left(\frac {\partial g}{\partial \bar {v}}\right) ^ {- 1} \times \frac {\partial g}{\partial \Phi_ {l} ^ {d}} \\ \end{array}
$$
Because the differentiation of sigmoid has exact forms,  $\frac{\partial g}{\partial\nu}$  and  $\frac{\partial g}{\partial\Phi_l^d}$  also have exact form. Therefore, we do not need autograd to compute the implicit gradient  $\frac{\partial\nu}{\partial\Phi_l^d}$ . The gradient of  $\hat{m}_l^d (\tau)$  w.r.t  $\Phi_l^d$  is computed as follows:
$$
= \frac {\partial \hat {m} _ {l} ^ {d} (\tau)}{\partial v} \times \frac {\partial v}{\partial \Phi_ {l} ^ {d}} + \frac {1}{\tau} \frac {\exp \left(\frac {g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i) + v}{\tau}\right)}{\left(1 + \exp \left(\frac {g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i) + v}{\tau}\right)\right) ^ {2}} \tag {7}
$$
In our algorithm, we solve (6) by binary search. The convergence of binary search is extremely fast and assured by the monotonicity of  $g(\nu, \Phi_l^d)$ . In our experiments, we set the search range to  $[-100, 100]$ .
Finally, we need prove that  $\lim_{\tau \to 0} \hat{m}_l^d(\tau) = \tilde{m}_l^d$ .
We assume  $g_{l}^{d}(i_{1}) + \Phi_{l}^{d}(i_{1}) > g_{l}^{d}(i_{2}) + \Phi_{l}^{d}(i_{2})>$
$$
\dots > g _ {l} ^ {d} \left(i _ {n _ {p}}\right) + \Phi_ {l} ^ {d} \left(i _ {n _ {p}}\right).
$$
$$
\begin{array}{l} \lim  _ {\tau \rightarrow 0} \sigma \left(\frac {g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i) + v}{\tau}\right) = \\ = \left\{ \begin{array}{l l} 1, & \text {i f} \tau > - (g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i)), \\ 0, & \text {i f} \tau <   - (g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i)), \\ \frac {1}{2} & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
$$
\sum_ {i = 1} ^ {n _ {p}} \sigma (\frac {g _ {l} ^ {d} (i) + \Phi_ {l} ^ {d} (i) + \nu}{\tau}) = k
$$
there exist  $\varepsilon$  such that  $\forall \tau < \varepsilon$ , the solution  $\bar{\nu}$  of (6) satisfies  $-(g_{l}^{d}(i_{k + 1}) + \Phi_{l}^{d}(i_{k + 1})) > \bar{\nu} > -(g_{l}^{d}(i_{k}) + \Phi_{l}^{d}(i_{k}))$ . Furthermore, because sigmoid is monotonically increasing,