SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1:  $B\gets$  number of permutations
2:  $\mathbf{X}^{-\mathrm{j}}\gets \mathbf{X}$  with j-th column removed
3: for  $\mathrm{i} = 1$  to  ${n}_{\text{test }}$  do
4:  $\hat{x}_i^j\gets \mathrm{RF}_j(\mathbf{x_i^{-j}})$
5: end for
6: Residuals  $\epsilon^{\mathbf{j}}\gets \mathbf{x}^{\mathbf{j}} - \hat{\mathbf{x}}^{\mathbf{j}}$
7: for  $\mathbf{b} = 1$  to  $\mathbf{B}$  do
8:  $\tilde{e}^{\mathrm{j},\mathrm{b}}\gets$  Random Shuffling(ej)
9:  $\tilde{\mathbf{x}}^{\mathbf{j},\mathbf{b}}\gets \hat{\mathbf{x}}^{\mathbf{j}} + \tilde{\epsilon}^{\mathbf{j},\mathbf{b}}$
10: for  $\mathrm{i} = 1$  to  ${n}_{\text{test }}$  do
11:  $\tilde{y}_i^b\gets \hat{\mu} (\tilde{\mathbf{x}}_{\dot{\mathbf{i}}}^{\mathbf{j},\mathbf{b}})$
12: compute  $l_i^{j,b}$
13: end for
14: end for
15: mean  $\left(\hat{m}_{CPI}^{j}\right) = \frac{1}{n_{test}}\frac{1}{B}\sum_{i = 1}^{n_{test}}\sum_{b = 1}^{B}l_{i}^{j,b}$
16:  $\tau_{i}^{j} = \left(\frac{1}{B}\sum_{b = 1}^{B}l_{i}^{j,b} - mean(\hat{m}_{CPI}^{j})\right)^{2}$
17:  $\mathbf{std}(\hat{m}_{CPI}^{j}) = \sqrt{\frac{1}{n_{test} - 1}\sum_{i = 1}^{n_{test}}\tau_{i}^{j}}$
18:  $z_{CPI}^{j} = \frac{\mathrm{mean}(\hat{m}_{CPI}^{j})}{\mathrm{std}(\hat{m}_{CPI}^{j})}$
19:  $p^j \gets 1 - cdf(z_{CPI}^j)$
B
Figure 1: CPI-DNN vs Perfit-DNN: Performance at detecting important variables on simulated data with  $n = 300$  and  $p = 100$ . (A): The type-I error quantifies to which extent the rate of low p-values ( $p < 0.05$ ) exceeds the nominal false positive rate. (B): The AUC score measures to which extent variables are ranked consistently with the ground truth. Dashed line: targeted type-I error rate. Solid line: chance level.
each block. In this experiment,  $p = 100$  and  $n = 300$ . The first variable of each of the first 5 blocks is chosen to predict the target  $y$  with the following model, where  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ :
$$
y _ {i} = x _ {i} ^ {1} + 2 \log (1 + 2 (x _ {i} ^ {1 1}) ^ {2} + (x _ {i} ^ {2 1} + 1) ^ {2}) + x _ {i} ^ {3 1} x _ {i} ^ {4 1} + \epsilon_ {i}, \forall i \in [   [ n ]   ]
$$
The AUC score and type-I error are presented in Fig. 1. Power and computation time are reported in the supplement Fig. 1 - S1. Based on the AUC scores, Permfit-DNN and CPI-DNN showed virtually identical performance. However, Permfit-DNN lost type-I error control when correlation in  $\mathbf{X}$  is increased, while CPI-DNN always controlled the type-I error at the targeted rate.
5.2 Experiment 2: Performance across different settings
In the second setup, we check if CPI-DNN and Permfit-DNN control the type-I error with an increasing total number of samples  $n$ . The data are generated as previously, with a correlation  $\rho = 0.8$ . We fix the number of variables  $p$  to 50 while the number of samples  $n$  increases from 100 to 1000 with a step size of 100. We use 5 different models to generate the outcome  $y$  from X: classification, Plain linear, Regression with ReLu, Interactions only and Main effects with interactions. Further details regarding each data-generating scenario can be found in supplement (section G).
Figure 2: Model comparisons across data-generating scenarios: The (A) type-I error and (B) AUC scores of Permfit-DNN and CPI-DNN are plotted as function of sample size for five different settings. The number  $n$  of samples increased from 100 to 1000 with a step size of 100. The number of variables  $p$  was set to 50. Dashed line: targeted type-I error rate. Solid line: chance level.
The AUC score and type-I error of Permfit-DNN and CPI-DNN are shown as a function of sample size in Fig. 2. The accuracy of the two methods was similar across data-generating scenarios, with a slight reduction in the AUC scores of Permfit-DNN as compared to CPI-DNN. Only CPI-DNN controlled the rate of type-I error in the different scenarios at the specified level of 0.05. Thus, CPI-DNN provided an accurate ranking of the variables according to their importance score while, at the same time, controlling for the type-I error in all scenarios.
5.3 Experiment 3: Performance benchmark across methods
In the third setup, we include Permfit-DNN and CPI-DNN in a benchmark with other state-of-the-art methods for variable importance using the same setting as in Experiment 2, while fixing the total number of samples  $n$  to 1000. We consider the following methods:
- Marginal Effects: A univariate linear model is fit to explain the response from each of the variables separately. The importance scores are then obtained from the ensuing p-values.
- Conditional-RF [Strobl et al., 2008]: A conditional variable importance approach based on a Random Forest model. This method provides p-values.
-  $\mathrm{d}_0\mathrm{CRT}$  [Liu et al., 2021, Nguyen et al., 2022]: The Conditional Randomization Test with distillation, using a sparse linear or logistic learner.
- Lazy VI [Gao et al., 2022].
- Permfit-DNN [Mi et al., 2021].
- LOCO [Lei et al., 2018]: This method applies the remove-and-retrain approach.
- cpi-knockoff [Watson and Wright, 2021]: Similar to CPI-RF, but permutation steps are replaced by a sampling step with a knockoff sampler.
- CPI-RF: This corresponds to the method in Alg. 1, where  $\hat{\mu}$  is a Random Forest.
CPI-DNN: This corresponds to the method in Alg. 1, where  $\hat{\mu}$  is a DNN.
The extensive benchmarks on baselines and competing methods that provide p-values are presented in Fig. 3. For type-I error,  $d_0CRT$ ,  $CPI-RF$ ,  $CPI-DNN$ ,  $LOCO$  and  $cpi$ -knockoff provided reliable control, whereas Marginal effects,  $Permfit-DNN$ ,  $Conditional-RF$  and  $LazyVI$  showed less consistent results across scenarios. For AUC, we observed that marginal effects performed poorly, as they do not use a proper predictive model.  $LOCO$  and  $cpi$ -knockoff behave similarly.  $d_0CRT$  performed well when the data-generating model was linear and did not include interaction effects.  $Conditional-RF$  and  $CPI-RF$  showed reasonable performance across scenarios. Finally,  $Permfit-DNN$  and  $CPI-DNN$  outperformed all the other methods, closely followed by  $LazyVI$ .
Additional benchmarks on popular methods that do not provide p-values, e.g. BART [Chipman et al., 2010] or local and instance-based methods such as Shapley values [Kumar et al., 2020], are reported in the supplement (section H). The performance of these methods in terms of power and computation
Scenario
$\bigcirc$  Main effects and Interactions
$\bigcirc$  Interactions only
Regression with ReLu
Plain linear
$\bigcirc$  Classification
A
Method
Figure 3: Extended model comparisons: CPI-DNN and Perfit-DNN were compared to baseline models (outer columns) and competing approaches across data-generating scenarios (inner columns). Prediction tasks were simulated with  $n = 1000$  and  $p = 50$ . (A): Type-I error. (B): AUC scores. Dashed line: targeted type-I error rate. Solid line: chance level.
time are reported in the supplement Figs. 3 - S2 & 3 - S3 respectively. Additional inspection of power showed that across data generating scenarios, CPI-DNN, Permfit-DNN and conditional-RF showed strong results. Marginal and dOCRT performed only well in scenarios without interaction effects. CPI-RF, cpi-knockoff, LOCO and Lazy VI performed poorly. Finally, to put estimated variable importance in perspective with model capacity, we benchmarked prediction performance of the underlying learning algorithms in the supplement Fig. 3 - S4.
5.4 Experiment 4: Permfit-DNN vs CPI-DNN on Real Dataset UKBB
Large-scale simulations comparing the performance of CPI-DNN and Permfit-DNN are conducted in supplement (section L). We conducted an empirical study of variable importance in a biomedical application using the non-conditional permutation approach Permfit-DNN (no statistical guarantees for correlated inputs) and the safer CPI-DNN approach. A recent real-world data analysis of the UK Biobank dataset reported successful machine learning analysis of individual characteristics. The UK Biobank project (UKBB) curates phenotypic and imaging data from a prospective cohort of volunteers drawn from the general population of the UK [Constantinescu et al., 2022]. The data is provided by the UKBB operating within the terms of an Ethics and Governance Framework. The work focused on age, cognitive function and mood from brain images and social variables and put the ensuing models in relation to individual life-style choices regarding sleep, exercise, alcohol and tobacco [Dadi et al., 2021].
A coarse analysis of variable importance was presented, in which entire blocks of features were removed. It suggested that variables measuring brain structure or brain activity were less important for explaining the predictions of cognitive or mood outcomes than socio-demographic characteristics. On the other hand, brain imaging phenotypes were highly predictive of the age of a person, in line with the brain-age literature [Cole and Franke, 2017]. In this benchmark, we explored variable-level importance rankings provided by the CPI-DNN and Perfit-DNN methods.
The real-world empirical benchmarks on predicting personal characteristics and life-style are summarized in Fig. 4. Results in panel (A) suggest that highest agreement for rankings between CPI-DNN and Permfit-DNN was achieved for social variables (bottom left, orange squares). At the same time, CPI-DNN flagged more brain-related variables as relevant (bottom right, circles). We next computed counts and percentage and broke down results by variable domain (Fig. 4, B). Naturally, the total relevance for brain versus social variables varied by outcome. However, as a tendency, CPI-DNN seemed more selective as it flagged fewer variables as important (blue) beyond those flagged as important by both methods (light blue). This was more pronounced for social variables where CPI-DNN
Figure 4: Real-world empirical benchmark: Prediction of personal characteristics (age, cognition, mood) and life-style habits (alcohol consumption, sleep, exercise & smoking) from various sociodemographic and brain-imaging derived phenotypes in a sample of  $n = 8357$  volunteers from the UK Biobank. (A) plots variable rankings for Permfit-DNN (x axis) versus CPI-DNN (y axis) across all outcomes. Color: variable domain (brain versus social). Shape: variables classified by both methods as important (squares), unimportant (crosses) or by only one of the methods, i.e., CPI-DNN (circles) or Permfit-DNN (triangles). (B) presents a detailed breakdown of percentage and counts of variable classification split by variable domain.
sometimes added no further variables. As expected by the impact of aging on brain structure and function, brain data was most important for age-prediction compared to other outcomes. Interestingly, most disagreements between the methods occurred in this setting as  $CPI$  rejected 16 out of 66 brain inputs that were found as important by Permfit. This outlines the importance of correlations between brain variables, that lead to spurious importance findings with Permfit. We further explored the utility of our approach for age-prediction from neuromagnetic recordings [Engemann et al., 2020] and observed that  $CPI$ -DNN readily selected relevant frequency bands without fine-tuning the approach (section M in the supplement).
In this work, we have developed a framework for studying the behavior of marginal and conditional permutation methods and proposed the CPI-DNN method, that was inspired by the limitations of the Permfit-DNN approach. Both methods build on top of an expressive DNN learner, and both methods turned out superior to competing methods at detecting relevant variables, leading to high AUC scores across various simulated scenarios. However, our theoretical results predicted that Permfit-DNN would not control type-I error with correlated data, which was precisely what our simulation-based analyzes confirmed for different data-generating scenarios (Fig. 1 - 2). Other popular methods (Fig. 3) showed similar failures of type-I error control across scenarios or only worked well in a subset of tasks. Instead, CPI-DNN achieved control of type-I errors by upgrading the permutation to conditional permutation. The consequences were pronounced for correlated predictive features arising from generative models with product terms, which was visible even with a small fraction of data points for model training. Among alternatives, the Lazy VI approach [Gao et al., 2022] obtained an accuracy almost as good as Permfit-DNN and CPI-DNN but with an unreliable type-I error control.
Taken together, our results suggest that CPI-DNN may be a practical default choice for variable importance estimation in predictive modeling. A practical validation of the standard normal distribution assumption for the non important variables can be found in supplement (section N). The CPI approach is generic and can be implemented for any combination of learning algorithms as a base learner or conditional means estimator. CPI-DNN has a linear and quadratic complexity in the number of samples and variables, respectively. This is of concern when modeling the conditional distribution of the variable of interest which lends itself to high computational complexity. In our work, Random Forests proceed to be useful default estimators as they are computationally lean and their model complexity, given reasonable default choices implemented in standard software, can be well controlled by tuning the tree depth. In fact, our supplementary analyses (section O) suggest that proper hyperparameter tuning was sufficient to obtain good calibration of p-values. As a potential limitation, it is noteworthy the current configuration of our approach uses a deep neural network as
the base learner. Therefore, in general, more samples might be needed for good model performance, hence, improved model interpretation.
Our real-world data analysis demonstrated that CPI-DNN is readily applicable, providing similar variable rankings as Perfit-DNN. The differences observed are hard to judge as the ground truth is not known in this setting. Moreover, accurate variable selection is important to obtain unbiased interpretations which are relevant for data-rich domains like econometrics, epidemiology, medicine, genetics or neuroscience. In that context, it is interesting that recent work raised doubts about the signal complexity in the UK biobank dataset [Schulz et al., 2020], which could mean that underlying predictive patterns are spread out over correlated variables. In the subset of the UK biobank that we analysed, most variables actually had low correlation values (Fig. E4), which would explain why CPI-DNN and Perfit-DNN showed similar results. Nevertheless, our empirical results seem compatible with our theoretical results as CPI-DNN flagged fewer variables as important, pointing at stricter control of type-I errors, which is a welcome property for biomarker discovery.
When considering two highly correlated variables  $\mathbf{x}_1$  and  $\mathbf{x}_2$ , the corresponding conditional importance of both variables is 0. This problem is linked to the very definition of conditional importance, and not to the CPI procedure itself. The only workaround is to eliminate, prior to importance analysis, degenerate cases where conditional importance cannot be defined. Therefore, possible future directions include inference on groups of variables, e.g., gene pathways, brain regions, while preserving statistical control offered by CPI-DNN.
Acknowledgement This work has been supported by Bertrand Thirion and is supported by the KARAIB AI chair (ANR-20-CHIA-0025-01), and the H2020 Research Infrastructures Grant EBRAIN-Health 101058516. D.E. is a full-time employee of F. Hoffmann-La Roche Ltd.
Marlene Sophie Altenmüller, Leonie Lucia Lange, and Mario Gollwitzer. When research is me-search: How researchers' motivation to pursue a topic affects laypeople's trust in science. PLoS One, 16 (7):e0253911, July 2021. ISSN 1932-6203. doi: 10.1371/journal.pone.0253911.
Andre Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. Permutation importance: A corrected feature importance measure. Bioinformatics, 26(10):1340-1347, May 2010. ISSN 1367-4803. doi: 10.1093/bioinformatics/btq134.
Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145-1159, July 1997. ISSN 0031-3203. doi: 10.1016/S0031-3203(96)00142-2.
Leo Breiman. *Random Forests. Machine Learning*, 45(1):5-32, October 2001. ISSN 1573-0565. doi: 10.1023/A:1010933404324.
Przemyslaw Biecek and Tomasz Burzykowski. Explanatory Model Analysis. December 2020.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection. arXiv:1610.02351 [math, stat], December 2017.
Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the Feature Importance for Black Box Models. In Michele Berlingerio, Francesco Bonchi, Thomas Gartner, Neil Hurley, and Georgiana Ifrim, editors, Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pages 655-670, Cham, 2019. Springer International Publishing. ISBN 978-3-030-10925-7. doi: 10.1007/978-3-030-10925-7_40.
Diego Castillo-Barnes, Javier Ramírez, Fermín Segovia, Francisco J. Martínez-Murcia, Diego Salas-Gonzalez, and Juan M. Górriz. Robust Ensemble Classification Methodology for I123-Ioflupane SPECT Images and Multiple Heterogeneous Biomarkers in the Diagnosis of Parkinson's Disease. Frontiers in Neuroinformatics, 12:53, 2018. ISSN 1662-5196. doi: 10.3389/fninf.2018.00053.
Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. BART: Bayesian additive regression trees. Ann. Appl. Stat., 4(1), March 2010. ISSN 1932-6157. doi: 10.1214/09-AOAS285.
Jieping Ye, Kewei Chen, Teresa Wu, Jing Li, Zheng Zhao, Rinkal Patel, Min Bae, Ravi Janardan, Huan Liu, Gene Alexander, and Eric Reiman. Heterogeneous data fusion for alzheimer's disease study. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, pages 1025-1033, New York, NY, USA, August 2008. Association for Computing Machinery. ISBN 978-1-60558-193-4. doi: 10.1145/1401890.1402012.
B. Zheng and A. Agresti. Summarizing the predictive power of a generalized linear model. Stat Med, 19(13):1771-1781, July 2000. ISSN 0277-6715. doi: 10.1002/1097-0258(20000715)19:13<1771:: aid-sim485>3.0.co;2-p.
A Supplement proof - getting from Eq. 2 to Eq. 3
$$
\begin{array}{l} \hat {m} ^ {j} = \frac {1}{n _ {t e s t}} \sum_ {i = 1} ^ {n _ {t e s t}} (\hat {\mu} (\mathbf {x _ {i}}) - \hat {\mu} (\mathbf {x _ {i} ^ {(j)}})) (2 \mu (\mathbf {x _ {i}}) - \hat {\mu} (\mathbf {x _ {i}}) - \hat {\mu} (\mathbf {x _ {i} ^ {(j)}}) + 2 \varepsilon_ {i}) 2 \\ = \frac {1}{n _ {t e s t}} \sum_ {i = 1} ^ {n _ {t e s t}} (\mathbf {x _ {i} ^ {- j} \hat {\mathbf {w}} ^ {- j}} + x _ {i} ^ {j} \hat {w} ^ {j} - \mathbf {x _ {i} ^ {- j} \hat {\mathbf {w}} ^ {- j}} - \{x _ {i} ^ {j} \} ^ {p e r m} \hat {w} ^ {j}) (2 \mathbf {x _ {i} ^ {- j} \mathbf {w} ^ {- j}} - 2 \mathbf {x _ {i} ^ {- j} \hat {\mathbf {w}} ^ {- j}} - (x _ {i} ^ {j} \hat {w} ^ {j} + \{x _ {i} ^ {j} \} ^ {p e r m} \hat {w} ^ {j}) + 2 \varepsilon_ {i}) \\ = \frac {2 \hat {w} ^ {j}}{n _ {t e s t}} \sum_ {i = 1} ^ {n _ {t e s t}} (x _ {i} ^ {j} - \{x _ {i} ^ {j} \} ^ {p e r m}) (\mathbf {x _ {i}} ^ {- \mathbf {j}} (\mathbf {w} ^ {- \mathbf {j}} - \hat {\mathbf {w}} ^ {- \mathbf {j}}) + \varepsilon_ {i}) - \underbrace {\hat {w} ^ {j} ((x _ {i} ^ {j}) ^ {2} - (\{x _ {i} ^ {j} \} ^ {p e r m}) ^ {2})} \\ = \frac {2 \hat {w} ^ {j}}{n _ {t e s t}} \sum_ {i = 1} ^ {n _ {t e s t}} \left(x _ {i} ^ {j} - \left\{x _ {i} ^ {j} \right\} ^ {p e r m}\right) \left(\mathbf {x} _ {\mathbf {i}} ^ {- \mathbf {j}} \left(\mathbf {w} ^ {- \mathbf {j}} - \hat {\mathbf {w}} ^ {- \mathbf {j}}\right) + \varepsilon_ {i}\right) 3 \\ \end{array}
$$
B Diagram of CPI constructions
Figure E1: CPI-DNN's constructions: Constructing the variable of interest  $\tilde{\mathbf{x}}^{\mathrm{j}}$  is done either (1) by the additive construction (top block) where a shuffled version of the residuals is added to the predicted version using the remaining predictors with the mean of a random forest (RF) or (2) by the sampling construction (bottom block) using a random forest (RF) model to fit  $\mathbf{x}^{\mathrm{j}}$  from  $\mathbf{X}^{-\mathrm{j}}$  and then sample the prediction within the leaves of the RF.
C Conditional Permutation Importance (CPI) Wald statistic asymptotically controls type-I errors: hypotheses, theorem and proof
Outline The proof relies on the observation that the importance score defined in (4) is 0 in the asymptotic regime, where the permutation procedure becomes a sampling step, under the assumption that variable  $j$  is not conditionally associated with  $y$ . Then all the proof focuses on the convergence of the finite-sample estimator to the population one. To study this, we use the framework developed in [Williamson et al., 2021]. Note that the major difference with respect to other contributions [Watson and Wright, 2021] is that the ensuing inference is no longer conditioned on the estimated learner  $\hat{\mu}$ . Next, we first restate the precise technical conditions under which the different importance scores considered are asymptotically valid, i.e. lead to a Wald-type statistic that behaves as a standard normal under the null hypothesis.
Notations Let  $\mathcal{F}$  represent the class of functions from which a learner  $\mu : \mathbf{x} \mapsto y$  is sought.
Let  $P_0$  be the data-generating distribution and  $P_n$  is the empirical data distribution observed after drawing  $n$  samples (noted  $n_{train}$  in the main text; in this section, we denote it  $n$  to simplify notations). The separation between train and test samples is actually only relevant to alleviate
some technical conditions on the class of learners used.  $\mathcal{M}$  is the general class of distributions from which  $P_{1},\ldots ,P_{n},P_{0}$  are drawn.  $\mathcal{R}:= \{c(P_1 - P_2):c\in [0,\infty),P_1,P_2\in \mathcal{M}\}$  is the space of finite signed measures generated by  $\mathcal{M}$ . Let  $l$  be the loss function used to obtain  $\mu$ . Given  $f\in \mathcal{F}$ ,  $l(f;P_0) = \int l(f(\mathbf{x}),y)P_0(\mathbf{z})d\mathbf{z}$ , where  $\mathbf{z} = (\mathbf{x},y)$ . Let  $\mu_0$  denote a population solution to the estimation problem  $\mu_0\in \mathrm{argmin}_{f\in \mathcal{F}}l(f;P_0)$  and  $\hat{\mu}_n$  a finite sample estimate  $\hat{\mu}_n\in \mathrm{argmin}_{f\in \mathcal{F}}l(f;P_n) = \frac{1}{n}\sum_{(\mathbf{x},y)\in P_n}l(f(\mathbf{x}),y)$ .
Let us denote by  $\dot{l} (\mu ,P_0;h)$  the Gâteaux derivative of  $P\mapsto l(\mu ,P)$  at  $P_{0}$  in the direction  $h\in \mathcal{R}$  and define the random function  $g_{n}:\mathbf{z}\mapsto \dot{l} (\hat{\mu}_{n},P_{0};\delta_{\mathbf{z}} - P_{0}) - \dot{l} (\mu_{0},P_{0};\delta_{\mathbf{z}} - P_{0})$  , where  $\delta_{\mathbf{z}}$  is the degenerate distribution on  $\mathbf{z} = (\mathbf{x},y)$
(A1) (Optimality) there exists some constant  $C > 0$ , such that for each sequence  $\mu_1, \mu_2, \dots \in \mathcal{F}$  given that  $\| \mu_n - \mu_0 \| \to 0$ ,  $|l(\mu_n, P_0) - l(\mu_0, P_0)| < C \| \mu_n - \mu_0 \|_{\mathcal{F}}^2$  for each  $n$  large enough.
(A2) (Differentiability) there exists some constant  $\kappa > 0$  such that for each sequence  $\epsilon_1, \epsilon_2, \dots \in \mathbb{R}$  and  $h_1, h_2, \dots \in \mathcal{R}$  satisfying  $\epsilon_n \to 0$  and  $\| h_n - h_\infty \| \to 0$ , it holds that
$$
\sup  _ {\mu \in \mathcal {F}: \| \mu - \mu_ {0} \| _ {\mathcal {F}} <   \kappa} \left| \frac {l (\mu , P _ {0} + \epsilon_ {n} h _ {n}) - l (\mu , P _ {0})}{\epsilon_ {n}} - \dot {l} (\mu , P _ {0}; h _ {n}) \right| \to 0.
$$