SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
N _ {T} = \widetilde {\mathcal {O}} \bigg (\inf  _ {\varepsilon} \Bigg \{H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H}{\lambda \varepsilon^ {2}} \sum_ {h = 1} ^ {H} \sum_ {m = 1} ^ {M} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h} ^ {m}; T\right) \cdot \mathfrak {E} \left(\mathcal {F} _ {h} ^ {m}, \varepsilon / 8; f _ {h} ^ {\star , m}\right) \Bigg \} \Bigg).
$$
In Appendix A, we evaluate our IL algorithm on the Cartpole environment, with single and multiple experts. We found that our algorithm can match the performance of passive querying algorithms while making a significantly lesser number of expert queries. Finally, note that setting  $H = 1$  in the above result, recovers an algorithm, and a similar result for selective sampling with multiple experts.
In this paper, or goal is to develop algorithms for online IL with active queries with small regret and query complexity bounds. Towards that end, we started by considering the selective sampling setting (IL with  $H = 1$ ), and provided a selective sampling algorithm that can work with general function classes  $\mathcal{F}$  and modeling assumptions, and relies on access to an online regression oracle w.r.t.  $\mathcal{F}$  to make its predictions (Section 3). The provided regret and query complexity bounds depend on the margin of the expert model. We then extended our selective sampling algorithm to interactive IL (Section 4). For IL, we showed that the margin term that appears in the regret and the query complexity depends on the margin of the expert on counterfactual trajectories that would have been observed on following the expert policy (that we wish to compare to), instead of the trajectories that the learner observes. Thus, if the expert always chooses actions that leads to states where it is confident (i.e. has less margin), the margin term will be smaller. We also considered extensions to learning with multiple experts.
AS thanks Sasha Rakhlin and Dylan Foster for helpful discussions. AS acknowledges support from the Simons Foundation and NSF through award DMS-2031883, as well as from the DOE through award DE-SC0022199. WS acknowledges support from NSF grant IIS-2154711. KS acknowledges support from NSF CAREER Award 1750575, and LinkedIn-Cornell grant.
Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In International Conference on Machine Learning, pages 1220-1228. PMLR, 2013.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning, pages 1732-1748. PMLR, 2022.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreement-regularized imitation learning. In International Conference on Learning Representations, 2019.
Kianté Brantley, Amr Sharaf, and Hal Daumé III. Active imitation learning with noisy guidance. arXiv preprint arXiv:2005.12801, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. In 2022 International Conference on Robotics and Automation (ICRA), pages 441-447. IEEE, 2022.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Nicolo Cesa-Bianchi, Gabor Lugosi, and Gilles Stoltz. Minimizing regret with label efficient prediction. IEEE Transactions on Information Theory, 51(6):2152-2162, 2005.
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning, pages 2058-2066. PMLR, 2015.
Ching-An Cheng and Byron Boots. Convergence of value aggregation for imitation learning. In International Conference on Artificial Intelligence and Statistics, pages 1801-1809. PMLR, 2018.
Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems, 33:5587-5598, 2020.
Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple experts. The Journal of Machine Learning Research, 13(1):2655-2697, 2012.
Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint arXiv:2304.08742, 2023.
Dylan Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning, pages 1539-1548. PMLR, 2018a.
Dylan J Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Conference On Learning Theory, pages 167-208. PMLR, 2018b.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
Stéphane Ross, Narek Melik-Barkhudarov, Kumar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J Andrew Bagnell, and Martial Hebert. Learning monocular reactive uav control in cluttered natural environments. In 2013 IEEE international conference on robotics and automation, pages 1765-1772. IEEE, 2013.
Daniel Russo and Benjamin Van Roy. eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in neural information processing systems, 23, 2010.
Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggravated: Differentiable imitation learning for sequential prediction. In International conference on machine learning, pages 3309-3318. PMLR, 2017.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.
Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics, 32(1):135-166, 2004.
Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprint arXiv:2204.00043, 2022.
A Experiments 15
B Further Discussion on Related Works 17
C Discussion on Computationally Efficiency 17
D Useful Tools and Notation 19
D.1 Basic Probabilistic Tools 19
D.2 Online Learning 19
D.3 Eluder Dimension, Disagreement Coefficient, and Star Number 20
E Selective Sampling: Learning from Single Expert 23
E.1 Comparison to Related Works 23
E.2 Proof Sketch for Selective Sampling and Binary Labels 23
E.3 Proof of Theorem 1 25
E.3.1 Supporting Technical Results 25
E.3.2 Regret Bound 28
E.3.3 Total Number of Queries 29
E.4 Proof of Theorem 2 30
E.4.1 Supporting Technical Results 32
E.4.2 Regret Bound 36
E.4.3 Total Number of Queries 38
E.5 Proof of Corollary 1 39
E.6 Proofs for Lower Bounds in Section 3.2 40
F Imitation Learning: Learning from Single Expert 42
F.1 Imitation Learning Tools 42
F.2 Proof of Proposition 1 42
F.3 Proof of Theorem 4 43
F.3.1 Supporting Technical Results 43
F.3.2 Regret Bound 45
F.3.3 Total Number of Queries 47
F.4 Proof for the Stochastic Setting 49
F.5 Proof of Theorem 5 50
F.5.1 Supporting Technical Results 52
F.5.2 Regret Bound 53
F.5.3 Total Number of Queries 54
We conduct experiments to verify our theory. To this end, we first introduce the simulator, Cart Pole [Barto et al., 1983, Brockman et al., 2016], and then explain the implementation of our algorithm and the baselines. Finally, we present the results.
Cart Pole. Cart Pole is a classical control problem, in which a pole is attached by an un-actuated joint to a cart. The goal is to balance the pole by applying force to the cart either towards the left or towards the right (so binary action). The episode is terminated once either the pole is out of balance or the cart deviates too far from the origin. A reward of 1 is obtained in each time step (however, the algorithm does not get any reward signal). The observations are four-dimensional, with the values representing the cart's position, velocity, the pole's angle, and angular velocity. The action is binary, indicating the force is either to the left or to the right.
Expert policies generation. We first generate an optimal policy  $\pi^{\star}$  (that attains the maximum possible reward of 500) by policy gradient. We notice that when running the optimal policy  $\pi^{\star}$ , the absolute value of the cart's position only lies in  $[0,2]$ . Hence, to generate  $M$  experts, we first divide this interval into  $M$  sub-intervals  $[a_0,a_1],[a_1,a_2],\ldots ,[a_{n - 1},a_M]$  ( $a_0 = 0$  and  $a_{M} = 2$ ) by geometric progression. For the  $i$ -th expert, it plays the same action as  $\pi^{\star}$  when the absolute value of the cart's position is in the interval  $[a_{i - 1},a_i]$  and plays uniformly at random outside of this interval. We find that using such generation, each expert individually cannot achieve a good performance (when  $M > 1$ ), while a proper combination of them can still be as strong as  $\pi^{\star}$ . We conduct experiment for  $M = 1,2,3,$  and 5, respectively. Given this design of expert generation, when the cart is in the sub-interval  $[a_{i - 1},a_i]$ , the only expert with non-zero margin is exactly the  $i$ -th expert.
Implementation. The algorithm is similar to Algorithm 4 but with some modification for practical purpose. First, we use a neural network (single hidden layer neural network,with 4 neurons in the hidden layer) as our function class  $\{\mathcal{F}_h^m\}_{h\leq H,m\leq M}$ . Second, we specify SelectAction to pick the action of the most confident expert, i.e.,
$$
\text {S e l e c t A c t i o n} \left(f _ {t, h} ^ {1} (x), \dots , f _ {t, h} ^ {M} (x)\right) := \operatorname {s i g n} \left(f _ {t, h} ^ {\hat {i}} (x)\right) \quad \text {w h e r e} \quad \hat {i} = \underset {i \in [ M ]} {\arg \max } | f _ {t, h} ^ {i} (x) |.
$$
Since we are considering binary action, we assume  $f_{t,h}^{i}(x) \in [-1,1]$ , and the action space is  $\{-1,1\}$ . Third, to compute  $\Delta_{t,h}^{m}$  efficiently, we apply the Lagrange multiplier to (60) to arrive at the following equivalent problem:
$$
\begin{array}{l} \Delta_{t,h}^{m}(x_{t,h}):= \min_{f\in \mathcal{F}_{h}^{m}}\max_{\alpha \geq 0} - \| f(x_{t,h}) - f_{t,h}^{m}(x_{t,h})\| \\ + \alpha \left(\sum_ {s = 1} ^ {t - 1} Z _ {s, h} \left\| f (x _ {s, h}) - f _ {s, h} ^ {m} (x _ {s, h}) \right\| ^ {2} - \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} _ {h} ^ {m}, T)\right). \\ \end{array}
$$
Then we treat the Lagrange multiplier  $\alpha$  as a constant, which converts the problem into the following:
$$
\Delta_ {t, h} ^ {m} \left(x _ {t, h}\right) := \min  _ {f \in \mathcal {F} _ {h} ^ {m}} - \| f \left(x _ {t, h}\right) - f _ {t, h} ^ {m} \left(x _ {t, h}\right) \| + \alpha \sum_ {s = 1} ^ {t - 1} Z _ {s, h} \| f \left(x _ {s, h}\right) - f _ {s, h} ^ {m} \left(x _ {s, h}\right) \| ^ {2}. \tag {9}
$$
The study of varying  $\alpha$  is shown in Figure 1. We found that small values (e.g.,  $\alpha = 1$ ) mostly lead to poor performance, while the results are fairly similar for large values. In our key experiments, we choose  $\alpha = 50$  when the number of experts is 1, 2 or 3, and choose 200 for 5-expert experiments. We note that since computing (9) for each time step involves repetitively fitting neural networks, which is time-consuming, we do a warm start at each round. In particular, we set the initial weights for the neural network of each round to be the weights of the trained network from the previous round. We also implemented early stopping that stops the iteration if the loss does not significantly decrease for multiple consecutive iterations. The online regression oracle Oracle is instantiated as applying gradient descent for certain steps on the mean squared loss over all data collected so far, using warm start for speedup as well.
We first conduct experiments on a single expert setting. In Figure 2 we plot the curves of return and number of queries with respect to iterations for our method, and compare to DAgger (which passively makes queries at every time step; Ross and Bagnell [2014]). We note that while our algorithm does
Figure 1: Learning curves of return with respect to the number of queries for different values of  $\alpha$  and different numbers of experts.
not converge to the optimal value as fast as DAgger, the number of queries made by our algorithm is significantly fewer, which means that our method is indeed balancing the speed of learning and the number of queries.
Figure 2: Learning curves of the return and the number of queries for 1 expert.
In additional to DAgger, we also compare to the following baselines:
- Passive learning. By passive learning, we mean running our algorithms with  $Z_{t,h} = 1$ , i.e., making queries whenever possible. Based on different styles of expert feedback, we divide the passive learning baselines into two: noisy experts and noiseless experts. For the former we get the noisy label  $y_{t,h}^{m}$  for  $x_{t,h}$  (generated by  $y_{t,h}^{m} \sim \phi(f_{h}^{\star,m}(x_{t,h}))$ ), and for the latter we directly get the action of the optimal policy (i.e. the action  $\pi_{h}^{\star}(x_{t,h})$ ). Intuitively, noiseless feedback is more helpful than the noisy one.
- MAMBA. We compare our algorithm with (a slight variant of) MAMBA [Cheng et al., 2020]. At each time step, it creates copies of the environment and run each expert policy on these copies, and then it selects the action of the expert policy with the highest return. For simplicity, we refer to this algorithm as MAMBA. Note that MAMBA assumes that one has access to the underlying reward function. Thus this baseline is using significantly more information than our approach.
- Best expert. We also compared our algorithm with the best expert policy.
The main results are shown in Figure 3. We first noticed that our algorithm outperforms passive learning with noisy experts in all settings. Moreover, we beat the noiseless version when there is only one expert. Intuitively, getting feedback from noiseless experts is a very strong assumption and it is not surprising to see that the performance is improved with this stronger feedback. Note that our algorithm is only getting noisy labels as feedback. We also note that, despite the fact that MAMBA achieves better results than the best expert policy (in terms of the value function), it is still worse than our algorithm. Indeed, MAMBA does not even learn a policy that can solve the task when  $M \geq 2$ . This is because by our construction of experts, there is no single expert that is capable of solving the task alone. Note that MAMBA performs well in the one expert case because in that case, the (single) expert can reliably solve the control task.
Figure 3: Learning curves of return with respect to the number of queries for different algorithms and numbers of experts.
B Further Discussion on Related Works
Selective Sampling. There is a large bank of both theoretical and empirical work for active learning and selective sampling. Perhaps the work closest to ours is the work of Zhu and Nowak [2022]. In this paper, the authors consider binary classification problem and provide bounds on number of queries and bound on excess risk in the active learning framework. Their algorithm also relies on regression oracle. However, there are many key differences: Firstly, their guarantees for regret for selective sampling problem (see for instance Theorem 10 on page 28 of Zhu and Nowak [2022]) has a dependence on disagreement coefficient in the regret bound as well as number of queries. On the contrary, as we show in our work, one only needs to pay for eluder dimension or disagreement coefficient in query complexity and not in regret bound. Furthermore, we supplement our result with lower bound showing that unless one has label complexity that depends on star number (and hence can be also related to worst case disagreement coefficient), one can not get a small enough regret bound. So the separation between regret bound (that is independent of eluder dimension/star number/disagreement coefficient) and query complexity (that depends on those quantities) is real. Secondly, the results in [Zhu and Nowak, 2022] dont automatically adapt to the margin region and in general there is no way to estimate the parameters of Tsybakov's noise condition. Finally, their regret bounds depend on pseudo dimension and are thus generally suboptimal for complex  $\mathcal{F}$ .
Imitation Learning. IL has enjoyed tremendous research from both theoretical and empirical perspective in the last decade; notable references include Ross et al. [2011], Ross and Bagnell [2014], Sun et al. [2017], Chang et al. [2015], Brantley et al. [2019, 2020], Nguyen and Daumé III [2020]. Ross et al. [2011] initiated research on using online regression oracles to model the expert feedback, and provided regret bounds for IL. The key differences between our work and prior theoretical works on IL are as follows: Firstly, we consider active querying, and provide query complexity bounds for our algorithms. Secondly, and more importantly, we consider interactive IL with noisy expert feedback whereas prior works was restricted to exact expert feedback. Finally, our regret and query complexity bounds scale with the number of times when the comparator policy (induced by the expert) goes to the states where expert has a small margin (instead of the number of times when the learner goes to such states). In many cases, the margin error term corresponding to the comparator policy could be much smaller. On the empirical side, there is a long line of research that provided algorithms and empirical heuristics for making IL sample efficient by modeling the experts in both single expert and multiple expert settings [Beliaev et al., 2022, Cao et al., 2022, Hejna and Sadigh, 2023, Du et al., 2023, Hao et al., 2022]; however most of these algorithms do not come with any rigorous guarantees.
C Discussion on Computationally Efficiency
Given the theoretical nature of this paper, our focus is to understand the statistically efficient (query complexity), and to develop algorithms, for selective sampling and imitation learning with general model classes  $\mathcal{F}$ , given access to an online regression oracle w.r.t.  $\mathcal{F}$ . For a general function class, our algorithm is computationally inefficient. However, in many cases, our algorithm (or its slight modification) can also be implemented efficiently. We describe some scenarios below:
- 1D-Linear models: When  $f: \mathcal{X} \mapsto \mathbb{R}$  is linear, the optimization objective in (5) in Algorithm 1 (or (6) in Algorithm 2) can be efficiently by instead solving the objectives
$$
\Delta_ {t} ^ {(1)} (x _ {t}) = \max  _ {f \in \mathcal {F}} f (x _ {t}) - f _ {t} (x _ {t}) \mathrm {s . t .} \sum_ {s = 1} ^ {t - 1} Z _ {s} | f (x _ {s}) - f _ {s} (x _ {s}) | \leq \Psi ,
$$
$$
\Delta_ {t} ^ {(2)} (x _ {t}) = \max _ {f \in \mathcal {F}} - \big (f (x _ {t}) - f _ {t} (x _ {t}) \big) \mathrm {s . t .} \sum_ {s = 1} ^ {t - 1} Z _ {s} | f (x _ {s}) - f _ {s} (x _ {s}) | \leq \Psi
$$
and then picking the maximum absolute value. Both of these new objectives are linear functions, with convex constraints, and thus can be solved efficiently using a standard solver (e.g. CvxOPT).
- Differentiable parameterizations: When class  $\mathcal{F}$  could be parameterized in a differentiable way, e.g. using a neural network, we can simply add the constraints as a penalty (with the appropriate multiplicative scale parameter) to convert (5) into an unconstrained optimization problem w.r.t the parameters  $\theta$  (of the differentiable parameterization) and then solve it using SGD algorithm. While this is only a heuristic, it works well in practice and is precisely what we do for our experiments in A with 2 layer neural networks.
- Efficient Implementation of (5) via calls to a Regression Oracles w.r.t.  $\mathcal{F}$ . Below we discuss some scenarios and minor modifications of our algorithms under which the computation of  $\Delta_t$  (as in (5) in Algorithm 1 or (6) in Algorithm 2) can be performed efficiently via calls to a regression oracle w.r.t.  $\mathcal{F}$ . Suppose that  $\mathcal{F}$  is closed under convexification. We consider two scenarios:
- Binary Actions Setting: In this case, we can simply choose  $\mathcal{F}$  to be a class of 1D functions, since we can model the expert using a function  $f^{\star}:\mathcal{X}\mapsto \mathbb{R}\in \mathcal{F}$  where for any  $x$ ,  $f^{\star}(x)$  denotes the probability of choosing the first action. The probability of choosing the second action would then be  $1 - f^{\star}(x)$ . In this case, (5) in Algorithm 1 simply reduces to
$$
\Delta_ {t} ^ {\prime} (x _ {t}) = \max _ {f \in \mathcal {F}} | f (x _ {t}) - f _ {t} (x _ {t}) | \mathrm {s . t .} \sum_ {s = 1} ^ {t - 1} Z _ {s} | f (x _ {s}) - f _ {s} (x _ {s}) | \leq \Psi .
$$
The above can be implemented efficiently using the techniques from Foster et al. [2018a]. In particular, let  $r_{\mathrm{max}}$  denote the maximum value that  $f(x)$  can take. We can solve the above objective using the BINSEARCH procedure in Foster et al. [2018a] where we perform a binary search over a weight parameter  $w$ , by solving for each  $w$  the optimization problems:
$$
\arg \min  _ {f \in \mathcal {F}} \left(w \cdot \left(f (x _ {t}) - f _ {t} (x _ {t}) - 2 r _ {\max }\right) ^ {2} + \sum_ {s = 1} ^ {t - 1} Z _ {s} | f (x _ {s}) - f _ {s} (x _ {s}) |\right)
$$
$$
\arg \min _ {f \in \mathcal {F}} \left(w \cdot \left(f (x _ {t}) - f _ {t} (x _ {t}) + 2 r _ {\max}\right) ^ {2} + \sum_ {s = 1} ^ {t - 1} Z _ {s} | f (x _ {s}) - f _ {s} (x _ {s}) |\right)
$$
both of which can be efficiently implemented using square loss regression oracles to  $\mathcal{F}$ . We refer to Foster et al. [2018a] for more details.
- Multiple Actions Setting: Suppose that  $\mathcal{F}$  is a product class of  $\{\mathcal{F}_k\}$  for different actions  $k \in [K]$ . We can get an oracle-efficient algorithm for a slight modification of (5), at a price of an extra multiplicative  $K$  factor in the query complexity bound. Consider the  $\Delta_t(x_t)$  given by  $\max_{f \in \mathcal{F}} \| f(x_t) - f_t(x_t) \|_\infty$  s.t.  $\sum_{s=1}^{t-1} Z_s \| f(x_s) - f_s(x_s) \|_2 \leq \Psi$ , which is equal to  $\max_{k \in [K]} \max_{f \in \mathcal{F}} |f_k(x_t) - f_{t,k}(x_t)|$  s.t.  $\sum_{s=1}^{t-1} Z_s \| f(x_s) - f_s(x_s) \|_2 \leq \Psi$ , which can again be implemented via calls to a square loss regression oracle w.r.t.  $\mathcal{F}$  by using BINSEARCH procedure in Foster et al. [2018a] (similar to what we did for the Binary actions case above).
D Useful Tools and Notation
Additional notation. Throughout the paper, we assume that the ties are broken arbitrarily but consistently. Vector-valued variables are denoted with small alphabets like  $u, v$ , etc, and matrix-valued variables are denoted with capital alphabets like  $F, G$ , etc. For any two distributions  $D_{1}$  and  $D_{2}$ , we define  $\mathrm{KL}(D_1\| D_2)$  to denote the KL divergence between  $D_{1}$  and  $D_{2}$ . Furthermore,  $\mathrm{kl}(b_1\| b_2)$  denotes the KL divergence between Bernoulli  $(b_{1})$  and Bernoulli  $(b_{2})$ . Finally, we assume that  $\| f(x)\| \leq B \leq 1$  for any  $f \in \mathcal{F}$  and  $x \in \mathcal{X}$ .
The following lemma is used throughout the appendix, and its proof is trivial.
Lemma 1. Let  $\mathcal{E}_1$  and  $\mathcal{E}_2$  be any two events such that  $\mathcal{E}_1 \Longrightarrow \mathcal{E}_2$  then  $\mathbf{1}\{\mathcal{E}_1\} \leq \mathbf{1}\{\mathcal{E}_2\}$ .
D.1 Basic Probabilistic Tools