Locality Sensitive Teaching

The emergence of the Internet-of-Things ( IoT ) sheds light on applying the machine teaching ( MT ) algorithms for online personalized education on home devices . This direction becomes more promising during the COVID-19 pandemic when in-person education becomes infeasible . However , as one of the most influential and practical MT paradigms , iterative machine teaching ( IMT ) is prohibited on IoT devices due to its inefficient and unscalable algorithms . IMT is a paradigm where a teacher feeds examples iteratively and intelligently based on the learner ’ s status . In each iteration , current IMT algorithms greedily traverse the whole training set to find an example for the learner , which is computationally expensive in practice . We propose a novel teaching framework , Locality Sensitive Teaching ( LST ) , based on locality sensitive sampling , to overcome these challenges . LST has provable near-constant time complexity , which is exponentially better than the existing baseline . With at most 425.12× speedups and 99.76 % energy savings over IMT , LST is the first algorithm that enables energy and time efficient machine teaching on IoT devices . Owing to LST ’ s substantial efficiency and scalability , it is readily applicable in real-world education scenarios . 1 Introduction . During the COVID-19 pandemic , there is an increasing demand for learning at home . Computerbased personalized education ( CBPE ) [ 1 , 2 , 3 ] on Internet-of-Things ( IoT ) devices become essential as it improves the accessibility of students to the educational resources and reduces the potential privacy risks . Due to the popularity of Coursera , Duolingo , and EdX , online on-device education has become increasingly more important nowadays . To better understand and improve the CBPE approaches in this context , we consider machine teaching ( MT ) [ 4 ] as a simplified yet helpful paradigm . Specifically , MT defines the problem where a machine teacher constructs a minimal set of examples that allows a student to learn a target concept repeatedly . However , MT on the IoT device is still prohibitive for two primary reasons : ( 1 ) MT technique does not allow real-time interaction between teachers and students . It is designed to work over a static set and can not incorporate realtime student feedback , ( 2 ) MT can not support on-device training due to the expensive cost spent in constructing this minimal teaching set . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As a result , a more practical paradigm – iterative machine teaching ( IMT ) [ 5 , 6 ] has been proposed to achieve state-of-the-art teaching performance . In IMT , the teacher interacts with the student in every iteration and aims to teach the target concept with a minimum number of iterations . Although IMT reduces both teaching set size and teaching iterations , it is still not efficient enough . The time and energy required by IMT are prohibitively expensive on IoT devices . There are two primary reasons : ( 1 ) IMT is not designed for real-time teaching . Current IMT uses a deterministic greedy algorithm , which traverses the entire dataset to find the optimal teaching example during each iteration . This linear time scan costs excessive time in real-world datasets , preventing more frequent interactions between machine teachers and students . ( 2 ) IMT has poor energy efficiency because the greedy scan requires scoring and ranking on the training examples . The energy budget of IoT devices such as integrated CPU-GPU System-on-Chips ( SoCs ) can not afford this expensive energy consumption in real-world teaching . Clearly , there is a gap between IMT and its deployment on IoT devices . Contemporary IMT literature focuses on developing and analyzing theories , models , or paradigms in teaching . Few studies are addressing the trade-offs between teachability and efficiency . Although some IMT algorithms manage their potential applications in crowd-sourcing [ 7 , 8 , 9 ] , personal education [ 4 ] , or online data poison [ 10 , 11 ] , they do not consider the edge computing and IoT settings and ignore the substantial potential impact behind it . Our work proposes a time and energy-efficient IMT algorithm , namely Locality Sensitive Teaching ( LST ) , that enables machine teaching on IoT devices . A real-life teaching observation inspires LST : a human teacher teaches a target concept to the student efficiently based on his or her knowledge in mind rather than a brutal-force scan over all the course materials . Similarly , an efficient IMT algorithm should provide optimal examples by looking up from a data structure rather than traversing the entire example set . Our LST algorithm effectively pre-indexes the teaching examples in hash tables . Then , given the student ’ s current state at each iteration , we regard it as a query and generate examples through efficient sampling using the hash tables [ 12 ] . Moreover , we significantly improve the time and energy efficiency by reducing the expensive linear scan into a lightweight lookup in hash tables . As a result , LST could be compatible with edge computing on IoT devices . Our proposal , LST , comes with three key questions : ( 1 ) how to reformulate the IMT problem as a sampling problem that involves efficient hash table data structures ? ( 2 ) how to maintain the identical teachability of IMT when utilizing the efficient hash table structures ? ( 3 ) how to provide an efficient LST implementation on IoT devices that reduce energy and time consumption ? At a high level , we tackle these challenges as follows : • We reformulate IMT as an adaptive inner product sampling problem . We partition the teaching materials into two parts : ( 1 ) query : current state of student and the optimal model possessed by the machine teacher , ( 2 ) data : teaching examples denoted as feature-label pairs . We apply an asymmetric transformation that projects both parts as vectors . Then , we argue that the original IMT formula can be formulated as an adaptive inner product sampling . Given the query vector , the task is to sample data vectors with large inner products . Therefore , locality sensitive sampling can be introduced for efficient teaching examples generation . • We demonstrate , both theoretically and empirically , that LST preserves the teachability of IMT . Theoretically , we prove the LST can achieve exponential teachability with high probability . Empirically , the experiments on real-world teaching indicate that LST matches or even exceed the teachability of original IMT . • We provide a novel LST system design on integrated CPU-GPU SoC platforms that performs time and energy-efficient teaching in real-world settings . We re-partition the locality sensitive sampling procedure into two parts : GPU-friendly random projection and CPU-friendly hash table lookups . Then , we exploit the benefits of fast matrix multiplications on GPU and efficient hash table lookups implemented on CPUs . Therefore , our LST system takes full advantage of the memory-constrained CPU-GPU SoCs . Through extensive experiments in real-world teaching scenarios , we demonstrate that LST performs exponential teachability that matches or even exceeds IMT while achieving at most 425.12× speedups and 99.76 % energy savings on IoT devices . On server-based evaluation , LST achieves more than 2000× speedups over IMT . 2 Related Work and Preliminaries . 2.1 Iterative Machine Teaching . In machine teaching , a machine teacher obtains a minimal training set for a student to learn a concept . [ 4 , 13 ] introduce a general teaching framework and build its connections to curriculum learning [ 14 ] . [ 15 ] studies the machine teaching for Bayesian learners in exponential family and provides the teaching example by solving an optimization problem . [ 16 ] gives the linear learners ’ teaching dimension . Machine teaching has been found useful in cybersecurity [ 17 ] , human-computer interaction [ 18 ] , and human education [ 19 ] . [ 20 , 6 , 21 ] study the teaching scenario where the teacher aims to teach multiple concepts to a forgetful learner . [ 22 , 6 ] assume that the learner is black-box and study how the teaching should be performed . [ 23 , 6 ] discuss how the teacher can teach multiple different learners . [ 24 , 25 , 26 ] connect the machine teaching problem with inverse reinforcement learning by studying an inverse reinforcement learner . [ 27 , 2 ] consider the machine teaching from an interpretable perspective . [ 28 , 29 ] discuss the teaching model with a version space learner . [ 3 ] provides machine teaching extensions in human-in-the-loop settings . [ 30 , 31 , 32 ] study the machine teaching in theory . Previous machine teaching works focus on the improvement of teachability . [ 5 ] proposes the iterative teaching paradigm . In this paradigm , an omniscient teacher model knows almost everything about the student . Then , the omniscient teacher provides training examples based on the student ’ s status . [ 33 ] further generalizes this teaching scenario and introduces an optimal control approach to address sequential machine teaching . Built upon the same framework as [ 5 ] , [ 6 ] looks into the IMT problem when the learner becomes black-box . [ 34 ] proposes to perform iterative teaching by label synthesis while still preserving the provable teaching speedup . In our work , we address the scalability bottleneck of IMT by leveraging the power of locality sensitive hashing . This paper follows the same teaching setting as [ 5 ] . Without loss of generality , we mainly discuss the omniscient teaching model where the teacher knows the learner ’ s full information . We apply our LST to speed up the omniscient teacher . We note that it is straightforward to improve the teaching quality and efficiency of more advanced teaching models using our proposed method . 2.2 Hashing-based Sampling . This section briefly describes the recent development of efficient sampling and estimation via locality sensitive hashing ( LSH ) [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ] . The intuition of LSH is to hash similar items into the a bucket of a hash table via functions such as random projection . For details of LSH , we refer to Appendix A . We denote p as the collision probability of LSH function . Therefore , given a query , an item will be retrieved with probability p. The precise form of p is determined by the LSH function family . This sampling view of LSH has been used in a wide range of applications , such as neural networks training [ 47 , 48 ] , kernel density estimation [ 49 ] , outlier detection [ 50 ] , and optimization [ 51 ] . The success of LSH in efficient inner product search sheds light on its application in adaptive sampling . More specifically , in this paper , given a collection C of input and a query Q , candidates S are drawn from C , in which each xi ∈ C is sampled with pi ≤ 1 , using LSH algorithm with special designed hashing [ 52 ] . Note here pi is a monotonically increasing function of Q · xi . Therefore , once we pay a linear preprocessing cost that maps C into hash tables , the adaptive sampling for query Q could be done via a few hash lookups . The LSH sampler component in Figure 1 demonstrates the sampling process using LSH . Similar to what has been mentioned in [ 51 ] , the sampling scheme is not a valid distribution , i.e. , ∑ xi∈C pi 6= 1 . Given a query , the probability of sampling xi is not independent of the probability of sampling xj ( i 6= j ) . However , we can still use it for unbiased estimation . pi is a monotonic function of Q · xi because pi = ( 1 − ( 1 − g ( Q · xi ) ) K ) L , where g ( Q · xi ) is the collision probability .

Summary: This paper considers an iterative machine learning task of how an omniscient teacher choosing learning data optimally such that the student can learn fast from the data via gradient descent. It proposes an adaptive inner product sampling (AIPS) algorithm that uses locality-sensitive hashing-based sampling to make the search much more efficient than greedy approaches. The authors show the exponential teachability of their approach and provide empirical studies.