Locality Sensitive Teaching

The emergence of the Internet-of-Things ( IoT ) sheds light on applying the machine teaching ( MT ) algorithms for online personalized education on home devices . This direction becomes more promising during the COVID-19 pandemic when in-person education becomes infeasible . However , as one of the most influential and practical MT paradigms , iterative machine teaching ( IMT ) is prohibited on IoT devices due to its inefficient and unscalable algorithms . IMT is a paradigm where a teacher feeds examples iteratively and intelligently based on the learner ’ s status . In each iteration , current IMT algorithms greedily traverse the whole training set to find an example for the learner , which is computationally expensive in practice . We propose a novel teaching framework , Locality Sensitive Teaching ( LST ) , based on locality sensitive sampling , to overcome these challenges . LST has provable near-constant time complexity , which is exponentially better than the existing baseline . With at most 425.12× speedups and 99.76 % energy savings over IMT , LST is the first algorithm that enables energy and time efficient machine teaching on IoT devices . Owing to LST ’ s substantial efficiency and scalability , it is readily applicable in real-world education scenarios . 1 Introduction . During the COVID-19 pandemic , there is an increasing demand for learning at home . Computerbased personalized education ( CBPE ) [ 1 , 2 , 3 ] on Internet-of-Things ( IoT ) devices become essential as it improves the accessibility of students to the educational resources and reduces the potential privacy risks . Due to the popularity of Coursera , Duolingo , and EdX , online on-device education has become increasingly more important nowadays . To better understand and improve the CBPE approaches in this context , we consider machine teaching ( MT ) [ 4 ] as a simplified yet helpful paradigm . Specifically , MT defines the problem where a machine teacher constructs a minimal set of examples that allows a student to learn a target concept repeatedly . However , MT on the IoT device is still prohibitive for two primary reasons : ( 1 ) MT technique does not allow real-time interaction between teachers and students . It is designed to work over a static set and can not incorporate realtime student feedback , ( 2 ) MT can not support on-device training due to the expensive cost spent in constructing this minimal teaching set . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As a result , a more practical paradigm – iterative machine teaching ( IMT ) [ 5 , 6 ] has been proposed to achieve state-of-the-art teaching performance . In IMT , the teacher interacts with the student in every iteration and aims to teach the target concept with a minimum number of iterations . Although IMT reduces both teaching set size and teaching iterations , it is still not efficient enough . The time and energy required by IMT are prohibitively expensive on IoT devices . There are two primary reasons : ( 1 ) IMT is not designed for real-time teaching . Current IMT uses a deterministic greedy algorithm , which traverses the entire dataset to find the optimal teaching example during each iteration . This linear time scan costs excessive time in real-world datasets , preventing more frequent interactions between machine teachers and students . ( 2 ) IMT has poor energy efficiency because the greedy scan requires scoring and ranking on the training examples . The energy budget of IoT devices such as integrated CPU-GPU System-on-Chips ( SoCs ) can not afford this expensive energy consumption in real-world teaching . Clearly , there is a gap between IMT and its deployment on IoT devices . Contemporary IMT literature focuses on developing and analyzing theories , models , or paradigms in teaching . Few studies are addressing the trade-offs between teachability and efficiency . Although some IMT algorithms manage their potential applications in crowd-sourcing [ 7 , 8 , 9 ] , personal education [ 4 ] , or online data poison [ 10 , 11 ] , they do not consider the edge computing and IoT settings and ignore the substantial potential impact behind it . Our work proposes a time and energy-efficient IMT algorithm , namely Locality Sensitive Teaching ( LST ) , that enables machine teaching on IoT devices . A real-life teaching observation inspires LST : a human teacher teaches a target concept to the student efficiently based on his or her knowledge in mind rather than a brutal-force scan over all the course materials . Similarly , an efficient IMT algorithm should provide optimal examples by looking up from a data structure rather than traversing the entire example set . Our LST algorithm effectively pre-indexes the teaching examples in hash tables . Then , given the student ’ s current state at each iteration , we regard it as a query and generate examples through efficient sampling using the hash tables [ 12 ] . Moreover , we significantly improve the time and energy efficiency by reducing the expensive linear scan into a lightweight lookup in hash tables . As a result , LST could be compatible with edge computing on IoT devices . Our proposal , LST , comes with three key questions : ( 1 ) how to reformulate the IMT problem as a sampling problem that involves efficient hash table data structures ? ( 2 ) how to maintain the identical teachability of IMT when utilizing the efficient hash table structures ? ( 3 ) how to provide an efficient LST implementation on IoT devices that reduce energy and time consumption ? At a high level , we tackle these challenges as follows : • We reformulate IMT as an adaptive inner product sampling problem . We partition the teaching materials into two parts : ( 1 ) query : current state of student and the optimal model possessed by the machine teacher , ( 2 ) data : teaching examples denoted as feature-label pairs . We apply an asymmetric transformation that projects both parts as vectors . Then , we argue that the original IMT formula can be formulated as an adaptive inner product sampling . Given the query vector , the task is to sample data vectors with large inner products . Therefore , locality sensitive sampling can be introduced for efficient teaching examples generation . • We demonstrate , both theoretically and empirically , that LST preserves the teachability of IMT . Theoretically , we prove the LST can achieve exponential teachability with high probability . Empirically , the experiments on real-world teaching indicate that LST matches or even exceed the teachability of original IMT . • We provide a novel LST system design on integrated CPU-GPU SoC platforms that performs time and energy-efficient teaching in real-world settings . We re-partition the locality sensitive sampling procedure into two parts : GPU-friendly random projection and CPU-friendly hash table lookups . Then , we exploit the benefits of fast matrix multiplications on GPU and efficient hash table lookups implemented on CPUs . Therefore , our LST system takes full advantage of the memory-constrained CPU-GPU SoCs . Through extensive experiments in real-world teaching scenarios , we demonstrate that LST performs exponential teachability that matches or even exceeds IMT while achieving at most 425.12× speedups and 99.76 % energy savings on IoT devices . On server-based evaluation , LST achieves more than 2000× speedups over IMT . 2 Related Work and Preliminaries . 2.1 Iterative Machine Teaching . In machine teaching , a machine teacher obtains a minimal training set for a student to learn a concept . [ 4 , 13 ] introduce a general teaching framework and build its connections to curriculum learning [ 14 ] . [ 15 ] studies the machine teaching for Bayesian learners in exponential family and provides the teaching example by solving an optimization problem . [ 16 ] gives the linear learners ’ teaching dimension . Machine teaching has been found useful in cybersecurity [ 17 ] , human-computer interaction [ 18 ] , and human education [ 19 ] . [ 20 , 6 , 21 ] study the teaching scenario where the teacher aims to teach multiple concepts to a forgetful learner . [ 22 , 6 ] assume that the learner is black-box and study how the teaching should be performed . [ 23 , 6 ] discuss how the teacher can teach multiple different learners . [ 24 , 25 , 26 ] connect the machine teaching problem with inverse reinforcement learning by studying an inverse reinforcement learner . [ 27 , 2 ] consider the machine teaching from an interpretable perspective . [ 28 , 29 ] discuss the teaching model with a version space learner . [ 3 ] provides machine teaching extensions in human-in-the-loop settings . [ 30 , 31 , 32 ] study the machine teaching in theory . Previous machine teaching works focus on the improvement of teachability . [ 5 ] proposes the iterative teaching paradigm . In this paradigm , an omniscient teacher model knows almost everything about the student . Then , the omniscient teacher provides training examples based on the student ’ s status . [ 33 ] further generalizes this teaching scenario and introduces an optimal control approach to address sequential machine teaching . Built upon the same framework as [ 5 ] , [ 6 ] looks into the IMT problem when the learner becomes black-box . [ 34 ] proposes to perform iterative teaching by label synthesis while still preserving the provable teaching speedup . In our work , we address the scalability bottleneck of IMT by leveraging the power of locality sensitive hashing . This paper follows the same teaching setting as [ 5 ] . Without loss of generality , we mainly discuss the omniscient teaching model where the teacher knows the learner ’ s full information . We apply our LST to speed up the omniscient teacher . We note that it is straightforward to improve the teaching quality and efficiency of more advanced teaching models using our proposed method . 2.2 Hashing-based Sampling . This section briefly describes the recent development of efficient sampling and estimation via locality sensitive hashing ( LSH ) [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ] . The intuition of LSH is to hash similar items into the a bucket of a hash table via functions such as random projection . For details of LSH , we refer to Appendix A . We denote p as the collision probability of LSH function . Therefore , given a query , an item will be retrieved with probability p. The precise form of p is determined by the LSH function family . This sampling view of LSH has been used in a wide range of applications , such as neural networks training [ 47 , 48 ] , kernel density estimation [ 49 ] , outlier detection [ 50 ] , and optimization [ 51 ] . The success of LSH in efficient inner product search sheds light on its application in adaptive sampling . More specifically , in this paper , given a collection C of input and a query Q , candidates S are drawn from C , in which each xi ∈ C is sampled with pi ≤ 1 , using LSH algorithm with special designed hashing [ 52 ] . Note here pi is a monotonically increasing function of Q · xi . Therefore , once we pay a linear preprocessing cost that maps C into hash tables , the adaptive sampling for query Q could be done via a few hash lookups . The LSH sampler component in Figure 1 demonstrates the sampling process using LSH . Similar to what has been mentioned in [ 51 ] , the sampling scheme is not a valid distribution , i.e. , ∑ xi∈C pi 6= 1 . Given a query , the probability of sampling xi is not independent of the probability of sampling xj ( i 6= j ) . However , we can still use it for unbiased estimation . pi is a monotonic function of Q · xi because pi = ( 1 − ( 1 − g ( Q · xi ) ) K ) L , where g ( Q · xi ) is the collision probability .

Summary: By reformulating Iterative Machine Teaching’s (IMT) optimization problem to an adaptive sampling problem, the papers proposes a Locality Sensitive Teaching (LST) algorithm. LST achieves exponential teachability and reduces the time complexity of each iteration from O(N), where N is the number of teaching examples, to few O(1) lookups in hash tables. They also attach the IoT perspective to the problem.