Title: TEACHING TO TEACH BY STRUCTURED DARK KNOWL-EDGE

Abstract: To educate hyper deep learners, Curriculum Learnings (CLs) require either human heuristic participation or self-deciding difficulties of training instances. These coaching manners are blind to the coherent structures among examples, categories and tasks, which are pregnant with more knowledgeable curriculum-routed teachers. In this paper, we propose a general methodology Teaching to Teach (T2T). T2T is facilitated by Structured Dark Knowledge (SDK) that constitutes a communication between structured knowledge prior and teaching strategies. On one hand, SDK adaptively extracts structured knowledge by selecting a training subset consistent with the previous teaching decisions. On the other hand, SDK teaches curriculumagnostic teachers by transferring these knowledge to update their teaching policy. This virtuous cycle can be flexibly-deployed in most existing CL platforms and more importantly, very generic across various structured knowledge characteristics, e.g., diversity, complementarity and causality. We evaluate T2T across different learners, teachers and tasks, which significantly demonstrates that structured knowledge can be inherited by the teachers to further benefit learners' training.

1. INTRODUCTION:

From an infant to a fully functional adult, human being requires years of highly advanced education. It purposively uses pedagogical instruments, demonstrates suitable examples and organizes targeted examinations, so as to reduce human being's time to equip with knowledge and skills. Drawing lessons from such social evolutionism, learning scientists proposed teaching Anderson et al. (1985) ; Goldman & Kearns (1995) , a coined terminology that broadly refers to the frameworks and algorithms guiding better training qualities for complicated machine learners, e.g., networks and agents. One way to teach them is to demonstrate examples through following the leitmotiv "from-easy-to-hard", famous as curriculum learning (CL) Bengio et al. (2009) . Specifically, Classical CLs (CCLs) Spitkovsky et al. (2010) ; Zaremba & Sutskever (2014) manage a syllabus (a dynamical training criteria) by ranking examples/tasks via increasing their difficulties from the perspective of human understanding1 . Then the difficulty threshold continuously updates to tolerate harder examples and tasks (Fig. 1 . a.upper). Sometimes, CCLs inevitablely entail human priors causing extra annotation and heuristically tuning, thus, are limited due to the algorithmic transferability. By contrast, Interactive CLs (ICLs) including SPL Kumar et al. (2010) ; SPCL Jiang et al. (2015) ;ACL Graves et al. (2017) ; TSCL Matiisen et al. (2017) ;MentorNet Jiang et al. (2017) ;sampling-based T2L Fan et al. (2018) and more, receive online training feedbacks as the difficulty signals to update their syllabuses. Their learner-teacher communication protocols reap the curriculum advantage without labor involvement (Fig. 1 . a.lower). So despite inconsistent performances across different learning scenarios Sachan & Xing (2016) ; Fan et al. (2018) , ICLs maintain prevalent in the frontier of CL researches.

CCLs pay attention to human-explainable teaching while ICLs believe that only black-box learners can bring up a machine-suited curriculum. These cutting-edge studies indulging the argument about these pedagogical ordering styles, rarely realize that, most knowledge emerges as a natural macrocosm rather than a pile of isolated or artificially ordered pieces. For instance, a biological taxonomy consists of levels of creature species in a hierarchy; sentences and paragraphs imply logic rules of writing; a knowledge base bridges concepts embedded in undirected graphs. These structures integrate sporadic show our T2T where CCLs and ICL perform by interacting with a structured dark knowledge cycle (Section.3.2 ). Then CCL and ICL determine their teaching decisions based on not only the curricula but also the structured knowledge instructions. pieces of knowledge, e.g., demonstrated examples, concepts (attributes and categories), multiple tasks, to reflect cognitive association characteristics, e.g., ambiguity, diversity, complementarity, causality, etc. It promises across-the-aboard education quality and has already grabbed a great amount of attentions in the field of pedagogical psychology Kirkpatrick & Epstein (1992) . Regretfully, retrospect to massive literatures related to CLs in details and we find that, seldom were proposed under this consideration. A few of studies directly treated the case as an evolving training subset selection with diversity, based upon self-paced learning Jiang et al. (2014) Zhou & Bilmes (2018) or some tricks for a concrete problem Sachan & Xing (2016) . They are situated under the difficulty-specific background and absent for generalization. Besides, their selected sets are directly used to optimize the learner, which implies the deterministic binary weights of training examples. It performs inferior compared with the other CL strategies promoting soft weighting or stochastic sampling technique.

Just as American novelist Ralph Ellison said " Education is all a matter of building bridges ", to bridge pieces of knowledge therefore rectify curriculum-based machine education, is what our work chases after. Note that, rather than crafting a specific CL algorithm or framework, we prefer teaching to teach (T2T), namely, distilling the substructure to dig out the possible coherence of training instances, e.g., examples, categories and tasks, then teach curriculum-routed teachers from existing CLs. As illustrated by the comic in Fig. 1 .b, our methodology extends CCLs and ICLs by a teacher of teacher mastering structured knowledge, which is transferred to curriculum-routed teachers within a virtuous cycle: On one hand, this teacher of teacher "comprehends" the previous CCL/ICL teaching decisions, thus adaptively selects a knowledge-based substructure to "instruct" the curriculum-routed teachers. On the other hand, the curriculum-routed teachers merge the curricula with their updated structured knowledge to iteratively polish their teaching strategies toward learners.

More specifically, let's go through the technical discussion roadmap of this paper. In Section.3.1, we first revisit CCL and ICL approaches and frame them into the background of progressive reweighting learning. Then we observe that, the weight of each training instance inferred by CCL/ICL is valued in [0, 1]. In a stochastic sampling circumstance, they also represent the instance-selection probabilities, thus, the inference can be regulated by KL divergence, similar to transferring dark knowledge Hinton et al. (2015) by matching their activation outputs. But how to construct the structured knowledge to properly regulate the probabilistic weight inferences ? In Section.3.2, we introduce a set function as our structured knowledge prior, namely, the teacher of teacher shown in Fig. 1 .b. The set function could be submodular Fujishige (2005) or just preserves submodular-like properties Das & Kempe (2011) Zhou & Spanos (2016b) (More specification refers to our Appendix.B). Our set function cooperates with the previous CL teaching decisions by matching their outputs, thus selects a subset of training instances to adaptively extract the structured knowledge. When curriculum-routed teaching strategy updates, this activation matching plays a role to transfer the structured knowledge to infer the curriculum teaching strategy. This cycle can be flexibly deployed in most existing CL strategies then incorporate structured knowledge to enhance their teaching performances.

Compared with previous work about CLs, our methodology embodies three apparent virtues:

• Generality for structured knowledge. Our methodology employs a generic set function as the structured knowledge prior to perform a constrained subset selection, where the structured dark knowledge only performs as a modular function. It is in harmony with diverse forms of subset selection, thus, refers to various structured knowledge among data. • Flexibility for curricula. Our methodology could be flexibly deployed on all curriculumbased teaching strategies driven by, e.g., models, algorithms, preset rules, even choices from human beings, as long as they obey the generalized CCL and ICL formulas in SubSection.3.1. • Simplicity for implementation. Our methodology connects teaching and teaching to teach with only a structured dark knowledge term conceptually simple for implementation.

Finally, we verify our methodology in three scenarios: classification, domain adaptation and sequence learning. We import the structured knowledge priors about diversity, complementarity and causality to their experimental setups. Empirical studies across diverse learners and teachers show that structured dark knowledge can be acquired by the teachers and helpful for them to educate the learners better.

2. RELATED LITERATURES:

Teaching has gradually become an attractive AI research direction Khan et al. (2011); Zhu (2015) ; Zhu (2013) . Our work keeps cohesive with two thriving trends about how to teach.

2.1. TEACHING TO LEARN WITH CURRICULUM:

Plenty of researches concerned the learning principle starting small Elman (1993). Bengio et al. (2009) suggested that a series of training criterion by increasing the sample-based learning difficulties are able to accelerate the trainings or improve the performances of networks. This cognition-steered ideology was interpreted as curriculum learning (CL). Though born as a heuristic for practitioners, CL attracts increasing interests of theoretician in explaining, e.g., extreme strategy in teaching dimension Khan et al. (2011) , relationship with importance sampling Katharopoulos & Fleuret (2018) . Under some circumstance, the CLs using ideal difficulty score Weinshall & Cohen (2018) are proved to boost linear regression learner's convergence rate. Doubtlessly, CLs inspired a line of subsequent deep-model-based AI investigations about how to imitate human behaviors, e.g., BCD number calculation Zaremba & Sutskever (2014) , game shooting from the first-person aspect Wu & Tian (2016) , etc, and especially suit robotic control Sanger (1994 ) Florensa et al. (2017) . Since these CLs are human-designed, we call them classical CLs (CCLs) in this paper.

Syllabuses in CCLs proceed in accordance with predefined schemes. For instance, a robotic arm is supposed to acquire the grasping motions from elementary to complex. By contrast, interactive curriculum learnings (ICLs) 

2.2. TEACHING TO LEARN WITH DARK KNOWLEDGE:

The development of dark knowledge originates from the mockup experiments about hyper energy physics, where the deep networks were leveraged to search for dark matter from the particle collider synthesis Sadowski et al. (2015) . Hinton et al. (2015) elaborated the method as knowledge distillation (KD), namely, seeking to transfer dark knowledge, the soft predictions containing not only correct but also meaty ambiguous wrong informations, from a sophisticated teacher to fresh student networks by matching their output activations. The primitive goal of dark knowledge is to compress a complicated model into a lightweight one while retaining comparative performance. There could be multiple teachers for one student Rusu et al. (2015) ; Ruder et al. (2016) , where the student is supposed to inherit all the teachers' capabilities. Recent researches discovered its potential to boost student network's performance Furlanello et al. (2018) . It explains the successes of its wide applications in the other areas Shin et al. (2017) ; Papernot et al. (2016); Lopez-Paz et al. (2015) .

Dark knowledge is broadly deemed as the gift from deep learning, whereas also surprisingly successful by hiring a teacher from the other ML areas. Hu et al. (2016) designed a rule-based teaching with soft first-order logic, then the student nets are iteratively updated to obey these principles. In Bayesian dark knowledge Korattikara et al. (2015) , a Monte Carlo posterior predictive density (e.g., stochastic gradient Lagevin dynamic) educates a student to endow a probabilistic model reaping the benefit of Bayesian tools yet only costing the same run time as those plug-in methods.

3. CURRICULUM MARRIES DARK KNOWLEDGE:

Let's begin with a typical supervised learning setting where

D = {(x i , y i )} |D| i=1

2 denotes training set and f θ (•) denote a machine learner w.r.t. θ. To include all situations in magnificent literatures about CL, we reconsider D as D ⊆ {z ∈ 2 |D| , |z| = m} and treats z as a single training instance. Correspondingly, L(z; θ) denotes a empirical risk based on a surrogate loss over ∀z ∈ D. In this case, z could be a training example (m = 1), or a group of samples belonging to the same class or task with the same size m. The primary objective is proposed as

EQUATION

where P (z) denotes the base training distribution over D and usually is uniform. Classical curriculum learning (CCL). A human prior mapping w λ z ∈ [0, 1] is employed by CCL to measure difficulty: Given a scalar λ > 0 per iteration and ∀z 1 , z 2 ∈ D, z 1 maintains harder than z 2 iff w λ (z1) ≤ w λ (z2) . Specify Q(z; λ) ∝ w λ (z)P (z) then Eq.1 evolves to

EQUATION

. The objective performs as a series of training criterion as λ iteratively increases. Q(z; λ) indicates the dynamical training distribution of samples. It is emphasized that, the increase of λ rigorously does not change the order of their weights. ∂w λ z ∂λ > 0 promises as CCL proceeds, training samples are ultimately accepted in a fair proportion. Due to the definition about training instance z, existing supervised learning frameworks that involve manual curriculum can be concluded as Eq.2 .

Interactive curriculum learning (ICL). CCL curates hand-crafted difficulty and update the training syllabus by increasing λ. In contrast, interactive curriculum learning (ICLs) receive training feedbacks to estimate the learner-oriented difficulty. Assume f θ (t-1) as the learner previously trained in the iteration t -1, then the current weight for z is written as w(z; θ (t-1) ) that represents the past training feedback. Using Q z; θ (t-1) ∝ w z; θ (t-1) P (z) ICL presents as min θ (t) L ICL (θ (t) ; θ (t-1) ) := E z∼Q(z;θ (t-1) )

L(z; θ (t) ).

(3)

Distinguished with the difficulties pre-planned by w λ z , Eq.3 develops the training criterion with w z, θ (t-1) where the difficulties are hidden. Conventionally, a well-performed training example in the previous iter is considered easy and beneficial to train the current model. It obtains a larger weight or higher stochastic sampling frequency in this training phase. Realize that, ICLs (Eq.3) should be implemented by specifying w(z, θ (t-1) ). Most ICLs implicitly contain their weight functions. In our Appendix.A, we elaborates them and show how to explicitly relate their strategy to w(z, θ (t-1) ).

Probabilistic shadows of curriculum-routed teachers. The curriculum-routed teachers of CCLs and ICLs might be explicit (updated by differentiable scheme functions Jiang et al. (2017) ), but more commonly are implicit (updated by an algorithm, a rule, an isolated agent or model). We observe that, their strategies take effects in training via w λ z or w z, θ (t-1) . These weights are located in [0, 1], then in a stochastic manner, approximately equal to the probabilities to select ∀z ∈ D. Precisely, we use P w to reconfigure the weights into a set of selection probabilities,

∀z i ∈ D, P w (g i ) = w i (1 -2 ) + if g i = 1 1 --w i (1 -2 ) if g i = 0 (4)

where g i denotes a binary random variable representing a selection flag of z i . w i unifies w λ z i and w z i , θ (t-1) for simplicity. 0 < << 1 prevents the arithmetic pathological problem when applying log operator. In fact, given → 0, it preserves P w (z i ) → w i .

The above analysis implicates two perspectives. First, transferring knowledge to regulate the weight update, equally acts on teaching the curriculum-routed teachers. Second, due to the probabilistic view of w λ (z) and w(z; θ t-1 ), this knowledge could be transferred by matching the activations between the teaching decision P w (g i ) and some other selection probabilistic decision of structured knowledge.

The second point of view is very similar to knowledge distillations (see more comparison in Table .1 ). For teaching to teach, we develop a new way of probability matching termed Structured Dark Knowledge. On one hand, it helps to discover structured knowledge, namely, selects a subset S ⊆ D that illustrates the structured coherence in D and at the same time, keeps the dark-knowledge-driven consistency with the teaching decisions P w (•). On the other hand, it has been re-applied to transfer structured knowledge into the teacher, so as to influence the teaching decision update. This cycle iteratively performs to guide a curriculum-routed teacher absorbing structured knowledge to teach a learner.

3.2. STRUCTURED DARK KNOWLEDGE CYCLE:



Structured knowledge prior.:

There is no such thing as free lunch and the structured knowledge is no exception. To bring the external knowledge into teaching, we should first import a set function F (S) (∀S ⊂ D) as a carrier to embed structured knowledge prior, because a set is the basis to present magnificent kind of structure representations Anderson et al. (1985) .

Structured dark knowledge. Provided a set function F (•) over D as our prior constraint, we expect to select a subset S ⊆ D based upon not only the structured knowledge but also the consistency with the teaching decision P w (•). The consistency is constructed by a knowledge distillation term matching their selection decisions in probability. To achieve this goal, we revisit ∀S ⊆ D from a probabilistic view like w i :

if z i ∈ S, P F (g i ; S) = 1 - g i = 1 g i = 0 , if z i ∈ D/S, P F (g i ; S) = g i = 1 1 - g i = 0 (5)

Note that, P F (•; S) means the probability under the background of subset S, but not conditional density. Obviously, the density set {P F (g i ; S)} |D| i=1 refers to the subset selection result S if → 0. Then we propose a structured dark knowledge term D SDK to correlate P F (g i ; S) and P w (g i ):

D SDK (z i , w; S) = D KL (P F (g i ; S)|P w (g i )) = -(1 -) log wi(1-2 )+ 1- -log (1-)-wi(1-2 ) if z i ∈ S -log wi(1-2 )+ -(1 -) log (1-)-wi(1-2 ) 1- if z i ∈ D/S (6)

where D KL indicates the Kullback-Leibler (KL) divergence.

Structured dark knowledge maintains two interpretations replaying in a cycle. In the context of subset selection, it presents as the consistency measure on each instance between the specified structured knowledge (namely, subset S) and the previous teaching decision P w . It leads to their instance-level disagreement. Hence we are able to define the knowledge-based subset selection as

max S⊆D,|S|≤k J (S) = F (S) + β zi∈S 1 - D SDK (z i , w; S) C (7)

where γ is a balance factor and |S| ≤ k is a cardinality constraint. C = -(1 -2 ) log 1- upperbounds D SDK (z i , w|S) when z i ∈ S, thus, promises the second term always greater than 03 . Provided z i and z i performing close in the structured knowledge prior (F (S ∪ {z i }/{z j }) ≈ F (S)), a smaller D SDK (z i , w|S) indicates a good matching between the knowledge and curriculum on instance z i , which is more preferable to be a candidate.

Algorithm 1 T2T by SDK: General Algorithm 1: for each epoch do

2: S ← arg max S⊆D,|S|≤k J (S)

3:

for each mini-batch do 4:

if w := w φ then 5: φ ← arg min φ O(φ; S), w ← w φ 6: else 7: ŵ ← arg min ŵ O( ŵ; S), w ← ŵ 8: end if 9: Formulate min θ L CCL (θ; λ) or min θ (t)

L ICL (θ (t) ; θ (t-1) ) by w. Solve it to obtain θ.

10::

end for 11: end for 12: return θ * = θ.

The above objective is mostly NP-hard without any specification of the structured knowledge prior F . Yet under some situations (e.g., F obeys submodularity), it can be solved with a provable approximate ratio. Due to the space limit, we are going to discuss more concrete definitions and examples of F in our Appendix.B, which embed different structured knowledge indicating, e.g., diversity, complementarity and causality. The algorithms corresponding to those characteristics are also provided to solve Eq.7 .

Suppose the structured knowledge has been distilled by S, we turn to transfer them into our curriculum-routed teaching strategy. Under this context, D SDK (z i |S) become a constraint to update P w .

Structured knowledge transfer. The main challenge is to transfer the structured knowledge into a teacher instead of a learner. A learner is usually a parameterized model or agent updated by a differentiable learning process, which our structured dark knowledge can directly join to regulate. However, due to an extensive array of studies about CL, their teachers can be an algorithm, a fix update rule even directly coming from intentional choices of human being. One outlet is to infer a new strategy to balance the decisions of curriculum and the structured knowledge. Employing ŵi = ŵ(z i ) (∀z i ∈ D) to present a new balance strategy. We propose the objective as

EQUATION

where the weights are inferred by gradient decent. However, the most desirable case is that the teacher is also a learner and optimized by the loss function with f θ 's training feedback. Hence we are able to transfer the knowledge by posing D SDK (z i |S) on the teacher learning objective. Given w φ as this learnable teacher with parameter φ, the objective presents min

EQUATION

where M(w φ (t) , f θ (t-1) ) indicates the original teacher objective, where the current teacher is updated by the feedbacks from the previous learner. If the teacher is a RL agent , M(w (t) , f θ (t-1) ) is the negative of the value function and trained under the MDP setting.

4. EXPERIMENTS:

In this section, we evaluate our methodology by comprehensive empirical studies. They include most existing CL approaches across diverse tasks, from transfer learning to computational finance.

4.1 BASIC EXPERIMENTAL SETTING.

Before introducing our empirical studies, let's specify some common experimental details. For the setting of our structured dark knowledge (SDK), = 0.04 if the targeted CL applies reweighting strategy (e.g., SPL family); and set = 0.1 if they apply stochastic sampling schemes. It provides more exploration freedom for the sampling-based CL approaches. For the cardinality constraint in Eq.7, k starts from selecting 15% instances to construct S, then after per epoch, linearly increases till k = |D|. The duration is decided by the speed of the original learner's convergence. The motivation is to promise, if F monotonically increases, all the samples will be finally included.

4.1.1 CURRICULA AS APPRENTICES.

• Preset Curriculum (PC). To thoroughly understand the benefit of structured knowledge, our experiments should include CCLs. But traditionally, their curricula are manually-planned, task-oriented and worst of all, subjective across people. For a fair comparison, we follow the recent study Weinshall & Cohen (2018) and train a "prophet" network (higher capacity than the learner, we apply ResNet101 He et al. (2016) in our classification experiment), then rank the training data by increasing their losses on this well-trained network. Then we use this schedule to perform stochastic sampling as the preset curriculum (PC) in CCL.

• SPL (Eq.12), SPCL (mixture) (Eq.15). SPL and its variants are the elementary ICL methods.

According to their teaching principles, the training instances with larger losses are assigned smaller weights, then a threshold grows to tolerate the large losses so that eventually, all training losses are assigned equal coefficients. For the robustness in a stochastic manner, we follow the routine in Jiang et al. (2014) : For the data in each mini-batch, SPL Kumar et al.

(2010) and SPCL (mixture) Jiang et al. (2015) infer their weights, thus, we filter out top K weight-smallest samples. K starts from a quarter of a mini-batch, then it linearly decreases to 0 after E epochs.

• ACL (Eq.16). ACL belongs to the sub-branch of ICLs using a stochastic bandit to sample multiple tasks and construct mini-batches online. Each slot in the bandit refers to a task, and the reward comes from diverse learning progress signals. We use self-prediction gain signal (Eq.21 ) that shows an unbias progress estimation during the stochastic learning. We directly borrow the hyperparameter setup in Graves et al. (2017) , which has already been good enough to present ACL's superiority. ACL is only applied in multi-task learning.

In our structured dark knowledge transfer, the weights of the above CL methods are refined by Eq.8, then the newly inferred weights replace the originals to execute their CL strategies.

• Learning to teach (L2T) (Eq.24) Sampling-based L2T Our evaluation is conducted on several widely-adopted standard benchmarks in ML researches. They do not naturally provide structured knowledge among data. To built up this knowledge, we accept the "prophet" setting in PC. Specifically for the first experiment, we train a ResNet101 on the targeted benchmark and extract the training data before the last activation. These features marked as f * lying on an oracle semantic space, are used to measure the similarities among training data. For our second experiment, we follow a similar routine on the source and target domains respectively. Evaluation. PC, SPL and L2T can be viewed as sampling-based curriculum strategies. To illustrate the data-efficiencies they bring about, we evaluate the accuracies of their underlying learners according to their growing consumption of training data. To effectively reveal this case, we borrow the principle in Fan et al. (2018) : During their teaching processes, the learners will not be updated till M untrained, thus, to accumulate the selected instances. It promises the convergence speed relying on the quality of selected instances, which provides a good observation to evaluate curriculum strategies, and their chemical reactions with structured dark knowledge. More detailedly, compared with CL, ICL obtain mild improvement margins, and L2Ts obtain more benefits than SPLs. It probably implies that, the way of using SDK by Eq.9 performs more harmony with the learnable teaching methods.

Figure 4 : The change of relative performance improvement as we alter the balance parameter β.

Finding the Schrodinger's cat of structured dark knowledge. Despite of a notion inspired from dark matter, structured dark knowledge (SDK) is not a Schrodinger's cat. We know that dark knowledge indicates the wrong information about groundtruth yet its ambiguity is potentially helpful for learning. Under our context, it is explained by the discrepancy between P w and P (; S) across training set, thus, the value of D SDK (z i , w; S)(∀z ∈ D). We investigate

DSDK(zi,w;S) C

in average, and see how it changes as the curriculum strategies (PC and SPL) proceed. As shown in Fig. 3 , we find that no matter of PC or SPL, they start with hight and vibrated SDK values at the very beginning. It means that, the initial subset selection results are quite different with the teaching decision from CLs. Interestingly, as the training proceeds, the values are progressively become small and steady. On one hand, it is explained by our our progression manner of increasing k. On the other hand, it also demonstrates that, the structured knowledge can be "absorbed" by the curriculum-routed teacher underlying SPL and PC.

Besides, since the hyper-parameter β controls the knowledge-curriculum balance, we are able to observe SDK via altering β in Eq.7 . Our hyper-parameter sensitivity analysis is shown in Fig. 4 and for a fair comparison, we specify the evaluation when the learner just consumes 2×10 6 training instances. The measure (Relative Improvement Ratio) demonstrates the enhanced magnitude compared with the original PC and SPL strategies. Specifically, we find that when β is small, CLs equipped with SDK shows negative performance gain, thus, indicates that SDK will be harmful to the learners in this situation. It is because that, when β → 0, the selected subset become totally data-driven. Vividly speaking, the "voice" from the curriculum-routed teacher could not be heard by the teacher of teacher. So directly urging the teaching to obey this knowledge might bring about the structured information yet not suitable to the learners. It potentially spoils the training procedure. 2014) with the identical hyper-parameter setup has been adopted during the stochastic optimization. We promise its reproduced results as close as possible to their report in the original paper.

Curricula setup. SPL used to be a powerful tool in DA Tang et al. (2012) . For this consideration, we choose SPL and SPCL as our experimental base teachers to evaluate our T2T. Their curricula act on the DDC's adversarial confusion losses instead of classification losses. It helps to discover whether curricula and knowledge take the positive effects in domain transfer rather than the classifiers.

Knowledge setup. Our DA experiment focuses on the structured knowledge including diversity and complementarity. In specific, Concept and BP function (Eq.39) are employed as the priors F (S). Eq.7 under these priors are solved by Local Search and GreedMAX (Algorithm.3) respectively.

EMPIRICAL RESULTS AND IMPLICATIONS.

To illustrate more information, the curriculum-routed teachers and their SDK-armed variants are evaluated on the target and source domains, respectively. All the DA results are presented in Besides, compared with screening source-domain information, SPL and SPCL are better in reweighting examples from the target domain (three in the four best results come from reweighting target data). In SVHN→MNIST, their performances on the target domain are close to the source-based results. In MNIST→USPS experiment, SPL and SPCL perform on the target domain by obvious margins. Under those situations, the complementarity can be a good partner of diversity in DA, which always boosts the learner solely preserving the diversity-based knowledge. Finally, we compare the CL-guided performances with some other famous DA baselines, e.g., RevGred Ganin & Lempitsky (2015) , MMD Long et al. (2015) , in Table .3. DCC is not born to be the best in these methods, however, by model-agnostic learning with the cooperation of curricula and knowledge, still possible to outperform the other baselines.

4.4. SEQUENCE LEARNING.:

In this section, T2T assists BCD number calculation Zaremba & Sutskever (2014) and stock price prediction Sun et al. (2014) . Different from the previous studies, the structured knowledge considered here comes from the existent facts instead of the oracle knowledge. More are detailed in Appendix.C.

Learner models. We employ ordinary Long-Short-Term-Memory (LSTMs) as our student networks.

For the calculation experiment, LSTM performs as a sequence-to-sequence model. For the stock price prediction, it receives the price window trajectory to regress the price in the future.

Curricula setup. Decimal number calculation can be treated as a case of multi-task learning, where the length of the output decides the task species. For the stock price regression task, we treat the stocks from different industries belonging to different tasks, which also indicates a multi-task setting. So we apply ACL as the only CL teacher algorithm in this two sequence learning cases.

Knowledge setup. Though we consider multi-task settings, our structured priors are still based on the relationships among training data. We follow the policy-weight interaction manner in Appendix.A, namely, ACL is used to select tasks and tune their examples' learning weights by Eq.8. As for our knowledge constructions, we apply the hamming distance between the decimal numbers then construct Concept among training instances. Since the stock price data might imply causality, we investigate them by formulating a casual subset selection using the structured prior in Eq.50 .

DECIMAL NUMBER CALCULATION.

The results of our human-imitated calculation is illustrated in Table .4 . As can be observed, curricula behave quite efficient to speed up the LSTM calculator training. Manual curriculum is very competitive, yet the automatic selection in ACL is even more impressive than the hand-crated difficulty.

Finally, SDK is able to help ACL perform the uppermost performance.

STOCK PRICE PREDICTION.:

The empirical studies of stock price prediction are generally based upon two evaluations. In the first evaluation, LSTM are designed to regress the daily Highest and Lowest prices, and we compare the regression error distributions between ACL and ACL+SDK. The results are exhibited in Appendix.C.

Our second evaluation is the backtesting in the real world. Roughly speaking, we specify a buy-andsell rule and apply it to simulate the trading according to the price predictions from the following three investment strategies: LSTM, LSTM+ACL, LSTM+ACL+SDK, and their portfolio return ratios are weekly accumulated in those years for testing. As illustrated in Fig. 5 , the vanilla LSTM always earns the least profits and not recommendable to investors. LSTM+ACL initially attains the highest return ratio, but in the longterm, LSTM+ACL+SDK always outperforms the other investment strategies. There are several branches of development about our research in the future. Firstly, T2T paradigm hitherto focuses on supervised learning, but lack of a glimpse about how to train an agent by exploring structured knowledge among the intelligent agents' experience. How to define a proper structured knowledge and use them to improve curriculum-based reinforcement learning will be quite intriguing. Moreover, though our paradigm attempts to discover sub-structured knowledge, the structured knowledge prior is still predefined. The invention of a complete knowledge discovery algorithm is very promoting. Finally, T2T is an innovative concept about machine education, which should not be limited to structured knowledge. Its development probably leads to more general and efficient teaching paradigms in the high level.

5. CONCLUSION AND FUTURE WORK:



SELF-PACED LEARNING (SPL) AND VARIANTS:

Self-paced learning (SPL) Kumar et al. (2010) accepts training loss L(z; θ) as the feedback and devise a thread of scheme functions to measure the weight. It typically presents as

EQUATION

where negative function G({w i } N i=1 ; λ) named self-paced functioin, controls the pace to incorporate X by increasing age parameter λ. Jiang et al. Jiang et al. (2015) provides a definition of G({w i } N i=1 ; λ) in convexity: Definition 5.1. (Self-paced function ) Controlling the age λ in the iterative process, regularizer G({w i } |D| i=1 ; λ) is a self-paced function, which satisfies three principles as follows:

(1). G({w i } |D| i=1 ; λ) is convex with respect to ∀i ∈ [N ], w i ∈ [0, 1].

(2). When all variables are fixed except for w i , w * i decreases with L(z i ; θ), and holds that lim

L(zi;θ)→0 w i = 1, lim L(zi;θ)→∞ w i = 0. (3). |{w i } |D| i=1 | increases with respect to λ, holding that ∀i ∈ [N ], lim λ→0 w * i = 0, lim λ→∞ w * i = 1.

where {w * i } i=1 denotes the optimal of latent weights at each step in SPL. The principles above state how Eq.( 10) operates in the iterative training process. Principle 2 indicates that when the age λ is certain, SPL inclines to conduct larger latent weight to easy data (with less losses) in favor of complicated ones (with larger losses). Principle 3 promises that as f gets more "mature" by reasonably increasing. λ, Eq.( 10) should accept more complex data into training. These two principles ensure the learning scheme follow in order and advance step by step (self-paced manner), and to some extent, the first principle drives the model to pursue a good {w i } N i=1 . When L(•) is convex in θ, Eq.( 10) turns into a biconvex optimization, and can be efficiently solved by alternate convex search (ACS).

Obviously, SPL implies the weight function as

w(z i ; f θ ) = arg min wi∈[0,1] {w i L(z; θ ) + G({w i } |D| i=1 ; λ)} (11)

to manage the training distribution. Using different scheme functions G({w i } N i=1 ; λ), we can further specify the weight function for implementation. 2018) employ a multiarmed bandit (MAB) as the sampling tool. In their settings, each group of samples or task has been treated as a slot, which is decided the sample to chose or not in each step mini-batch sampling. Although these branch of method consider task as selection instance. In our scenario, it can be viewed as a sample and task pair to merge in our framework, thus, we define w z; f θ as the task-specific weight function. Training instances in the same task are assigned the same weight value.

EQUATION

Automated curriculum learning (ACL). Graves et al. (2017) first leverages adversarial MAB algorithm to solve the predefined curricula issues. Specifically, they treat each slot in the bandit as a task4 , then apply EXPS.3 algorithm to select them to ensemble mini-batches. Concretely, suppose we have M tasks for consideration and ∀i ∈ [M ], the D i corresponds to its subset (

EQUATION

)

where µ ∈ (0, 1) is the exploration ratio and w

(i)

t (θ) is the accumulated reward that.

EQUATION

where ∀i ∈ [M ], w i,0 = 0. The t th iteration reward r i,t-1 (z, θ ) in ACL comes from diverse learning progress signals v. They are calculated online, trimmed into the range [-1, 1] by the following rule:

EQUATION

)

where τ t (z) denotes the longest time to process ∀z ∈ D in the iteration t; q lo t and q hi t denote the trimmed reward lower and upper bounds in the t th iteration (the authors set 20% and 80% quantile of

{ vi τ (zi) } |D| i=1

). The learning progress signals are mainly classified into two branches: loss-driven and complexity-driven. The former focuses on the loss reduction on each instance; the latter consider the model complexity variation. In this paper, we focus on the loss-driven progress signals empirically showing more superior than complexity-driven progress signals in Graves et al. (2017) .

• Prediction Gain (PG). PG aims to estimate the change of loss of instance z. The more decrease indicates z is easier to learn in this step:

EQUATION

• Gradient Prediction Gain (GPG). PG requires two forward propagation to obtain the current loss and previous loss of the considered instances. Developed from the first-order Taylor series approximation to PG, GPG only needs one forward operation:

v GPG = ||∇ θ L(z; θ)|| 2 (20)

• Self-Prediction Gain (SPG). PG and GPG are the biased estimate of the change of loss when the parameter change from θ to θ is caused by z. To mitigate this issue, SPG select the other sample z from the same task to replace z:

EQUATION

where D i(z) indicates task-specific or category-specific training instance subset.

To bridge the weight w i to π i , it needs to clarify the structured knowledge is built on the relations among instances or taskes.

If it is built on the relations among instances, since ACL performs on multiple tasks or categories, where each instance is uniformly sampled. So we need to do just set w i = 1(∀z i ∈ D) in P w and update the balance strategy through Eq.8. To perform the refined curriculum, we first apply π EXP3.S i,t

to choose a task slot, then uniformly sample an instance and assign its weight by the balance strategy close-form result.

If it is built on the relations among tasks or categories, then each instance corresponds to a task or category. In this case, w i might be directly treated as π EXP3.S i,t

.

In our evaluation of ACL, we only consider the first case.

Teacher-student curriculum learning (TSCL). TSCL can be viewed as a simple version of ACL:

log π i z; f θ = αr i (z, f θ ) + (1 -α) log π i z; f θ

where π i z; f θ is the bandit sampling strategic probability; r(z, f θ ) is the sum of in the evaluation scores of task-based instance z. α denotes the discounted ratio to calculate the moving average of r(z, f θ ). Obviously, the teacher-student strategy focus on task selection and training instances in the same task are uniformly collected.

w z; f θ connects π i z; f θ under the same rule of ACL.

LEARNABLE ICLS.:

Recent triumphs in advanced machine learning raise a question to machine teaching methods: Is it possible to devise a guiding curriculum via a neural network? MentorNet Jiang et al. ( 2017), ScreenNet Kim & Choi (2018) and L2T Fan et al. (2018) 

w φ g m (z i ) L(z i ; θ) + G({w φ g m (z i ) } |D| i=1 ; λ) (22)

Obviously, this objective is developed from Eq.11. But G might refer to non-convex property. After imported structured dark knowledge D SDK , the objective is refined as:

EQUATION

then S keeps alternatively updating as our Algorithm performs.

Sampling-based L2T. Different from MentorNet, L2T directly employ a policy network to select training instance. It first specifies a state feature g s (z) of instance z, then formulate the curriculumbased teaching as a binary classification decision on each instance. These decisions are made by a policy network (w φ in our context) trained through a policy gradient algorithm REINFORCE Williams (1992) .

Concretely, suppose the final activation of w φ is a sigmoid function, thus, P olicy φ (a|g

s (z)) = aw φ (g s (z)) + (1 -a)(1 -w φ (g s (z))

, where a ∈ {0, 1} indicates the selection decision. Then the policy gradient update performs as

EQUATION

where Ξ denotes the training episode, r(a, g s (z)) denotes the sampled estimation of the teaching reward from one episode execution (More details refer to Fan et al. (2018) ). For structured dark knowledge transfer, the update gradient has turned into

EQUATION

APPENDIX.B

In this Appendix, we specify our structured knowledge prior F , namely, a set function over D.

Assume the training set D is the ground set we consider. Set function F : 2 D → R can be viewed as a combinatorial optimization objective, which searches a subset S ⊆ D to maximize/minimize the value of F (S). In specific in Eq.7, we consider the subset selection with cardinality constraint, where |S| ≤ k. Moreover, we only consider the maximization, due to minimization can be achieved by setting -F (S).

SUBMODULARITY AND SUPERMODULARITY.:

General subset selection is a NP-hard problem, yet when F satisfies submodularity Fujishige (2005), there are many effective algorithms providing the sub-optimal results with -approximation Nemhauser et al. (1978 ) Zhou & Spanos (2016b) . Namely, suppose S * denotes the global optima, the -approximate sub-optima Ŝ maintains S * ≤ Ŝ. Definition 5.2 (submodularity). Set function

EQUATION

Definition 5.3 (submodularity (deminishing return)).

EQUATION

The definitions above are equivalent. Besides, if F (•) is submodular, then -F (•) is supermodular. F can be submodular and supermodular at the meantime, then we get

F (S 1 ) + F (S 2 ) = F (S 1 ∪ S 2 ) + F (S 1 ∩ S 2 ) and F (•) is termed modular function. F (•) is normalized if F (∅) = 0; is monotonically increases if F (S 1 ) < F (S 2 ) iff ∀S 1 ⊂ S 2 ; monotonically decreases if F (S 1 ) < F (S 2 ) iff ∀S 2 ⊂ S 1 . Remark 5.1. If F (S)

is submodular/supermodular, then Eq.7 is submodular/supermodular maximization under the cardinality constraints.

Since the second term of Eq.7 is modular, the remark is obvious. Remark 5.2. If F (S) is supermodular, Eq.7 turns to a submodular minimization problem under cardinality constraint. Remark 5.3. If ∀j ∈ [J], F j (S) is a submodular/supermodular function, then

F (S) = J j=1 α j F j (S), ∀j ∈ [J], α j > 0 (28)

is also a submodular/supermodular function.

Diversity and Complementarity. Submodularity and supermodularity convey different structured properties. Under the context of maximization, submodular function enforces the subset selection presenting diversity and summarization. For example, if there is a document to be summarized by k sentences, we can maximize a submodular function about its organized structure to achieve the goal. On the other hand, supermodularity indicates cooperation and similarity, namely, the group of data appear more coherent are more probably selected together.

If F (S) is submodular, then max 

J (S) by J (S * ) ≤ ( 1-e -κ J κ J )J ( Ŝ).

where κ J denotes the curvature of J : Definition 5.4 (submodular curvature Fujishige ( 2005)). κ F is the curvature of submodular function F :

EQUATION

) A0 ← ∅, D, k, J (•). for t = 1 to k do z (t) ← arg max z∈D/A t J (At ∪ {z}) -J (At) . At ← At-1 end for return A k . Algorithm 3 GREEDMAX Bai & Bilmes (2018) A0 ← ∅, B0 ← D, k, F1(•), F2(•). for t = 1 to k do z * = arg maxz∈B t h(z) + F1(At-1 ∪ {z}) - F1(At-1) + F2(At-1 ∪ {z}) -F2(At-1) At ← At-1 ∪ {z * }, Bt ← Bt-1/{z * } end for return A k . If F (S) is supermodular, max S⊆D,|S|≤k J (S) is reduced to size-constrained submodular minimization.

This problem can be approximately solved by a sampling-based algorithm Svitkina & Fleischer (2011) while the constant factor polynomial-time approximation algorithm is non-exist in this problem. But in our setting, we pay more interest in the constrained suBmodular+suPermodular (BP) maximization Bai & Bilmes (2018) rather than a pure supermodular maximization. BP maximization is more realistic and can be easily extended from a pure supermodular maximization. Definition 5.5 (BP maximization). Given a normalized monotonically increasing submodular function F 1 and supermodular function F 2 , BP maximization refers to solve a following objective max S⊆D,|S|≤k

EQUATION

The meaning of BP maximization is developed from diversity. In structured knowledge with diversity, we hope to choose data distributed along with the structure, thus, as sparse as possible. But in some situations, we also hope some certain groups of data are selected together. The cooperation of F 1 and F 2 is able to produce this effect.

Remark 5.5. Provided F as Definition.5.5, max

S⊆D,|S|≤k:

J (S) is a BP maximization.

Proof. We know the second term of J (S) is modular and monotonically increases. then

J (S) = F 1 (S) + F 2 (S) + β zi∈S h(z i ) = F 1 (S) + β zi∈S h(z i ) normalized monotonically increasing submodular function +F 2 (S)

where h(z i ) = 1 -DSDK(zi,w;S)

C:

. Proof is concluded.

Remark 5.6. BP maximization solved by GreedMAX (Algorithm.3) leads to an approximate outcome as.

(

EQUATION

where κ F1 and κ F2 respectively indicate the curvatures of submodular function F 1 and supermodular function F 2 : Definition 5.6 (supermodular curvature). Given a monotonically increasing supermodular function F 2 , the curvature of F 2 κ F2 is defined as

EQUATION

Proof. Remark.5.6 comes from the theoretical result of Bai & Bilmes (2018) :

Lemma 5.1. Given the maximization problem in Eq.30, GreedMAX obtain a suboptimal Ŝ that maintains an approximation toward S * :

EQUATION

then by Remark.5.5, we have

EQUATION

where κ F1+β hi denotes the submodular curvature of F 1 (S) + β zi∈S h(z i ). Then

κ F1+β hi = 1 -min zi∈S βh(z i ) + F 1 (S) -F 1 (S/{z i }) βh(z i ) + F 1 ({z i }) = max zi∈S F 1 ({z i }) -[F 1 (S) -F 1 (S/{z i })] βh(z i ) + F 1 ({z i }) = max zi∈S 1 -[ F1(S)-F1(S/{zi}) F1({zi}) ] β h(zi) F1({zi}) + 1 ≤ κ F1 βc + 1 , c = min zi∈S h(z i ) F 1 ({z i }) (35)

Besides, we observe the inequality that ∀x ∈ [0, 1],

EQUATION

Combine Eq.34, 35 and we have

EQUATION

. Conclude the proof.

Here we specify some kind of F (S). They are applied in our first and second experiments to show structured knowledge prior about diversity and complementarity among data.

• Concepts. Suppose D contains J concepts to describe training instances. Each concept can be modeled by a subset V j ⊆ D(∀j ∈ [J]). Then we can formulate this structured knowledge via a positive linear combination of concave over modular functions.

F (S) = J j=1 α j |S ∩ V j | µ , ∀j ∈ [J], V j ⊆ D, J ∪ j=1 V j = D, α j > 0, 0 < µ < 1 (38)

The implication of this structured knowledge is proposed to evenly absorb the J concepts.

The selection criterion of V j is based on the experiment setting.

• BP function. Different from the above submodular set functions, BP function is neither submodular nor supermodular, which indicates the balance between diversity and complementarity. We follow the example in Bai & Bilmes (2018) and present this structured knowledge by

EQUATION

where the first and second terms indicate diversity and complementarity respectively. Specifically, V j denotes a set of instance containing certain semantic meaning, W i denotes a set of ambiguous instances nearby the optimal decision boundary.

SUBMODULAR INDEX.:

Although submodular functions are widely applied in structured inference, there are always some structures that can not be modeled by this property. Hence we are going to discuss a more general conceptual standard named Submodular Index (SI). SI provides a view to describe a set function how close to be submodular. It leads to an unified approximate method to solve generic subset selection.

The following theoretical results totally come from Zhou & Spanos (2016a) . Definition 5.7 (Submodular Index (SI) ). For a set function F : 2 D → R the submodularity index (SI) for a location set L and a cardinality k, denoted by Ω f (L, k), is defined as

EQUATION

where ϕ f (S, A) indicates local submodular index (LSI): Definition 5.8 (Local submodular Index (LSI)).

EQUATION

where F ({x}|A) = F ({x} ∪ A) -F (A) and F (S|A) = F (S ∪ A) -F (A). Based on this definition, Algorithm.4 is provided to solve generic subset selection. Theorem 5.1. For a general (possibly non-monotonic, non-submodular) set function F , let the optimal solution of the cardinality-constrained maximization be denoted as S * , and the solution Ŝ of random greedy algorithm satisfying

E[F ( Ŝ)] ≥ 1 e - ξ f Ŝ,k E[F ( Ŝ)] F (S * ) (42) Algorithm 4 Random Greedy Zhou & Spanos (2016a) A0 ← ∅, D, k, J (•). for t = 1 to k do z (t) ← arg max M t ⊂D/A t-1 ,|M t |=k u∈M t F (u|At). Draw u uniformly from Mt. At ← At-1 ∪ {u} end for return A k . where ξ f Ŝ,k = Ω f Ŝ,k + k(k-1) 2 max{0, Ω f Ŝ,2 }

There are two extensions from the theorem above: Corollary 5.1. For monotonic set functions in general, random greedy algorithm achieves

E[F ( Ŝ)] ≥ 1 - 1 e + Ω f ( Ŝ, k) E[F ( Ŝ)] F (S * ) (43)

and deterministic version of Algorithm.4 maintains

EQUATION

where

EQUATION

Corollary 5.2. For submodular function that are not necessarily monotonic, random greedy algorithm has performance

EQUATION

Here we specify some cases of structured knowledge priors F (S) that can be solved by Algorithm.4.

• Asymmeric Graph Cut. Unordered graphs are common structures to represent data knowledge. Suppose vertices denote samples in D and edges reflect the pairwise relations among them, F (S) presents a subconnection based on a RBF kernel variant K (det(K > 0), i.e., ∀z i , z j ∈ D, K i,j = exp(-

||f θ * (zi)-f θ * (zj )|| 2 2 σ 2

).

EQUATION

)

where σ is the bandwidth that we set 0.4 in our experiment. f θ * (z) indicates a oracle knowledge embedding of z. Note that, Eq.47 is not monotonic, thus, it can not be solved by Local Search. In the implementation, it would be more favorable to choose the sparse nearest neighbors and normalize them to construct K.

• Granger Causality (GC). Consider the random processes X (N ) and Y (N ) , where

X (i) = {X 1 , X 2 , • • • , X i } (1 ≤ i ≤ N indicates time index).

Directed information is a term defined by

EQUATION

The formula can be viewed as the aggregated dependence between the history of X and the current value of process Y , given the past observations of Y . Then in a specific iteration t, it conveys a causal relation that given the past Y t-1 , X t on Y t should be unique. A subset selection problem based on Eq.48 is so-called causal subset selection. There are two following specifications about this problems: Definition 5.9 (Causal Sensor Placement).

EQUATION

Definition 5.10 (Casual Covariates Selection).

EQUATION

. The first knowledge prior (Eq.49) is unsupervised, yet the second knowledge prior (Eq.50) presents a target process Y (N ) that the selected subset should consider. More information about them can be found in Zhou & Spanos (2016a) .

APPENDIX.C:

MORE SPECIFIC IMPLEMENTATION OF CLS.

DETAILS IN PC.:

For training a learner, PC performs as a stochastic sampling process under the distribution constantly changing in a fixed step. Initially it focuses on easiest examples, then the weights of more difficult examples gradually increase so that finally the distribution become converge to the training prior P (z).

In our implementation, we schedule the PC decreasing process according to the minimum epochs for the leaner's convergence. Then it linearly changes the proportion of the sampling distributions.

If w i is updated, PC presents either a reweighting scheme or a renewed sampling distribution. Both of them are available in our methodology. In order to reveal the data mining efficiency compared with the other CL baselines, PC performs in a stochastic sampling style in our experiment. Namely, the mini-batches are sampled by Q λ (z) (λ indicates the proceeding schedule).

To implement our method on PC, we use w to reweight their instance sampling distribution, and employ the updated training distribution to sample training instances.

DETAILS IN L2T.:

Our L2T is reimplemented under the tensorflow platform. We follow the experimental details in the original paper. Then for the hyper-parameter setting, state feature is concatnated by the data, model and combined features, and the evaluation rate threshold τ is set 0.84.

Two evaluation settings are demonstrated in L2T. Accordingly, their teaching schedules are quite different from the other CL approaches. The authors claimed that the teacher in L2T co-evolves with the student, yet in fact, teacher requires a pre-training on a student to perform better education quality. So we employ different learning manners to incorporate L2T into our T2T. In the first setting when the teacher is trained on CIFAR-10, our SDK regulates its policy learning, then when the teacher is fixed to teach the student, SDK is deactivated. In the second setting when the teacher is trained on MNIST, our SDK do not performe at this pre-training. Then when the teacher performs teaching on CIFAR-10, SDK performs to finetune the teaching strategy.

TINY IMAGE CLASSIFICATION.

Benchmark and learner. The first experiment is conducted on CIFAR-10. CIFAR-10 is widely-used benchmark for visual classification. It contains 60, 000 RGB images with size 32 across 10 classes.

The data has been partitioned into a training set with 50, 000 images and test set with 10, 000 images. During our training, data augmentation is applied: each images has been padded with 4 pixels to each side and get cropped into 32 × 32. We apply ResNet32 models in tensorflow implementation and the size of mini batch is set 128. We follow the optimization strategy in He et al. (2016) . Namely, we employ Momentum-SGD as the solver and set the initial learning rate as 0.1, which is multiplied by 0.1 after the 32, 000 th and 48, 000 th update. The training leads to 93.26% in a test accuracy.

Curricula specification. We evaluate PC, SPL and L2T in CIFAR10. ACL is not born for single task learning. SPCL is a reweighting approach, yet in this classification experiment, we expect to compare the data selection efficiency. Hence we choose the binary SPL as the representative in SPL family. Their implementations have been previously detailed.

Knowledge engineering. The structured knowledge prior is built upon Eq.38 where each index j ∈ [J] denotes a concept among data. More specifically, ∀j ∈ [M ] we set α j = 1, and we determine V j by a simple feature selection principle: we collect all training data and perform feature-level J-means clustering (the feature lie on oracle knowledge space, namely, extracted from a pre-trained ResNet110). Then each example will be assigned to some of those J clusters that the feature entries of this example belong to. We set J = 64 in our experiment and employ the technique in Arthur & Vassilvitskii (2007) to initiate J seeds before clustering.

DOMAIN ADAPTATION.:

Curricula specification. To ensure the robustness in adversarial learning, SPL and SPCL applied in our DA obey three extra learning principles. First, they only perform the weight inferences during the feature extractors update. Second, at each time when the discriminator update finishes, K is reset, thus, rearranges the mini-batch self-paced process narrated in SubSection.4.1.1 (E is set as maximum the iterations of each feature extractor updating process). Our strategy separately operates on the training sets from source and target domains. Structured dark knowledge is also based on this separation. Different from the classification experiment, the selected training instances are directly used to construct the mini-batch online, thus, the mini-batch-based self-paced selection would leads to the mini-batch size changing during the entire learning procedure. To promise a fixed mini-batch size, the K ousteds are replaced by the other samples randomly selected from training set and uniformly assigned the weights in [0, 1]. It is inspired by the tricks in Sachan & Xing (2016) .

Knowledge engineering. In DA experiment, we specify two kind of structured knowledge based on Concept (Eq.38) and BP function (Eq.39) . As what we have mentioned, the former indicates diversity where the subset are supposed to be selected to achieve sparsity across different conceptual groups {V j } J j . The latter also consider data diversity (the first term in Eq.39), yet simultaneously perceive the complementarity (the second term in Eq.39) when the subset selection is performed. It means that, each group of {W i } I i=1 are tend to be selected together.

Compared with the Concepts constructed in classification, the Concepts in DA are engineered within different manner. Specifically for source domain, J concepts in DA are branched into two kinds. The first kind is based on categories. So there are 10 groups of concept (each concept refers to a specific class in the dataset). The second kind is constructed within a group of samples that belongs to the same category. We perform a similar feature-level clustering in each class and choose 5 clusters.

Then totally we have J = 10 + 5 * 10 = 60 to construct the Concept structured knowledge in DA. We apply the same routine to engineer the structured knowledge in the target domain.

SEQUENCE LEARNING.

DECIMAL NUMBER CALCULATION.:

This task belongs to human-imitation task from AI perspective and is well-known that requires CCL methods to boost learning speed. In the basic setting, a sequence-to-sequence model (mostly, RNN) is employed to receive two-decimal-coded numbers separated by the sign "+", then produce the sum of those numbers in decimal coding (we only consider plus operation in this paper.). The evaluation is the minimum cost time to obtain the test accuracy more than 99%.

Our setup is similar to Zaremba & Sutskever (2014) , yet the implementation is based on PyTorch. In our sequence-to-sequence models, the encoder and decoder are LSTMs with the same architecture containing 128 units. During training, all the CL baselines learn on 40,960 samples. Validation set consists of 4,096 examples and our batch size is set the same number.

Curricula specification. Decimal number addition includes a manual curriculum and ACL algorithm.

The former is proposed based on the demonstration in Matiisen et al. (2017) , ACL is directly applied to screen and select multiple tasks.

Knowledge engineering. We insist on diversity knowledge prior term Concept, while under the different knowledge engineering manner. Detailedly, we employ Hamming distance as the measure between digit examples then assign them into J = 64 clusters.

Our stock price data come from the daily data of the Chinese A share markets from July, 2007 to July, 2017. It is offered by a stock data service provider (www.wind.com.cn). The daily data for each stock contains six indexes including the Open Price, the Highest Price, the Lowest Price, the Close Price, the Turnover Rate and Volume. For our empirical study, we only use the Highest Price and Lowest Price for regression. MIN-MAX Normalized technique is used to promise all data value stay in the same order of magnitude.

We employ the opportunity window technique in Appel (2005) , then 9423 windows of interest (at weekly granularity) are extracted by this strategy. Then LSTMs taught by different teachers are feed these time series at the daily granularity to predict the price. Our LSTM learner also consider multi-scale information, thus, the pricing information of the past 5, 10 and 20 days are encoded as a concatenated feature for regression.

Curricula specification. Different from decimal number addition, we have no information about which day or stock is easy for a LSTM model to learn. We use ACL as the only curriculum strategy to evaluate our T2T.

Knowledge engineering. The knowledge prior in stock price data should imply causality. In specific, we consider the prior as the set function in Eq.50 . Then all the opportunity windows from a specific stock, naturally maintain the time-based dependencies. We select the series within three months so that detrend the possible longterm information that are not relevant for our daily analysis. Then these time-based aligned information constructs the target processes Y in Eq.50 .

During implementation, we employ a trick to the casual subset selection. As can be observed, the samples not in the range of a specific time series are totally without causality. So instead of considering over all training data, we tend to consider each causality-based process respectively, then select k training instances in total. Concretely, suppose we have N s aligned processes, then we separately select k Ns training instances within each process. Future price regression. We perform the price regression task where the 9423 windows of interest are trained to predict the lowest and highest prices within the next two weeks. The outcomes from ACL and ACL+SDK are demonstrated in Table .5 . As can be observed, ACL+SDK generally outperforms ACL, which also explains the higher accumulated profit in Fig. 5 . 

:

Example/task with less difficulty owns more priority in training, which presents as sampling with more frequency, or larger loss coefficient.

xi ∈ X over data space in R d , and yi ∈ Y over label space in N+ for classification or R for regression. For multi-task and structured learning, Y turn to be high-dimensional.

The positive subset selection objective is quite welcome in many combinatorial optimization setup.

The original paper consider two cases: multi-task learning and single-task learning by previously learning the other tasks as the bridge. Here we only consider the first case.

Figures and Tables:

Figure 1: Overview of the iterative machine teaching procedures using CCL and ICL schemes on learner f θ . a). show the situations where CCLs and ICL normally perform. CCL increases difficulty threshold λ to incorporete more training examples (training weight w λ (z) constantly increases); ICL builds up a teacher-learner communication protocol by training weight w(z; θ) to facilitate the alternative updating of teacher and learner. b). show our T2T where CCLs and ICL perform by interacting with a structured dark knowledge cycle (Section.3.2 ). Then CCL and ICL determine their teaching decisions based on not only the curricula but also the structured knowledge instructions.

CURRICULA GENERALLY REVISITED CLs advocate the teaching through reconfiguring the training prior P (z) based upon the instances' difficulties: Less difficult instance earns larger weight or sampling frequency during training. Distinguished from the update rules, they basically are categorized into two branches: CCL and ICL.

Figure 2: Overview of three comparison settings for CLs. The first evaluates PC+SDK by comparing the original PC, random teaching (No CL) and anti-curriculum (Anti-PC, weights inversely proceeds accorinding to PC). The second and third evaluate SDKs acting on ICL methods, e.g., SPL and L2T.

Figure 3: The scaled averaged SDK values of SPL and PC as the training proceeds.

Fig.3.1 illustrate three circumstances: The sub-figures from left to right, indicate the evaluations under CCL, ICL(the first test setting in L2T), ICL(the first test setting in L2T). SDKs basically perform envelopes that improve the targeted PC, SPLs and L2Ts.

Provided a source domain with labeled data, domain adaptation (DA) aims to transfer their semantic to classify the unlabeled data from a target domain (non-i.i.d with the source)Long et al. (2015). Benchmarks. Our DA considers two transfer tasks: SVHNNetzer et al. (2011)→MNISTLeCun et al. (1998) and MNIST→USPS. For each transfer, training sets in source and target are used and the evaluation is based on their target test sets.DA methods and learner model. There are plenty of adversarial DA approaches and we choose DDCTzeng et al. (2017) as our based DA method to present the combination power of curriculum and

Teaching to Teach (T2T) by Structured Dark Knowledge (SDK) concerns how to use structured knowledge among training instances to generally enhance existing CL approaches. It connects with a variety of structures represented by generic set functions and refers to diverse model-agnostic curriculum-routed teaching paradigms. Our empirical studies have demonstrated its efficiency across different learning tasks.

Figure 5: The cumulative return ratios in the market backtest of the following investment strategies: LSTM, LSTM+ACL and LSTM+ACL+SDK.

15) ICLS WITH POLICY SAMPLING Policy sampling CL Graves et al. (2017) Matiisen et al. (2017) Doan et al. (

Now we turn to introduce how to construct BP function in DA. For a ablation study, the first term indicating diversity in BP function is completely the same of the Concept we previously mentioned. Then for the second term, we tend to choose the ambiguous examples into the same group. We train a prophet network then extract the classification activations across all training data, thus, use them to calculate the ambiguous semantic among training data. Then for each class, we sample a representative (the example closet to the centric) and collect 9 most similar examples from the other classes, respectively. So we have I = 10 in Eq.39 and each group contains 10 examples.

Comparison of dark knoweldge (DK) approaches.

Table content: <table><tr><td/><td/><td colspan="2">DK approaches</td></tr><tr><td/><td>Standard DK</td><td>Bayesian DK</td><td>Structured DK (Ours)</td></tr><tr><td>Teacher</td><td>Networks</td><td>MC Posterior Density</td><td>Training Subset</td></tr><tr><td>Student</td><td>Networks</td><td>Networks</td><td>CL Strategies (can be networks)</td></tr><tr><td colspan="2">Update Fixed Teacher</td><td>Alternative Update</td><td>Alternative Update</td></tr></table>

Fan et al. (2018) specifies a RLbased teacher agent to select training instances. Different with the above methods, L2T suits to verify whether structured dark knowledge can benefit the learnable CLs 9. In our setup, γD SDK (z i , w φ (t) ; S) constrains L2T's binary policy decision of its teacher network (Under this context, w φ (t) denotes the teacher policy network with binary classification).

Table content: <table><tr><td>More implementations about PC and L2T are deferred in Appendix.C.</td></tr><tr><td>4.1.2 ORACLE STRUCTURED KNOWLEDGE.</td></tr></table>

DCC-based Digit DA across curricula and structured dark knowledge (SDK). (C) and (BP) denote Concept and BP function as the structured dark knowledge priors.The result in bracket indicates the reproduced performances.

Table content: <table><tr><td/><td/><td colspan="2">SVHN→MNIST</td><td/><td colspan="2">MNIST→USPS</td><td/></tr><tr><td>SDK</td><td/><td colspan="6">No curriculum SPL SPCL No curriculum SPL SPCL</td></tr><tr><td>Non</td><td/><td>68.1(69.2)</td><td>67.5</td><td>70.3</td><td>79.1(79.0)</td><td>74.4</td><td>78.5</td></tr><tr><td>(C)</td><td>source target</td><td>--</td><td>68.9 69.2</td><td>71.2 71.3</td><td>--</td><td>76.9 79.9</td><td>77.9 81.8</td></tr><tr><td>(BP)</td><td>source target</td><td>--</td><td>70.4 71.7</td><td>73.2 73.0</td><td>--</td><td>76.4 78.7</td><td>79.4 82.6</td></tr></table>

The accuracies (%) of Curriculum-Improved DCC and other DA baselines.

Table content: <table><tr><td/><td colspan="2">Other DA baselines</td><td/><td colspan="3">DCC+Curricula+SDK (ours)</td><td/></tr><tr><td/><td colspan="2">RevGred MMD</td><td colspan="5">No SPL(C) SPL(BP) SPCL(C) SPCL(BP)</td></tr><tr><td>SVHN→MNIST</td><td>71.1</td><td>71.1</td><td>68.1</td><td>69.2</td><td>71.7</td><td>71.3</td><td>73.2 (+5.1)</td></tr><tr><td>MNIST→USPS</td><td>77.1</td><td>-</td><td>79.1</td><td>79.9</td><td>78.7</td><td>81.8</td><td>82.6 (+3.5)</td></tr><tr><td colspan="4">4.3 UNSUPERVISED DOMAIN ADAPTATION.</td><td/><td/><td/><td/></tr></table>

LSTM net performs sequence2sequence task to calculate decimal numbers. The evaluation is the neccesary epochs to provide the first shot evaluation accuracy more than 99%. The number is the average of five times repeated evaluation. Less means better. knowledge. To be convenient of curricula and structured knowledge implementation, we reproduce DDC in PyTorch platform. We employ the same architectures of feature extractor, discriminator and classifier, and Adam Kingma & Ba (

Table content: <table><tr><td>Methods Random Manual</td><td>ACL</td><td>ACL+SDK(our)</td></tr><tr><td colspan="2">24894.6 14583.2 13357.9</td><td>13178.6</td></tr></table>

Table.2, 3 . From Table.2 we discover that, SPL and SPCLs perform even worse than the learners without using curricula. It means that, CL methods (vanilla SPL at least ) might not always take positive effects to guide better training. Such observation is consistent with some previous findingsSachan & Xing (2016);Fan et al. (2018) yet seemingly collides withTang et al. (2012). One possible explanation is that, DA inTang et al. (2012) does not involve deep representation learning. So the training loss as difficulty index might be more trustful than those obtained in the modern cases. Thankfully, after injected by structured knowledge among data, SPL and SPCL have been made great again in DA.

Table content: <table/>

The stock price regression results of ACL and ACL+SDK.

Table content: <table><tr><td/><td/><td/><td>ACL</td><td/><td/><td/><td colspan="2">ACL+SDK</td><td/></tr><tr><td/><td>Errors</td><td colspan="8">0 -10% 10 -20% 20 -40% &gt; 40% 0 -10% 10 -20% 20 -40% &gt; 40%</td></tr><tr><td colspan="2">Price (Yuan)</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0 -5</td><td>0%</td><td>0%</td><td>0.47%</td><td>99.53%</td><td>0%</td><td>0%</td><td>0.56%</td><td>99.44%</td></tr><tr><td/><td>5 -10</td><td>1.26%</td><td>2.67%</td><td>16.96%</td><td>79.11%</td><td>5.98%</td><td>9.32%</td><td>25.47%</td><td>69.23%</td></tr><tr><td>Highest</td><td>10 -15</td><td>32.27%</td><td>28.68%</td><td>29.34%</td><td>9.7%</td><td>31.26%</td><td>31.29%</td><td>25.95%</td><td>8.7%</td></tr><tr><td/><td>15 -20</td><td>20.02%</td><td>29.66%</td><td>48.82%</td><td>1.49%</td><td>21.52%</td><td>34.16%</td><td>41.82%</td><td>2.5%</td></tr><tr><td/><td>&gt; 20</td><td>4.05%</td><td>4.05%</td><td>32.83%</td><td>58.08%</td><td>7.5%</td><td>8.5%</td><td>34.38%</td><td>50.62%</td></tr><tr><td/><td>0 -5</td><td>0%</td><td>0%</td><td>0%</td><td>100%</td><td>0%</td><td>0%</td><td>0%</td><td>100%</td></tr><tr><td/><td>5 -10</td><td>1.67%</td><td>4.73%</td><td>16.48%</td><td>77.12%</td><td>1.67%</td><td>4.12%</td><td>18.76%</td><td>75.45%</td></tr><tr><td>Lowest</td><td>10 -15</td><td>31.24%</td><td>35.79%</td><td>21.65%</td><td colspan="2">11.32% 28.43%</td><td>36.69%</td><td>26.75%</td><td>8.13%</td></tr><tr><td/><td>15 -20</td><td>22.6%</td><td>27.59%</td><td>42.40%</td><td>7.4%</td><td>10.05%</td><td>34.95%</td><td>46.44%</td><td>7.56%</td></tr><tr><td/><td>&gt; 20</td><td>4.01%</td><td>6.09%</td><td>20.65%</td><td>69.25%</td><td>5.05%</td><td>5.05%</td><td>25.15%</td><td>64.75%</td></tr></table>

Peer review (iclr2019):
 Summarization: This paper studies how to inject structured prior knowledge into the teaching model for machine learning. The authors propose a very general framework called `teach to teach’, in which 1) the knowledge is distilled via subset selection that matches the teacher decision distribution 2) the distilled knowledge is then transferred into the teacher model by reweighting the contributions of teacher objective and coherence constraints. Extensive experiments are conducted on image classification, unsupervised domain adaptation and sequence learning. Questions: 1) Can the teacher models, like those in L2T, be successfully transferred? For example, the teacher model trained with task 1 (with prior knowledge 1) successfully applied to task 2? 2) I’m not that clear about the relationship with knowledge distillation (Hinton et. al 2015). Per my understanding, the authors seem to make the distribution of the knowledge (specified by the set prior function F) coherent with teacher model and let the both influence each other (in section 3.2). In that sense I do not know what is the `dark’ knowledge here. Pros: In general I think this paper is a decent work that the structural prior knowledge is elegantly combined with teaching strategy (a.k.a. the teacher models in curriculum learning). The proposed method is intuitive and natural. The empirical verifications are deep and comprehensive to demonstrate the effectiveness of the `teaching to teach’ framework. Cons: 1) I think the authors should compare with self-paced learning with diversity (SPLD) since you also take diversity as a form of structural knowledge. 2) The writing needs to be significantly polished. First, please simply the writing both in terms of general logic and language. I spent quite a few efforts in figuring out the meaning of some notations and complicated terms such as `curriculum-routed’ and `g_i’. Furthermore, I see no reason of putting so much fancy decorations on an essentially iterative algorithm (the bottom part of page 5 and all page 6). Second, I suggest the authors give more intuitive and concrete examples towards what is the structural prior knowledge at the earlier phase of the paper, rather than putting most of them into appendix. Last but not least, please use more clear citation formats: currently quite a few citations are missing of publishing venues such as Fan et.al 2018 and Furlanello et.al 2018.