Learning With Single Teacher Multi Student
Venue: AAAI
Year: 2018

In this paper we study a new learning problem defined as "Single Teacher Multi Student" (STMS) problem, which investigates how to learn a series of student (simple and specific) models from a single teacher (complex and universal) model. Taking the multiclass and binary classification for example, we focus on learning multiple binary classifiers from a single multiclass classifier, where each of binary classifier is responsible for a certain class. This actually derives from some realistic problems, such as identifying the suspect based on a comprehensive face recognition system. By treating the already trained multiclass classifier as the teacher, and multiple binary classifiers as the students, we propose a gated support vector machine (gSVM) as a solution. A series of gSVMs are learned with the help of single teacher multiclass classifier. The teacher's help is two fold; first, the teacher's score provides the gated values for students' decision; second, the teacher can guide the students to accommodate training examples with different difficulty degrees. Extensive experiments on real datasets validate its effectiveness.  Multiclass classification (MC) (Aly 2005; Bishop 2006; Nie, Wang, and Huang 2017) considers that an instance can be affiliated to one of multiple 2) classes. Many real applications can be cast into MC problem. Taking the face recognition (He et al. 2017) for example, the task is required to distinguish multiple people based on their face photos, e.g. 126 individuals in AR Face Database (Martinez 1998) Other MC applications refer to image classification (Guo 2017) text categorization (Xu et al. 2017) information retrieval (Yang et al. 2017) multi view classification (Xu, Tao, and Xu 2013) et.al. To accommodate multiple classes, the multiclass classifiers usually adopt a score based prediction mechanism. In detail, they can generate a score vector in terms of all classes as the output, then the class with the maximum score is regarded as the prediction. Some of the widely used multiclass classifiers actually follow this mechanism, such as One vs All (Vapnik 2013) structural SVM (Tsochantaridis et al. 2004) rank SVM (Joachims 2002) et.al. As a result, the multiclass classifiers usually have larger model size than the binary classifiers, and more expensive inference cost as well.  Suppose we have a multiclass classifier (e.g. structural SVM) which has been already trained well. It allows us to identify a class from multiple candidates. In some real circumstances, however, the task in hand may only require a binary classification task, i.e. identifying whether a test instance belongs to one specific class instead of what class it belongs to. For example, in news categorization a multiclass classifier is implemented to recognize what category a news item is related to, such as tech news, entertainment news, economic news, et.al. However, when a user only requests for the tech news, the system only needs to classify whether the news items are tech related or not. Another example refers to the suspect tracking in criminal detection. Usually, a face recognition system has already existed whose function is to identify a person's ID according to his/her face. This system may be capable of distinguishing millions of people. Nevertheless, for suspect tracking, we only concern whether the testing faces belong to the suspect or not. Of course, we can still use the multiclass classifier to directly recognize whether the instance is a tech news item (or the suspect) If the instance is related to other news categories (or persons) we think it is no tech news (or suspect) However, this practice will always involve the inference of all the other classes; thus for a binary classification, the inference cost is too expensive. Or just using the training data to learn the binary classifiers is also an intuitive method. Nevertheless, this can not take advantage of the existing multiclass classifier which may provide some useful information. Therefore, it is of the need to learn a new binary classifier with much faster inference speed than the multiclass classifier, and higher classification accuracy than that of simple learning with training data. In our paper, we suggest learning the binary classifiers by virtue of the existing multiclass classifier. In fact, this can be likened vividly to the basketball team training by a single teacher (or coach) (He, Eisner, and Daume 2012; Hinton, Vinyals, and Dean 2015; You et al. 2017b; Zhu, Liu, and Lopes 2017) An experienced teacher has extensive knowledge about how to be a good forward, center and guard. What the teacher wants is to train a team and each player has his/her own speciality; some specialize in the forward while others are more suitable for centers or guards. Our problem can also be regarded as learning from a single teacher (i.e. the multiclass classifier) to obtain multiple students (i.e. binary The Thirty Second AAAI Conference on Artificial Intelligence (AAAI 18)