Minimum Enclosing Ball-based Learner Independent Knowledge Transfer for Correlated Multitask Learning

Multi-Task Learning (MTL), as opposed to Single Task Learning (STL), has become a hot topic in machine learning research. For many real world problems in application areas such as medical decision making, pattern recognition, and finance forecasting – MTL has shown significant advantage to STL because of its ability to faciliate knowledge sharing between tasks. This thesis presents our recent studies on Knowledge Transfer (KT) – the process of transferring knowledge from one task to another, which is at the core of MTL. The novelly proposed KT algorithm for correlation multi-task machine learning adapts learner independence into MTL, thus empowering any ordinary classifier for MTL. The improvement in the learning rate of MTL as compared with STL relies on KT between tasks. Given two correlated learning tasks T , T , previous approaches to MTL are often modeled as a learner dependent model: ML = L(T , KTL(T , T )), for i, j = 1, 2, i 6= j, where L is a specific learner such as an k Nearest Neighbour (kNN), Support Vector Machine (SVM), or Multi-Layer Perceptron (MLP), andML is the resultant learning model built from task T i and transferred knowledge KTL from T j to T . Because representation of the knowledge transferred in KTL depends on the learner, it follows that the transferred knowledge KT is incompatible for learners other than L. To remove the learner-dependent restriction, we propose a learner independent MTL scheme such that ML = L(T , KT (T , T )), for i, j = 1, 2, i 6= j, where KT is independent to the learner L, and MTL is conducted for arbitrary learner combinations. In the proposed solution, we use Minimum Enclosing Balls (MEBs) as knowledge carriers to extract and transfer knowledge from one task to another. Since the knowledge presented in MEB can be decomposed as raw data, it can be incorporated into any learner as additional training data for a new learning task and thus improve its learning rate. The proposed MEB-based KT is on the basis that in the feature space, the two correlated tasks share some common input data that lie on the overlapping regions of the feature spaces in-between the two correlated tasks. The main idea is to find the correlating knowledge – overlapping regions of the two tasks – and transfer the related data regardless of the learner employed. KT is done by building a correlation space via MEBs and transferring the enclosed instances from the primary task to the secondary task. The extent of KT depends on the amount of overlapping instances between two tasks. The effectiveness and robustness of the proposed KT is evaluated on multi-task pattern recognition (MTPR) problems derived from synthetic datasets, University of California at Irvine (UCI) datasets, and real world face datasets, using classifiers from different disciplines for MTL. We evaluate the learner independent KT model on three measures: (A) KT capability – how much knowledge can the method transfer; (B) contribution of KT – how beneficial is the transferred knowledge to the new task; and (C) adaptability of KT – how could KT adapt to learning tasks with varied relevance. It is shown that multi-task learners using KT via MEB carriers perform better than learners without-KT, and it is successfully applied to all type of classifiers. On the other hand, in cases where the two learning tasks are semantically irrelevant, negative KT could happen to the proposed KT. For example, face recognition is not able to be supported by knowledge from hand writing recognition – simply overlapping feature domains does not work. How to discover correlated knowledge for MTL between unrelated topics remains a challenging problem and is left for future work.