title: Evaluation methods and decision theory for classification of streaming data with temporal dependence

authors: Žliobaitė, Indrė

subjects: Classification

abstract: Predictive modeling on data streams plays an important role in modern data analysis, where data arrives continuously and needs to be mined in real time. In the stream setting the data distribution is often evolving over time, and models that update themselves during operation are becoming the state-of-the-art. This paper formalizes a learning and evaluation scheme of such predictive models. We theoretically analyze evaluation of classifiers on streaming data with temporal dependence. Our findings suggest that the commonly accepted data stream classification measures, such as classification accuracy and Kappa statistic, fail to diagnose cases of poor performance when temporal dependence is present, therefore they should not be used as sole performance indicators. Moreover, classification accuracy can be misleading if used as a proxy for evaluating change detectors with datasets that have temporal dependence. We formulate the decision theory for streaming data classification with temporal dependence and develop a new evaluation methodology for data stream classification that takes temporal dependence into account. We propose a combined measure for classification performance, that takes into account temporal dependence, and we recommend using it as the main performance measure in classification of streaming data.

text: Evaluation methods and decision theory for classification of streaming data with temporal dependence Predictive modeling on data streams plays an important role in modern data analysis, where data arrives continuously and needs to be mined in real time. In the stream setting the data distribution is often evolving over time, and models that update themselves during operation are becoming the state-of-the-art. This paper formalizes a learning and evaluation scheme of such predictive models. We theoretically analyze evaluation of classifiers on streaming data with temporal dependence. Our findings suggest that the commonly accepted data stream classification measures, such as classification accuracy and Kappa statistic, fail to diagnose cases of poor performance when temporal dependence is present, therefore they should not be used as sole performance indicators. Moreover, classification accuracy can be misleading if used as a proxy for evaluating change detectors with datasets that have temporal dependence. We formulate the decision theory for streaming data classification with temporal dependence and develop a new evaluation methodology for data stream classification that takes temporal dependence into account. We propose a combined measure for classification performance, that takes into account temporal dependence, and we recommend using it as the main performance measure in classification of streaming data.

year: 2015

title: Ensembles of nested dichotomies for multi-class problems

authors: Frank, Eibe

abstract: Nested dichotomies are a standard statistical technique for tackling certain polytomous classification problems with logistic regression. They can be represented as binary trees that recursively split a multi-class classification task into a system of dichotomies and provide a statistically sound way of applying two-class learning algorithms to multi-class problems (assuming these algorithms generate class probability estimates). However, there are usually many candidate trees for a given problem and in the standard approach the choice of a particular tree is based on domain knowledge that may not be available in practice. An alternative is to treat every system of nested dichotomies as equally likely and to form an ensemble classifier based on this assumption. We show that this approach produces more accurate classifications than applying C4.5 and logistic regression directly to multi-class problems. Our results also show that ensembles of nested dichotomies produce more accurate classifiers than pairwise classification if both techniques are used with C4.5, and comparable results for logistic regression. Compared to error-correcting output codes, they are preferable if logistic regression is used, and comparable in the case of C4.5. An additional benefit is that they generate class probability estimates. Consequently they appear to be a good general-purpose method for applying binary classifiers to multi-class problems.

text: Ensembles of nested dichotomies for multi-class problems Nested dichotomies are a standard statistical technique for tackling certain polytomous classification problems with logistic regression. They can be represented as binary trees that recursively split a multi-class classification task into a system of dichotomies and provide a statistically sound way of applying two-class learning algorithms to multi-class problems (assuming these algorithms generate class probability estimates). However, there are usually many candidate trees for a given problem and in the standard approach the choice of a particular tree is based on domain knowledge that may not be available in practice. An alternative is to treat every system of nested dichotomies as equally likely and to form an ensemble classifier based on this assumption. We show that this approach produces more accurate classifications than applying C4.5 and logistic regression directly to multi-class problems. Our results also show that ensembles of nested dichotomies produce more accurate classifiers than pairwise classification if both techniques are used with C4.5, and comparable results for logistic regression. Compared to error-correcting output codes, they are preferable if logistic regression is used, and comparable in the case of C4.5. An additional benefit is that they generate class probability estimates. Consequently they appear to be a good general-purpose method for applying binary classifiers to multi-class problems.

year: 2004-07-04

title: Horowaewae

authors: Maxwell, Te Kahautu

abstract: Horowaewae is dedicated to Horowaewae Maxwell a Te Matatini Lifemember. Horowaewae is from the Rotorua area and the Te Arawa iwi. He is a foundation member of the champion kapa haka Ngāti Rangiwewehi. Horowaewae has been a performer, a male leader, the Te Arawa delegate to the Te Matatini National Committee. He has served kapa haka in its development since the inception of Te Matatini in 1972 and prior to this performing in the tourism industry in Rotorua. Horowaewae was performed by Ōpōtiki Mai Tawhiti at the 2013 Te Matatini World Championship Kapa Haka Competition.

text: Horowaewae Horowaewae is dedicated to Horowaewae Maxwell a Te Matatini Lifemember. Horowaewae is from the Rotorua area and the Te Arawa iwi. He is a foundation member of the champion kapa haka Ngāti Rangiwewehi. Horowaewae has been a performer, a male leader, the Te Arawa delegate to the Te Matatini National Committee. He has served kapa haka in its development since the inception of Te Matatini in 1972 and prior to this performing in the tourism industry in Rotorua. Horowaewae was performed by Ōpōtiki Mai Tawhiti at the 2013 Te Matatini World Championship Kapa Haka Competition.

year: 2012-09-13