document: Detecting anomalous records in categorical datasets We consider the problem of detecting anomalies in high aritycategorical datasets . In most applications , anomalies are defined as datapoints that are `` abnormal '' . Quite often we have access to data which consists mostly of normal records , a long with a small percentage of unlabelled anomalous records . We are interested in the problem of unsupervised anomaly detection , where we use the unlabelled data for training , and detect records that do not follow the definition of normality . A standard approach is to create a model of normal data , and compare test records against it . A probabilistic approach builds a likelihood model from the training data . Records are tested for anomalies based on the complete record likelihood given the probability model . For categorical attributes , bayes nets give a standard representation of the likelihood . While this approach is good at finding outliers in the dataset , it often tends to detect records with attribute values that are rare . Sometimes , just detecting rare values of an attribute is not desired and such outliers are not considered as anomalies in that context . We present an alternative definition of anomalies , and propose an approach of comparing against marginal distribution of attribute subsets . We show that this is a more meaningful way of detecting anomalies , and has a better performance over semi-synthetic as well as real world datasets .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O