Bag of Machine Learning Concepts for Visual Concept Recognition in Images

My thesis deals with the recognition of visual concepts on images using statistica l machine learning. Recognition is treated here as classification task with continu ous predictions. The continuous predictions can be used to generate a ra nking of images and thus will be often evaluated in a ranking setting. Ranking means that for a given visual concept the set of all test images will be sorted accor ding to the prediction in a descending order and evaluated using a ranking measure. This dissertation treats the general case of visual concepts in which concepts ar defined explicitly by a set of images. The aim is multi-label classification in which for one image all present concepts are to be predicted. The challenge compared to highly specialized tasks such as face recognition is the ability to deal with a generic s et of visual concepts which are defined by the training data. Classification is based on kernel methods such as extensions of support vect machines. The features are predominantly bag of visual words (BoW) whic h yield superior results for visual concept recognition on images with generic co n epts as demonstrated constantly over the last years by the results of international be nchmark competitions such as Pascal VOC classification and ImageCLEF Photo an notation. The problem of classification and ranking of a generic set of vis ual concepts can be divided into three subtasks: Formulation of the problem and design or choice of a corresponding loss function , theLearning of feature combinations given a loss functionand theDesign of Features . My publication record contains co-authored work on all subtasks. This dissertation contains contributions for the first two subtasks. In the first part of the dissertation I consider (for the aspect of Formulation of the problem and design or choice of a corresponding loss function ) models which are capable of minimizing hierarchical loss functions which are induced by ta xonomies over the set of all visual concepts. The idea is that a taxonomy defi nes a prioritization of classification and ranking errors. The goal is to avoid er rors which originate from confusing concepts which are distant under the giv en taxonomy. One example is a system which annotates images such that it returns for a request of dogs in case of absence of dogs or in case of error rathe images of cats than images of cars. In contrast to preceding publications the focus lies not on speed during te sting time but on improved classification and ranking performance under the hiera rchical loss. The developed model aggregates the votes of all edges in the tax onomy, not only those of the locally best or shortest path. Furthermore the hierar chic l models are generalized such that they can be predict multiple labels for multilabel ranking problems in which each image can have more than one visual co ncept. Previous approaches based on greedy walks along the edges of th hierarchy are able to predict only the most likely concept. In the context of multi-label ra nking we define also a ranking measure which incorporates taxonomical infor mati n. The developed model is compared against one-versus-all and structur ed prediction baselines. In the second part of the dissertation I analyze (for the aspect of Learning of feature combinations given a loss function ) the non-sparse multiple kernel learning (MKL) for multi-label ranking of images. It is compared against average k ernel support vector machines (SVMs) and sparse l1-norm MKL. For the empirical part I evaluate the performance of these methods on the Pascal VOC2009 Class ification and ImageCLEF2010 Photo Annotation datasets. It is shown that when u sing model selection in a practical setup, non-sparse MKL yields equal or bette r r sults compared to the average kernel SVM which does not learn feature combin ations, in contrast to sparse l1-norm MKL which yields worse results. For the theoretical part we identify limiting and promoting factors for the performance gains o f non-sparse MKL when compared to the other methods. The dissertation is closed by an outlook section.