TASK DEFINITION: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
PROBLEM: In this research we address the problem of capturing recurring concepts in a data stream environment. Recurrence capture enables the re-use of previously learned classifiers without the need for re-learning while providing for better accuracy during the concept recurrence interval. We capture concepts by applying the Discrete Fourier Transform (DFT) to Decision Tree classifiers to obtain highly compressed versions of the trees at concept drift points in the stream and store such trees in a repository for future use. Our empirical results on real world and synthetic data exhibiting varying degrees of recurrence show that the Fourier compressed trees are more robust to noise and are able to capture recurring concepts with higher precision than a meta learning approach that chooses to re-use classifiers in their originally occurring form.

SOLUTION: Mining Recurrent Concepts in Data Streams using the Discrete Fourier Transform

PROBLEM: Latent tree analysis seeks to model the correlations among a set of random variables using a tree of latent variables. It was proposed as an improvement to latent class analysis — a method widely used in social sciences and medicine to identify homogeneous subgroups in a population. It provides new and fruitful perspectives on a number of machine learning areas, including cluster analysis, topic detection, and deep probabilistic modeling. This paper gives an overview of the research on latent tree analysis and various ways it is used in practice. Much of machine learning is about modeling and utilizing correlations among variables. In classification, the task is to establish relationships between attributes and class variables so that unseen data can be classified accurately. In Bayesian networks, dependencies among variables are represented as directed acyclic graphs and the graphs are used to facilitate efficient probabilistic inference. In topic models, word cooccurrences are accounted for by assuming that all words are generated probabilistically from the same set of topics, and the generation process is reverted via statistical inference to determine the topics. In deep belief networks, correlations among observed units are modeled using multiple levels of hidden units, and the top-level hidden units are used as a representation of the data for further analysis. Latent tree analysis (LTA) seeks to model the correlations among a set of observed variables using a tree model, called latent tree model (LTM), where the leaf nodes represent observed variables and the internal nodes represent latent variables.The dependence between two observed variables is explained by the path between them. Despite their simplicity, LTMs subsume two classes of models widely used in academic research. The first one is latent class models (LCMs) (Lazarsfeld and Henry, 1968; Knott and Bartholomew, 1999), which are LTMs with a single latent variable. They are used for categorical data clustering in social sciences and medicine. The second class is probabilistic phylogenetic trees (Durbin et al., 1998), which are a tool for determining the evolution history of a set of species. Phylogenetic trees are special LTMs where the model structures are binary (bifurcating) trees and all the Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. variables have the same number of possible states. LTA also provides new and fruitful perspectives on a number of machine learning areas. One area is cluster analysis. Here finite mixture models such as LCMs are commonly used. A finite mixture model has one latent variable and consequently it gives one soft partition of data. An LTM typically has multiple latent variables and hence LTA yields multiple soft partitions of data simultaneously. In other words, LTA performs multidimensional clustering (Chen et al., 2012; Liu et al., 2013). It is interesting because complex data usually have multiple facets and can be meaningfully clustered in multiple ways. Another area is topic detection. Applying LTA to text data, we can partition a collection of documents in multiple ways. The document clusters in the partitions can be interpreted as topics. Furthermore, it is possible to learn hierarchical LTMs where the latent variables are organized into multiple layers. This leads to an alternative method for hierarchical topic detection (Liu, Zhang, and Chen, 2014; Chen et al., 2016a), which has been shown to find more meaningful topics and topic hierarchies than the state-of-the-art method based on latent Dirichlet allocation (Paisley et al., 2015). The third area is deep probabilistic modeling. Hierarchical LTM and deep belief network (DBN) (Hinton, Osindero, and Teh, 2006) are similar in that they both consist of multiple layers of variables, with an observed layer at the bottom and multiple layers of hidden units on top of it. One difference is that, in DBN, units from adjacent layers are fully connected, while HLTM is tree-structured. It would be interesting to explore the middle ground between the two extreme and develop algorithms for learning what might be called sparse DBNs. Learning structures for deep models is an interesting open problem. Extension of LTA might offer one solution (Chen et al., 2016b). The concept of latent tree models was introduced in (Zhang, 2002, 2004), where they were referred to as hierarchical latent class models. The term “latent tree models” first appeared in (Zhang et al., 2008; Wang, Zhang, and Chen, 2008). Mourad et al. (2013) surveyed the research on latent tree models as of 2012 in details. This paper provides a concise overview of the methodology. The exposition are more conceptual and less technical than (Mourad et al., 2013). Developments after 2012 are also included. ar X iv :1 61 0. 00 08 5v 1 [ cs .L G ] 1 O ct 2 01 6

SOLUTION: Latent Tree Analysis

PROBLEM: We study the connection between the highly<lb>non-convex loss function of a simple model of<lb>the fully-connected feed-forward neural net-<lb>work and the Hamiltonian of the spherical<lb>spin-glass model under the assumptions of:<lb>i) variable independence, ii) redundancy in<lb>network parametrization, and iii) uniformity.<lb>These assumptions enable us to explain the<lb>complexity of the fully decoupled neural net-<lb>work through the prism of the results from<lb>the random matrix theory. We show that for<lb>large-size decoupled networks the lowest crit-<lb>ical values of the random loss function are<lb>located in a well-defined narrow band lower-<lb>bounded by the global minimum. Further-<lb>more, they form a layered structure. We<lb>show that the number of local minima out-<lb>side the narrow band diminishes exponen-<lb>tially with the size of the network. We em-<lb>pirically demonstrate that the mathemati-<lb>cal model exhibits similar behavior as the<lb>computer simulations, despite the presence<lb>of high dependencies in real networks. We<lb>conjecture that both simulated annealing and<lb>SGD converge to the band containing the<lb>largest number of critical points, and that<lb>all critical points found there are local min-<lb>ima and correspond to the same high learn-<lb>ing quality measured by the test error. This<lb>emphasizes a major difference between large-<lb>and small-size networks where for the lat-<lb>ter poor quality local minima have non-zero<lb>probability of being recovered. Simultane-<lb>ously we prove that recovering the global<lb>minimum becomes harder as the network size<lb>increases and that it is in practice irrelevant<lb>as global minimum often leads to overfitting.<lb>

SOLUTION:
The Loss Surface of Multilayer Networks