Modeling language with structured penalties

Modeling natural language is among fundamental challenges of artificial intelligence and the design of interactive machines. It has applications spanning across various domains, such as dialogue systems, text generation and machine translation among others and has been studied extensively in the past decades. Among problems that are common to most applications of language is to model the distribution over a vocabulary of words that could follow a given sequence of words or the context. The simplest of models in the literature are based on frequency counts. Since the size of the vocabulary is large, many possible sentences are never observed leading to sparse sampling or data sparsity. This causes overestimation of observed sequences and underestimation of the unobserved ones. Thus it is necessary to use prior knowledge to appropriately modify the distribution. Frequency counts are commonly modified using smoothing techniques. Learning-based methods are an alternative to frequency-based methods that fit an appropriate model to the data. Numerous learning techniques have been applied to build language models; generative models (Dirichlet, Pitman-Yor and hierarchical PitmanYor processes), discriminative models (maximum entropy, conditional random fields) and distributional representations (neural networks, Boltzman machines, log-bilinear models). We propose a discriminatively trained log-linear model to learn the distribution of words following a given context. Due to data sparsity, it is necessary to appropriately regularize the model using a penalty term. However, simple choices for regularization (like `2 and `1 penalties) fail to capture long range dependencies. They overfit severely with increasing context length because they are agnostic to the sequential structure of the problem. We show that it is necessary to design a penalty term that properly encodes the structure of the feature space to avoid overfitting and improve generalization while appropriately capturing long range dependencies. Some nice properties of specific structured penalties can be used to reduce the number of parameters required to encode the model. The outcome is an efficient model that suitably captures long dependencies in language without a significant increase in time or space requirements. In a log-linear model, both training and testing become increasingly expensive with growing number of classes. This is because the normalization factor involved in computing the probabilities grows linearly with the number of classes or categories in the problem. The number of classes in a language model is the size of the vocabulary which is typically very large. A common trick is to cluster classes and apply the model in two-steps; the first step picks the most probable cluster and the second picks the most probable word from the chosen cluster. This idea can be generalized to a hierarchy of larger depth with multiple levels of clustering. However, the performance of the resulting hierarchical classifier depends on the suitability of the clustering to the problem and building a ‘good’ hierarchy is non-trivial. We study different strategies to build the hierarchy of categories from their