niryuu/sni-each-converted huggingface dataset:
A new approach called SKYSET (Synthetic Knowledge Yield Social Entities Translation) is proposed to validate completeness and to reduce ambiguity from written instructional documentation. SKYSET utilizes a quintuple set of standardized categories, which differs from traditional approaches that typically use triples. The SKYSET System defines the categories required to form a standard template for representing information that is portable across different domains. It provides a standardized framework that enables sentences from written instructions to be translated into sets of category typed entities on a table or database. The SKYSET entities contain conceptual units or phrases that represent information from the original source documentation. SKYSET enables information concatenation where multiple documents from different domains can be translated and combined into a single common filterable and searchable table of entities. Index Terms – Cognitive Linguistics, Dual Intentional Semantic Decomposition and Reconstruction (DISDR), Frame Semantics, Information Concatenation, Information Integration, Information Summarization, Knowledge Architecture, Knowledge Management Internal External (KMIE), Metalinguistic Awareness, Synthetic Knowledge Yield Social Entities Translation (SKYSET).
Output: Introducing SKYSET - a Quintuple Approach for Improving Instructions
We study the problem of multiclass classification with an extremely large number<lb>of classes, with the goal of obtaining train and test time complexity logarithmic<lb>in the number of classes. We develop top-down tree construction approaches for<lb>constructing logarithmic depth trees. On the theoretical front, we formulate a new<lb>objective function, which is optimized at each node of the tree and creates dy-<lb>namic partitions of the data which are both pure (in terms of class labels) and<lb>balanced. We demonstrate that under favorable conditions, we can construct loga-<lb>rithmic depth trees that have leaves with low label entropy. However, the objective<lb>function at the nodes is challenging to optimize computationally. We address the<lb>empirical problem with a new online decision tree construction procedure. Exper-<lb>iments demonstrate that this online algorithm quickly achieves small error rates<lb>relative to more common O(k) approaches and simultaneously achieves signif-<lb>icant improvement in test error compared to other logarithmic training time ap-<lb>proaches.
Output: Logarithmic Time Online Multiclass prediction
In this work we present a new approach to the treatment of property graphs using neural encoding techniques derived from machine learning. Specifically, we will deal with the problem of embedding property graphs in vector spaces. Throughout this paper we will use the term embedding as an operation that allows to consider a mathematical structure, X, inside another structure Y , through a function, f : X → Y . We are interested on embeddings capable of capturing, within the characteristics of a vector space (distance, linearity, clustering, etc.), the interesting features of the graph. For example, it would be interesting to get embeddings that, when projecting the nodes of the graph into points of a vector space, keep edges with the same type of the graph into the same vectors. In this way, we can interpret that the semantic associated to the relation has been captured by the embedding. Another option is to check if the embedding verifies clustering properties with respect to the types of nodes, types of edges, properties, or some of the metrics that can be measured on the graph. Subsequently, we will use these good embedding features to try to obtain prediction / classification / discovery tools on the original graph. This paper is structured as follows: we will start by giving some preliminary definitions necessary for the presentation of our proposal and a brief introduction to the use of artificial neural networks as encoding machines. After this review, we will present our embedding proposal based on neural encoders, and we will verify if the topological and semantic characteristics of the original graph have been maintained in the new representation. After evaluating the properties of the new representation, it will be used to carry out machine learning and discovery tasks on real databases. Finally, we will present some conclusions and future work proposals that have arisen during the implementation of this work.
Output: Semantic Preserving Embeddings for Generalized Graphs
Video captioning has been attracting broad research attention in multimedia community. However, most existing approaches either ignore temporal information among video frames or just employ local contextual temporal knowledge. In this work, we propose a novel video captioning framework, termed as Bidirectional Long-Short Term Memory (BiLSTM), which deeply captures bidirectional global temporal structure in video. Specifically, we first devise a joint visual modelling approach to encode video data by combining a forward LSTM pass, a backward LSTM pass, together with visual features from Convolutional Neural Networks (CNNs). Then, we inject the derived video representation into the subsequent language model for initialization. The benefits are in two folds: 1) comprehensively preserving sequential and visual information; and 2) adaptively learning dense visual features and sparse semantic representations for videos and sentences, respectively. We verify the effectiveness of our proposed video captioning framework on a commonlyused benchmark, i.e., Microsoft Video Description (MSVD) corpus, and the experimental results demonstrate that the superiority of the proposed approach as compared to several state-of-the-art methods.
Output: Bidirectional Long-Short Term Memory for Video Description
We show that matrix completion with tracenorm regularization can be significantly hurt when entries of the matrix are sampled nonuniformly. We introduce a weighted version of the trace-norm regularizer that works well also with non-uniform sampling. Our experimental results demonstrate that the weighted trace-norm regularization indeed yields significant gains on the (highly non-uniformly sampled) Netflix dataset.
Output: Collaborative Filtering in a Non-Uniform World:  Learning with the Weighted Trace Norm
The success of kernel methods has initiated the design of novel positive semidefinite functions, in particular for structured data. A leading design paradigm for this is the convolution kernel, which decomposes structured objects into their parts and sums over all pairs of parts. Assignment kernels, in contrast, are obtained from an optimal bijection between parts, which can provide a more valid notion of similarity. In general however, optimal assignments yield indefinite functions, which complicates their use in kernel methods. We characterize a class of base kernels used to compare parts that guarantees positive semidefinite optimal assignment kernels. These base kernels give rise to hierarchies from which the optimal assignment kernels are computed in linear time by histogram intersection. We apply these results by developing the Weisfeiler-Lehman optimal assignment kernel for graphs. It provides high classification accuracy on widely-used benchmark data sets improving over the original Weisfeiler-Lehman kernel.
Output: On Valid Optimal Assignment Kernels and Applications to Graph Classification
We document a connection between constraint reasoning and probabilistic reasoning. We present an algorithm, called probabilistic arc consistency, which is both a generalization of a well known algorithm for arc consistency used in constraint reasoning, and a specialization of the belief updating algorithm for singly-connected networks. Our algorithm is exact for singly­ connected constraint problems, but can work well as an approximation for arbitrary problems. We briefly discuss some empirical results, and re­ lated methods.
Output: Probabilistic Arc Consistency: A Connection between Constraint Reasoning and Probabilistic Reasoning
The ability to predict which patterns are formed in brain scans when imagining a celery or an airplane, based on how these concepts as words co-occur in texts, suggests that it is possible to model mental representations based on word statistics. Whether counting how frequently nouns and verbs combine in Google search queries, or extracting eigenvectors from matrices made up of Wikipedia lines and Shakespeare plots, these latent semantics approximate the associative links that form concepts. However, cognition is fundamentally intertwined with action; even passively reading verbs has been shown to activate the same motor circuits as when we tap a finger or observe actual movements. If languages evolved by adapting to the brain, sensorimotor constraints linking articulatory gestures with aspects of motion might also be reflected in the statistics of word co-occurrences. To probe this hypothesis 3 × 20 emotion, face, and hand related verbs known to activate premotor areas in the brain were selected, and latent semantic analysis LSA was applied to create a weighted adjacency matrix. Hierarchically clustering the verbs and modeling their connectivity within a force directed graph, they divide into modules of mouth and hand motion, facial expressions and negative emotions. Transforming the verbs into their constituent phonemes, the corresponding consonant vowel transitions can be represented in an articulatory space defined by tongue height and formant frequencies. Here the vowels appear positioned along a front to back continuum reflecting aspects of size and intensity related to the actions described by the verbs. More forceful verbs combine plosives and sonorants with fricatives characterized by sustained turbulent airflows, while positive and negative emotional expressions tend to incorporate upor downwards shifts in formant frequencies. Suggesting, that articulatory gestures reflect parameters of size and intensity which might be retrieved from the latent semantics of action verbs.
Output: When push comes to shove verbs literally shake due to latent semantic parameters of size and intensity
Previous studies have demonstrated that encoding a Bayesian network into a SAT formula and then performing weighted model counting using a backtracking search algorithm can be an effective method for exact inference. In this paper, we present techniques for improving this approach for Bayesian networks with noisy-OR and noisy-MAX relations— two relations that are widely used in practice as they can dramatically reduce the number of probabilities one needs to specify. In particular, we present two SAT encodings for noisy-OR and two encodings for noisy-MAX that exploit the structure or semantics of the relations to improve both time and space efficiency, and we prove the correctness of the encodings. We experimentally evaluated our techniques on large-scale real and randomly generated Bayesian networks. On these benchmarks, our techniques gave speedups of up to two orders of magnitude over the best previous approaches for networks with noisyOR/MAX relations and scaled up to larger networks. As well, our techniques extend the weighted model counting approach for exact inference to networks that were previously intractable for the approach.
Output: Exploiting Structure in Weighted Model Counting Approaches to Probabilistic Inference
The number of computers, tablets and smartphones is increasing rapidly, which entails the ownership and use of multiple devices to perform online tasks. As people move across devices to complete these tasks, their identities becomes fragmented. Understanding the usage and transition between those devices is essential to develop efficient applications in a multi-device world. In this paper we present a solution to deal with the cross-device identification of users based on semi-supervised machine learning methods to identify which cookies belong to an individual using a device. The method proposed in this paper scored third in the ICDM 2015 Drawbridge Cross-Device Connections challenge proving its good performance.
Output: Cross-Device Tracking: Matching Devices and Cookies
We describe the class of convexified convolutional neural networks (CCNNs), which capture the parameter sharing of convolutional neural networks in a convex manner. By representing the nonlinear convolutional filters as vectors in a reproducing kernel Hilbert space, the CNN parameters can be represented as a low-rank matrix, which can be relaxed to obtain a convex optimization problem. For learning two-layer convolutional neural networks, we prove that the generalization error obtained by a convexified CNN converges to that of the best possible CNN. For learning deeper networks, we train CCNNs in a layer-wise manner. Empirically, CCNNs achieve performance competitive with CNNs trained by backpropagation, SVMs, fully-connected neural networks, stacked denoising auto-encoders, and other baseline methods.
Output: Convexified Convolutional Neural Networks
Short text messages such as tweets are very noisy and sparse in their use of vocabulary. Traditional textual representations, such as tf-idf, have difficulty grasping the semantic meaning of such texts, which is important in applications such as event detection, opinion mining, news recommendation, etc. We constructed a method based on semantic word embeddings and frequency information to arrive at low-dimensional representations for short texts designed to capture semantic similarity. For this purpose we designed a weight-based model and a learning procedure based on a novel median-based loss function. This paper discusses the details of our model and the optimization methods, together with the experimental results on both Wikipedia and Twitter data. We find that our method outperforms the baseline approaches in the experiments, and that it generalizes well on different word embeddings without retraining. Our method is therefore capable of retaining most of the semantic information in the text, and is applicable out-of-the-box. c © 2016 Elsevier Ltd. All rights reserved.
Output: Representation learning for very short texts using weighted word embedding aggregation
Distributed dense word vectors have been shown to be effective at capturing tokenlevel semantic and syntactic regularities in language, while topic models can form interpretable representations over documents. In this work, we describe lda2vec, a model that learns dense word vectors jointly with Dirichlet-distributed latent document-level mixtures of topic vectors. In contrast to continuous dense document representations, this formulation produces sparse, interpretable document mixtures through a non-negative simplex constraint. Our method is simple to incorporate into existing automatic differentiation frameworks and allows for unsupervised document representations geared for use by scientists while simultaneously learning word vectors and the linear relationships between them.
Output: Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec
We address the problem of general supervised learning when data can only be ac-<lb>cessed through an (indefinite) similarity function between data points. Existing<lb>work on learning with indefinite kernels has concentrated solely on binary/multi-<lb>class classification problems. We propose a model that is generic enough to handle<lb>any supervised learning task and also subsumes the model previously proposed for<lb>classification. We give a “goodness” criterion for similarity functions w.r.t. a given<lb>supervised learning task and then adapt a well-known landmarking technique to<lb>provide efficient algorithms for supervised learning using “good” similarity func-<lb>tions. We demonstrate the effectiveness of our model on three important super-<lb>vised learning problems: a) real-valued regression, b) ordinal regression and c)<lb>ranking where we show that our method guarantees bounded generalization error.<lb>Furthermore, for the case of real-valued regression, we give a natural goodness<lb>definition that, when used in conjunction with a recent result in sparse vector re-<lb>covery, guarantees a sparse predictor with bounded generalization error. Finally,<lb>we report results of our learning algorithms on regression and ordinal regression<lb>tasks using non-PSD similarity functions and demonstrate the effectiveness of<lb>our algorithms, especially that of the sparse landmark selection algorithm that<lb>achieves significantly higher accuracies than the baseline methods while offering<lb>reduced computational costs.
Output: Supervised Learning with Similarity Functions
We study the problem of identifying individuals based on their characteristic gaze patterns during reading of arbitrary text. The motivation for this problem is an unobtrusive biometric setting in which a user is observed during access to a document, but no specific challenge protocol requiring the user’s time and attention is carried out. Existing models of individual differences in gaze control during reading are either based on simple aggregate features of eye movements, or rely on parametric density models to describe, for instance, saccade amplitudes or word fixation durations. We develop flexible semiparametric models of eye movements during reading in which densities are inferred under a Gaussian process prior centered at a parametric distribution family that is expected to approximate the true distribution well. An empirical study on reading data from 251 individuals shows significant improvements over the state of the art.
Output: A Semiparametric Model for Bayesian Reader Identification
Generating optimal plans in highly dynamic environments is challenging. Plans are predicated on an assumed initial state, but this state can change unexpectedly during plan generation, potentially invalidating the planning effort. In this paper we make three contributions: (1) We propose a novel algorithm for generating optimal plans in settings where frequent, unexpected events interfere with planning. It is able to quickly distinguish relevant from irrelevant state changes, and to update the existing planning search tree if necessary. (2) We argue for a new criterion for evaluating plan adaptation techniques: the relative running time compared to the “size” of changes. This is significant since during recovery more changes may occur that need to be recovered from subsequently, and in order for this process of repeated recovery to terminate, recovery time has to converge. (3) We show empirically that our approach can converge and find optimal plans in environments that would ordinarily defy planning due to their high dynamics.
Output: Generating Optimal Plans in Highly-Dynamic Domains
In this paper, we claim that Vector Cosine – which is generally considered one of the most efficient unsupervised measures for identifying word similarity in Vector Space Models – can be outperformed by a completely unsupervised measure that evaluates the extent of the intersection among the most associated contexts of two target words, weighting such intersection according to the rank of the shared contexts in the dependency ranked lists. This claim comes from the hypothesis that similar words do not simply occur in similar contexts, but they share a larger portion of their most relevant contexts compared to other related words. To prove it, we describe and evaluate APSyn, a variant of Average Precision that – independently of the adopted parameters – outperforms the Vector Cosine and the co-occurrence on the ESL and TOEFL test sets. In the best setting, APSyn reaches 0.73 accuracy on the ESL dataset and 0.70 accuracy in the TOEFL dataset, beating therefore the non-English US college applicants (whose average, as reported in the literature, is 64.50%) and several state-of-the-art approaches.
Output: What a Nerd! Beating Students and Vector Cosine in the ESL and TOEFL Datasets
We provide several applications of Optimistic Mirror Descent, an online learning algorithm based on the idea<lb>of predictable sequences. First, we recover theMirror Prox algorithm for offline optimization, prove an extension<lb>to Hölder-smooth functions, and apply the results to saddle-point type problems. Next, we prove that a version<lb>of Optimistic Mirror Descent (which has a close relation to the Exponential Weights algorithm) can be used by<lb>two strongly-uncoupled players in a finite zero-summatrix game to converge to the minimax equilibrium at the<lb>rate ofO((logT )/T ). This addresses a question of Daskalakis et al [6]. Further, we consider a partial information<lb>version of the problem. We then apply the results to convex programming and exhibit a simple algorithm for the<lb>approximateMax Flow problem.
Output: Optimization, Learning, and Games with Predictable Sequences
Screening rules allow to early discard irrelevant variables from the optimization in Lasso problems, or its derivatives, making solvers faster. In this paper, we propose new versions of the socalled safe rules for the Lasso. Based on duality gap considerations, our new rules create safe test regions whose diameters converge to zero, provided that one relies on a converging solver. This property helps screening out more variables, for a wider range of regularization parameter values. In addition to faster convergence, we prove that we correctly identify the active sets (supports) of the solutions in finite time. While our proposed strategy can cope with any solver, its performance is demonstrated using a coordinate descent algorithm particularly adapted to machine learning use cases. Significant computing time reductions are obtained with respect to previous safe rules.
Output: Mind the duality gap: safer rules for the Lasso
Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.
Output: Scikit-learn: Machine Learning in Python
Self-paced learning (SPL) has been attracting increasing attention in machine learning and computer vision. Albeit empirically substantiated to be effective, the investigation on its theoretical insight is still a blank. It is even unknown that what objective a general SPL regime converges to. To this issue, this study attempts to initially provide some new insights under this “heuristic” learning scheme. Specifically, we prove that the solving strategy on SPL exactly accords with a majorization minimization algorithm, a well known technique in optimization and machine learning, implemented on a latent objective. A more interesting finding is that, the loss function contained in this latent objective has a similar configuration with non-convex regularized penalty, an attractive topic in statistics and machine learning. In particular, we show that the previous hard and linear self-paced regularizers are equivalent to the capped norm and minimax concave plus penalties, respectively, both being widely investigated in statistics. Such connections between SPL and previous known researches enhance new insightful comprehension on SPL, like convergence and parameter setting rationality. The correctness of the proposed theory is substantiated by experimental results on synthetic and UCI data sets.
Output: What Objective Does Self-paced Learning Indeed Optimize?
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent neural networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.
Output: ENTROPY-SGD: BIASING GRADIENT DESCENT INTO WIDE VALLEYS
Many prediction domains, such as ad placement, recommendation, trajectory prediction, and document summarization, require predicting a set or list of options. Such lists are often evaluated using submodular reward functions that measure both quality and diversity. We propose a simple, efficient, and provably near-optimal approach to optimizing such prediction problems based on noregret learning. Our method leverages a surprising result from online submodular optimization: a single no-regret online learner can compete with an optimal sequence of predictions. Compared to previous work, which either learn a sequence of classifiers or rely on stronger assumptions such as realizability, we ensure both data-efficiency as well as performance guarantees in the fully agnostic setting. Experiments validate the efficiency and applicability of the approach on a wide range of problems including manipulator trajectory optimization, news recommendation and document summarization.
Output: Learning Policies for Contextual Submodular Prediction
P lanning problems where effects of actions are non-deterministic can be modeled a8 Markov decision processes. Planning prob­ lems are usually goal-directed. This paper proposes several techniques for exploiting the goal-directedness to accelerate value itera­ tion, a standard algorithm for solving Markov decision processes. Empirical studies have shown that the techniques can bring about significant speedups.
Output: Fast Value Iteration for Goal-Directed Markov Decision Processes
Artificial neural networks have gone through a recent rise in popularity, achieving state-of-the-art results in various fields, including image classification, speech recognition, and automated control. Both the performance and computational complexity of such models are heavily dependant on the design of characteristic hyper-parameters (e.g., number of hidden layers, nodes per layer, or choice of activation functions), which have traditionally been optimized manually. With machine learning penetrating low-power mobile and embedded areas, the need to optimize not only for performance (accuracy), but also for implementation complexity, becomes paramount. In this work, we present a multi-objective design space exploration method that reduces the number of solution networks trained and evaluated through response surface modelling. Given spaces which can easily exceed 10 solutions, manually designing a near-optimal architecture is unlikely as opportunities to reduce network complexity, while maintaining performance, may be overlooked. This problem is exacerbated by the fact that hyper-parameters which perform well on specific datasets may yield sub-par results on others, and must therefore be designed on a per-application basis. In our work, machine learning is leveraged by training an artificial neural network to predict the performance of future candidate networks. The method is evaluated on the MNIST and CIFAR-10 image datasets, optimizing for both recognition accuracy and computational complexity. Experimental results demonstrate that the proposed method can closely approximate the Pareto-optimal front, while only exploring a small fraction of the design space.
Output: Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization
We present a new algorithm to model and investigate the learning process of a learner mastering a set of grammatical rules from an inconsistent source. The compelling interest of human language acquisition is that the learning succeeds in virtually every case, despite the fact that the input data are formally inadequate to explain the success of learning. Our model explains how a learner can successfully learn from or even surpass its imperfect source without possessing any additional biases or constraints about the types of patterns that exist in the language. We use the data collected by Singleton & Newport (2004) on the performance of a 7-year boy Simon, who mastered the American Sign Language (ASL) by learning it from his parents, both of whom were imperfect speakers of ASL. We show that the algorithm possesses a frequency-boosting property, whereby the frequency of the most common form of the source is increased by the learner. We also explain several key features of Simon’s ASL.
Output: When learners surpass their models: mathematical modeling of learning from an inconsistent source
Square grids are commonly used in robotics and game development to model an agent’s environment, and well known in Artificial Intelligence heuristic search algorithms (A*, JPS, Theta* etc.) are utilized for grid path planning. A lot of research in this area has been focused so far on finding the shortest paths while in many applications producing smooth paths is preferable. In our work, we study the problem of generating smooth grid paths and concentrate on angle constrained path planning. We put angle constrained path planning problem formally and present a new algorithm of solving it – LIAN. We examine LIAN both theoretically and empirically. On the theoretical side, we prove that LIAN is sound and complete (under well-defined restrictions). On the experimental side, we show that LIAN significantly outperforms competitors in ability to find solutions under tough resource constraints and in computational efficiency.
Output: Grid-based angle-constrained path planning
Normalized web distance (NWD) is a similarity or normalized semantic distance based on the World Wide Web or any other large electronic database, for instance Wikipedia, and a search engine that returns reliable aggregate page counts. For sets of search terms the NWD gives a similarity on a scale from 0 (identical) to 1 (completely different). The NWD approximates the similarity according to all (upper semi)computable properties. We develop the theory and give applications. The derivation of the NWD method is based on Kolmogorov complexity.
Output: Web Similarity
Clustering evaluation measures are frequently used to evaluate the performance of algorithms. However, most measures are not properly normalized and ignore some information in the inherent structure of clusterings. We model the relation between two clusterings as a bipartite graph and propose a general component-based decomposition formula based on the components of the graph. Most existing measures are examples of this formula. In order to satisfy consistency in the component, we further propose a split-merge framework for comparing clusterings of different data sets. Our framework gives measures that are conditionally normalized, and it can make use of data point information, such as feature vectors and pairwise distances. We use an entropy-based instance of the framework and a coreference resolution data set to demonstrate empirically the utility of our framework over other measures.
Output: A Split-Merge Framework for Comparing Clusterings
Recently, there has been a growing interest in modeling planning with information constraints. Accordingly, an agent maximizes a regularized expected utility known as the free energy, where the regularizer is given by the information divergence from a prior to a posterior policy. While this approach can be justified in various ways, including from statistical mechanics and information theory, it is still unclear how it relates to decisionmaking against adversarial environments. This connection has previously been suggested in work relating the free energy to risk-sensitive control and to extensive form games. Here, we show that a single-agent free energy optimization is equivalent to a game between the agent and an imaginary adversary. The adversary can, by paying an exponential penalty, generate costs that diminish the decision maker’s payoffs. It turns out that the optimal strategy of the adversary consists in choosing costs so as to render the decision maker indifferent among its choices, which is a definining property of a Nash equilibrium, thus tightening the connection between free energy optimization and game theory.
Output: An Adversarial Interpretation of Information-Theoretic Bounded Rationality
We present an asymptotic analysis of Viterbi Training (VT) and contrast it with a more conventional Maximum Likelihood (ML) approach to parameter estimation in Hidden Markov Models. While ML estimator works by (locally) maximizing the likelihood of the observed data, VT seeks to maximize the probability of the most likely hidden state sequence. We develop an analytical framework based on a generating function formalism and illustrate it on an exactly solvable model of HMM with one unambiguous symbol. For this particular model the ML objective function is continuously degenerate. VT objective, in contrast, is shown to have only finite degeneracy. Furthermore, VT converges faster and results in sparser (simpler) models, thus realizing an automatic Occam’s razor for HMM learning. For more general scenario VT can be worse compared to ML but still capable of correctly recovering most of the parameters.
Output: Comparative Analysis of Viterbi Training and Maximum Likelihood Estimation for HMMs
Well-established automatic analyses of texts mainly consider frequencies of linguistic units, e.g. letters, words and bigrams, while methods based on cooccurrence networks consider the structure of texts regardless of the nodes label (i.e. the words semantics). In this paper, we reconcile these distinct viewpoints by introducing a generalized similarity measure to compare texts which accounts for both the network structure of texts and the role of individual words in the networks. We use the similarity measure for authorship attribution of three collections of books, each composed of 8 authors and 10 books per author. High accuracy rates were obtained with typical values from 90% to 98.75%, much higher than with the traditional the TF-IDF approach for the same collections. These accuracies are also higher than taking only the topology of networks into account. We conclude that the different properties of specific words on the macroscopic scale structure of a whole text are as relevant as their frequency of appearance; conversely, considering the identity of nodes brings further knowledge about a piece of text represented as a network.
Output: On the role of words in the network structure of texts: application to authorship attribution
Deep neural networks have dramatically advanced the state of the art for many<lb>areas of machine learning. Recently they have been shown to have a remarkable<lb>ability to generate highly complex visual artifacts such as images and text rather<lb>than simply recognize them.<lb>In this work we use neural networks to effectively invert low-dimensional face<lb>embeddings while producing realistically looking consistent images. Our contri-<lb>bution is twofold, first we show that a gradient ascent style approaches can be<lb>used to reproduce consistent images, with a help of a guiding image. Second, we<lb>demonstrate that we can train a separate neural network to effectively solve the<lb>minimization problem in one pass, and generate images in real-time. We then<lb>evaluate the loss imposed by using a neural network instead of the gradient descent<lb>by comparing the final values of the minimized loss function.
Output: Inverting face embeddings with convolutional neural networks
The Abstract Meaning Representation (AMR) is a representation for opendomain rich semantics, with potential use in fields like event extraction and machine translation. Node generation, typically done using a simple dictionary lookup, is currently an important limiting factor in AMR parsing. We propose a small set of actions that derive AMR subgraphs by transformations on spans of text, which allows for more robust learning of this stage. Our set of construction actions generalize better than the previous approach, and can be learned with a simple classifier. We improve on the previous state-of-the-art result for AMR parsing, boosting end-to-end performance by 3 F1 on both the LDC2013E117 and LDC2014T12 datasets.
Output: Robust Subgraph Generation Improves Abstract Meaning Representation Parsing
More and more of the information on the web is dialogic, from Facebook newsfeeds, to forum conversations, to comment threads on news articles. In contrast to traditional, monologic resources such as news, highly social dialogue is very frequent in social media, as illustrated in the snippets in Fig. 1 from the publicly available Internet Argument Corpus (IAC) (Walker et al., 2012). Utterances are frequently sarcastic, e.g., Really? Well, when I have a kid, I’ll be sure to just leave it in the woods, since it can apparently care for itself (R2 in Fig. 1 as well as Q1 and R1), and are often nasty, (R2 in Fig. 1). Note also the frequent use of dialogue specific discourse cues, e.g. the use of No in R1, Really? Well in R2, and okay, well in Q3 in Fig. 1 (Fox Tree and Schrock, 1999; Bryant and Fox Tree, 2002; Fox Tree, 2010).
Output: Identifying Subjective and Figurative Language in Online Dialogue
The increasing amount of available Linked Data resources is laying the foundations for more advanced Semantic Web applications. One of their main limitations, however, remains the general low level of data quality. In this paper we focus on a measure of quality which is negatively affected by the increase of the available resources. We propose a measure of semantic richness of Linked Data concepts and we demonstrate our hypothesis that the more a concept is reused, the less semantically rich it becomes. This is a significant scalability issue, as one of the core aspects of Linked Data is the propagation of semantic information on the Web by reusing common terms. We prove our hypothesis with respect to our measure of semantic richness and we validate our model empirically. Finally, we suggest possible future directions to address this scalability problem.
Output: A Linked Data Scalability Challenge: Concept Reuse Leads to Semantic Decay
The paper considers the class of information systems capable of solving heuristic problems on basis of formal theory that was termed modal and vector theory of formal intelligent systems (FIS). The paper justifies the construction of FIS resolution algorithm, defines the main features of these systems and proves theorems that underlie the theory. The principle of representation diversity of FIS construction is formulated. The paper deals with the main principles of constructing and functioning formal intelligent system (FIS) on basis of FIS modal and vector theory. The following phenomena are considered: modular architecture of FIS presentation sub-system, algorithms of data processing at every step of the stage of creating presentations. Besides the paper suggests the structure of neural elements, i.e. zone detectors and processors that are the basis for FIS construction. Subjects: Artificial Intelligence (cs. AI)
Output: Principles of modal and vector theory of formal intelligence systems
The Copyright of this work is owned by the Association for Computational Linguistics (ACL). However, each of the authors and the employers for whom the work was performed reserve all other rights, specifically including the following: ... (4) The right to make copies of the work for internal distribution within the author’s organization and for external distribution as a preprint, reprint, technical report, or related class of document. The original paper is to appear in: ACL 2014: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics We introduce a novel approach for building language models based on a systematic, recursive exploration of skip n-gram models which are interpolated using modified KneserNey smoothing. Our approach generalizes language models as it contains the classical interpolation with lower order models as a special case. In this paper we motivate, formalize and present our approach. In an extensive empirical experiment over English text corpora we demonstrate that our generalized language models lead to a substantial reduction of perplexity between 3.1% and 12.7% in comparison to traditional language models using modified Kneser-Ney smoothing. Furthermore, we investigate the behaviour over three other languages and a domain specific corpus where we observed consistent improvements. Finally, we also show that the strength of our approach lies in its ability to cope in particular with sparse training data. Using a very small training data set of only 736 KB text we yield improvements of even 25.7% reduction of perplexity.
Output: A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser-Ney Smoothing
Recent years have witnessed the increase of competition in science. While promoting the quality of research in many cases, an intense competition among scientists can also trigger unethical scientific behaviors. To increase the total number of published papers, some authors even resort to software tools that are able to produce grammatical, but meaningless scientific manuscripts. Because automatically generated papers can be misunderstood as real papers, it becomes of paramount importance to develop means to identify these scientific frauds. In this paper, I devise a methodology to distinguish real manuscripts from those generated with SCIGen, an automatic paper generator. Upon modeling texts as complex networks (CN), it was possible to discriminate real from fake papers with at least 89% of accuracy. A systematic analysis of features relevance revealed that the accessibility and betweenness were useful in particular cases, even though the relevance depended upon the dataset. The successful application of the methods described here show, as a proof of principle, that network features can be used to identify scientific gibberish papers. In addition, the CN-based approach can be combined in a straightforward fashion with traditional statistical language processing methods to improve the performance in identifying artificially generated papers.
Output: Comparing the topological properties of real and artificially generated scientific manuscripts
Most real-world dynamic systems are composed of different components that often evolve at very different rates. In traditional temporal graphical models, such as dynamic Bayesian networks, time is modeled at a fixed granularity, generally selected based on the rate at which the fastest component evolves. Inference must then be performed at this fastest granularity, potentially at significant computational cost. Continuous Time Bayesian Networks (CTBNs) avoid time-slicing in the representation by modeling the system as evolving continuously over time. The expectation-propagation (EP) inference algorithm of Nodelman et al. (2005) can then vary the inference granularity over time, but the granularity is uniform across all parts of the system, and must be selected in advance. In this paper, we provide a new EP algorithm that utilizes a general cluster graph architecture where clusters contain distributions that can overlap in both space (set of variables) and time. This architecture allows different parts of the system to be modeled at very different time granularities, according to their current rate of evolution. We also provide an information-theoretic criterion for dynamically re-partitioning the clusters during inference to tune the level of approximation to the current rate of evolution. This avoids the need to hand-select the appropriate granularity, and allows the granularity to adapt as information is transmitted across the network. We present experiments demonstrating that this approach can result in significant computational savings.
Output: Reasoning at the Right Time Granularity
In this paper, we study the use of recurrent neural networks (RNNs) for modeling and forecasting time series. We first illustrate the fact that standard sequence-to-sequence RNNs neither capture well periods in time series nor handle well missing values, even though many real life times series are periodic and contain missing values. We then propose an extended attention mechanism that can be deployed on top of any RNN and that is designed to capture periods and make the RNN more robust to missing values. We show the effectiveness of this novel model through extensive experiments with multiple univariate and multivariate datasets.
Output: Time Series Forecasting using RNNs: an Extended Attention Mechanism to Model Periods and Handle Missing Values
Bayesian Optimisation (BO) is a technique used in optimising a D-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on D even though the function depends on all D dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.
Output: High Dimensional Bayesian Optimisation and Bandits via Additive Models
The field of machine learning has taken a dramatic twist in recent times, with the rise of the Artificial Neural Network (ANN). These biologically inspired computational models are able to far exceed the performance of previous forms of artificial intelligence in common machine learning tasks. One of the most impressive forms of ANN architecture is that of the Convolutional Neural Network (CNN). CNNs are primarily used to solve difficult image-driven pattern recognition tasks and with their precise yet simple architecture, offers a simplified method of getting started with ANNs. This document provides a brief introduction to CNNs, discussing recently published papers and newly formed techniques in developing these brilliantly fantastic image recognition models. This introduction assumes you are familiar with the fundamentals of ANNs and machine learning.
Output: An Introduction to Convolutional Neural Networks