niryuu/sni-each-converted huggingface dataset:
The computational mechanisms by which nonlinear recurrent neural networks (RNNs) achieve their goals remains an open question. There exist many problem domains where intelligibility of the network model is crucial for deployment. Here we introduce a recurrent architecture composed of input-switched affine transformations, in other words an RNN without any nonlinearity and with one set of weights per input. We show that this architecture achieves near identical performance to traditional architectures on language modeling of Wikipedia text, for the same number of model parameters. It can obtain this performance with the potential for computational speedup compared to existing methods, by precomputing the composed affine transformations corresponding to longer input sequences. As our architecture is affine, we are able to understand the mechanisms by which it functions using linear methods. For example, we show how the network linearly combines contributions from the past to make predictions at the current time step. We show how representations for words can be combined in order to understand how context is transferred across word boundaries. Finally, we demonstrate how the system can be executed and analyzed in arbitrary bases to aid understanding.
Output: INTELLIGIBLE LANGUAGE MODELING WITH INPUT SWITCHED AFFINE NETWORKS
We study the classic online learning problem of predicting with expert advice, and propose a truly parameter-free and adaptive algorithm that achieves several objectives simultaneously without using any prior information. The main component of this work is an improved version of the NormalHedge.DT algorithm [Luo and Schapire, 2014], called AdaNormalHedge. On one hand, this new algorithm ensures small regret when the competitor has small loss and almost constant regret when the losses are stochastic. On the other hand, the algorithm is able to compete with any convex combination of the experts simultaneously, with a regret in terms of the relative entropy of the prior and the competitor. This resolves an open problem proposed by Chaudhuri et al. [2009] and Chernov and Vovk [2010]. Moreover, we extend the results to the sleeping expert setting and provide two applications to illustrate the power of AdaNormalHedge: 1) competing with time-varying unknown competitors and 2) predicting almost as well as the best pruning tree. Our results on these applications significantly improve previous work from different aspects, and a special case of the first application resolves another open problem proposed by Warmuth and Koolen [2014] on whether one can simultaneously achieve optimal shifting regret for both adversarial and stochastic losses.
Output: Achieving All with No Parameters: Adaptive NormalHedge
Events and entities are closely related; entities are often actors or participants in events and events without entities are uncommon. The interpretation of events and entities is highly contextually dependent. Existing work in information extraction typically models events separately from entities, and performs inference at the sentence level, ignoring the rest of the document. In this paper, we propose a novel approach that models the dependencies among variables of events, entities, and their relations, and performs joint inference of these variables across a document. The goal is to enable access to document-level contextual information and facilitate contextaware predictions. We demonstrate that our approach substantially outperforms the stateof-the-art methods for event extraction as well as a strong baseline for entity extraction.
Output: Joint Extraction of Events and Entities within a Document Context
In a variety of applications, one desires to detect groups of anomalous data samples, with a group potentially manifesting its atypicality (relative to a reference model) on a low-dimensional subset of the full measured set of features. Samples may only be weakly atypical individually, whereas they may be strongly atypical when considered jointly. What makes this group anomaly detection problem quite challenging is that it is a priori unknown which subset of features jointly manifests a particular group of anomalies. Moreover, it is unknown how many anomalous groups are present in a given data batch. In this work, we develop a group anomaly detection (GAD) scheme to identify the subset of samples and subset of features that jointly specify an anomalous cluster. We apply our approach to network intrusion detection to detect BotNet and peer-to-peer flow clusters. Unlike previous studies, our approach captures and exploits statistical dependencies that may exist between the measured features. Experiments on real world network traffic data demonstrate the advantage of our proposed system, and highlight the importance of exploiting feature dependency structure, compared to the feature (or test) independence assumption made in previous studies.
Output: Detecting Clusters of Anomalies on Low-Dimensional Feature Subsets with Application to Network Traffic Flow Data
Knowledge graph embedding aims to embed entities and relations of knowledge graphs into low-dimensional vector spaces. Translating embedding methods regard relations as the translation from head entities to tail entities, which achieve the state-of-the-art results among knowledge graph embedding methods. However, a major limitation of these methods is the time consuming training process, which may take several days or even weeks for large knowledge graphs, and result in great diculty in practical applications. In this paper, we propose an ecient parallel framework for translating embedding methods, called ParTrans-X, which enables the methods to be paralleled without locks by utilizing the distinguished structures of knowledge graphs. Experiments on two datasets with three typical translating embedding methods, i.e., TransE [3], TransH [19], and a more ecient variant TransEAdaGrad [11] validate that ParTrans-X can speed up the training process by more than an order of magnitude.
Output: Eicient Parallel Translating Embedding For Knowledge Graphs
Since Alan Turing envisioned Artificial Intelligence (AI) [1], a major driving force behind technical progress has been competition with human cognition. Historical milestones have been frequently associated with computers matching or outperforming humans in difficult cognitive tasks (e.g. face recognition [2], personality classification [3], driving cars [4], or playing video games [5]), or defeating humans in strategic zero-sum encounters (e.g. Chess [6], Checkers [7], Jeopardy! [8], Poker [9], or Go [10]). In contrast, less attention has been given to developing autonomous machines that establish mutually cooperative relationships with people who may not share the machine’s preferences. A main challenge has been that human cooperation does not require sheer computational power, but rather relies on intuition [11], cultural norms [12], emotions and signals [13, 14, 15, 16], and pre-evolved dispositions toward cooperation [17], common-sense mechanisms that are difficult to encode in machines for arbitrary contexts. Here, we combine a state-of-the-art machine-learning algorithm with novel mechanisms for generating and acting on signals to produce a new learning algorithm that cooperates with people and other machines at levels that rival human cooperation in a variety of two-player repeated stochastic games. This is the first general-purpose algorithm that is capable, given a description of a previously unseen game environment, of learning to cooperate with people within short timescales in scenarios previously unanticipated by algorithm designers. This is achieved without complex opponent modeling or higher-order theories of mind, thus showing that flexible, fast, and general human-machine cooperation is computationally achievable using a non-trivial, but ultimately simple, set of algorithmic mechanisms. ∗Correspondence should be addressed to crandall@cs.byu.edu and irahwan@mit.edu ar X iv :1 70 3. 06 20 7v 1 [ cs .A I] 1 7 M ar 2 01 7
Output: Cooperating with Machines
Vulnerability of state-of-the-art deep neural networks to adversarial attacks has been attracting a lot of attention recently. In this work we propose a new algorithm for constructing universal adversarial perturbations. Our approach is based on computing the so called (p, q)-singular vectors of the Jacobian matrices of hidden layers of a network. Resulting perturbations present interesting visual patterns and by using a batch of just 64 images we can construct adversarial perturbations with relatively high fooling rate. We also investigate a correlation between the singular values of the Jacobian matrices and the fooling rate of a corresponding singular vector.
Output: Art of singular vectors and universal adversarial perturbations
A new computationally efficient dependence measure, and an adaptive statistical test of independence, are proposed. The dependence measure is the difference between analytic embeddings of the joint distribution and the product of the marginals, evaluated at a finite set of locations (features). These features are chosen so as to maximize a lower bound on the test power, resulting in a test that is data-efficient, and that runs in linear time (with respect to the sample size n). The optimized features can be interpreted as evidence to reject the null hypothesis, indicating regions in the joint domain where the joint distribution and the product of the marginals differ most. Consistency of the independence test is established, for an appropriate choice of features. In real-world benchmarks, independence tests using the optimized features perform comparably to the state-of-the-art quadratictime HSIC test, and outperform competing O(n) and O(n log n) tests.
Output: An Adaptive Test of Independence with Analytic Kernel Embeddings
In this paper, we provide all information to participate to the Second International Nurse Rostering Competition (INRC-II). First, we describe the problem formulation, which, differently from INRC-I, is a multi-stage procedure. Second, we illustrate all the necessary infrastructure do be used together with the participant’s solver, including the testbed, the file formats, and the validation/simulation tools. Finally, we state the rules of the competition. All update-to-date information about the competition is available at http://mobiz.vives.be/inrc2/.
Output: Second International Nurse Rostering Competition (INRC-II) — Problem Description and Rules —
To be considered for the 2017 IEEE Jack Keil Wolf ISIT Student Paper Award. Interaction information is one of the multivariate generalizations of mutual information, which expresses the amount information shared among a set of variables, beyond the information, which is shared in any proper subset of those variables. Unlike (conditional) mutual information, which is always non-negative, interaction information can be negative. We utilize this property to find the direction of causal influences among variables in a triangle topology under some mild assumptions.
Output: Interaction Information for Causal Inference: The Case of Directed Triangle
We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with M ≥ 3 components. Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical Gaussians. We prove that the log-likelihood value of these bad local maxima can be arbitrarily worse than that of any global optimum, thereby resolving an open question of Srebro [21]. Our second main result shows that the EM algorithm (or a first-order variant of it) with random initialization will converge to bad critical points with probability at least 1 − e. We further establish that a first-order variant of EM will not converge to strict saddle points almost surely, indicating that the poor performance of the first-order method can be attributed to the existence of bad local maxima rather than bad saddle points. Overall, our results highlight the necessity of careful initialization when using the EM algorithm in practice, even when applied in highly favorable settings.
Output: Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences
Inspired by the hierarchical hidden Markov models (HHMM), we present the hierarchical semi-Markov conditional random field (HSCRF), a generalisation of embedded undirected Markov chains to model complex hierarchical, nested Markov processes. It is parameterised in a discriminative framework and has polynomial time algorithms for learning and inference. Importantly, we consider partiallysupervised learning and propose algorithms for generalised partially-supervised learning and constrained inference. We demonstrate the HSCRF in two applications: (i) recognising human activities of daily living (ADLs) from indoor surveillance cameras, and (ii) noun-phrase chunking. We show that the HSCRF is capable of learning rich hierarchical models with reasonable accuracy in both fully and partially observed data cases.
Output: Hierarchical Semi-Markov Conditional Random Fields for Recursive Sequential Data
Probabilistic Linear Discriminant Analysis (PLDA) has become state-of-the-art method for modeling i-vector space in speaker recognition task. However the performance degradation is observed if enrollment data size differs from one speaker to another. This paper presents a solution to such problem by introducing new PLDA scoring normalization technique. Normalization parameters are derived in a blind way, so that, unlike traditional ZT-norm, no extra development data is required. Moreover, proposed method has shown to be optimal in terms of detection cost function. The experiments conducted on NIST SRE 2014 database demonstrate an improved accuracy in a mixed enrollment number condition.
Output: Blind score normalization method for PLDA based speaker recognition
We introduce a simple and accurate neural model for dependency-based semantic role labeling. Our model predicts predicate-argument dependencies relying on states of a bidirectional LSTM encoder. The semantic role labeler achieves respectable performance on English even without any kind of syntactic information and only using local inference. However, when automatically predicted part-of-speech tags are provided as input, it substantially outperforms all previous local models and approaches the best reported results on the CoNLL-2009 dataset. Syntactic parsers are unreliable on out-of-domain data, so standard (i.e. syntactically-informed) SRL models are hindered when tested in this setting. Our syntax-agnostic model appears more robust, resulting in the best reported results on the standard out-of-domain test set.1
Output: A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling
The significant computational costs of deploying neural networks in large-scale or resource constrained environments, such as data centers and mobile devices, has spurred interest in model compression, which can achieve a reduction in both arithmetic operations and storage memory. Several techniques have been proposed for reducing or compressing the parameters for feed-forward and convolutional neural networks, but less is understood about the effect of parameter compression on recurrent neural networks (RNN). In particular, the extent to which the recurrent parameters can be compressed and the impact on short-term memory performance, is not well understood. In this paper, we study the effect of complexity reduction, through singular value decomposition rank reduction, on RNN and minimal gated recurrent unit (MGRU) networks for several tasks. We show that considerable rank reduction is possible when compressing recurrent weights, even without fine tuning. Furthermore, we propose a perturbation model for the effect of general perturbations, such as a compression, on the recurrent parameters of RNNs. The model is tested against a noiseless memorization experiment that elucidates the short-term memory performance. In this way, we demonstrate that the effect of compression of recurrent parameters is dependent on the degree of temporal coherence present in the data and task. This work can guide on-the-fly RNN compression for novel environments or tasks, and provides insight for applying RNN compression in low-power devices, such as hearing aids. Keywords—compression; recurrent neural networks; complexity reduction; SVD; RNN; MGRU; MRU
Output: Parameter Compression of Recurrent Neural Networks and Degredation of Short-term Memory
We define a notion of rational closure for the logic SHIQ, which does not enjoys the finite model property, building on the notion of rational closure introduced by Lehmann and Magidor in [23]. We provide a semantic characterization of rational closure in SHIQ in terms of a preferential semantics, based on a finite rank characterization of minimal models. We show that the rational closure of a TBox can be computed in EXPTIME using entailment in SHIQ.
Output: Rational closure in SHIQ
To what extent is the success of deep visualization due to the training? Could we do deep visualization using untrained, random weight networks? To address this issue, we explore new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks. First we invert representations in feature spaces and reconstruct images from white noise inputs. The reconstruction quality is statistically higher than that of the same method applied on well trained networks with the same architecture. Next we synthesize textures using scaled correlations of representations in multiple layers and our results are almost indistinguishable with the original natural texture and the synthesized textures based on the trained network. Third, by recasting the content of an image in the style of various artworks, we create artistic images with high perceptual quality, highly competitive to the prior work of Gatys et al. on pretrained networks. To our knowledge this is the first demonstration of image representations using untrained deep neural networks. Our work provides a new and fascinating tool to study the representation of deep network architecture and sheds light on new understandings on deep visualization.
Output: A Powerful Generative Model Using Random Weights for the Deep Image Representation
Estimating student proficiency is an important task for computer based learning systems. We compare a family of IRTbased proficiency estimation methods to Deep Knowledge Tracing (DKT), a recently proposed recurrent neural network model with promising initial results. We evaluate how well each model predicts a student’s future response given previous responses using two publicly available and one proprietary data set. We find that IRT-based methods consistently matched or outperformed DKT across all data sets at the finest level of content granularity that was tractable for them to be trained on. A hierarchical extension of IRT that captured item grouping structure performed best overall. When data sets included non-trivial autocorrelations in student response patterns, a temporal extension of IRT improved performance over standard IRT while the RNNbased method did not. We conclude that IRT-based models provide a simpler, better-performing alternative to existing RNN-based models of student interaction data while also affording more interpretability and guarantees due to their formulation as Bayesian probabilistic models.
Output: Back to the basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation
In this paper, we propose several improvements on the block-coordinate Frank-Wolfe (BCFW) algorithm from Lacoste-Julien et al. (2013) recently used to optimize the structured support vector machine (SSVM) objective in the context of structured prediction, though it has wider applications. The key intuition behind our improvements is that the estimates of block gaps maintained by BCFW reveal the block suboptimality that can be used as an adaptive criterion. First, we sample objects at each iteration of BCFW in an adaptive non-uniform way via gapbased sampling. Second, we incorporate pairwise and away-step variants of Frank-Wolfe into the block-coordinate setting. Third, we cache oracle calls with a cache-hit criterion based on the block gaps. Fourth, we provide the first method to compute an approximate regularization path for SSVM. Finally, we provide an exhaustive empirical evaluation of all our methods on four structured prediction datasets.
Output: Minding the Gaps for Block Frank-Wolfe Optimization of Structured SVMs
Machine learning relies on the assumption that unseen test instances of a classification problem follow the same distribution as observed training data. However, this principle can break down when machine learning is used to make important decisions about the welfare (employment, education, health) of strategic individuals. Knowing information about the classifier, such individuals may manipulate their attributes in order to obtain a better classification outcome. As a result of this behavior—often referred to as gaming—the performance of the classifier may deteriorate sharply. Indeed, gaming is a well-known obstacle for using machine learning methods in practice; in financial policy-making, the problem is widely known as Goodhart’s law. In this paper, we formalize the problem, and pursue algorithms for learning classifiers that are robust to gaming. We model classification as a sequential game between a player named “Jury” and a player named “Contestant.” Jury designs a classifier, and Contestant receives an input to the classifier drawn from a distribution. Before being classified, Contestant may change his input based on Jury’s classifier. However, Contestant incurs a cost for these changes according to a cost function. Jury’s goal is to achieve high classification accuracy with respect to Contestant’s original input and some underlying target classification function, assuming Contestant plays best response. Contestant’s goal is to achieve a favorable classification outcome while taking into account the cost of achieving it. For a natural class of separable cost functions, and certain generalizations, we obtain computationally efficient learning algorithms which are near optimal, achieving a classification error that is arbitrarily close to the theoretical minimum. Surprisingly, our algorithms are efficient even on concept classes that are computationally hard to learn. For general cost functions, designing an approximately optimal strategy-proof classifier, for inverse-polynomial approximation, is NP-hard. ar X iv :1 50 6. 06 98 0v 1 [ cs .L G ] 2 3 Ju n 20 15
Output: Strategic Classification
We apply multilayer bootstrap network (MBN), a recent pro-<lb>posed unsupervised learning method, to unsupervised speaker<lb>recognition. The proposed method first extracts supervectors<lb>from an unsupervised universal background model, then re-<lb>duces the dimension of the high-dimensional supervectors by<lb>multilayer bootstrap network, and finally conducts unsuper-<lb>vised speaker recognition by clustering the low-dimensional<lb>data. The comparison results with 2 unsupervised and 1 su-<lb>pervised speaker recognition techniques demonstrate the ef-<lb>fectiveness and robustness of the proposed method.
Output: MULTILAYER BOOTSTRAP NETWORK FOR UNSUPERVISED SPEAKER RECOGNITION
We present an annotation schema as part of an effort to create a manually annotated corpus for Arabic dialogue language understanding including spoken dialogue and written „chat‟ dialogue for inquiryanswer domain. The proposed schema handles mainly the request and response acts that occurs frequently in inquiry-answer debate conversations expressing request services, suggests, and offers. We applied the proposed schema on 83 Arabic inquiry-answer dialogues.
Output: Arabic Inquiry-Answer Dialogue Acts Annotation Schema
The development of machine learning in particular and artificial intelligent in general has been strongly conditioned by the lack of an appropriate interface layer between deduction, abduction and induction [1]. In this work we extend traditional algebraic specification methods [2] in this direction. Here we assume that such interface for AI emerges from an adequate Neural-Symbolic integration [3]. This integration is made for universe of discourse described on a Topos[4] governed by a many-valued Lukasiewicz logic. Sentences are integrated in a symbolic knowledge base describing the problem domain, codified using a graphicbased language, wherein every logic connective is defined by a neuron in an artificial network. This allows the integration of first-order formulas into a network architecture as background knowledge, and simplifies symbolic rule extraction from trained networks. For the train of such neural networks we changed the Levenderg-Marquardt algorithm [5], restricting the knowledge dissemination in the network structure using soft crystallization. This procedure reduces neural network plasticity without drastically damaging the learning performance, allowing the emergence of symbolic patterns. This makes the descriptive power of produced neural networks similar to the descriptive power of Lukasiewicz logic language, reducing the information lost on translation between symbolic and connectionist structures. We tested this method on the extraction of knowledge from specified structures. For it, we present the notion of fuzzy state automata, and we use automata behaviour to infer its structure. We use this type of automata on the generation of models for relations specified as symbolic background knowledge. Using the involved automata behaviour as data sets, we used our learning methodology, to extract new insights about the models, and inject them into the specification. This allows the improvement about the problem domain knowledge. ———————————————————————–
Output: KNOWLEDGE EXTRACTION AND KNOWLEDGE INTEGRATION GOVERNED BY LUKASIEWICZ LOGICS
The user equilibrium in traffic assignment problem is based on the fact that travelers choose the minimum-cost path between every origin-destination pair and on the assumption that such a behavior will lead to an equilibrium of the traffic network. In this paper, we consider this problem when the traffic network links are fuzzy cost. Therefore, a Physarum-type algorithm is developed to unify the Physarum network and the traffic network for taking full of advantage of Physarum Polycephalum’s adaptivity in network design to solve the user equilibrium problem. Eventually, some experiments are used to test the performance of this method. The results demonstrate that our approach is competitive when compared with other existing algorithms.
Output: A bio-inspired algorithm for fuzzy user equilibrium problem by aid of Physarum Polycephalum
Neural machine translation has become a major alternative to widely used phrase-based statistical machine translation. We notice however that much of research on neural machine translation has focused on European languages despite its language agnostic nature. In this paper, we apply neural machine translation to the task of Arabic translation (Ar↔En) and compare it against a standard phrase-based translation system. We run extensive comparison using various configurations in preprocessing Arabic script and show that the phrase-based and neural translation systems perform comparably to each other and that proper preprocessing of Arabic script has a similar effect on both of the systems. We however observe that the neural machine translation significantly outperform the phrase-based system on an out-of-domain test set, making it attractive for real-world deployment.
Output: First Result on Arabic Neural Machine Translation
A key goal of research in conversational systems is to train an interactive agent to help a user with a task. Human conversation, however, is notoriously incomplete, ambiguous, and full of extraneous detail. To operate effectively, the agent must not only understand what was explicitly conveyed but also be able to reason in the presence of missing or unclear information. When unable to resolve ambiguities on its own, the agent must be able to ask the user for the necessary clarifications and incorporate the response in its reasoning. Motivated by this problem we introduce QRAQ (Query, Reason, and Answer Questions), a new synthetic domain, in which a User gives an Agent a short story and asks a challenge question. These problems are designed to test the reasoning and interaction capabilities of a learningbased Agent in a setting that requires multiple conversational turns. A good Agent should ask only non-deducible, relevant questions until it has enough information to correctly answer the User’s question. We use standard and improved reinforcement learning based memory-network architectures to solve QRAQ problems in the difficult setting where the reward signal only tells the Agent if its final answer to the challenge question is correct or not. To provide an upper-bound to the RL results we also train the same architectures using supervised information that tells the Agent during training which variables to query and the answer to the challenge question. We evaluate our architectures on four QRAQ dataset types, and scale the complexity for each along multiple dimensions.
Output: Learning to Query, Reason, and Answer Questions On Ambiguous Texts
We propose a novel framework for the analysis of learning algorithms that allows us to say when such algorithms can and cannot generalize certain patterns from training data to test data. In particular we focus on situations where the rule that must be learned concerns two components of a stimulus being identical. We call such a basis for discrimination an identitybased rule. Identity-based rules have proven to be difficult or impossible for certain types of learning algorithms to acquire from limited datasets. This is in contrast to human behaviour on similar tasks. Here we provide a framework for rigorously establishing which learning algorithms will fail at generalizing identity-based rules to novel stimuli. We use this framework to show that such algorithms are unable to generalize identitybased rules to novel inputs unless trained on virtually all possible inputs. We demonstrate these results computationally with a multilayer feedforward neural network.
Output: Which Learning Algorithms Can Generalize Identity-Based Rules to Novel Inputs?
Predicting user responses, such as clicks and conversions, is of great importance and has found its usage in many Web applications including recommender systems, web search and online advertising. The data in those applications is mostly categorical and contains multiple fields; a typical representation is to transform it into a high-dimensional sparse binary feature representation via one-hot encoding. Facing with the extreme sparsity, traditional models may limit their capacity of mining shallow patterns from the data, i.e. low-order feature combinations. Deep models like deep neural networks, on the other hand, cannot be directly applied for the high-dimensional input because of the huge feature space. In this paper, we propose a Product-based Neural Networks (PNN) with an embedding layer to learn a distributed representation of the categorical data, a product layer to capture interactive patterns between interfield categories, and further fully connected layers to explore high-order feature interactions. Our experimental results on two large-scale real-world ad click datasets demonstrate that PNNs consistently outperform the state-of-the-art models on various metrics.
Output: Product-based Neural Networks for User Response Prediction
Truth discovery is to resolve conflicts and find the truth from multiple-source statements. Conventional methods mostly research based on the mutual effect between the reliability of sources and the credibility of statements, however, pay no attention to the mutual effect among the credibility of statements about the same object. We propose memory network based models to incorporate these two ideas to do the truth discovery. We use feedforward memory network and feedback memory network to learn the representation of the credibility of statements which are about the same object. Specially, we adopt memory mechanism to learn source reliability and use it through truth prediction. During learning models, we use multiple types of data (categorical data and continuous data) by assigning different weights automatically in the loss function based on their own effect on truth discovery prediction. The experiment results show that the memory network based models much outperform the state-of-the-art method and other baseline methods.
Output: Truth Discovery with Memory Network
Stacked denoising auto encoders (DAEs) are well known to learn useful deep representations, which can be used to improve supervised training by initializing a deep network. We investigate a training scheme of a deep DAE, where DAE layers are gradually added and keep adapting as additional layers are added. We show that in the regime of mid-sized datasets, this gradual training provides a small but consistent improvement over stacked training in both reconstruction quality and classification error over stacked training on MNIST and CIFAR datasets. 1 GRADUAL TRAINING OF DENOISING AUTOENCODERS We test here gradual training of deep denoising auto encoders, training the network layer-by-layer, but lower layers keep adapting throughout training. To allow lower layers to adapt continuously, noise is injected at the input level. This training procedure differs from stack-training of auto encoders (Vincent et al., 2010) More specifically, in gradual training, the first layer of the deep DAE is trained as in stacked training, producing a layer of weights w1. Then, when adding the second layer autoencoder, its weights w2 are tuned jointly with the already-trained weights w1. Given a training sample x, we generate a noisy version x̃, feed it to the 2-layered DAE, and compute the activation at the subsequent layers h1 = Sigmoid(w > 1 x), h2 = Sigmoid(w > 2 h1) and y = Sigmoid(w ′> 3 h2). Importantly, the loss function is computed over the input x, and is used to update all the weights including w1. Similarly, if a 3rd layer is trained, it involves tuning w1 and w2 in addition to w3 and w′ 4. 2 EXPERIMENTAL PROCEDURES We compare the performance of gradual and stacked training in two learning setups: an unsupervised denoising task, and a supervised classification task initialized using the weights learned in an unsupervised way. Evaluations were made on three benchmarks: MNIST, CIFAR-10 and CIFAR100, but only show here MNIST results due to space constraints. We used a test subset of 10,000 samples and several sizes of training-set all maintaining the uniform distribution over classes. Hyper parameters were selected using a second level of cross validation, including the learning rate, SGD batch size, momentum and weight decay. In the supervised experiments, training was ’early stopped’ after 35 epochs without improvement. The results reported below are averages over 3 train-validation splits. Since gradual training involves updating lower layers, every presentation of a sample involves more weight updates than in a single-layered DAE. To compare stacked and gradual training on a common ground, we limited gradual training to use the same budget of weight update steps as stacked training. For example, when training the second layer for n epochs in gradual training, we allocate 2n training epochs for stacked training (details in the full paper). 1 ar X iv :1 50 4. 02 90 2v 1 [ cs .L G ] 1 1 A pr 2 01 5 Accepted as a workshop contribution at ICLR 2015 a) Unsupervised Training b) Supervised training 0 0.25 0.5 10.4 10.5 10.6 10.7
Output: GRADUAL TRAINING METHOD FOR DENOISING AUTO ENCODERS
We present a method for implementing an Efficient Unitary Neural Network (EUNN) whose computational complexity is merelyO(1) per parameter and has full tunability, from spanning part of unitary space to all of it. We apply the EUNN in Recurrent Neural Networks, and test its performance on the standard copying task and the MNIST digit recognition benchmark, finding that it significantly outperforms a non-unitary RNN, an LSTM network, an exclusively partial space URNN and a projective URNN with comparable parameter numbers.
Output: Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNN
Without discourse connectives, classifying implicit discourse relations is a challenging task and a bottleneck for building a practical discourse parser. Previous research usually makes use of one kind of discourse framework such as PDTB or RST to improve the classification performance on discourse relations. Actually, under different discourse annotation frameworks, there exist multiple corpora which have internal connections. To exploit the combination of different discourse corpora, we design related discourse classification tasks specific to a corpus, and propose a novel Convolutional Neural Network embedded multi-task learning system to synthesize these tasks by learning both unique and shared representations for each task. The experimental results on the PDTB implicit discourse relation classification task demonstrate that our model achieves significant gains over baseline systems.
Output: Implicit Discourse Relation Classification via Multi-Task Neural Networks
Seven years on from OWL becoming a W3C recommendation, and two years on from the more recent OWL 2 W3C recommendation, OWL has still experienced only patchy uptake on the Web. Although certain OWL features (like owl:sameAs) are very popular, other features of OWL are largely neglected by publishers in the Linked Data world. This may suggest that despite the promise of easy implementations and the proposal of tractable profiles suggested in OWL’s second version, there is still no “right” standard fragment for the Linked Data community. In this paper, we (1) analyse uptake of OWL on the Web of Data, (2) gain insights into the OWL fragment that is actually used/usable on the Web, where we arrive at the conclusion that this fragment is likely to be a simplified profile based on OWL RL, (3) propose and discuss such a new fragment, which we call OWL LD (for Linked Data).
Output: OWL: Yet to arrive on the Web of Data?
LP relaxation-based message passing algorithms provide an effective tool for MAP inference over Probabilistic Graphical Models. However, different LP relaxations often have different objective functions and variables of differing dimensions, which presents a barrier to effective comparison and analysis. In addition, the computational complexity of LP relaxation-based methods grows quickly with the number of constraints. Reducing the number of constraints without sacrificing the quality of the solutions is thus desirable. We propose a unified formulation under which existing MAP LP relaxations may be compared and analysed. Furthermore, we propose a new tool called Marginal Polytope Diagrams. Some properties of Marginal Polytope Diagrams are exploited such as node redundancy and edge equivalence. We show that using Marginal Polytope Diagrams allows the number of constraints to be reduced without loosening the LP relaxations. Then, using Marginal Polytope Diagrams and constraint reduction, we develop three novel message passing algorithms, and demonstrate that two of these show a significant improvement in speed over state-of-art algorithms while delivering a competitive, and sometimes higher, quality of solution.
Output: Constraint Reduction using Marginal Polytope Diagrams for MAP LP Relaxations
Approximate Markov chain Monte Carlo (MCMC) offers the promise of more rapid sampling at the cost of more biased inference. Since standard MCMC diagnostics fail to detect these biases, researchers have developed computable Stein discrepancy measures that provably determine the convergence of a sample to its target distribution. This approach was recently combined with the theory of reproducing kernels to define a closed-form kernel Stein discrepancy (KSD) computable by summing kernel evaluations across pairs of sample points. We develop a theory of weak convergence for KSDs based on Stein’s method, demonstrate that commonly used KSDs fail to detect non-convergence even for Gaussian targets, and show that kernels with slowly decaying tails provably determine convergence for a large class of target distributions. The resulting convergence-determining KSDs are suitable for comparing biased, exact, and deterministic sample sequences and simpler to compute and parallelize than alternative Stein discrepancies. We use our tools to compare biased samplers, select sampler hyperparameters, and improve upon existing KSD approaches to one-sample hypothesis testing and sample quality improvement.
Output: Measuring Sample Quality with Kernels
Algorithm portfolios represent a strategy of composing multiple heuristic algorithms, each suited to a different class of problems, within a single general solver that will choose the best suited algorithm for each input. This approach recently gained popularity especially for solving combinatoric problems, but optimization applications are still emerging. The COCO platform [6] [5] of the BBOB workshop series is the current standard way to measure performance of continuous black-box optimization algorithms. As an extension to the COCO platform, we present the Python-based COCOpf framework that allows composing portfolios of optimization algorithms and running experiments with different selection strategies. In our framework, we focus on black-box algorithm portfolio and online adaptive selection. As a demonstration, we measure the performance of stock SciPy [8] optimization algorithms and the popular CMA algorithm [4] alone and in a portfolio with two simple selection strategies. We confirm that even a naive selection strategy can provide improved performance across problem classes.
Output: COCOpf: An Algorithm Portfolio Framework
This paper addresses the problem of detecting coherent motions in crowd scenes and presents its two applications in crowd scene understanding: semantic region detection and recurrent activity mining. It processes input motion fields (e.g., optical flow fields) and produces a coherent motion filed, named as thermal energy field. The thermal energy field is able to capture both motion correlation among particles and the motion trends of individual particles which are helpful to discover coherency among them. We further introduce a two-step clustering process to construct stable semantic regions from the extracted time-varying coherent motions. These semantic regions can be used to recognize pre-defined activities in crowd scenes. Finally, we introduce a cluster-and-merge process which automatically discovers recurrent activities in crowd scenes by clustering and merging the extracted coherent motions. Experiments on various videos demonstrate the effectiveness of our approach.
Output: A Diffusion and Clustering-based Approach for Finding Coherent Motions and Understanding Crowd Scenes
We consider the problem of efficient exploration in finite horizon MDPs. We show that an optimistic modification to modelbased value iteration, can achieve a regret bound Õ( √ HSAT+HSA+H √ T )whereH is the time horizon, S the number of states, A the number of actions and T the time elapsed. This result improves over the best previous known bound Õ(HS √ AT ) achieved by the UCRL2 algorithm Jaksch et al. (2010a). The key significance of our new results is that when T ≥ HSA and SA ≥ H , it leads to a regret of Õ( √ HSAT ) that matches the established lower bounds of Ω( √ HSAT ) up to a logarithmic factor. Our analysis contain two key insights. We use careful application of concentration inequalities to the optimal value function as a whole, rather than to the transitions probabilities (to improve scaling in S), and we use "exploration bonuses" based on Bernstein’s inequality, together with using a recursive -Bellman-typeLaw of Total Variance (to improve scaling in H).
Output: Minimax Regret Bounds for Reinforcement Learning
In this work we describe and evaluate methods to learn musical embeddings. Each embedding is a vector that represents four contiguous beats of music and is derived from a symbolic representation. We consider autoencoding-based methods including denoising autoencoders, and context reconstruction, and evaluate the resulting embeddings on a forward prediction and a classification task.
Output: A Unit Selection Methodology for Music Generation Using Deep Neural Networks
Our objective is to efficiently design a robust projection matrix Φ for the Compressive Sensing (CS) systems when applied to the signals that are not exactly sparse. The optimal projection matrix is obtained by mainly minimizing the average coherence of the equivalent dictionary. In order to drop the requirement of the sparse representation error (SRE) for a set of training data as in [15] [16], we introduce a novel penalty function independent of a particular SRE matrix. Without requiring of training data, we can efficiently design the robust projection matrix and apply it for most of CS systems, like a CS system for image processing with a conventional wavelet dictionary in which the SRE matrix is generally not available. Simulation results demonstrate the efficiency and effectiveness of the proposed approach compared with the state-of-the-art methods. In addition, we experimentally demonstrate with natural images that under similar compression rate, a CS system with a learned dictionary in high dimensions outperforms the one in low dimensions in terms of reconstruction accuracy. This together with the fact that our proposed method can efficiently work in high dimension suggests that a CS system can be potentially implemented beyond the small patches in sparsity-based image processing.
Output: An Efficient Method for Robust Projection Matrix Design
Answer sentence selection is the task of identifying sentences that contain the answer to a given question. This is an important problem in its own right as well as in the larger context of open domain question answering. We propose a novel approach to solving this task via means of distributed representations, and learn to match questions with answers by considering their semantic encoding. This contrasts prior work on this task, which typically relies on classifiers with large numbers of hand-crafted syntactic and semantic features and various external resources. Our approach does not require any feature engineering nor does it involve specialist linguistic data, making this model easily applicable to a wide range of domains and languages. Experimental results on a standard benchmark dataset from TREC demonstrate that—despite its simplicity—our model matches state of the art performance on the answer sentence selection task.
Output: Deep Learning for Answer Sentence Selection
Network models play an important role in studying complex systems in many scientific disciplines. Graph signal processing is receiving growing interest as to design novel tools to combine the analysis of topology and signals. The graph Fourier transform, defined as the eigendecomposition of the graph Laplacian, allows extending conventional signal-processing operations to graphs. One main feature is to let emerge global organization from local interactions; i.e., the Fiedler vector has the smallest non-zero eigenvalue and is key for Laplacian embedding and graph clustering. Here, we introduce the design of Slepian graph signals, by maximizing energy concentration in a predefined subgraph for a given spectral bandlimit. We also establish a link with classical Laplacian embedding and graph clustering, for which the graph Slepian design can serve as a generalization.
Output: When Slepian Meets Fiedler: Putting a Focus on the Graph Spectrum