niryuu/sni-each-converted huggingface dataset:
Relation classification is an important research arena in the field of natural language processing (NLP). In this paper, we present SDP-LSTM, a novel neural network to classify the relation of two entities in a sentence. Our neural architecture leverages the shortest dependency path (SDP) between two entities; multichannel recurrent neural networks, with long short term memory (LSTM) units, pick up heterogeneous information along the SDP. Our proposed model has several distinct features: (1) The shortest dependency paths retain most relevant information (to relation classification), while eliminating irrelevant words in the sentence. (2) The multichannel LSTM networks allow effective information integration from heterogeneous sources over the dependency paths. (3) A customized dropout strategy regularizes the neural network to alleviate overfitting. We test our model on the SemEval 2010 relation classification task, and achieve an F1-score of 83.7%, higher than competing methods in the literature.
Output: Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths
We present a work-in-progress snapshot of learning with a 15 billion parameter deep learning network on HPC architectures applied to the largest publicly available natural image and video dataset released to-date. Recent advancements in unsupervised deep neural networks suggest that scaling up such networks in both model and training dataset size can yield significant improvements in the learning of concepts at the highest layers. We train our three-layer deep neural network on the Yahoo! Flickr Creative Commons 100M dataset. The dataset comprises approximately 99.2 million images and 800, 000 user-created videos from Yahoo’s Flickr image and video sharing platform. Training of our network takes eight days on 98 GPU nodes at the High Performance Computing Center at Lawrence Livermore National Laboratory. Encouraging preliminary results and future research directions are presented and discussed.
Output: LARGE-SCALE DEEP LEARNING ON THE YFCC100M DATASET
Texts are complex structures emerging from an intricate system consisting of syntactical constraints and semantical relationships. While the complete modeling of such structures is impractical owing to the high level of complexity inherent to linguistic constructions, under a limited domain, certain tasks can still be performed. Recently, statistical techniques aiming at analysis of texts, referred to as text analytics, have departed from the use of simple word count statistics towards a new paradigm. Text mining now hinges on a more sophisticate set of methods, including the representation of texts as complex networks. In this perspective, networks represent a set of textual elements, typically words; and links are established via adjacency relationships. While current word-adjacency (co-occurrence) methods successfully grasp syntactical and stylistic features of written texts, they are unable to represent important aspects of textual data, such as its topical structure. As a consequence, the mesoscopic structure of texts is often overlooked by current methodologies. In order to grasp mesoscopic characteristics of semantical content in written texts, we devised a network approach which is able to analyze documents in a multi-scale, mesoscopic fashion. In the proposed model, a limited amount of adjacent paragraphs are represented as nodes, which are connected whenever they share a minimum semantical content. To illustrate the capabilities of our model, we present, as a use case, a qualitative analysis of “Alice’s Adventures in Wonderland”, a novel by Lewis Carroll. We show that the mesoscopic structure of documents modeled as networks reveals many semantic traits of texts, a feature that could be explored in a myriad of semantic-based applications.
Output: Mesoscopic representation of texts as complex networks
We give an interpretation of the Maxi­ mum Entropy (MaxEnt) Principle in game­ theoretic terms. Based on this interpretation, we make a formal distinction between differ­ ent ways of applying Maximum Entropy dis­ tributions. MaxEnt has frequently been crit­ icized on the grounds that it leads to highly representation dependent results. Our dis­ tinction allows us to avoid this problem in many cases.
Output: Maximum Entropy and the Glasses You are Looking Through
Language learning is thought to be a highly complex process. One of the hurdles in learning a language is to learn the rules of syntax of the language. Rules of syntax are often ordered in that before one rule can applied one must apply another. It has been thought that to learn the order of n rules one must go through all n! permutations. Thus to learn the order of 27 rules would require 27! steps or 1.08889x10 steps. This number is much greater than the number of seconds since the beginning of the universe! In an insightful analysis the linguist Block ([Block 86], pp. 62-63, p.238) showed that with the assumption of transitivity this vast number of learning steps reduces to a mere 377 steps. We present a mathematical analysis of the complexity of Block’s algorithm. The algorithm has a complexity of order n given n rules. In addition, we improve Block’s results exponentially, by introducing an algorithm that has complexity of order less than n log n.
Output: On the complexity of learning a language: An improvement of Block’s algorithm
Partial monitoring games are repeated games where the learner receives feedback that might be different from adversary’s move or even the reward gained by the learner. Recently, a general model of combinatorial partial monitoring (CPM) games was proposed [1], where the learner’s action space can be exponentially large and adversary samples its moves from a bounded, continuous space, according to a fixed distribution. The paper gave a confidence bound based algorithm (GCB) that achieves O(T 2/3 logT ) distribution independent and O(log T ) distribution dependent regret bounds. The implementation of their algorithm depends on two separate offline oracles and the distribution dependent regret additionally requires existence of a unique optimal action for the learner. Adopting their CPM model, our first contribution is a Phased Exploration with Greedy Exploitation (PEGE) algorithmic framework for the problem. Different algorithms within the framework achieve O(T 2/3 √ logT ) distribution independent and O(log T ) distribution dependent regret respectively. Crucially, our framework needs only the simpler “argmax” oracle from GCB and the distribution dependent regret does not require existence of a unique optimal action. Our second contribution is another algorithm, PEGE2, which combines gap estimation with a PEGE algorithm, to achieve an O(log T ) regret bound, matching the GCB guarantee but removing the dependence on size of the learner’s action space. However, like GCB, PEGE2 requires access to both offline oracles and the existence of a unique optimal action. Finally, we discuss how our algorithm can be efficiently applied to a CPM problem of practical interest: namely, online ranking with feedback at the top.
Output: Phased Exploration with Greedy Exploitation in Stochastic Combinatorial Partial Monitoring Games
Word embeddings are used with success for a variety of tasks involving lexical semantic similarities between individual words. Using unsupervised methods and just cosine similarity, encouraging results were obtained for analogical similarities. In this paper, we explore the potential of pre-trained word embeddings to identify generic types of semantic relations in an unsupervised experiment. We propose a new relational similarity measure based on the combination of word2vec’s CBOW input and output vectors which outperforms concurrent vector representations, when used for unsupervised clustering on SemEval 2010 Relation Classification data.
Output: Exploring Vector Spaces for Semantic Relations
Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with such “deep" transition functions remain difficult to train, even when using Long Short-Term Memory networks. We introduce a novel theoretical analysis of recurrent networks based on Geršgorin’s circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks (RHN), which are long not only in time but also in space, generalizing LSTMs to larger step-to-step depths. Experiments indicate that the proposed architecture results in complex but efficient models, beating previous models for character prediction on the Hutter Prize dataset with less than half of the parameters.
Output: Recurrent Highway Networks
Many popular knowledge graphs such as Freebase, YAGO or DBPedia maintain a list of non-discrete aributes for each entity. Intuitively, these aributes such as height, price or population count are able to richly characterize entities in knowledge graphs. is additional source of information may help to alleviate the inherent sparsity and incompleteness problem that are prevalent in knowledge graphs. Unfortunately, many state-of-the-art relational learning models ignore this information due to the challenging nature of dealing with non-discrete data types in the inherently binary-natured knowledge graphs. In this paper, we propose a novel multi-task neural network approach for both encoding and prediction of non-discrete aribute information in a relational setting. Specically, we train a neural network for triplet prediction along with a separate network for aribute value regression. Via multi-task learning, we are able to learn representations of entities, relations and aributes that encode information about both tasks. Moreover, such aributes are not only central to many predictive tasks as an information source but also as a prediction target. erefore, models that are able to encode, incorporate and predict such information in a relational learning context are highly aractive as well. We show that our approach outperforms many state-ofthe-art methods for the tasks of relational triplet classication and aribute value prediction.
Output: Multi-Task Neural Network for Non-discrete Aribute Prediction in Knowledge Graphs
Critical Infrastructures like power and communication networks are highly interdependent on each other for their full functionality. Many significant research have been pursued to model the interdependency and failure analysis of these interdependent networks. However most of these models fail to capture the complex interdependencies that might actually exist between the infrastructures. The Implicative Interdependency Model that utilizes Boolean Logic to capture complex interdependencies was recently proposed which overcome the limitations of the existing models. A number of problems were studies based on this model. In this paper we study the Robustness problem in Interdependent Power and Communication Network. The robustness is defined with respect to two parameters K ∈ I ∪ {0} and ρ ∈ (0, 1]. We utilized the Implicative Interdependency Model model to capture the complex interdependency between the two networks. The model classifies the interdependency relations into four cases. Computational complexity of the problem is analyzed for each of these cases. A polynomial time algorithm is designed for the first case that outputs the optimal solution. All the other cases are proved to be NP-complete. An in-approximability bound is provided for the third case. For the general case we formulate an Integer Linear Program to get the optimal solution and a polynomial time heuristic. The applicability of the heuristic is evaluated using power and communication network data of Maricopa County, Arizona. The experimental results showed that the heuristic almost always produced near optimal value of parameter K for ρ < 0.42.
Output: On Robustness in Multilayer Interdependent Network
This paper describes several results of Wimmics, a research lab which names stands for: web-instrumented man-machine interactions, communities, and semantics. The approaches introduced here rely on graph-oriented knowledge representation, reasoning and operationalization to model and support actors, actions and interactions in web-based epistemic communities. The research results are applied to support and foster interactions in online communities and manage their resources.
Output: Challenges in bridging Social Semantics and Formal Semanticson the Web
External neural memory structures have recently become a popular tool for algorithmic deep learning (Graves et al., 2014; Weston et al., 2014). These models generally utilize differentiable versions of traditional discrete memory-access structures (random access, stacks, tapes) to provide the storage necessary for computational tasks. In this work, we argue that these neural memory systems lack specific structure important for relative indexing, and propose an alternative model, Lieaccess memory, that is explicitly designed for the neural setting. In this paradigm, memory is accessed using a continuous head in a key-space manifold. The head is moved via Lie group actions, such as shifts or rotations, generated by a controller, and memory access is performed by linear smoothing in key space. We argue that Lie groups provide a natural generalization of discrete memory structures, such as Turing machines, as they provide inverse and identity operators while maintaining differentiability. To experiment with this approach, we implement a simplified Lie-access neural Turing machine (LANTM) with different Lie groups. We find that this approach is able to perform well on a range of algorithmic tasks.
Output: LIE-ACCESS NEURAL TURING MACHINES
In this paper we propose a simple yet powerful method for learning representations in supervised learning scenarios where an input datapoint is described by a set of feature vectors and its associated output may be given by soft labels indicating, for example, class probabilities. We represent an input datapoint as a K-dimensional vector, where each component is a mixture of probabilities over its corresponding set of feature vectors. Each probability indicates how likely a feature vector is to belong to one-out-of-K unknown prototype patterns. We propose a probabilistic model that parameterizes these prototype patterns in terms of hidden variables and therefore it can be trained with conventional approaches based on likelihood maximization. More importantly, both the model parameters and the prototype patterns can be learned from data in a discriminative way. We show that our model can be seen as a probabilistic generalization of learning vector quantization (LVQ). We apply our method to the problems of shape classification, hyperspectral imaging classification and people’s work class categorization, showing the superior performance of our method compared to the standard prototype-based classification approach and other competitive benchmarks.
Output: Discriminative Probabilistic Prototype Learning
Ever growing number of image documents available on the Internet continuously motivates research in better annotation models and more efficient retrieval methods. Formal knowledge representation of objects and events in pictures, their interaction as well as context complexity becomes no longer an option for a quality image repository, but a necessity. We present an ontologybased online image annotation tool WNtags and demonstrate its usefulness in several typical multimedia retrieval tasks using International Affective Picture System emotionally annotated image database. WNtags is built around WordNet lexical ontology but considers Suggested Upper Merged Ontology as the preferred labeling formalism. WNtags uses sets of weighted WordNet synsets as high-level image semantic descriptors and query matching is performed with word stemming and node distance metrics. We also elaborate our near future plans to expand image content description with induced affect as in stimuli for research of human emotion and attention.
Output: WNtags: A Web-Based Tool For Image Labeling And Retrieval With Lexical Ontologies
Whereas acausal Bayesian networks rep­ resent probabilistic independence, causal Bayesian networks represent causal relation­ ships. In this paper, we examine Bayesian methods for learning both types of networks. Bayesian methods for learning acausal net­ works are fairly well developed. These meth­ ods often employ assumptions to facilitate the construction of priors, including the as­ sumptions of parameter independence, pa­ rameter modularity, and likelihood equiva­ lence. We show that although these assump­ tions also can be appropriate for learning causal networks, we need additional assump­ tions in order to learn causal networks. We introduce two sufficient assumptions, called mechanism independence and component in­ dependence. We show that these new as­ sumptions, when combined with parame­ ter independence, parameter modularity, and likelihood equivalence, allow us to apply methods for learning acausal networks to learn causal networks.
Output: A Bayesian Approach to Learning Causal Networks
The folksonomy is the result of free personal information or assignment of tags to an object (determined by the URI) in order to find them. The practice of tagging is done in a collective environment. Folksonomies are self constructed, based on co-occurrence of definitions, rather than a hierarchical structure of the data. The downside of this was that a few sites and applications are able to successfully exploit the sharing of bookmarks. The need for tools that are able to resolve the ambiguity of the definitions is becoming urgent as the need of simple instruments for their visualization, editing and exploitation in web applications still hinders their diffusion and wide adoption. An intelligent interactive interface design for folksonomies should consider the contextual design and inquiry based on a concurrent interaction for a perceptual user interfaces. To represent folksonomies a new concept structure called "Folksodriven" is used in this paper. While it is presented the Folksodriven Structure Network (FSN) to resolve the ambiguity of definitions of folksonomy tags suggestions for the user. On this base a Human-Computer Interactive (HCI) systems is developed for the visualization, navigation, updating and maintenance of folksonomies Knowledge Bases – the FSN – through the web. System functionalities as well as its internal architecture will be introduced.
Output: Intelligent Interface Architectures for Folksonomy Driven Structure Network
We introduce the value iteration network: a fully differentiable neural network with a ‘planning module’ embedded within. Value iteration networks are suitable for making predictions about outcomes that involve planning-based reasoning, such as predicting a desired trajectory from an observation of a map. Key to our approach is a novel differentiable approximation of the valueiteration algorithm, which can be represented as a convolutional neural network, and trained endto-end using standard backpropagation. We evaluate our value iteration networks on the task of predicting optimal obstacle-avoiding trajectories from an image of a landscape, both on synthetic data, and on challenging raw images of the Mars terrain.
Output: Value Iteration Networks
The popular Alternating Least Squares (ALS) algorithm for tensor decomposition is extremely efficient, but often converges to poor local optima, particularly when the weights of the factors are non-uniform. We propose a modification of the ALS approach that is as efficient as standard ALS, but provably recovers the true factors with random initialization under standard incoherence assumptions on the factors of the tensor. We demonstrate the significant practical superiority of our approach over traditional ALS (with both random initialization and SVDbased initialization) for a variety of tasks on synthetic data—including tensor factorization on exact, noisy and over-complete tensors, as well as tensor completion—and for computing word embeddings from a third-order word tri-occurrence tensor.
Output: Orthogonalized ALS: A Theoretically Principled Tensor Decomposition Algorithm for Practical Use
We present a concrete design for Solomonoff’s incremental machine learning system suitable for desktop computers. We use R5RS Scheme and its standard library with a few omissions as the reference machine. We introduce a Levin Search variant based on a stochastic Context Free Grammar together with new update algorithms that use the same grammar as a guiding probability distribution for incremental machine learning. The updates include adjusting production probabilities, re-using previous solutions, learning programming idioms and discovery of frequent subprograms. The issues of extending the a priori probability distribution and bootstrapping are discussed. We have implemented a good portion of the proposed algorithms. Experiments with toy problems show that the update algorithms work as expected.
Output: Gigamachine: incremental machine learning on desktop computers
Kernel canonical correlation analysis (KCCA) is a nonlinear multi-view representation learning technique with broad applicability in statistics and machine learning. Although there is a closed-form solution for the KCCA objective, it involves solving an N × N eigenvalue system where N is the training set size, making its computational requirements in both memory and time prohibitive for large-scale problems. Various approximation techniques have been developed for KCCA. A commonly used approach is to first transform the original inputs to an M -dimensional random feature space so that inner products in the feature space approximate kernel evaluations, and then apply linear CCA to the transformed inputs. In many applications, however, the dimensionality M of the random feature space may need to be very large in order to obtain a sufficiently good approximation; it then becomes challenging to perform the linear CCA step on the resulting very high-dimensional data matrices. We show how to use a stochastic optimization algorithm, recently proposed for linear CCA and its neuralnetwork extension, to further alleviate the computation requirements of approximate KCCA. This approach allows us to run approximate KCCA on a speech dataset with 1.4 million training samples and a random feature space of dimensionality M = 100000 on a typical workstation.
Output: LARGE-SCALE APPROXIMATE KERNEL CANONICAL CORRELATION ANALYSIS
The 2002 Trading Agent Competition (TAC) presented a challenging market game in the domain of travel shopping. One of the pivotal issues in this domain is uncertainty about hotel prices, which have a significant influence on the relative cost of alternative trip schedules. Thus, virtually all participants employ some method for predicting hotel prices. We survey approaches employed in the tournament, finding that agents apply an interesting diversity of techniques, taking into account differing sources of evidence bearing on prices. Based on data provided by entrants on their agents’ actual predictions in the TAC-02 finals and semifinals, we analyze the relative efficacy of these approaches. The results show that taking into account game-specific information about flight prices is a major distinguishing factor. Machine learning methods effectively induce the relationship between flight and hotel prices from game data, and a purely analytical approach based on competitive equilibrium analysis achieves equal accuracy with no historical data. Employing a new measure of prediction quality, we relate absolute accuracy to bottom-line performance in the game.
Output: Price Prediction in a Trading Agent Competition
A Support Vector Machine (SVM) has become a very popular machine learning method for text classification. One reason for this relates to the range of existing kernels which allow for classifying data that is not linearly separable. The linear, polynomial and RBF (Gaussian Radial Basis Function) kernel are commonly used and serve as a basis of comparison in our study. We show how to derive the primal form of the quadratic Power Kernel (PK) – also called the Negative Euclidean Distance Kernel (NDK) – by means of complex numbers. We exemplify the NDK in the framework of text categorization using the Dewey Document Classification (DDC) as the target scheme. Our evaluation shows that the power kernel produces F-scores that are comparable to the reference kernels, but is – except for the linear kernel – faster to compute. Finally, we show how to extend the NDK-approach by including the Mahalanobis distance. Keywords—SVM, kernel function, text categorization
Output: Complex Decomposition of the Negative Distance Kernel
Deep neural networks (DNNs) are powerful types of artificial neural networks (ANNs) that use several hidden layers. They have recently gained considerable attention in the speech transcription and image recognition community (Krizhevsky et al., 2012) for their superior predictive properties including robustness to overfitting. However their application to algorithmic trading has not been previously researched, partly because of their computational complexity. This paper describes the application of DNNs to predicting financial market movement directions. In particular we describe the configuration and training approach and then demonstrate their application to backtesting a simple trading strategy over 43 different Commodity and FX future mid-prices at 5-minute intervals. All results in this paper are generated using a C++ implementation on the Intel Xeon Phi co-processor which is 11.4x faster than the serial version and a Python strategy backtesting environment both of which are available as open source code written by the authors.
Output: Classification-based Financial Markets Prediction using Deep Neural Networks
We consider relative error low rank approximation of tensors with respect to the Frobenius norm. Namely, given an order-q tensor A ∈ R ∏q i=1 ni , output a rank-k tensor B for which ‖A − B‖F ≤ (1 + ) OPT, where OPT = infrank-k A′ ‖A − A‖F . Despite much success on obtaining relative error low rank approximations for matrices, no such results were known for tensors. One structural issue is that there may be no rank-k tensor Ak achieving the above infinum. Another, computational issue, is that an efficient relative error low rank approximation algorithm for tensors would allow one to compute the rank of a tensor, which is NP-hard. We bypass these two issues via (1) bicriteria and (2) parameterized complexity solutions: 1. We give an algorithm which outputs a rank k′ = O((k/ )q−1) tensor B for which ‖A − B‖F ≤ (1+ ) OPT in nnz(A)+n ·poly(k/ ) time in the real RAM model, whenever either Ak exists or OPT > 0. Here nnz(A) denotes the number of non-zero entries in A. If both Ak does not exist and OPT = 0, then B instead satisfies ‖A − B‖F < γ, where γ is any positive, arbitrarily small function of n. 2. We give an algorithm for any δ > 0 which outputs a rank k tensor B for which ‖A−B‖F ≤ (1+ ) OPT and runs in (nnz(A)+n poly(k/ )+exp(k/ )) ·nδ time in the unit cost RAM model, whenever OPT > 2−O(n ) and there is a rank-k tensor B = ∑k i=1 ui ⊗ vi ⊗ wi for which ‖A − B‖F ≤ (1 + /2) OPT and ‖ui‖2, ‖vi‖2, ‖wi‖2 ≤ 2 δ). If OPT ≤ 2−Ω(nδ), then B instead satisfies ‖A−B‖F ≤ 2−Ω(n δ). Our first result is polynomial time, and in fact input sparsity time, in n, k, and 1/ , for any k ≥ 1 and any 0 < < 1, while our second result is fixed parameter tractable in k and 1/ . For outputting a rank-k tensor, or even a bicriteria solution with rank-Ck for a certain constant C > 1, we show a 2 1−o(1)) time lower bound under the Exponential Time Hypothesis. Our results are based on an “iterative existential argument”, and give the first relative error low rank approximations for tensors for a large number of error measures for which nothing was known. In particular, we give the first relative error approximation algorithms on tensors for: column row and tube subset selection, entrywise `p-low rank approximation for 1 ≤ p < 2, low rank approximation with respect to sum of Euclidean norms of faces or tubes, weighted low rank approximation, and low rank approximation in distributed and streaming models. We also obtain several new results for matrices, such as nnz(A)-time CUR decompositions, improving the previous nnz(A) log n-time CUR decompositions, which may be of independent interest. ∗Work done while visiting IBM Almaden, and supported in part by UTCS TAship (CS361 Spring 17 Introduction to Computer Security). †Supported in part by Simons Foundation, and NSF CCF-1617955. ar X iv :1 70 4. 08 24 6v 1 [ cs .D S] 2 6 A pr 2 01 7
Output: Relative Error Tensor Low Rank Approximation
In this paper we propose a unified framework for structured prediction with latent variables which includes hidden conditional random fields and latent structured support vector machines as special cases. We describe a local entropy approximation for this general formulation using duality, and derive an efficient message passing algorithm that is guaranteed to converge. We demonstrate its effectiveness in the tasks of image segmentation as well as 3D indoor scene understanding from single images, showing that our approach is superior to latent structured support vector machines and hidden conditional random fields.
Output: Efficient Structured Prediction with Latent Variables for General Graphical Models
This paper presents the input convex neural network architecture. These are scalar-valued (potentially deep) neural networks with constraints on the network parameters such that the output of the network is a convex function of (some of) the inputs. The networks allow for efficient inference via optimization over some inputs to the network given others, and can be applied to settings including structured prediction, data imputation, reinforcement learning, and others. In this paper we lay the basic groundwork for these models, proposing methods for inference, optimization and learning, and analyze their representational power. We show that many existing neural network architectures can be made input-convex with only minor modification, and develop specialized optimization algorithms tailored to this setting. Finally, we highlight the performance of the methods on multi-label prediction, image completion, and reinforcement learning problems, where we show improvement over the existing state of the art in many cases.
Output: Input Convex Neural Networks
Abstract. We study the learning algorithm corresponding to the incremental gradient descent defined by the empirical risk over an infinite dimensional hypotheses space. We consider a statistical learning setting and show that, provided with a universal step-size and a suitable early stopping rule, the learning algorithm thus obtained is universally consistent and derive finite sample bounds. Our results provide a theoretical foundation for considering early stopping in online learning algorithms and shed light on the effect of allowing for multiple passes over the data.
Output: REGULARIZATION BY EARLY STOPPING FOR ONLINE LEARNING ALGORITHMS
The role of kernels is central to machine learning. Motivated by the importance of power law distributions in modeling, simulation and learning, in this paper, we propose a powerlaw generalization of the Gaussian kernel. This generalization is based on q-Gaussian distribution, which is a power-law distribution studied in context of nonextensive statistical mechanics. We prove that the proposed kernel is positive definite, and provide some insights regarding the corresponding Reproducing Kernel Hilbert Space (RKHS). We also study practical significance of qGaussian kernels in classification, regression and clustering, and present some simulation results.
Output: On q-Gaussian kernel and its Reproducing Kernel Hilbert Space
Existing deep embedding methods in vision tasks are capable of learning a compact Euclidean space from images, where Euclidean distances correspond to a similarity metric. To make learning more effective and efficient, hard sample mining is usually employed, with samples identified through computing the Euclidean feature distance. However, the global Euclidean distance cannot faithfully characterize the true feature similarity in a complex visual feature space, where the intraclass distance in a high-density region may be larger than the interclass distance in low-density regions. In this paper, we introduce a Position-Dependent Deep Metric (PDDM) unit, which is capable of learning a similarity metric adaptive to local feature structure. The metric can be used to select genuinely hard samples in a local neighborhood to guide the deep embedding learning in an online and robust manner. The new layer is appealing in that it is pluggable to any convolutional networks and is trained end-to-end. Our local similarity-aware feature embedding not only demonstrates faster convergence and boosted performance on two complex image retrieval datasets, its large margin nature also leads to superior generalization results under the large and open set scenarios of transfer learning and zero-shot learning on ImageNet 2010 and ImageNet-10K datasets.
Output: Local Similarity-Aware Deep Feature Embedding
<lb>We consider the problem of subspace clustering: given points that lie on or near the union of<lb>many low-dimensional linear subspaces, recover the subspaces. To this end, one first identifies<lb>sets of points close to the same subspace and uses the sets to estimate the subspaces. As the<lb>geometric structure of the clusters (linear subspaces) forbids proper performance of general<lb>distance based approaches such as K-means, many model-specific methods have been proposed.<lb>In this paper, we provide new simple and efficient algorithms for this problem. Our statistical<lb>analysis shows that the algorithms are guaranteed exact (perfect) clustering performance under<lb>certain conditions on the number of points and the affinity between subspaces. These conditions<lb>are weaker than those considered in the standard statistical literature. Experimental results on<lb>synthetic data generated from the standard unions of subspaces model demonstrate our theory.<lb>We also show that our algorithm performs competitively against state-of-the-art algorithms on<lb>real-world applications such as motion segmentation and face clustering, with much simpler<lb>implementation and lower computational cost.
Output: Greedy Subspace Clustering
In this paper, we propose to employ the convolutional neural network (CNN) for the image question answering (QA). Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: one image CNN to encode the image content, one sentence CNN to compose the words of the question, and one multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on the DAQUAR and COCO-QA datasets, which are two benchmark datasets for the image QA, with the performances significantly outperforming the state-of-the-art.
Output: Learning to Answer Questions From Image Using Convolutional Neural Network
The amount of data available in the world is growing faster and bigger than our ability to deal with it. However, if we take advantage of the internal structure, data may become much smaller for machine learning purposes. In this paper we focus on one of the most fundamental machine learning tasks, empirical risk minimization (ERM), and provide faster algorithms with the help from the clustering structure of the data. We introduce a simple notion of raw clustering that can be efficiently obtained with just one pass of the data, and propose two algorithms. Our variance-reduction based algorithm ClusterSVRG introduces a new gradient estimator using the clustering information, and our accelerated algorithm ClusterACDM is built on a novel Haar transformation applied to the dual space of each cluster. Our algorithms outperform their classical counterparts both in theory and practice.
Output: Exploiting the Structure: Stochastic Gradient Methods Using Raw Clusters
We consider the problem of off-policy evaluation—estimating the value of a target policy using data collected by another policy—under the contextual bandit model. We establish a minimax lower bound on the mean squared error (MSE), and show that it is matched up to constant factors by the inverse propensity scoring (IPS) estimator. Since in the multi-armed bandit problem the IPS is suboptimal [Li et al., 2015], our result highlights the difficulty of the contextual setting with non-degenerate context distributions. We further consider improvements on this minimax MSE bound, given access to a reward model. We show that the existing doubly robust approach, which utilizes such a reward model, may continue to suffer from high variance even when the reward model is perfect. We propose a new estimator called SWITCH which more effectively uses the reward model and achieves a superior bias-variance tradeoff compared with prior work. We prove an upper bound on its MSE and demonstrate its benefits empirically on a diverse collection of datasets, often seeing orders of magnitude improvements over a number of baselines.
Output: Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
Electronic health records capture patient information using structured controlled vocabularies and unstructured narrative text. While structured data typically encodes lab values, encounters and medication lists, unstructured data captures the physician’s interpretation of the patient’s condition, prognosis, and response to therapeutic intervention. In this paper, we demonstrate that information extraction from unstructured clinical narratives is essential to most clinical applications. We perform an empirical study to validate the argument and show that structured data alone is insufficient in resolving eligibility criteria for recruiting patients onto clinical trials for chronic lymphocytic leukemia (CLL) and prostate cancer. Unstructured data is essential to solving 59% of the CLL trial criteria and 77% of the prostate cancer trial criteria. More specifically, for resolving eligibility criteria with temporal constraints, we show the need for temporal reasoning and information integration with medical events within and across unstructured clinical narratives and structured data.
Output: How essential are unstructured clinical narratives and information fusion to clinical trial recruitment?
We revisit the leaderboard problem introduced by Blum and Hardt (2015) in an effort to reduce overfitting in machine learning benchmarks. We show that a randomized version of their Ladder algorithm achieves leaderboard error O(1/n0.4) compared with the previous best rate of O(1/n1/3). Short of proving that our algorithm is optimal, we point out a major obstacle toward further progress. Specifically, any improvement to our upper bound would lead to asymptotic improvements in the general adaptive estimation setting as have remained elusive in recent years. This connection also directly leads to lower bounds for specific classes of algorithms. In particular, we exhibit a new attack on the leaderboard algorithm that both theoretically and empirically distinguishes between our algorithm and previous leaderboard algorithms.
Output: Climbing a shaky ladder: Better adaptive risk estimation
Deep Neural Networks (DNNs) are presently the state-of-the-art for image classification tasks. However, recent works have shown that these systems can be easily fooled to misidentify images by modifying the image in particular ways, often rendering them practically useless. Moreover, defense mechanisms proposed in the literature so far are mostly attack-specific and prove to be ineffective against new attacks. Indeed, recent work on universal perturbations can generate a single modification for all test images that is able to make existing networks misclassify 90% of the time. Presently, to our knowledge, no defense mechanisms are effective in preventing this. As such, the design of a general defense strategy against a wide range of attacks for Neural Networks becomes a challenging problem. In this paper, we derive inspiration from recent advances in the field of cybersecurity and multi-agent systems, and propose to use the concept of Moving Target Defense (MTD) for increasing the robustness of well-known deep networks trained on the ImageNet dataset towards such adversarial attacks. In using this technique, we formalize and exploit the notion of differential immunity of different networks to specific attacks. To classify a single test image, we pick one of the trained networks each time and then use its classification output. To ensure maximum robustness, we generate an effective strategy by formulating this interaction as a Repeated Bayesian Stackelberg Game (BSG) with a Defender (who hosts the classification networks) and Users (both Legitimate users and Attackers). As a network switching strategy, we compute a Strong Stackelberg Equilibrium that optimizes the accuracy of prediction while at the same time reduces the misclassification rate on adversarial modification of test images. We show that while our approach produces an accuracy of 92.79% for the legitimate users, attackers can only misclassify images 58% (instead of 93.7%) of the time even when they select the best attack available to them. This is at least twice as good, to sometimes even an order of magnitude better, compared the accuracy rates of the worst affected networks.
Output: Securing Deep Neural Nets against Adversarial Attacks with Moving Target Defense
One of the better studied properties for operators in judgment aggregation is independence, which essentially dictates that the collective judgment on one issue should not depend on the individual judgments given on some other issue(s) in the same agenda. Independence, although considered a desirable property, is too strong, because together with mild additional conditions it implies dictatorship. We propose here a weakening of independence, named agenda separability: a judgment aggregation rule satisfies it if, whenever the agenda is composed of several independent sub-agendas, the resulting collective judgment sets can be computed separately for each sub-agenda and then put together. We show that this property is discriminant, in the sense that among judgment aggregation rules so far studied in the literature, some satisfy it and some do not. We briefly discuss the implications of agenda separability on the computation of judgment aggregation rules.
Output: Agenda Separability in Judgment Aggregation
We prove in this paper that the expected value of the objective function of the k-means++ algorithm for samples converges to population expected value. As k-means++, for samples, provides with constant factor approximation for k-means objectives, such an approximation can be achieved for the population with increase of the sample size. This result is of potential practical relevance when one is considering using subsampling when clustering large data sets (large data bases).
Output: On the Consistency of k-means++ algorithm
In this paper, we propose a novel neural network model called RNN Encoder–Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.
Output: Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Based on NFL game data we try to predict the outcome of a play in multiple different ways including Decision and Classification Trees, Nearest Neighbors, Naive Bayes, Linear Discriminant Analysis, Support Vector Machines and Regression, and Artificial Neural Networks. An application of this is the following: by plugging in various play options one could determine the best play for a given situation in real time. While the outcome of a play can be described in many ways we had the most promising results with a newly defined measure that we call progress. We see this work as a first step to include predictive analysis into NFL playcalling.
Output: NFL Play Prediction
Deep neural networks are capable of modelling highly nonlinear functions by capturing different levels of abstraction of data hierarchically. While training deep networks, first the system is initialized near a good optimum by greedy layer-wise unsupervised pre-training. However, with burgeoning data and increasing dimensions of the architecture, the time complexity of this approach becomes enormous. Also, greedy pre-training of the layers often turns detrimental by over-training a layer causing it to lose harmony with the rest of the network. In this paper a synchronized parallel algorithm for pre-training deep networks on multi-core machines has been proposed. Different layers are trained by parallel threads running on different cores with regular synchronization. Thus the pre-training process becomes faster and chances of overtraining are reduced. This is experimentally validated using a stacked autoencoder for dimensionality reduction of MNIST handwritten digit database. The proposed algorithm achieved 26% speed-up compared to greedy layer-wise pre-training for achieving the same reconstruction accuracy substantiating its potential as an alternative.
Output: Faster learning of deep stacked autoencoders on multi-core systems using synchronized layer-wise pre-training
In this paper we discuss a novel framework for multiclass learning, defined by a suitable coding/decoding strategy, namely the simplex coding, that allows to generalize to multiple classes a relaxation approach commonly used in binary classification. In this framework, a relaxation error analysis can be developed avoiding constraints on the considered hypotheses class. Moreover, we show that in this setting it is possible to derive the first provably consistent regularized method with training/tuning complexity which is independent to the number of classes. Tools from convex analysis are introduced that can be used beyond the scope of this paper.
Output: Multiclass Learning with Simplex Coding
A pseudo independent (PI) model is a proba­ bilistic domain model (PDM) where proper subsets of a set of collectively dependent variables display marginal independence. PI models cannot be learned correctly by many algorithms that rely on a single link search. Earlier work on learning PI models has sug­ gested a straightforward multi-link search al­ gorithm. However, when a domain contains recursively embedded PI submodels, it may escape the detection of such an algorithm. In this paper, we propose an improved al­ gorithm that ensures the learning of all em­ bedded PI submodels whose sizes are upper bounded by a predetermined parameter. We show that this improved learning capability only increases the complexity slightly beyond that of the previous algorithm. The perfor­ mance of the new algorithm is demonstrated through experiment.
Output: Learning Belief Networks in Domains with Recursively Embedded Pseudo Independent Submodels