Query: Commit2Vec: Learning Distributed Representations of Code Changes

title: Commit2Vec: Learning Distributed Representations of Code Changes
abstract: Deep learning methods, which have found successful applications in fields like image classification and natural language processing, have recently been applied to source code analysis too, due to the enormous amount of freely available source code (e.g. from open source software repositories) In this work, we elaborate upon a state of the art approach to the representation of source code that uses information about its syntactic structure, and we adapt it to represent source changes (i.e. commits) We use this representation to classify security relevant commits. Because our method uses transfer learning (that is, we train a network on a "pretext task" for which abundant labeled data is available, and then we use such network for the target task of commit classification, for which fewer labeled instances are available) we studied the impact of pre training the network using two different pretext tasks versus a randomly initialized model. Our results indicate that representations that leverage the structural information obtained through code syntax outperform token based representations. Furthermore, the performance metrics obtained when pre training on a loosely related pretext task with a very large dataset >10^6$ samples) were surpassed when pretraining on a smaller dataset >10^4$ samples) but for a pretext task that is more closely related to the target task.

title: Commit2Vec: Learning Distributed Representations of Code Changes [PRE PRINT]
abstract: Deep learning methods, which have found successful applications in fields like image classification and natural language processing, have recently been applied to source code analysis too, due to the enormous amount of freely available source code (e.g. from open source software repositories) In this work, we elaborate upon a state of the art approach to the representation of source code that uses information about its syntactic structure, and we adapt it to represent source code changes (i.e. commits) We use this representation to classify security relevant commits. Because our method uses transfer learning (that is, we train a network on a "pretext task" for which abundant labeled data is available, and then we use such network for the target task of commit classification, for which fewer labeled instances are available) we studied the impact of pre training the network using two different pretext tasks versus a randomly initialized model. Our results indicate that representations that leverage the structural information obtained through code syntax outperform token based representations. Furthermore, the performance metrics obtained when pre training on a loosely related pretext task with a very large dataset 106 samples) were surpassed when pretraining on a smaller dataset 104 samples) but for a pretext task that is more closely related to the target task.

title: patch2vec: Distributed Representation of Code Changes
abstract: Deep learning methods, which have found successful applications in fields like image classification and natural language processing, have recently been applied to source code analysis too, due to the enormous amount of freely available source code (e.g. from open source software repositories) In this work, we elaborate upon a state of the art approach to the representation of source code that uses information about its syntactic structure, and we adapt it to represent source changes (i.e. commits) We use this representation to classify security relevant commits. Because our method uses transfer learning (that is, we train a network on a "pretext task" for which abundant labeled data is available, and then we use such network for the target task of commit classification, for which fewer labeled instances are available) we studied the impact of pre training the network using two different pretext tasks versus a randomly initialized model. Our results indicate that representations that leverage the structural information obtained through code syntax outperform token based representations. Furthermore, the performance metrics obtained when pre training on a loosely related pretext task with a very large dataset >10^6$ samples) were surpassed when pretraining on a smaller dataset >10^4$ samples) but for a pretext task that is more closely related to the target task.

title: A neurally plausible model learns successor representations in partially observable environments
abstract: Animals need to devise strategies to maximize returns while interacting with their environment based on incoming noisy sensory observations. Task relevant states, such as the agent's location within an environment or the presence of a predator, are often not directly observable but must be inferred using available sensory information. Successor representations (SR) have been proposed as a middle ground between model based and model free reinforcement learning strategies, allowing for fast value computation and rapid adaptation to changes in the reward function or goal locations. Indeed, recent studies suggest that features of neural responses are consistent with the SR framework. However, it is not clear how such representations might be learned and computed in partially observed, noisy environments. Here, we introduce a neurally plausible model using distributional successor features, which builds on the distributed distributional code for the representation and computation of uncertainty, and which allows for efficient value function computation in partially observed environments via the successor representation. We show that distributional successor features can support reinforcement learning in noisy environments in which direct learning of successful policies is infeasible.

title: The Unreasonable Effectiveness of Machine Learning in Neuroscience: Understanding High dimensional Neural Representations with Realistic Synthetic Stimuli
abstract: Author(s) Thielk, Marvin Advisor(s) Gentner, Timothy; Sharpee,  Parametrizing complex natural stimuli is a difficult and long standing challenge. We used a generative deep convergent network to represent and parametrize a large corpus of song from European starlings, a songbird species, into a compressed low dimensional space. We applied psychophysical methods to probe categorical perception of natural starling song syllables, which reveal a shared categorical perceptual space. Some categorical boundaries are sensitive to the category assignment of training syllables, indicating that the consensus is context dependent and that underlying dimensions of the space are not independent. We record simultaneous firing from populations of 10's of neurons in a secondary auditory cortical region of anesthetized starlings. By estimating how fast population level neural representation change with respect to the stimuli, we produce a measure along a path in stimuli space that is shared between birds and descriptive of the psychophysically determined parameters in other birds. Consistent with this, we predict the behavioral psychometric function along one dimension by fitting the behavior for other dimensions to the population level neural activity. Thus, knowing how the animal responds in one sub region of the parametrized space informs responses in other sub regions. Our results implicate the importance of experience in shaping shared perceptual boundaries among complex communication signals and suggest the categorical representation of natural signals in secondary sensory cortices is distributed much more densely than predicted by traditional hierarchical object recognition models. This thesis also explores other applications of machine learning to solve neuroscience problems, in particular, the curse of dimensionality and exploring predictive coding and surprise. A model explicitly designed to predict future states allows the compression of high dimensional time varying signals into a lower dimensional representation encoding exclusively predictive and predictable information and has many practical applications.

title: Learning modulation of odor representations: new findings from Arc indexed networks
abstract: We first review our understanding of odor representations in rodent olfactory bulb (OB) and anterior piriform cortex (APC) We then consider learning induced representation changes. Finally we describe the perspective on network representations gained from examining Arc indexed odor networks of awake rats. Arc indexed networks are sparse and distributed, consistent with current views. However Arc provides representations of repeated odors. Arc indexed repeated odor representations are quite variable. Sparse representations are assumed to be compact and reliable memory codes. Arc suggests this is not necessarily the case. The variability seen is consistent with electrophysiology in awake animals and may reflect top down cortical modulation of context. Arc indexing shows that distinct odors share larger than predicted neuron pools. These may be low threshold neuronal subsets. Learning's effect on Arc indexed representations is to increase the stable or overlapping component of rewarded odor representations. This component can decrease for similar odors when their discrimination is rewarded. The learning effects seen are supported by electrophysiology, but mechanisms remain to be elucidated.

title: Unsupervised t Distributed Video Hashing and Its Deep Hashing Extension
abstract: In this paper, a novel unsupervised hashing algorithm, referred to as t USMVH, and its extension to unsupervised deep hashing, referred to as t UDH, are proposed to support large scale video to video retrieval. To improve robustness of the unsupervised learning, the t USMVH combines multiple types of feature representations and effectively fuses them by examining a continuous relevance score based on a Gaussian estimation over pairwise distances, and also a discrete neighbor score based on the cardinality of reciprocal neighbors. To reduce sensitivity to scale changes for mapping objects that are far apart from each other, Student t distribution is used to estimate the similarity between the relaxed hash code vectors for keyframes. This results in more accurate preservation of the desired unsupervised similarity structure in the hash code space. By adapting the corresponding optimization objective and constructing the hash mapping function via a deep neural network, we develop a robust unsupervised training strategy for a deep hashing network. The efficiency and effectiveness of the proposed methods are evaluated on two public video collections via comparisons against multiple classical and the state of the art methods.

title: Superposition catastrophe 1 Running head SUPERPOSITION CATASTROPHE PDP Networks Learn Local Grandmother Cell Representations in Order to Code for Multiple Things at the Same Time
abstract: One potential limitation with distributed representation is that it is difficult to co activate multiple things at the same time over the same set of processing units. That is, a superposition of co active distributed representations results in a blend pattern that is ambiguous; the socalled superposition catastrophe. Thus it is striking that M. M. Botvinick and D. C. Plaut (2006) developed a PDP model of short term memory that recalls lists of letters based on the superposition of distributed letter codes. Their finding suggests that the distributed representations can solve the superposition catastrophe. However we show that the model's success does not mitigate against this constraint. Under the appropriate training conditions a version of their model can indeed solve (avoid) the superposition catastrophe, but only by learning localist as opposed to distributed representations. Given that many cognitive systems need to code multiple things at the same time, the pressure to learn localist "grandmother cell" representations is widespread. Superposition catastrophe 3 PDP Networks Learn Local (Grandmother Cell) Representations in Order to Code for Multiple Things at the Same Time Neural networks can code for information in two qualitatively different ways. Some models rely on localist representations, such that words, objects, faces, and lexical concepts are coded distinctly, with their own dedicated representations. For example, in localist models of written word identification, the words dog and log are coded with distinct orthographic units, and a word is identified when a single unit (e.g. dog) is activated beyond some threshold (e.g. McClelland Rumelhart, 1981) Other models rely on distributed representations, such that the words dog and hog are coded as patterns of activation over collections of units, with each unit involved in coding multiple words (e.g. Seidenberg McClelland, 1989) In fact, various different types of distributed representations can be distinguished, ranging from dense distributed representations in which many hidden units are co activated in response to a single input and each hidden unit is involved in coding many different things, to sparse distributed coding in which each input activates relatively few units, and each unit is involved in coding relatively few things (but more than one) Nevertheless all versions of distributed coding share one defining feature; that is, it is not possible to interpret the activation of a single unit unambiguously. The debate regarding the relative merits of localist and distributed coding has taken a variety of forms, including their relative biological plausibility (e.g. Bowers, 2009, 2010ab; Plaut McClelland, 2010ab; Quian Quiroga Kreiman, 2010ab) their empirical successes (e.g. Coltheart, Rastle, Perry, Langdon, Ziegler, 2001; Seidenberg Plaut, 2006) and their computational capacities (e.g. Botvinick Plaut, 2009ab; Bowers, Damian, Davis, 2009ab, Page, 2000; Plaut McClelland, 2000) In the present paper we focus on the computational capacity of the distributed representations typically learned in PDP models, and argue that they are ill suited to support the co activation of multiple items at the same time over the same set of units. That is, we claim that these distributed representations cannot overcome the superposition catastrophe, contrary to some recent findings that suggest otherwise (Botvinick Plaut, 2006) We argue that the functional requirement to co activate multiple things at the same time provides a computational pressure to develop representations that respond highly selectively to inputs. Indeed, we show that PDP models sometimes learn local "grandmother cell" representations in response to this constraint.  There is already one well known computational limitation of the dense distributed representations often learned in PDP models; namely these representations are poor at supporting rapid one trial learning, as required for episodic memory. The problem is that each unit in a dense distributed representation is involved in coding many things, and as a consequence, learning something new impacts on pre existing knowledge (specifically, knowledge coded with overlapping units and connection weights) The greater the learning the greater the impact on pre existing knowledge, and in the case of rapid one trial learning, new knowledge can erase prior knowledge, a phenomenon known as catastrophic interference (McClosky Cohen, 1989; Ratcliff, 1990) or the stability plasticity dilemma (Grossberg, 1976) According to McClelland, McNaughton, and O'Reilly (1995) the brain's solution to this problem was to develop sparse coding in the hippocampus and related structures, with relatively few units active at any one point, and each unit involved in coding relatively few things. These sparse codes are not considered local (each unit would be involved in coding for more than one thing) but nevertheless, the solution to catastrophic interference is to reduce the level of distribution. In this way, large changes in connections weights (in response to new learning) have minimal impact on prior knowledge. Superposition catastrophe 4 However, this solution is itself limited. PDP models generalize on the basis of new inputs activating hidden units that are involved in coding old knowledge, and this overlap is often reduced when knowledge is coded sparsely. This makes the sparse codes in the hippocampus a poor medium for generalizing. For this reason, McClelland et al. (1995) argue that the brain relies on both dense and sparse distributed coding schemes: dense distributed representations in the cortex that support various forms of generalization required for perception, language, and semantic memory, and sparse distributed codes in the hippocampus (and related structures) for episodic memory. This dual solution to fast learning and generalization is the so called complementary learning systems hypothesis. The key claim we make here is that there is yet another computational constraint that undermines the use of dense distributed representations in the cortex as well. That is, the dense distributed representations learned in PDP networks are not only poor at fast learning, but also poorly suited to the task of co activating multiple things at the same time. Accordingly, the challenge is to develop neural networks that can both generalize and co activate multiple things. We show that PDP networks that learn localist representations of letters in response to the superposition constraint can indeed generalize to novel words. The Superposition Catastrophe PDP models support generalization in a variety of domains (e.g. reading and semantics) but until recently, there have been no demonstrations that these models can co activate multiple things at the same time over the same set of units. Given that various cognitive systems support the co activation of multiple items at the same time (perhaps 4 1; Cowan, 2001) this is a striking omission. Particularly so given that it has been claimed that distributed representations are ill suited for supporting this function, due to the superposition catastrophe (von der Malsburg, 1986) The superposition catastrophe refers to the observation that co activating items in a distributed system leads to blend patterns that are ambiguous with regards to what produced the blend in the first place. For example, consider Figure 1, adapted from Page (2000) that depicts distributed patterns for the names Paul, John, George, Ringo, Mick, and Keith. The patterns are distributed given that each name is coded as a pattern of activation of two of four units, and the identity of a person cannot be determined by the activation of a single unit. Although the identity of a name can be determined based on the pattern of activation over the four units, the coding scheme is ambiguous when more than one person is coded at the same time over the same units. For example, the combination of MICK and KEITH co activates all four units, but so do JOHN and GEORGE. The tempting conclusion is that it is difficult to code for more than one thing at a time over a given set of processing units when relying on distributed representations. By contrast, ambiguity is eliminated with localist coding schemes. For example, as can be seen in Figure 2, if localist Mick and Keith units are co active, the constituent patterns that produced the blend could only be Mick and Keith. Bowers (2002) used the superposition catastrophe as an argument in favor of localist coding models. The ambiguity associated with a superposition of dense distributed representations is in fact worse than the example above. In Figure 1, the ambiguity is the product of blending together two familiar names (e.g. Mick and Keith) but blends might just as well be produced by combinations of novel inputs. If novel inputs are free to contribute to blends, then there is an infinity of different combinations of novel constituents that will produce the same blend. In this sense the superposition catastrophe is an ill posed problem; it is logically impossible to determine the constituent patterns that produced a given blend. This would seem to rule out dense distributed representations as a medium for supporting cognitive processes that deal with more than one thing at a time (that is, most of cognition) Nevertheless, it is important to note that the brain faces and often solves other ill posed problems. The classic example is the so called inverse problem in vision, in which an infinite Superposition catastrophe 5 number of possible shapes in the 3D world are consistent with a given 2D image projected on the retina. The conclusion is not that we cannot perceive in 3D, but rather, that the visual system needs to adopt biases about which 3D shapes are most likely to have produced a given (ambiguous) 2D image on the retina. The biases c

title: Solving Multiclass Learning Problems via Error Correcting Output Codes
abstract: Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k 2 values (i.e. k "classes" The definition is acquired by studying collections of training examples of the form (xi, f(xi) Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision tree pruning. Finally, we show that like the other methods the error correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error correcting output codes provide a general purpose method for improving the performance of inductive learning programs on multiclass problems.

title: The HTM Spatial Pooler A Neocortical Algorithm for Online Sparse Distributed Coding
abstract: Hierarchical temporal memory (HTM) provides a theoretical framework that models several key computational principles of the neocortex. In this paper, we analyze an important component of HTM, the HTM spatial pooler (SP) The SP models how neurons learn feedforward connections and form efficient representations of the input. It converts arbitrary binary input patterns into sparse distributed representations (SDRs) using a combination of competitive Hebbian learning rules and homeostatic excitability control. We describe a number of key properties of the SP, including fast adaptation to changing input statistics, improved noise robustness through learning, efficient use of cells, and robustness to cell death. In order to quantify these properties we develop a set of metrics that can be directly computed from the SP outputs. We show how the properties are met using these metrics and targeted artificial simulations. We then demonstrate the value of the SP in a complete end to end real world HTM system. We discuss the relationship with neuroscience and previous studies of sparse coding. The HTM spatial pooler represents a neurally inspired algorithm for learning sparse representations from noisy data streams in an online fashion.