Dynamic Alignment Kernels

There is much current interest in kernel methods for classi cation re gression PCA and other linear methods of data analysis Kernel methods may be particularly valuable for problems in which the input data is not readily described by explicit feature vectors One such problem is where input data consists of symbol sequences of di erent lengths and the re lationships between sequences are best captured by dynamic alignment scores This paper shows that the scores produced by certain dynamic align ment algorithms for sequences are in fact valid kernel functions This is proved by expressing the alignment scores explicitly as dot products Alignment kernels are potentially applicable to biological sequence data speech data and time series data The kernel construction may be extended from pair HMMs to pair probabilistic context free grammars Introduction Linear Methods using Kernel Functions Introduction Linear Methods using Kernel Functions In many types of machine learning the learner is given a training set of cases or examples a al A A denotes the set of all possible cases cases may be vectors pieces of text biological sequences sentences etc For supervised learning the cases are accompanied by a set of corresponding labels or values y yl The cases are mapped to feature vectors x xl X where the X is a real vector space termed the feature space The mapping from A to X is denoted by so that xi ai Sometimes the cases are given as feature vectors to start with in which case may be the identity mapping otherwise denotes the method of assigning numeric feature values to a case Once a feature vector xi has been de ned for each case ai it becomes pos sible to apply a wide range of linear methods such as support vector machines linear regression principal components analysis PCA and k means cluster analysis As shown in Vap for SV machines in for example Wah for linear re gression and in SSM for PCA and k means cluster analysis the calculations for all of these linear methods may be carried out using a dual rather than a primal formulation of the problem For example in linear least squares regression the primal formulation is to nd a coe cient vector that minimises kX yk whereX is the design matrix an l by d matrix in which the ith row is xi and each xi has d elements If l is larger than d the usual method of nding is to solve the normal equations XX Xy This requires the solution of a set of linear equations with coe cients given by the d d matrix XX The dual formulation is to nd a coe cient vector that minimises kXX yk so that one coe cient i is found for each case vector xi This requires the solution of a set of linear equations with coe cients given by the l l matrix XX Both methods lead to the same predicted value y for a new case x If there are more cases than features that is if l d the primal method is more economical because the d d matrix XX is smaller than the l l matrix XX For example if there are cases each described by a vector of measurements then the primal method requires solving a by system of linear equations while the dual method requires solving a by system which will have rank at most For such a problem the dual method has no advantage The potential advantage of the dual method for regression is that it can be applied to very large feature vectors The coe cient matrix XX contains the dot products of pairs of feature vectors the ijth element of XX is xi xj In the dual calculation it is only dot products of feature vectors that are used feature vectors never appear on their own As the feature vectors xi ai appear only in dot products it is often possible to avoid computing the feature vectors and to compute dot products directly in some economical fashion from the case descriptions ai instead A kernel is a function k that computes a dot product of feature vectors from the corresponding cases Applying Linear Methods to Structured Objects De nition A kernel is a function k such that for all a b A