Nystrom Methods for Efficient Kernel-Based Methods for Community Question Answering
Authors: Danilo Croce, Simone Filice, Roberto Basili (Università di Roma Tor Vergata, 2016)
Topic: Distributional Semantics

Abstract
Expressive but complex kernel functions, such as Sequence or Tree kernels, are usually underemployed in NLP tasks, \eg in community Question Answering (cQA), as for their significant complexity in both learning and classification stages. Recently, the \Nystrom methodology for data embedding has been proposed as a viable solution to scalability problems. By mapping data into low-dimensional approximations of kernel spaces, it positively increases scalability through compact linear representations for highly structured data. In this paper, we show that \Nystrom methodology can be effectively used to apply a kernel-based method in the cQA task, achieving state-of-the-art results by reducing the computational cost of orders of magnitude.

Introduction
Kernel methods AUTHOR have been employed in several Machine Learning algorithms AUTHOR achieving state-of-the-art performances in many classification tasks. Recently, the kernel based approach presented in AUTHOR has been applied in the community Question Answering (cQA) challenge at SemEval 2016 AUTHOR obtaining state-of-the-art results. Unfortunately, when large data volumes are involved, time and space complexity required in learning and classification may prevent the adoption of expressive but complex kernel functions, such as Sequence AUTHOR or Tree kernels AUTHOR. In particular, the classification cost required by a kernel-based model crucially depends on its number of support vectors: classifying a new instance requires a kernel computation against all support vectors. This scalability issue is evident in many NLP and IR applications, such as in re-ranking answers in question answering AUTHOR, where the number of support vectors is typically very large. Some approaches have been defined to bound the complexity of kernel-based methods, such as AUTHOR, but they are still specific to kernel formulations and learning algorithms. In AUTHOR it has been shown that a viable and more general solution to the above scalability issues is the \Nystrom methodology, a dimensionality reduction technique that has been applied also in kernel-based methods since AUTHOR. This methodology has been designed to approximate the Gram Matrix derivable by a kernel function, enabling the projections of examples into low-dimensional spaces. The \Nystrom projection function is generated by using some examples called landmarks, whose number directly impacts on the embeddings quality; dually, costs of projecting a new example in the embedding space rise linearly with the number of landmarks, that is usually of orders of magnitude lower with respect of the number of possible support vectors that can be derived from a learning process. Once each example is projected in the dense low-dimensional space, the application of efficient linear learning methods is enabled, such as AUTHOR, preserving at the same time the expressiveness and effectiveness of kernel methods. This approach is highly applicable to different input data as well as to different kernels or learning algorithms, as discussed in AUTHOR. In this paper we show that the \Nystrom method can be effectively used in the cQA task, by adopting the same kernel functions proposed in AUTHOR and obtaining the same results w.r.t. the metrics adopted in the SemEval task, by reducing the computational cost of orders of magnitude. In Section Nystrom, we demonstrate the viability of the \Nystrom method to reduce the computational costs of kernel machines. Experimental results (obtained by adopting efficient SVM learning over the cQA task) are discussed in Section Exp. Finally, Section sec:relwork describes related work, while in Section sec:concl conclusions are derived.

Conclusion
This paper discussed the application of \Nystrom method for a significant reduction of computational costs in kernel-based classifications in the cQA task. By projecting examples into low-dimensional embeddings, \Nystrom enables the adoption of efficient linear classifier, and drastically reduces the overall computational cost. Experimental results demonstrate that the proposed approach leads to a cost reduction higher than 90\%, with a negligible performance drop. Future research will be devoted to the definition of a principled strategy to estimate the optimal number of layers, as well as the size of embeddings at each layer.