Distributed Smoothed Tree Kernel
Authors: Lorenzo Ferrone, Fabio Massimo Zanzotto (Università di Roma Tor Vergata, 2014)
Topic: Distributional Semantics

Abstract
In this paper we explore the possibility to merge the world of Compositional Distributional Semantic Models (CDSM) with Tree Kernels (TK). In particular, we will introduce a specific tree kernel (smoothed tree ker-nel, or STK) and then show that is possibile to approximate such kernel with the dot product of two vectors obtained compositionally from the sen-tences, creating in such a way a new CDSM.

Introduction
Compositional distributional semantics is a flourishing research area that leverages dis-tributional semantics (see Baroni and Lenci (2010)) to produce meaning of simple phrases and full sentences (hereafter called text frag-ments). The aim is to scale up the success of word-level relatedness detection to longer fragments of text. Determining similarity or relatedness among sentences is useful for many applications, such as multi-document summar-ization, recognizing textual entailment (Dagan et al., 2013), and semantic textual similarity detection (Agirre et al., 2013; Jurgens et al., 2014). Compositional distributional semantics models (CDSMs) are functions mapping text fragments to vectors (or higher-order tensors). Functions for simple phrases directly map distributional vectors of words to distributional vectors for the phrases (Mitchell and Lapata, 2008; Baroni and Zamparelli, 2010; Zanzotto et al., 2010). Functions for full sentences are generally de ned as recursive functions over the ones for phrases (Socher et al., 2011). Distributional vectors for text fragments are then used as inner layers in neural networks, or to compute similarity among text fragments via dot product. CDSMs generally exploit structured representations tx of text fragments x to derive their meaning f(tx), but the structural information, although extremely important, is obfuscated in the nal vectors. Structure and meaning can interact in unexpected ways when computing cosine similarity (or dot product) between vectors of two text fragments, as shown for full additive models in (Ferrone and Zanzotto, 2013). Smoothed tree kernels (STK) (Croce et al., 2011) instead realize a clearer interaction between structural information and distributional meaning. STKs are speci c realizations of convolution kernels (Haussler, 1999) where the similarity function is recursively (and, thus, compositionally) computed. Distributional vectors are used to represent word meaning in computing the similarity among nodes. STKs, however, are not considered part of the CDSMs family. As usual in kernel machines (Cristianini and Shawe-Taylor, 2000), STKs directly compute the similarity between two text fragments x and y over their tree representations tx and ty, that is, STK(tx; ty). The function f that maps trees into vectors is only implicitly used, and, thus, STK(tx; ty) is not explicitly expressed as the dot product or the cosine between f(tx) and f(ty). Such a function f, which is the underlying reproducing function of the kernel (Aronszajn, 1950), is a CDSM since it maps trees to vectors by using distributional meaning. However, the huge nality of Rn (since it has to represent the set of all possible subtrees) prevents to actually compute the function f(t), which thus can only remain implicit. Distributed tree kernels (DTK) (Zanzotto and Dell'Arciprete, 2012) partially solve the last problem. DTKs approximate standard tree kernels (such as (Collins and Du y, 2002)) by de ning an explicit function DT that maps trees to vectors in Rm where m n and Rn is the explicit space for tree kernels. DTKs approximate standard tree kernels (TK), that is, hDT(tx);DT(ty)i TK(tx; ty), by approximating the corresponding reproducing function. Thus, these distributed trees are small vectors that encode structural information. In DTKs tree nodes u and v are represented by nearly orthonormal vectors, that is, vectors !u and !v such that h !u ; !v i ( !u ; !v ) where is the Kroneker's delta. This is in contrast with distributional semantics vectors where h !u ; !v i is allowed to be any value in [0; 1] according to the similarity between the words v and u. In this paper, leveraging on distributed trees, we present a novel class of CDSMs that encode both structure and distributional meaning: the distributed smoothed trees (DST). DSTs carry structure and distributional meaning on a rank-2 tensor (a matrix): one dimension encodes the structure and one dimension encodes the meaning. By using DSTs to compute the similarity among sentences with a generalized dot product (or cosine), we implicitly de ne the distributed smoothed tree kernels (DSTK) which approximate the corresponding STKs. We present two DSTs along with the two smoothed tree kernels (STKs) that they approximate. We experiment with our DSTs to show that their generalized dot products approximate STKs by directly comparing the produced similarities and by comparing their performances on two tasks: recognizing textual entailment (RTE) and semantic similarity detection (STS). Both experiments show that the dot product on DSTs approximates STKs and, thus, DSTs encode both structural and distributional semantics of text fragments in tractable rank-2 tensors. Experiments on STS and RTE show that distributional semantics encoded in DSTs increases performance over structure-only kernels. DSTs are the rst positive way of taking into account both structure and distributional meaning in CDSMs. The rest of the paper is organized as follows. Section 2.1 introduces the basic notation used in the paper. Section 2 describe our distributed smoothed trees as compositional distributional semantic models that can represent both structural and semantic information. Section 4 reports on the experiments. Finally, Section 5 draws some conclusions.

Conclusion
Distributed Smoothed Trees (DST) are a novel class of Compositional Distributional Se-mantics Models (CDSM) that effectively en-code structural information and distributional semantics in tractable rank-2 tensors, as ex-periments show. The paper shows that DSTs contribute to close the gap between two appar-ently different approaches: CDSMs and convo-lution kernels. This contribute to start a dis-cussion on a deeper understanding of the rep-resentation power of structural information of existing CDSMs.