# _Pat-in-the-Loop_: Declarative Knowledge for Controlling Neural Networks
## 3 The Model
### Distributed Tree Encoders for Exploiting Parse Trees in Neural Networks  
Distributed tree encoders are the encoders used in Pat-in-the-Loop to directly exploit parse trees in neural networks. These encoders, stemming from tree kernels [28] and distributed tree kernels [29], give the possibility to represent parse trees in vector spaces \(\mathbb{R}^{d}\) that embed huge spaces of subtrees \(\mathbb{R}^{n}\). Tree kernels [28] have offered an important opportunity to fully exploit parse trees in learning with kernel machines [30, 31]. Tree kernels are functions implicitly computing the similarity among parse trees \(\mathcal{T}\) mapped in vectors \(\mathbf{x}^{\mathcal{T}}\in\mathbb{R}^{n}\) where dimensions are subtrees \(\tau\). For example, the 52629-th dimension of \(\mathbf{x}^{\mathcal{T}}\in\mathbb{R}^{n}\) can represent the subtree \(\tau^{(52629)}=\)(_SQ,[(VBD,[did]),NP,VP]_) (see Table 2). Vectors \(\mathbf{x}^{\mathcal{T}}\) for parse trees \(T\) generally have:  
\[x_{i}^{\mathcal{T}}=\left\{ \begin{array}{ll}\lambda^{\frac{|\tau^{(i)}|}{2} }&\text{ if }\tau^{(i)}\in S(T)\\ 0&\text{ if }\tau^{(i)}\
otin S(T)\end{array} \right.\]  
where \(S(\mathcal{T})\) is the set of valid subtrees of \(\mathcal{T}\), \(0<\lambda<1\) is a decay factor penalizing large subtrees, and \(|\tau^{(i)}|\) is the size of the node set of \(\tau^{(i)}\). Valid subtrees \(\tau\in S(\mathcal{T})\) in [28] are connected subtrees of \(\mathcal{T}\) of at least two nodes and, if \(\tau\) contains a node \(c\), it should contains all the siblings of the node \(c\) in \(\mathcal{T}\). For example, \(x_{52629}^{\mathcal{T}}=\lambda^{\frac{5}{2}}\) for the parse tree in Figure 1 since \(\tau^{(52629)}\) is a valid subtree of \(\mathcal{T}_{e}\). The power of these tree kernels is that parse trees are are never explicitly represented as vectors \(\mathbf{x}^{\mathcal{T}}\) but the tree kernel functions implicitly compute their dot product. Distributed tree kernels [29] may transfer the opportunity given by tree kernels [28] within neural networks since distributed tree kernels implicitly embed vectors \(\mathbf{x}^{\mathcal{T}}\in\mathbb{R}^{n}\) into a reduced space \(\mathbb{R}^{d}\) in the context of support vector machines. Distributed tree kernels build on Johnson-Lindenstrauss Transformation [32] and holographic reduced representations (HRR) [33]. Building on distributed tree kernels, we propose _distributed tree encoders_ that may be seen as linear transformations \(\mathbf{W}_{\mathbf{dt}}\in\mathbb{R}^{d\times n}\) (similarly to Johnson-Lindenstrauss Transformation [32]). These linear transformations embed vectors \(\mathbf{x}^{\mathcal{T}}\in\mathbb{R}^{n}\) in the space of tree kernels in smaller vectors \(\mathbf{y}^{\mathcal{T}}\in\mathbb{R}^{d}\):  
\[\mathbf{y}^{\mathcal{T}}=\mathbf{W}_{\mathbf{dt}}\mathbf{x}^{\mathcal{T}}\]  
Columns \(\mathbf{w}_{i}\) of \(\mathbf{W}_{\mathbf{dt}}\) encode subtree \(\tau^{(i)}\) and are computed with an encoding function \(\mathbf{w}_{i}=E(\tau^{(i)})\) as follows:  
\[E(\tau^{(i)})=\left\{ \begin{array}{ll}\mathbf{r}&\text{ if }\tau^{(i)}=(r,[])\\ \mathbf{r}\otimes E(\tau^{(i)}_{1})\otimes\ldots\otimes E(\tau^{(i)}_{k})\\ &\text{ if }\tau^{(i)}=(r,[\tau^{(i)}_{1},\ldots,\tau^{(i)}_{k}])\end{array} \right. \tag{1}\]