# A General Model for Side Information in Neural Networks
## 3 A General Method for Incorporating Side Information
To clarify how side information is incorporated into an NN, we give, in Algorithm 1, high-level pseudo-code corresponding to the architecture for side information, as shown in Figure 1.  
```
1:\(x_{t}\) is the input to the network at time step \(t\) during the train or test phase
2:\(\lambda\leftarrow\) layer of the network that the side information is fed into
3:\(\mathcal{C}\leftarrow\) the set of constraints on the knowledge base
4:for time step \(t=1,2,\ldots,n\)do
5:\(x_{t}\) is fed as input to the network
6:\(s_{t}\leftarrow\mathcal{KB}(x_{t},\mathcal{C})\)
7: an encoding of \(s_{t}\) is fed into the network at layer \(\lambda\)
8:\(\hat{y}_{t}\leftarrow\) output of the network
9:endfor
```  
**Algorithm 1** Training or testing with side information  
We stipulate that \(\mathcal{KB}(x_{t},\mathcal{C})=\mathcal{KB}_{\mathcal{C}}(x_{t})\), i.e., we can either first compute the result of applying \(\mathcal{KB}\) to \(x_{t}\), resulting in \(\mathcal{KB}(x_{t})\), and then constrain \(\mathcal{KB}(x_{t})\) to \(\mathcal{C}\), or, alternatively, we can first constrain \(\mathcal{KB}\) to \(\mathcal{C}\), resulting in \(\mathcal{KB}_{\mathcal{C}}\), and then apply \(\mathcal{KB}_{\mathcal{C}}\) to \(x_{t}\). For this reason, from now on, we prefer to write \(\mathcal{KB}_{\mathcal{C}}(x_{t})\) instead of \(\mathcal{KB}(x_{t},\mathcal{C})\), noting that when \(\mathcal{C}\) is empty, i.e., \(\mathcal{C}=\varnothing\), then \(\mathcal{KB}_{\mathcal{C}}=\mathcal{KB}_{\varnothing}=\mathcal{KB}\). We leave the internal working of \(\mathcal{KB}_{\mathcal{C}}(x_{t})\) unspecified, although we note that it is important that the computational cost of \(\mathcal{KB}_{\mathcal{C}}(x_{t})\) is low. The set of constraints \(\mathcal{C}\) may be application-dependent, for example, on whether \(x_{t}\) is text- or image-based. Moreover, for a particular dataset we may constrain knowledge-based queries to a small number of one or more classes. To give a more concrete example, if \(x_{t}=image_{i}\), i.e., the input to the network is an image, and we have constrained the knowledge base to the class _green_, then its output will be of the form \(s_{t}=(p,image_{i},green)\). In this case, the knowledge base matches the input image precisely and returns in \(s_{t}\) its  
Figure 1: Architecture of a neural network with side information. The arrow from additional side information nodes to the hidden layers indicates that there are weighted links between them. The architecture is instantiated once it is specified which links from side information nodes to the network are active, and furthermore, we do not discount the possibility of having active links directly from side information nodes to the output node. probability of being \(green\). In the more general case, the knowledge base will only match part of the input. More specifically, we let \(s_{t}=\{s_{t1},s_{t2},\ldots,s_{tm}\}\) be the result of applying the knowledge base \(\mathcal{KB}_{\mathcal{C}}\) to the input \(x_{t}\), where each \(s_{tj}\), for \(j=1,2,\ldots,m\), encodes a fact \((p_{tj},z_{tj},c_{tj})\), so that the probability, \(p_{tj}=P(z_{tj}|c_{tj})\), for \(j=1,2,\ldots,m\), will be fed to the network as side information. From a practical perspective it will be effective to set a non-negative threshold \(\theta\) such that \(s_{tj}=(p_{tj},z_{tj},c_{tj})\) is included in \(s_{t}\) only when \(p_{tj}>\theta\). We also assume for convenience that the number of facts returned by \(s_{t}=\mathcal{KB}_{\mathcal{C}}(x_{t})\) that are fed into the network as side information is fixed at a maximum, say \(m\), and that they all pass the threshold, i.e., \(p_{tj}>\theta\). Now, if there are more than \(m\) facts that pass the threshold, then we only consider the top \(m\) facts with the highest probabilities, where ties are broken randomly. On the other hand, if there are less than \(m\) facts that pass the threshold, say \(k<m\), then we consider the values for the \(m-k\) facts that did not pass the threshold as missing.