# A General Model for Side Information in Neural Networks
## 3 A General Method for Incorporating Side Information  
To simplify the model under discussion, let us assume a generic neural network (NN) with input and output layers and some hidden layers in-between. We use the following terminology: \(x_{t}\) is the input to the NN at time step \(t\), \(y_{t}\) is the true target value (or simply the target value), \(\hat{y}_{t}\) is the estimated target value (or the NN's output value), and \(s_{t}\) is the side information, defined more precisely below. The side information comes from a _knowledge base_, \(\mathcal{KB}\), containing both _facts_ (probabilistic statements) and _rules_ (_if-then_ statements). To start off, herein, we will restrict ourselves to side information in the form of facts represented as triples, \((p,z,c)\), with \(p\) being the probability that the instance \(z\) is a member of class \(c\), and where a class may be viewed as a collection of facts. In other words, the triple \((p,z,c)\) can be written as \(p=P(z|c)\), which is the likelihood of \(z\) given \(c\). We note that facts may not necessarily be atomic units, and that side information such as \(s_{t}\) is generally viewed as a conjunction of one or more facts. Let us examine some examples of side information facts, demonstrating that the knowledge base \(\mathcal{KB}\) can, in principle, contain very general information:  
1. \((p,headache,symptom)\) is the probability of a _headache_ being a \(symptom\). Here, _headache_ is a textual instance and the likelihood is \(P(headache|symptom)\). 2. \((p,image_{i},green)\) is the probability that the background of \(image_{i}\) is \(green\); in this case the probability could denote the proportion of the background of the image. Here, \(image_{i}\) is an image instance and the likelihood is \(P(image_{i}|green)\). 3. \((p,ceg_{i},irregular)\) is the probability that \(ceg_{i}\) is \(irregular\). Here, \(ceg_{i}\) is a segment of an ECG time series instance and the likelihood is \(P(ceg_{i}|irregular)\). The architecture of the neural network incorporating side information is shown in Figure 1. We assume that the NN operates in discrete time steps, and that at the current time step, \(t\), instance \(x_{t}\) is fed into the network and \(\hat{y}_{t}\) is its output. As stated above, we will concentrate on a knowledge base of facts, and detail what preprocessing may occur to incorporate facts into the network as side information. It is important that the preprocessing step at time \(t\) can only access the side information and the input presented to the network at that time, i.e., it cannot look into the past or the future. It is, of course, sensible to assume that one cannot access inputs that will be presented to the network in the future at times greater than \(t\), and regarding the past, at times less than \(t\), the reasoning being that this information is already encoded in the network. Moreover, the learner may not be allowed to look into the past due to privacy considerations. On a high level, this is similar to continual learning (CL) frameworks where a CL model is updated at each time step using only the new data and the old model, without access to data from previous tasks due to speed, security, and privacy constraints [17]. As noted in the caption of Figure 1, side information may be connected to the rest of the network in several ways through additional nodes. The decision of whether, for example, to connect the side information to the first or second hidden layers and/or also to the output node can be made on the basis of experimental results. We emphasise that in our model, side information is available in both the training and test phases. Let \(x_{t}\) be the input instance to the network at time step \(t\) and \(s_{t}=\mathcal{KB}(x_{t},\mathcal{C})\) represent the side information at time \(t\), where \(\mathcal{C}\) is a set of constraints which may be empty.