# A General Model for Side Information in Neural Networks
## 4 A Formalism for Side Information  
Our formalism for assessing the benefits of side information is based on quantifying the difference between the loss of the neural network without and with side information; see (1) below. Additionally, we provide detailed analysis for two of the most common loss functions, _squared loss_ and _cross-entropy loss_[20]; see (2) and (3), respectively, below. Moreover, in (5) we present the notion of being _individually useful_ at time step \(t\), which is the smallest non-negative \(\delta\) such that the probability that the difference in loss at \(t\) when side information is absent and present is larger than a given non-negative margin, \(\epsilon\), which is greater than or equal to \(1-\delta\). When side information is individually useful for all time steps it is said to be _globally useful_. Furthermore, in (6) we modify (5) to allow for the concept of side information being _useful on average_. To clarify the notation, we assume that \(x_{t}\) is the input to the NN at time \(t\), \(y_{t}\) is the target value given input \(x_{t}\), \(\hat{y}_{t}\) is the estimated target value given \(x_{t}\), i.e., the output of the NN when no side information is available for \(x_{t}\), and \(\hat{y}_{s_{t}}\) is the estimated target value given \(x_{t}\) and side information \(s_{t}\), i.e., the output of the NN when side information \(s_{t}\) is present for \(x_{t}\). To formalise the contribution of the side information, let \(\mathcal{L}(y_{t},\hat{y}_{t})\) be the NN loss on input \(x_{t}\) and \(\mathcal{L}(y_{t},\hat{y}_{s_{t}})\) be the NN loss on input \(x_{t}\) with side information \(s_{t}\). The difference between the loss from \(\hat{y}_{t}\) and the loss from \(\hat{y}_{s_{t}}\), where \(0\leq y_{t}\leq 1\) and \(0<\hat{y}_{t},\hat{y}_{s_{t}}<1\), is given by  
\[\mathcal{L}(y_{t},\hat{y}_{t})-\mathcal{L}(y_{t},\hat{y}_{s_{t}}). \tag{1}\]  
Now assuming squared loss for a regression problem, we obtain  
\[\mathcal{L}(y_{t},\hat{y}_{t})-\mathcal{L}(y_{t},\hat{y}_{s_{t}}) =(y_{t}-\hat{y}_{t})^{2}-(y_{t}-\hat{y}_{s_{t}})^{2} \tag{2}\] \[=\hat{y}_{t}^{2}-\hat{y}_{s_{t}}^{2}-2y_{t}(\hat{y}_{t}-\hat{y}_{ s_{t}})\] \[=(\hat{y}_{t}+\hat{y}_{s_{t}})(\hat{y}_{t}-\hat{y}_{s_{t}})-2y_{t }(\hat{y}_{t}-\hat{y}_{s_{t}})\] \[=(\hat{y}_{t}-\hat{y}_{s_{t}})(\hat{y}_{t}+\hat{y}_{s_{t}}-2y_{t}),\]which has a positive solution when \(\hat{y}_{t}<\hat{y}_{s_{t}}<2y_{t}-\hat{y}_{t}\), i.e., both terms in the last equation are negative, or \(\hat{y}_{t}>\hat{y}_{s_{t}}>2y_{t}-\hat{y}_{t}\), i.e., both these terms are positive. If we restrict ourselves to binary classifiers, then squared loss reduces to the well-known Brier score [21]. Inspecting (2) for this case, when \(y_{t}=1\) we have the constraint \(\hat{y}_{t}<\hat{y}_{s_{t}}\), ensuring both terms are negative. Correspondingly, when \(y_{t}=0\), we have the constraint \(\hat{y}_{t}>\hat{y}_{s_{t}}\), ensuring that both terms are positive; note that in this case, \(\hat{y}_{s_{t}}<\hat{y}_{t}\) is intuitive since the classifier returns the probability of \(y_{t}\) being 1, i.e., estimating the chance of a positive outcome. On the other hand, assuming cross-entropy loss for a classification problem, we obtain  
\[\mathcal{L}(y_{t},\hat{y}_{t})-\mathcal{L}(y_{t},\hat{y}_{s_{t}}) =-y_{t}\log(\hat{y}_{t})+y_{t}\log(\hat{y}_{s_{t}}) \tag{3}\] \[=y_{t}\log\left(\frac{\hat{y}_{s_{t}}}{\hat{y}_{t}}\right),\]  
which has a positive solution when \(\hat{y}_{t}<\hat{y}_{s_{t}}\). In this case, if we restrict ourselves to binary classifiers, then cross-entropy loss reduces to the well-known logarithmic score [21]. Here, we need to note that a positive solution always exists, since the binary cross-entropy loss is given by  
\[-y_{t}\log(\hat{y}_{t})-(1-y_{t})\log(1-\hat{y}_{t}), \tag{4}\]  
when the estimated target value is \(\hat{y}_{t}\), and similarly replacing \(\hat{y}_{t}\) with \(\hat{y}_{s_{t}}\) when the estimated target value is \(\hat{y}_{s_{t}}\).