# From Scores to Predictions in Multi-Label Classification: Neural Thresholding Strategies
## 4 Neural Network Models for Thresholding Prediction Scores
### ThresNet--Neural Thresholding Model with Label Embeddings  
To overcome the space complexity problem of the above naive model, we propose a different architecture, presented in Figure 2, inspired by a FastText model [43]. In this model, each label is associated with a trainable label embedding (such as word embeddings in FastText), which is a vector of \(d\) floating point values. For a given input, which is a scaled score vector \(\mathbf{ss}\), the model calculates the _effective multi-label embedding_\(\mathbf{h}\), which can be described as a result of sparse vector-dense matrix multiplication: \(\mathbf{h}=\mathbf{ss}\cdot\mathbf{Emb}\), where \(\mathbf{Emb}\) is an \(L\times d\) matrix with the \(i\)-th label embedding in its \(i\)-th row. In other words, the effective multi-label embedding is a weighted sum of the label embeddings (not a simple average of the embeddings, as in the original FastText), with label scores as the weights of the components. This way, it includes information about which labels are involved in a given input as well as the intensity of these labels in the mix, as judged by a source scorer.  
This multi-label embedding could be directly used to produce prediction vectors via a dense layer with \(L\) output units. After initial experiments, we found that slightly better results are possible with a deeper model. The final model has two additional hidden layers, both having \(d\) units (the same sizes as embeddings), and using the popular ReLU [44] activation function (\(f_{\text{ReLU}}(z)=\max(0,z)\)). The final model realizes the following function:  
\[\text{ThresNet}(\mathbf{ss})=\sigma(\text{dense}(\text{dense}_{\text{ReLU}}( \text{dense}_{\text{ReLU}}(\mathbf{ss}\cdot\mathbf{Emb})))) \tag{14}\]  
For training purposes, the Heaviside is replaced by _sigmoid_\(\sigma(\cdot)\) (\(\sigma(z)=1/(1+\exp(-z))\)), which enables a standard gradient-based training. During inference, the Heaviside function is used again to achieve binary predictions.  
The presented architecture enables complex, nonlinear, and non-obvious thresholding, while having \(O(dL)\) complexity, where \(d<<L\). Dimensionality of label embeddings\(d\) (which is also the hidden layer size), is the main hyperparameter for controlling the complexity of the model.  
ThresNet, while inspired by FastText, is considerably different. First, the purposes of these models are different: thresholding for ThresNet versus representation learning or end-to-end classification for FastText. The main consequence of that is the nature of the model inputs: ThresNet obtains label scores and not raw feature vectors (especially not features from a fixed-sized moving context window). Second, ThresNet calculates a weighted average of label embeddings, while FastText calculates a simple average of feature embeddings. Third, ThresNet includes a couple of bottleneck hidden layers to mitigate the O (\(L^{2}\)) complexity and improve performance, while FastText maps averaged embeddings directly into the output space. Finally, the ThresNet\({}_{CSS}\) variant, described below, is the enhancement that does not have any equivalent counterpart in FastText (and which is rather impossible, because input and output spaces are different in sizes there).