## 2 Method
### Feature Mapping Network  
In this module, we aim to train a network to learn a transformation function that could later be used to map the testing set to a feature space with improved properties of separability and compactness. Our approach is to learn a regression function that transforms the distribution of the embeddings of each class to a more compact distribution. As we shortly show, regressors with high bias and low variance tend to output a value close to the average of the target. Following this observation, we want to force the regression model even more to learn a value close to the average of each feature embedding. Bias-variance tradeoff is a widely studied problem in the field of machine learning [57; 58]. The dilemma was first formally introduced in [59], and it refers to a model property where the estimation error of the model can be decomposed into two terms, known as bias and variance, thus, either bias or variance can affect the performance of a model. The bias error is the difference between the expected (average) prediction of our model and the target correct value which results from oversimplifying the estimated function and missing the regularities in the training data. The variance error is the difference between the average prediction of our model for a given data point which results from the sensitivity of the model to irregularities in the training data. As illustrated in Figure 2, there is often a tradeoff between the contribution of bias and variance to the estimation error as conflict arises from trying to simultaneously minimize the two error terms. The variance error can be reduced through smoothing the signal which will in effect introduce a high bias as the model will be oversmoothed and will miss the specifics of the underlying data points resulting in an underfitting scenario where both the training and test errors are high. On the other hand, the high bias can be reduced by increasing the complexity (number of fitting parameters) of the model, but then the variance of the model will increase as the model will have more capacity to fit the noise and irregularities of the underlying training data points and not generalizing to the target signal. This scenario is referred to as overfitting, giving a low training error while degrading the predictive performance on the test data. As we observe, the MLP regressor models for the bridging network learn to predict the average value of the target feature values for the same class. The intuition behind that observation is that the model is not complex enough to fit the underlying data (underfitting to data points), which means the model has too much bias. According to the bias-variance tradeoff, the high bias model will have a low variance to reduce the whole mean-squared error (\(MSE\)), which decomposes into the two terms of \(Bias\) and \(Variance\):  
\[MSE=Bias^{2}+Variance \tag{4}\]  
Figure 2: Bias and variance as a function of model complexity. In the book _Doing Bayesian data analysis_[60], the author stated that the mean of a distribution is the value that minimizes the expected squared deviation. As can be mathematically proven (see Appendix A), the mean-squared error has a lower bound value as the variance of \(Y\) (the target output) when \(\hat{Y}\) (the predicted output) equals the average of \(Y\). Therefore, the model will predict values closer to the average, trying to reach the lower bound of the \(MSE\). We also found that if we train the regressor with different outputs for the same set of corresponding inputs, the regressor will minimize its error by outputting a value close to the average of the presented target values. A plausible explanation for this behavior is that having different target values for the same input features resembles adding noise to the output. Adding noise to the input or the output (less common) is a well-known regularization method to aid generalization (lowering the variance) and avoid overfitting [61; 62; 63].