# Improving Re-Identification by Estimating and Utilizing Diverse Uncertainty Types for Embeddings
## 3 Uncertainty Estimation in Neural Networks -- Mathematics and Algorithm
### Uncertainty Estimation for Embedding Vectors
#### 3.3.1 Bayesian Module  
The first of these modules is the Bayesian module \(\psi_{\iota\theta}\), which acts as a pre-processing unit for the two embedding heads and enables the estimation of model uncertainty in the original approach as well as the estimation of distributional uncertainty with our  
Figure 3: Overview of our architecture to estimate the three types of uncertainty for an embedding. First, the backbone \(f\) produces a feature tensor \(F\) from the input \(x\). Next, the parameters \({}^{k}\theta\) of the Bayesian module are sampled \(K\) times from the weight distribution \(q_{\pi}(\theta)\), thus producing \(K\) versions of the Bayesian module, each of which processes the feature tensor \(F\), producing a total of \(K\) resulting feature tensors \(\left\{{}^{k}F\right\}_{k=1}^{K}\). Each of these resulting feature tensors \({}^{k}F\) is processed separately by the variance head \(\varphi^{(\Sigma)}\), yielding the variance features \({}^{k}S\), and by the mean head \(\varphi^{(\mu)}\), yielding the raw mean features \({}^{k}\bar{M}\), which are divided element-wise by the variance features to obtain the mean features \({}^{k}\bar{M}\). The mean and variance operations shown as ellipses are applied element-wise across the \(K\) versions of the raw mean vector \({}^{k}\bar{\mu}\), the mean vector \({}^{k}\mu\), and the variance vector \({}^{k}\Sigma\), all of which are obtained through global average pooling (GAP) from their precursors. Computing the variance of the raw mean vector in this way yields the estimate for model uncertainty \(\Sigma^{(M)}\). The mean of the mean vector yields the feature vector \(\mu\). The mean of the variance vector yields the estimate for data uncertainty \(\Sigma^{(D)}\). And finally, the _variance_ of the _variance_ vector yields the estimate for distributional uncertainty \(\Sigma^{(V)}\). Together, the feature vector and the data uncertainty estimate define the embedding distribution \(\mathcal{N}(\mu,\Sigma)\), where \(\Sigma=\text{diag}(\Sigma^{(D)})\). During training, we additionally need a classifier \(g\) to map the feature vector \(\mu\) to a categorical output \(y\), which encodes different individuals in the training dataset. The feature vector \(\mu\), the data uncertainty estimate \(\Sigma^{(D)}\), and the logits \(y\), all contribute to the Loss \(\mathcal{L}\). The shape of all named Tensors is sketched above the architecture for the respective sections. Mathematical and algorithmic details are explained in Section 3.3.  
modification. It is essentially a three-layered bottleneck block (Figure 1(a)), whose weights \({}^{k}\theta\) are sampled \(K\) times for each input, as described in Algorithm 1. The resulting \(K\) versions of the module each process the output of the backbone \(F\) to compute a feature tensor \({}^{k}F\in\mathbb{R}^{d\times 16\times 8}\) of the same size (Equation (4)). \[{}^{k}F=\psi_{\theta}(F),\quad k\in\{1,\dots,K\} \tag{4}\]  
The weight sampling can be more easily understood as dropping each entry of the full weight tensors \(\widehat{\theta}\), which are learned by ordinary backpropagation, to 0 with probability \(p=0.3\), which is known as DropConnect. The number of weight samples \(K\) is 1 during training but is increased to 30 during inference which then allows the uncertainty estimation. This is identical to the hyperparameters used by [PERSON] et al. (2018).