## 4 Training Method Design
### Stages
Thus, the following combined loss function is suggested for supervised learning  
\[\mathrm{L_{S}}=\lambda_{\mathrm{SCE}}\,\overline{\mathrm{L_{SCE}}}+\lambda_{ \mathrm{CCL}}\,\,\overline{\mathrm{L_{CCL}}}+\lambda_{\mathrm{CSD}}\,\, \overline{\mathrm{L_{CSD}}}+\lambda_{\mathrm{FSD}}\,\,\overline{\mathrm{L_{ FSD}}}+\lambda_{\mathrm{D}}\,\,\overline{\mathrm{L_{D}}}, \tag{4}\]  
where \(\,\overline{\mathrm{L_{SCE}}}\) and \(\,\overline{\mathrm{L_{CCL}}}\) represent the values of the respective loss functions \(\mathrm{L_{SCE}}\) and \(\mathrm{L_{CCL}}\) averaged over S-sections and H-levels of the hierarchy of classes of the classification model;  
\(\overline{\mathrm{L_{FSD}}}\) and \(\,\overline{\mathrm{L_{CSD}}}\) represent the values of the respective loss functions \(\mathrm{L_{FSD}}\) and \(\mathrm{L_{CSD}}\) averaged over \(\mathrm{(S-1)}\)-sections and H-levels of the hierarchy of classes of the classification model;  
\(\overline{\mathrm{L_{D}}}\) is averaged by S-outputs (sections) of the classification model, including the output of the last layer and the value of the loss function \(\mathrm{L_{D}}\);  
\(\lambda_{\mathrm{SCE}},\,\lambda_{\mathrm{CCL}},\,\,\lambda_{\mathrm{FSD}}, \,\lambda_{\mathrm{CSD}},\) and \(\,\lambda_{\mathrm{D}}\) are the coefficients for regulating the influence of the components of the resulting loss function. The classifier module is proposed to be built on the basis of class prototypes, which are the centers of hyperspherical containers. In this case, the cross-entropy loss \(\mathrm{L_{SCE}}\) is calculated at the output of the classifier by the formula  
\[\mathrm{L_{SCE}}=\mathrm{CE}(\mathrm{q}(\mathrm{z_{i}},\,\tau=1),\,\mathrm{y_{i}}), \tag{5}\]  
where \(\mathrm{CE}(\cdot)\) is the cross-entropy loss function;  
\(\mathrm{y_{i}}\) is a one-hot encoded vector of the target variable for the i-th input sample;  
\(\mathrm{q}(\cdot)\) is an assessment of the probability of belonging to the feature representation of the i-th sample to a particular class container, which for the k-th class is calculated by the formula  
\[\mathrm{q_{k}}(\mathrm{z_{i}},\,\tau)=\mathrm{softmax}(\mathrm{leaky\_relu}( \mathrm{\mu}(\mathrm{z_{i}})/\tau))_{\mathrm{k}}=\frac{\exp(\mathrm{leaky\_relu}( \mathrm{\mu_{k}}(\mathrm{z_{i}})/\tau))}{\sum_{\mathrm{c}=1}^{\mathrm{K}}\exp( \mathrm{leaky\_relu}(\mathrm{\mu_{c}}(\mathrm{z_{i}})/\tau))}, \tag{6}\]  
where K is a size of the class set;  
leaky_relu is an improved version of the ReLU function with a small slope (\(\mathrm{alpha}=0.02\)) for negative values. Contrastive-center loss function \(\mathrm{L_{CCL}}\), which impacts the optimization efficiency of each class prototype, is calculated on a labeled training set by the formula [10]  
\[\mathrm{L_{CCL}}=\frac{\mathrm{dist}\big{(}\mathrm{z_{i}},\,\mathrm{\bar{z}_{ y_{i}}}\big{)}}{\sum_{\mathrm{k}=1,\,\mathrm{k}\
eq\mathrm{y_{i}}}^{\mathrm{K}} \mathrm{dist}(\mathrm{z_{i}},\,\mathrm{\bar{z}_{k}})+1}. \tag{7}\]  
Self-knowledge distillation components, such as \(\mathrm{L_{FSD}}\) in the form of \(\mathrm{L_{2}}\) loss from hints and \(\mathrm{L_{CSD}}\) in the form of Kullback-Leibler divergence loss, are calculated based on the S-th (last) output of the model and the s-th output (intermediate) of the model  
\[\mathrm{L_{FSD}}=\mathrm{dist}\Big{(}\mathrm{z_{i}^{s}},\,\mathrm{z_{i}^{S}} \Big{)}, \tag{8}\]  
\[\mathrm{L_{CSD}}=\mathrm{KL}(\mathrm{q}(\mathrm{z_{i}^{s}},\tau),\,\mathrm{q}( \mathrm{z_{i}^{S}},\tau)), \tag{9}\]  
where \(\mathrm{z_{i}^{s}},\,\mathrm{z_{i}^{S}}\) is the feature representation at the output of the s-th section (intermediate output) of the model and at the S output (last output) model. A regularization component \(\mathrm{L_{D}}\) implements the information bottleneck by penalizing the discretization error of feature representation  
\[\mathrm{L_{D}}=\mathrm{z_{i}}^{\mathrm{T}}(\mathrm{e}-\mathrm{z_{i}}), \tag{10}\]  
where \(\mathrm{e}\) is a unit vector. To speed up adaptation to changes, unlabeled data examples can be used in consistency regularization [50].