# Classification and Fast Few-Shot Learning of Steel Surface Defects with Randomized Network
## 3 Proposed Methods
### Few-Shot Learning of Defects
#### 3.2.1 Randomized Weights for Generalization and Fast Tuning  
It is well known that randomizing weights in neural networks can result in improved accuracy. In ref. [50], the input data of a single (hidden)-layer feed-forward neural network (SLFN) were weighted with random weights. Then, in the output layer, a fully connected network with bias was applied, where its weights could be calculated by solving a linear set of equations by standard numerical methods. In ref. [51], the random vector functional links network (RVFL) was proposed, where beside the input patterns, their randomized version is also generated and fed to the output using the standard weighted connections. Extreme learning machines (ELM) [52] have subtle variations to these randomized networks (no direct connection between inputs and outputs like RVFL, other usage of bias than in ref. [50]) but became more popular recently, in spite of reports that RVFL gives better performance than ELM in some cases [53]. For more information, we propose to read ref. [54], a review on neural networks with random weights.  
Figure 3: The proposed architecture (EffNet+RC) for few-shot learning. In phase 0, we train the backbone through a large number of samples of base classes. In phase 1 (and further phases), we use features extracted by the previously trained backbone. Here, only the weights W are computed with the help of a few samples, while weights R are random and fixed.  
A formal description of the applied randomized back-end network follows. Consider a set of \(N\) distinct training samples \((x_{i},y_{i})\), \(i=1,\ldots,N\). Then, a SLFN with \(L\) hidden neurons has the following output equation:  
\[\mathbf{t}(x_{i})=\sum_{j=1}^{L}\mathbf{w}_{j}\phi(\mathbf{r}_{j}\mathbf{x}_{i }+b_{j}), \tag{1}\]  
where \(\phi\) is a sigmoid function, \(\mathbf{r}_{j}\) are the random- and fixed-input weight vectors, \(b_{j}\) are the biases, \(\mathbf{w}_{i}\) are the output weight vectors to be tuned, and \(\mathbf{t}\) is the target vector (the outputs for the different classes). \(\mathbf{R}\) and \(\mathbf{W}\) in Figure 3 correspond to the weights here. Now, let us see how to compute \(\mathbf{W}\) if \(\mathbf{R}\) is set randomly. In practice, closed-form solutions are used to find \(\mathbf{w}_{i}\) in a matrix form. Thus, we can shorten the equation:  
\[\mathbf{T}=\mathbf{H}\mathbf{W}, \tag{2}\]  
as the outputs of all hidden neurons are gathered into the matrix \(\mathbf{H}\):  
\[\mathbf{H}=\begin{bmatrix}\phi(\mathbf{r}_{1}\mathbf{x}_{1}+b_{1})&\cdots& \phi(\mathbf{r}_{L}\mathbf{x}_{1}+b_{L})\\ \vdots&\ddots&\vdots\\ \phi(\mathbf{r}_{1}\mathbf{x}_{N}+b_{1})&\cdots&\phi(\mathbf{r}_{L}\mathbf{x} _{N}+b_{L})\end{bmatrix}, \tag{3}\]  
given \(\mathbf{W}=\left(\mathbf{w}_{1}^{T}\cdots\mathbf{w}_{L}^{T}\right)^{T}\), and \(\mathbf{T}=\left(\mathbf{t}_{1}^{T}\cdots\mathbf{t}_{N}^{T}\right)^{T}\).  
A unique solution for this system can be given by using the Moore-Penrose generalized inverse (pseudoinverse) [55] of the matrix \(\mathbf{H}\), denoted as \(\mathbf{H}^{\dagger}\). From Equation (2):  
\[\mathbf{W}=\mathbf{H}^{-1}\mathbf{T}. \tag{4}\]  
To find the \"best fit\" (least squares) solution to the system of linear equations the pseudoinverse is computed [55]:  
\[\mathbf{H}^{\dagger}=(\mathbf{H}^{T}\mathbf{H})^{-1}\mathbf{H}^{T}. \tag{5}\]  
Finally, we get the weights:  
\[\mathbf{W}:=\mathbf{H}^{\dagger}\mathbf{T}. \tag{6}\]  
This explicit calculation of weights enables us to achieve an instantaneous fine-tuning of the architecture for new incoming classes in incremental learning.