# Helping the Visually Impaired See via Image Multi-labeling Based on SqueezeNet CNN
## 2 Materials and Methods
### Image Multi-Labeling Based on a Pre-Trained CNN  
Typically, early deep leaning solutions use a pre-trained CNN as an extractor of high level features. The complete learning model then uses extra network layers (usually fully connected layers) to adapt the network to another problem. Given the deep nature of a CNN, one can extract features at different layers. Thus, given an image \(\mathbf{I}_{i}\), we can generate a feature representation vector \(\boldsymbol{x}_{i}\in\mathcal{R}^{D}\) of dimension \(D\) as follows:  
\[\boldsymbol{x}_{i}=\mathrm{f}_{k}^{\mathrm{CNN}}\big{(}\dots\mathrm{f}_{2}^{ \mathrm{CNN}}\big{(}\mathrm{f}_{1}^{\mathrm{CNN}}(\mathbf{I}_{i})\big{)} \big{)},\ i=1,\dots,n \tag{1}\]  
where k is a layer within the network. For example, we can use the output of the hidden fully connected (FC) layer just before the last output layer (which is used to produce a classification result in the original CNN framework), to represent the training and test images. In this case, k = L-1, where L is the total number of layers in the network. Next, the extracted feature is fed into a fully connected (dense) layer that has free trainable weights that can be trained on the new dataset at hand. Figure 2 illustrates this typical architecture. Figure 1: Overview of the image multi-labeling problem. In the extra hidden FC layer, the input \(\mathbf{x}_{i}\in\Re^{D}\) is mapped to the hidden representation \(\mathbf{h}_{i}\in\Re^{S}\) of dimension \(S\) through the nonlinear activation function \(f\) as follows:  
\[\mathbf{h}_{i}=\text{f}(\mathbf{W}^{(h)}\mathbf{x}_{i}+\mathbf{b}^{(h)}\ \text{)} \tag{2}\]  
where \(\mathbf{W}^{(h)}\in\Re^{S\times D}\) is the weight matrix, and \(\mathbf{b}^{(h)}\in\Re^{S}\) is the bias vector. Similarly, the output vector \(\mathbf{y}_{i}\), is computed through a nonlinear activation function F as follows:  
\[\mathbf{y}_{i}=\text{f}(\mathbf{W}^{(o)}\mathbf{.}\mathbf{h}_{i}+\mathbf{b}^{(o)}\text{)} \tag{3}\]  
Typical choice of the activation function is the sigmoid function, i.e., \(\text{f}(v)=1/(1+\exp(-v))\), or the more advanced functions called ReLU or LeakyReLU shown in Figure 3.  
In order to utilize this pre-trained CNN to solve a new classification problem, we need to build the output layer so that it has as many neurons as the number of classes (typically the original CNN has an output layer with size 1000). Furthermore, for a typical classification problem, the output layer uses a softmax activation function and the categorical cross entropy loss function. Given the outputs \(\mathbf{h}_{i}^{(1)}\) of the last layer before the output layer, the softmax activation function produces an estimate of the posterior probability for each class label \(c=1,2,\ldots,C\) as in Equation (4):  
\[p\Big{(}\hat{y}_{i}=c|\mathbf{x}_{i}\Big{)}=\frac{\exp\!\left(\!\left(w_{c}^{(2)} \right)^{\text{T}}\!\mathbf{h}_{i}^{(1)}\!\right)}{\sum_{j=1}^{C}\exp\!\left(\! \left(w_{j}^{(2)}\right)^{\text{T}}\!\mathbf{h}_{i}^{(1)}\!\right)} \tag{4}\]  
Figure 3: State-of-the-art activation functions. (**a**) ReLU activation function; (**b**) LeakyReLU activation functions. Figure 2: Feature extraction using a pre-trained Convolutional Neural Network (CNN) model, where a new fully connected layer followed by a softmax layer with C neurons (equal to number of classes) are added for learning. where \(\mathbf{W}^{(2)}=\left[\mathbf{w}_{1}^{(2)}\ \mathbf{w}_{2}^{(2)}\dots\mathbf{w}_{C}^{(2)}\right] \in\mathbf{\Re}^{d\times C}\) are the weights of the softmax layer and the superscript \((\cdot)^{\text{T}}\) refers to the transpose operation. This softmax activation function turns the outputs of the neurons in the last layer into probabilities by ensuring that their total sum is equal to one. This is important for classification problems because we expect only one neuron to output one (because each input image must be assigned to one class) while the rest will be zero.