# Photo Identification of Individual _Salmo trutta_ Based on Deep Learning
## 2 Materials and Methods  
### Identification Method  
Research in deep learning has contributed to advances in a number of applications; those related to this paper include animal tracking [35] and animal recognition [36; 37]. Our starting point was similar to recognition in other applications [38], where we extracted features from different layers of the encoder network. An encoder network is a deep neural network that takes an input image and generates a high-dimensional feature vector. The fish recognition model is shown in Figure 1. The input to the identification method was a pair of fish images (Fish A and Fish B in Figure 1). The encoder was composed of stacked convolutional layers. Encoder features were extracted with EfficientNet [39], since networks that performed better on the ImageNet dataset were capable of learning better transferable representations [40]. The layers at the beginning of the encoder network captured primitive image features, such as edges and textures, while deeper layers capture rich global information. To further explore the spatial features and improve fish recognition performance, the automated identification method needs to incorporate high-level information about the fish and semantic information in the scene, and low-level information about the details in the textures, patterns, as well as an integration of various contexts extracted at different scales. Bidirectional feature pyramid networks (BiFPNs) [41] were extracted with the EfficientNet (\(\phi=0\)) encoder network (Figure 1). BiFPN leveraged a convolutional neural network (CNN) to extract bidirectional feature maps with different resolutions. The feature maps in earlier layers captured the textural and color detail information in the local regions, while the feature maps in deep layers captured the fish semantic information of the whole fish image. Based on the BiFPN representations of a fish, the problem of fish identification was reduced to a problem of matching low-level and high-level features between the two input images. It is necessary to identify a given fish correctly when it swims from right to left with viewpoint, scale, size, and appearance variations (Figure 2). For this, we employed the cosine similarity \(\phi_{i,j}\) between the corresponding BiFPN of the left and right input images (Figure 1). Cosine similarity has been shown to be effective in [42]. We used fish \(i\) and \(j\) to explain how the distance features were computed. Given the encoder network, BiFPN features at scale \(s\) were extracted for fish \(i\), \(f_{i}^{s}\), and \(j\), \(f_{j}^{s}\) (Figure 1). The size of each feature \(f\) depended on the depth of the network and we used the last five features for compact representation. Given the above notation, the cosine distance feature at each scale was given by Equation (1):  
\[\phi_{i,j}^{s}=\frac{\langle f_{i}^{s},f_{j}^{s}\rangle}{\langle f_{i}^{s},f_{ i}^{s}\rangle^{\frac{1}{2}}\langle f_{i}^{s},f_{j}^{s}\rangle^{\frac{1}{2}}} \tag{1}\]  
where \(\phi_{i,j}^{s}\) was the spatial cosine distance at each scale \(s\). We concatenated the corresponding cosine distance from each scale \(\Phi_{i,j}=\{\phi_{i,j}^{0}\}_{s=0}^{s=4}\) to form the final fused correspondence representation \(\Phi_{i,j}\). We applied fully connected layers to encode the correspondence  
Figure 1: Fish recognition block diagram. Given input images A and B, image features were extracted at different layers of the deep neural network. The similarity between Fish A and B features was computed using the cosine similarity metric \(\phi\), which indicated the feature distance between Fish A and B at each scale of the network. FC was a fully connected layer followed by rectified linear activation function (ReLU). The output of ReLU was passed through a Sigmoid function to give a value between 0 and 1, indicating non-identical or identical fishes, respectively. representation \(\Phi_{i,j}\) with a vector of size \(64\times 4\) and was then passed to a sigmoid layer.