## 2 Materials and Methods
### Convolutional Feature Extraction  
After the egg detection step, each cropped egg patch was fed into a convolutional neural network to extract the image features for egg classification. To train such a network, we needed to overcome the limitation of the small and imbalanced training dataset. Our study involved only a few hundreds of samples of zebrafish eggs, which were not enough for training a deep neural network. Compared to the popular ImageNet [14] dataset of over ten million sample images, the size of our datasets is at least four orders of magnitude less. When the number of weights to be trained in a neural network is far more than the number of training samples, the problem of overfitting is likely to occur, and the network will have poor generalization ability. Another problem with our training set is that the sample numbers of different categories were seriously imbalanced. The ratio between fertilized and unfertil eggs was almost 6:1 in our dataset. Imbalanced training data could potentially diminish the specificity of the network, making the network incompetent to recognize the relatively smaller category, i.e., the unfertilized eggs. To overcome the limitation of the small and imbalanced training set, we used the image augmentation strategy to increase the training set size and to balance the training sample numbers of different categories. Image augmentation is the process to increase the training set by creating altered versions of the existing sample images, and it is proved to be an effective solution to prevent overfitting [14]. Typical ways of data augmentation include rotation, translation, zooming, flipping, scaling, color perturbation, and adding random noise. Figure 3: The workflow of egg detection. (**a**) The egg template is created by averaging K patches of egg samples; (**b**) The egg template is rotated into different directions, and each rotated patch is moved through the target image to find its matched egg. In our study, image augmentation strategies were carefully chosen according to the characteristics of our datasets (as shown in Figure 4). Since the eggs were captured at the same time point, they had similar sizes. All the images were captured under the same environmental light condition so that the grayscale level of different eggs was similar. The most possible variation of the eggs is the different orientation caused by random placement. Therefore, we used image rotation and flipping to simulate possible deviation of egg orientations. In order to improve the balance of the dataset, we augmented the unfertilized eggs more than the fertilized eggs. Each fertilized patch was rotated three times with 60 degrees interval, while each unfertilized patch was rotated 18 times with 10 degrees interval. All the rotated patches were also flipped vertically to simulate the effect of different illumination orientations of the environmental light. As a result, the ratio between fertilized and unfertilized eggs was close to 1:1 after the augmentation. Based on the augmented training dataset, a convolutional neural network was trained. The network used the architecture of VGG-16 [15], the winner of the 2014 Large Scale Visual Recognition Challenge (ILSVRC). As shown in Figure 5, this architecture consisted of 5 blocks of 13 convolutional layers. For each convolutional layer, a convolution kernel of size \(3\times 3\) was convolved with the layer input to produce a tensor of outputs. The output tensor of the convolutional layer was then transferred into a finite value by an activation function of Rectified Linear Unit (ReLu), i.e., F(\(x\)) = \(x\) for \(x>0\) and 0 otherwise. At the end of each block of the convolutional layers, there was a max-pooling layer to perform down-sampling by dividing the output feature map from each block into \(2\times 2\) pooling regions and computing the maximum of each region. The down-sampled feature map from each max-pooling layer was then fed into the next convolutional layers as an input.