dataesr/scientific-paragraphs-categorization huggingface dataset:
, x N so that the elements of the N-dimensional output vector lie in the range [0,1] and sum to 1. It is defined for the ith element of X as:  On the other hand, among the proposed solutions to avoid the problem of limited training data and to improve the training of CNNs, there is a data augmentation technique that aims to artificially enlarge the training data. This technique is introduced to obtain better generalization by applying transformations on the existing data. In related works, some works applied data augmentation on all training data, even when the dataset is imbalanced. Others enlarged the training data for classes whose sample number is less than a threshold in order to balance the dataset. We propose, in this work, a new criterion for enlarging training data based on training and validation loss curves.
First, we train the model on the whole dataset and calculate, in parallel, the loss for each class. Then, we inspect training and validation loss curves for each class. Then, we augment data only for classes whose loss curves are not converged. For example, Figure  4  illustrates two train/validation loss curves for two different classes. When the difference between train and validation loss curves is low, we do not augment data for this class (Figure  4a ); on the other hand, we augment data only when the difference becomes higher (Figure  4b ). So f tmax(x i ) = exp (x i ) ∑ N j exp(x j ) (1.
Root node training: First, the root node is a pretrained CNN such as AlexNet  [14] , VGGNet  [41] , GoogleNet  [42] , ResNet  [43] , or ResNeXt 101  [33]. In this work, we use ResNeXt 101. It is efficient due to deeper layers and using the cardinality dimension, which make this network the 1st Runner Up of the ILSVRC 2017 classification task. So, we retrain ResNeXt on the examples of family taxon to extract the global features that are the input to each leaf node. This allows the leaf nodes to focus more on training the local features of each species. This root node uses the Softmax layer to learn the correlation between the input examples and classify them into the family taxon. At the end of this step, the parameters of the convolutional layers of this node are kept unchanged. The root node uses the same cross entropy loss defined in Equation (  2 ) considering N the number of families in the dataset.
Results We use two benchmark underwater datasets for evaluating the effectiveness of the proposed approaches for fish species classification. The two datasets contain images of fish of different colors, textures, positions, scales and orientations. Both are issued from the European project Fish4Knowledge (F4k) (www. fish4knowledge. eu) [48]. During this project of five years, a large dataset of over 700,000 unconstrained underwater videos with more than 3000 fish species were collected in Taiwan, the largest fish biodiversity environment in the world. We used a computer system equipped with an Intel Core-i5 processor with Geforce GTX 1050 Ti GPU and 2 Go GPU memory. We implemented the proposed approaches in Python using Pytorch. In Section 3. 1, we describe both benchmark datasets used in this paper.
The Fish Recognition Ground-Truth (FRGT) Benchmark Dataset The FRGT dataset is an underwater live fish dataset that contains 27,370 fish images of 23 different species and their masks. The fish species are manually labeled by following instructions from marine biologists. Figure  6  shows examples of the 23 fish species and Table  1  gives the distribution of the fish species in the dataset. We note that the distribution of the dataset is imbalanced as the number of majority class instances is about 1000 times more than minority class instances. The fish images in this dataset have various sizes ranging from about 20 × 20 to about 200 × 200 pixels. There is no test set in this dataset; in order to evaluate the performances of the proposed methods, we use 7-fold cross-validation as in  [7, 25].
LifeClef 2015 Fish (LCF-15) Benchmark Dataset The LCF-15 is an underwater live fish dataset. The training set consists of 20 annotated videos and more than 22,000 annotated sample images. In this dataset, we have 15 different fish species. Figure  7  shows examples of the 15 fish species, and Table  2  gives the distribution of the fish species in the dataset. Each video is manually labeled and agreed on by two specialist experts. The dataset is imbalanced in the number of instances of different species ; for example, the number of the species 'Dascyllus reticulates' is about 40 times more than the species 'Chaetodon speculum'. Like the FRGT dataset, the fish images also have various sizes ranging from about 20 × 20 to about 200 × 200 pixels. In experiments, we divide this training set into two subsets: 80% for training and 20% for validation (Table  2 ).
The test set has 73 annotated videos. We note that for three fish species there are no occurrences in the test set (Table  2 ). This is conducted to evaluate the method's capability to reject false positives. Compared with the first dataset, LCF-15 dataset provides challenging underwater videos marked by more noisy and blurry environments, complex and dynamic backgrounds and poor lighting conditions  [34]. Finally, we note that fish images can be extracted from videos using the available ground-truth of fish bounding boxes.
Evaluation Metrics We evaluate the effectiveness of both our approaches by adopting three measures: Precision of each class (P i ), where i is the index of class, Average Precision (AP) and Average Count or Accuracy (AC): P i = TruePositive i TruePositive i + FalsePositive i (4) AP = 1 N N ∑ i=1 P i (5) AC = ∑ N i=1 TruePositive i ∑ N i=1 (TruePositive i + FalsePositive i ) (6) where N refers to the number of dataset classes.
CNN Transfer Learning and Targeted Data Augmentation Approach In this section, we evaluate the proposed approach of CNN transfer learning on both datasets with and without targeted data augmentation. First, we note that inspecting training/validation loss curves is not enough to conclude if the model has learned all classes well. Figure  8  illustrates the loss function per epochs of fine-tuning ResNeXt on FRGT (a) and LCF-15 (b) datasets. We can observe that the model has globally well converged and does not suffer from overfitting. In order to define classes that need to enlarge their training data, we inspect training/validation loss for each species, as illustrated in Figure  9  for the FRGT dataset. 3  show results of three categories of species into the FRGT dataset. The first category contains the more representative species (DR, PD, CC, AC and CL).
We proceed with the same technique for the LCF-15 dataset. We inspect training/validation loss for each species and we obtain only two categories: species with well-converged training/validation loss curves (AC, CL, CS, DA, DR, HM and PV) and species whose training/validation loss curves suffer from irregularities (AN, AV, CC, CT, MK, NN, PD and ZS). We proceed with data augmentation for the second category of species. Figure  10  illustrates the confusion matrix of our approach on the test set of the LCF-15 dataset before and after applying targeted data augmentation. Each row of the matrix represents the instances in a target class, while each column represents the instances in a predicted class. To perform data augmentation, we proceed as follows. We flip each fish sample horizontally to simulate a new sample where a fish is swimming in the opposite direction; then, we scale each fish image to different scales (tinier and larger).
We also crop the images by removing one quarter from each side to eliminate parts of background. Finally, we rotate the fish images with angles -20 • , -10 • , 10 • and 20 • for the invariant rotation fish recognition issue. From the loss curves, after applying the targeted data augmentation technique (Figure  9d ), we can observe that this technique reduces overfitting and improves generalization. As a result, the performances of the model were improved. For the FRGT dataset (Table  3 ), the precision was significantly improved for some difficult species: AN is improved by 14. 31%, ZS by 2. 86% and NN by 21. 87%. For the LCF-15 dataset (Figure  10 ), the precision of some species are improved, such as for AN by 38. 99%, AV by 22. 93%, CC by 19. 99% and NN by 12. 39%. Table  4  shows the effects of targeted data augmentation on the AC and AP measures on both datasets.
With targeted data augmentation, we reach the highest accuracy and precision on both datasets. For the FRGT dataset, we achieve AC and AP of 99. 86% and 98. 59%. We have, respectively, AC and AP of 78. 47% and 69. 46% on the LCF-15 dataset. We conclude that generic input image transformations such as flipping, cropping, scaling and rotation are helpful for reducing overfitting on some classes and may substantially improve generalization.
Hierarchical Classification Approach We evaluate our hierarchical classification approach on the LCF-15 dataset, where accuracy was not high enough with CNN transfer learning. Figure  11  illustrates the taxonomic fish classification of the LCF-15 dataset. We can see that the 15 species in this dataset can be grouped into 6 families. The Pomacentridae family is the largest one, it contains 7 species. The Chaetodontidae family contains 3 species and the Acanthuridae family contains 2 species. Finally, the Holocentridae, Labridae and Pempheridae families contain one species each. The root node therefore has 6 outputs corresponding to the 6 families. In a hierarchical classification, the first nodes must perform well because if they misclassify the input images from the beginning, the child nodes will also misclassify them.
Finally, all fish of the Holocentridae, Labridae and Pempheridae families are well-classified. So, we have in this dataset a misclassification between some families. In order to improve the classification performance, we add a class 'Others' in the leaf nodes of the Acanthuridae, Chaetodontiade and Pomacentridae families. The class 'Others' contains fish samples from the Pomacentridae family for the nodes Acanthuridae and Chaetodontidae and contains fish samples from Acanthuridae and Chaetodontidae for the node Pomacentridae. The results are reported in Table  6. Figure  13  shows the confusion matrices for each leaf node. From Table  6  and Figure  13 , we see that the performance of the Acanthuridae node is the worst, but looking at the confusion matrices of the nodes, we notice that the classifiers identify their own species very well but misclassify species of the 'Others' class.
Discussion and Comparative Study Nowadays, underwater video systems are largely used by marine ecologists to study the biodiversity in underwater environments. Indeed, traditional techniques are destructive, affect fish behavior and demand time and labor costs. Many works proposed convolutional neural networks for fish species identification. However, these networks require large datasets due to the huge number of parameters to be trained, especially in deeper networks. In many real applications, there are not sufficient annotated training data, particularly in some specific tasks such as fish species identification. However, annotated training data is widely available for general computer vision tasks such as object recognition. Transferring network parameters learned on a general task to a specific task can solve the problem of lack of data, memory and processor resources, leading to better results.
In this study, we proposed to use the power of CNN transfer learning to recognize fish species in unconstrained underwater environments with low-resolution video images. Underwater environments are difficult and pose great challenges for computer vision due to luminosity condition, visibility, water turbidity and complex backgrounds together with the free movement of fish, fish overlapping and similarity in shape and patterns among fish. Training a deep network from scratch with insufficient training data of low contrast and resolution undoubtedly results in poor performance. In  [25] , the authors trained the AlexNet architecture from scratch on the FRGT benchmark dataset, and they achieved a low AP value of 48. 55%. Even with data augmentation, they reached an AP value of 61. 54%, which is far from enough.
Using transfer learning with data augmentation, they significantly improved the AP value to 99. 64%  [25]. This essentially shows the effectiveness of transfer learning for fish species recognition in underwater imagery. The most important contribution of our work is the high accuracy of fish species identification we reach. With only using transfer learning, we are able to achieve an AC value of 99. 21% on the FRGT benchmark dataset and an AC value of 76. 89% on LCF-15 benchmark dataset. Data augmentation techniques aim to expand training data by applying some transformation methods on these data in order to improve the generalization and regularize the learning models. Transformation techniques include a range of image processing operations such as flipping, cropping, shifting, rotation and much more.
This means we can simulate new and plausible fish samples at different scales, directions and luminosities. For example, the horizontal flip of fish samples produces a new fish sample swimming in the opposite direction. In  [25] , authors applied data augmentation on all training fish data, although the dataset is imbalanced. In  [7] , authors enlarged the training data for species whose sample number is fewer than 300. The latter solution may be not necessary for classes that are easy to be recognized. For these reasons, we proposed to inspect training and validation loss curves for each class to distinguish the difficult classes (Figures 9); we could differentiate between easy and difficult species. The loss curves for easy species have well converged without overfitting, while the loss curves for difficult species are not fully converged.
So, applying data augmentation on these species can reduce overfitting and improve generalization. We reached an AC value of 99. 86% and 78. 47% on FRGT and LCF-15 datasets, respectively. To overcome the problem of flat classification, one possible solution is to organize datasets hierarchically. In the fish recognition task, Huang et al. [4]  proposed a Balance-Guaranteed Optimized Tree (BGOT) approach. This approach is designed to use the interclass similarities. BGOT uses hand-crafted methods to extract features and binary SVM classifiers and recursively chooses the best binary splitting by exhaustively searching all possible combinations. In our proposed CNN hierarchical classification approach, we divided the fish dataset hierarchically by using taxonomic fish classification. Our approach classifies fish at the root node into the family level and then into species categories at leaf nodes.
Grouping fish that have similarities and common characteristics in the same subset then building a classifier for each subset improves classification performances. Indeed, the complicated task is divided in multiple easy subtasks. With this strategy, we improved the AC by 3. 90% and AP by 20. 79% on the LCF-15 dataset compared with the CNN transfer learning approach. Tables  7  and 8  show the comparison performances of our proposed approaches with the state-of-the-art methods on LCF-15 and FRGT datasets, respectively. Sz űcs et al. [13]  used the sped-up robust features SURF method to extract features from fish images in order to feed an SVM classifier. Sun et al. [8]  applied two deep archi-tectures, PCANet  [49]  and NIN  [50]  to extract features from underwater images. A linear SVM classifier is used for classification.
Iqbal et al. [27]  proposed a modified AlexNet model. It is a reduced version of the AlexNet model comprised of four convolutional layers (instead of five convolutional layers) and two fully connected layers. The output of the first fully connected layer is of 9216 units instead of 4096 units as in the original AlexNet. Mathur et al. [31]  fine-tuned the ResNet50 model by retraining only the last fully connected layers without any data augmentation. They used Adamx as the optimizer. We implemented the modified AlexNet and FishResNet models by using the same provided parameters in their papers. We also implemented the approach of Jalal et al. [22]  based on the second branch of their hybrid approach, where they used YOLOv3 to detect and classify fish images. Zhang et al. [32]  proposed AdvFish, which addresses the noisy background problem.
They fine-tuned the ResNet50 model by adding a new term in the loss function. This term encourages the network to automatically differentiate the fish regions from the noisy background and pay more attention to the fish regions. Jäger et al. [24]  used features extracted from the activations of the 7th hidden layer of the pretrained AlexNet model and fed a multiclass SVM classifier. From Table  7 , we observe that approaches based on deep learning perform better than the hand-crafted method. We can also see that CNN trained without fine-tuning is not efficient (CNN-SVM  [24] : 66 %) compared with CNNs trained with transfer learning (ResNeXt 78. 47%). Our proposed hierarchical classification outperforms the-state-of-theart methods. These results confirm the challenging nature of this benchmark dataset that is marked by highly blurry images with background confusion with fish and higher degradation in terms of light intensity.
We note that Salman et al  [34]  reached an accuracy of 93. 65% by testing their model on 7500 fish images issued from the LCF-15 dataset, but these fish images are not from the original test set provided in the dataset. At the moment that they tested their method, they did not use original training or the test split provided in the LCF-15 dataset. Moreover, Jalal et al. [22]  did not use the original test set provided in the LCF-15 benchmark dataset. Instead, they merged the training and test sets, then they took 70% of the samples for training and 30% for testing. The test set of the LCF-15 benchmark dataset is highly blurry compared with the training set, which explains their high accuracy in their article compared with ours. From Table  8 , in Deep-CNN  [6] , a CNN with three convolutional layers is created and trained from scratch.
64%. We tested the approach of FishResNet  [31]  by using the same test set, we obtained 95. 62%. We used the provided code in AdvFish  [32] , and we trained ResNet50 by using 7-Fold cross-validation; we achieved 90. 99%. In our work, without data augmentation, we obtained an AC value of 99. 21% and AP value of 95. 38%. With targeted data augmentation only applied to species that are difficult to be recognized, and without using any other classifier, we obtained an AC of 99. 86% and AP of 98. 59%. We can also conclude that networks that are trained with transfer learning give better results than networks trained from scratch. An important remark is that with our targeted data augmentation approach precision for others species, especially less representative species, are improved (CT, MK, HF, NS, AV, CV, S, PV and ZC) or unchanged (PM, LF, SB and SF), even though we did not apply data augmentation on these species.
Conclusions In this paper, we presented two CNN approaches for the live fish recognition task in an unconstrained underwater environment. First, we proposed to transfer knowledge from the pretrained ResNeXt 101 network and retrained it on underwater fish image datasets. We have shown that the fine-tuned ResNeXt improves the model performances. We also analyzed the effects of data augmentation. As well, we have not augmented the data uniformly for all classes, however, we have proposed a targeted data augmentation technique based on training/validation loss curves for better performance. Second, inspired from taxonomic fish classification, we proposed a hierarchical classification that classifies fish first by family then by species. Experiments on two underwater live fish benchmark datasets, namely the Fish Recognition Ground-Truth dataset and LifeClef 2015 Fish dataset, demonstrated that our proposed approaches outperform various state-of-the-art methods for fish species identification and improved our previous results.
We generated the training data using thousands of GECKO-A box simulations sampled from a broad range of initial environmental conditions, and focused on three representative SOA precursors: the oxidation by OH of two anthropogenic (toluene, dodecane), and the oxidation by O 3 of one biogenic VOC (α-pinene). We compare several neural models and quantify their underlying uncertainty and robustness. These are promising results, suggesting that neural network models could be applied to predict SOA in chemistry-climate models, limited however to the range of environmental conditions that were considered in the training datasets. Plain Language Summary Detailed and accurate representation of organic aerosol chemistry is needed to predict the effect of atmospheric aerosols formed from natural and anthropogenic sources on both human health and climate.
In Section 3, we characterize the two models' performance relative to the GECKO-A data sets for different precursor species, and compared to each other to assess the overall strengths and weaknesses of each model architecture. Both model types are also tested on data sets that help assess the ability of the models to generalize into new domains. Sections 4 and 5 provide a brief discussion about the pros and cons of these different model architectures and the ongoing challenges regarding numerical stability, computational cost, and interpretability for emulating SOA production with ML.
GECKO-A Generated Training Datasets We ran the GECKO-A model for a variety of initial conditions and physical parameters to generate the data set used to train the machine learning emulator. At this stage we focus on three representative SOA precursors: toluene, dodecane, and α-pinene. Toluene and dodecane are emitted from a wide range of anthropogenic sources, while α-pinene is one of the major SOA precursors emitted by vegetation. These compounds together contribute substantially to the global SOA burden. We generate chemical mechanisms and corresponding datasets for these precursors with their dominant oxidants, including the OH oxidation mechanisms of toluene, dodecane, and the ozonolysis mechanism of α-pinene. Reactions of the precursor with nitrate radical (NO 3 ) were not considered.
Hyperparameter Optimization At different stages in training an emulator model, from data post-processing to selecting an architecture, there are hyper parameters that need to be set that can affect the performance outcome of a trained model. They may include, for example, the learning rate used to update the model weights during training, or the size and number of the hidden layers in an MLP or GRU model. As the main objective is to minimize the difference between the model predictions and the test experiments in box simulations, we want to understand how the models performance depends on the hyper parameter choices. From such an understanding, an informed choice can be made in selecting potentially optimal parameter values. To estimate such a dependency, we use the package Earth Computing Hyperparameter Optimization (ECHO) developed by the authors at NCAR  (Schreck & Gagne, 2021) , and perform hyper parameter optimization given an objective metric for the three species for both MLP and GRU models.
GECKO-A Emulator Evaluation With External Datasets The performance abilities of both MLP and GRU models were tested by expanding the data sets to include additional simulations, (a) for 10 times (X10, = 100 ppt) and 100 times (X100, = 1 ppb) higher initial concentrations of the precursor, which are more representative of somewhat polluted atmospheric conditions, and (b) for simulating the diurnal variation in the precursor levels, that was not present in the original data sets which did not include the daily variability on the emissions.