dataesr/scientific-paragraphs-categorization huggingface dataset:
Sensors 2021, 21, 7268 By using key feature detection within the bone image mask, distances between ischium and femur bone cross-sections were calculated to determine the geometric phenotype of gigot length and width. Phenotype values determined using the computer vision pipeline were on average 101. 4 % that of the ground truth value with a standard deviation of 12. 90% (n = 32), indicating a high level of accuracy across the population. One of the potential limitations of this study was the small training dataset (m = 126) as development of neural networks typically uses datasets numbering in the thousands. However, this limited dataset did not cause any major issues in accuracy as ground truth and predicted images showed an SSI of 0. 98 ± 0. 0035 and were indistinguishable by eye. One possible reason for such high accuracy with this limited dataset was that all subjects within the CT scans were constrained to similar postures.
This hypothesis was later confirmed by re-introducing artificial random movement (such as rotation or vertical/horizontal shifts) into the images used for GAN training, resulting in a higher validation loss, poorer network performance and blurry resultant images (Supplementary Figures  S1-S9 ). Unfortunately, using this limited dataset resulted in one of the unseen images containing a small amount of testis tissue following processing with the GAN which was then incorrectly quantified as fat tissue. In the future, as more images are integrated into the model, we believe that the accuracy of the GAN shall improve which shall directly improve the precision of the CV phenotype determination pipeline.
Conclusions In summary, we believe this research represents the first case of using an automated phenotype detection pipeline on agricultural animal medical images. This was achieved by using a combined GAN-CV pipeline to analyse agricultural medical images in a fully automated fashion. By feeding a paired image dataset into a GAN, we were able to perform the various image processing steps needed to produce a predicted image, containing only the relevant tissues, with accuracies of 98% which rivalled that of manual processing and at a fraction of the cost. Phenotypes were then extracted or calculated from these predicted images by applying CV techniques as part of an automated pipeline. We hope to immediately expand this highly accurate GAN-CV pipeline to process and extract phenotypes from other key CT scan sections such as the 8th thoracic vertebra and 5th lumbar vertebra positions.
Data Labelling Labelme software (https://github. com/CSAILVision/LabelMeAnnotationTool (accessed on 22 October 2020) was used to annotate the data and build data sets in COCO format. Because the activities of the cows led to different degrees of deformation of their tags, the polygon mode was selected to label the target in an image. For the cascaded instance segmentation model in this paper, two steps of image annotation were required. Step (1): The tags in the original image were labelled to train and test the ID tag detection model. Step (  2 ): The detection model trained in step (1) was used to detect ID tags in the training set and crop the detected bounding boxes. The cropped images were labelled and taken as the training set for the segmentation model. We performed the same for the validation set and test set of the detection model to obtain the validation set and test set of the segmentation model, respectively.
According to the statistics of the tag size in the scaled image, the anchor sizes were determined to be 4, 8, 16, 32, and 64. The K-means clustering algorithm was used to calculate the anchor ratios suitable for our dataset, which were (0. 7, 1. 4), (1. 0, 1. 0), and (1. 4, 0. 7). The same training environment and training parameters were used to train Efficient-D0-D5 based on the pretraining model. After training, the performance of different EfficientDet models was evaluated with test sets. The training set and validation set constructed in step (2) of Section 2. 2 were used to train the ID tag segmentation model YOLACT++. SGD was selected as the optimizer for model training, and the batch size was set to 4. The initial learning rate was set to 1 × 10 -4. The weight decay coefficient and the momentum coefficient were set to 1 × 10 -4 and 0.
The character recognition model is constructed based on LeNet-5  [34]. We changed the C5 layer of LeNet-5 from fully connected layer to convolution layer to obtain the character recognition model. The reason for this change is to reduce the redundant parameters and enrich the features. Table  2  shows that about 76% of ID tags in the dataset are rotated and twisted, so the corresponding detected bounding box will contain different background areas. In the binarization of pixels, the pale white body area of cows and grass in the background are misjudged as characters, which considerably interferes with the implementation of subsequent steps. If the brightness of the background exceeds that of the character, some characters will be lost due to the high threshold in the binary segmentation. Figure  15a  depicts the character segmentation results of partially unsegmented images, and Figure  15b  displays the character segmentation result of partially segmented images.
To do so, we firstly prepare the dataset from video sequences taken during the three days before calving. After collecting the video sequences, the target cow regions are manually extracted by Visual Geometry Group (VGG) annotator  [13]  to remove the background, and to obtain the cow contour regions by using image processing techniques. We used an approach based on statistical analysis to predict calving time. Cow behavioral activities were recorded by human observers performing a direct visual observation of each individual cow in the calving barn. The four types of conditions include two postures and two transitions. They are defined as follows.
From Figure  8  of the confusion matrices, the total number of observations is 360 in the testing dataset for 5 cows. In this dataset, the 2 classes are calving and not-calving, with a total of 15 calving responses and 345 not-calving responses. The best accuracy obtained using our proposed method is 100%. From Figure  8  of the confusion matrices, the total number of observations is 360 in the testing dataset for 5 cows. In this dataset, the 2 classes are calving and not-calving, with a total of 15 calving responses and 345 not-calving responses. The best accuracy obtained using our proposed method is 100%.
e. , extracting the same features without distinction of modalities, often caused interference between multimodal information due to their distribution gap  [18]. The result fusion scheme was suboptimal since rich modality information was gradually compressed and lost in separate processes, ignoring the intermodality correlations. As a better choice, the feature fusion strategy fuses the intermediate information of multiple modalities, which avoids the distribution gap problem and achieves intermodality interaction simultaneously  [19, 20]. However, feature fusion is often limited to linear fusion (e. g. , simple concatenation and addition) and fails to explore deep multi-modality interactions and achieve complementary-redundant information combinations between multiple modalities  [17]. In addition, the collected sensor datasets often present class imbalance problems due to the inconsistent frequency and duration of each activity resulting from specific Sensors 2021, 21, 5818 animal physiology.
Deep learning methods trained on imbalanced datasets tend to be biased toward majority classes and away from minority classes, which easily causes poor modal generalization ability and high classification error rates for rare categories  [21]. Commonly used methods on imbalanced datasets mainly involve two techniques, namely, resampling and reweighting. Resampling attempts to sample the data to obtain an evenly distributed dataset, e. g. , oversampling and undersampling  [22]. However, oversampling and undersampling come with high potential risks of overfitting and information loss, respectively  [21]. Reweighting is more flexible and convenient by directly assigning a weight for the loss function per training sample to alleviate the sensitivity of the model to data distribution  [23]. This method is further divided into class-level and sample-level reweighting.
The CMI-Net consisted of a dual CNN trunk architecture and a joint cross-modality interaction module (CMIM). Specifically, the dual CNN trunk architecture extracted modality-specific features for accelerometer and gyroscope data, respectively, and the CMIM based on attention mechanism adaptively recalibrated the importance of the elements in the two modality-specific feature maps by leveraging multi-modal knowledge. The attention mechanism has been widely utilized in different tasks using multi-modal datasets such as RGB-D images  [17, 29]. It has also been adopted to focus on important elements along with channels and spatial dimensions of the same input feature  [30, 31]. The favorable performance presented in these studies with the attention mechanism indicated the rationality of our proposed CMIM.
In our method, softmax cross-entropy (CE) loss was initially used to supervise the training of CMI-Net. However, softmax CE loss suffered from inferior classification performance, especially for monitory classes  [23]. In contrast, CB focal loss, by adding a CB term to focal loss, focuses more on minor-class samples and hard-classified samples and can alleviate the class imbalance problem. Therefore, a CB focal loss  [25]  was also adopted. In this study, the CMI-Net was trained based on an extensively labeled dataset  [32]  to automatically recognize equine activities including eating, standing, trotting, galloping, walking-rider (walking while carrying a rider), and walking-natural (walking with no rider). The leaveone-out cross-validation (LOOCV) method was applied to test the generalization ability of our model, and the results were then compared to the existing algorithms.
• We devised a novel attention module, i. e. , CMIM, to achieve deep intermodality interaction. The CMIM combined spatial information from two-stream feature maps using basic CNN to produce two spatial attention maps with respect to their importance, which could adaptively recalibrate temporal-and axis-wise features in each modality. To the best of our knowledge, the attention mechanism was employed for the first time in animal activity recognition based on multi-modal data yielded by multiple wearable sensors. • We adopted a CB focal loss to supervise the training of CMI-Net to mitigate the influence of imbalanced datasets on overall classification performance. The CB focal loss can pay more attention not only to samples of minority classes, diminishing their influence from being overwhelmed during optimization, but also to samples that are hard to distinguish.
Data Description The dataset used in this study was a public dataset created by Kamminga et al. [32]. In this dataset, more than 1. 2 million 2 s data samples were collected from 18 individual equines using neck-attached IMUs. The sampling rate was set to 100 Hz for both the triaxial accelerometer and gyroscope and 12 Hz for the triaxial magnetometer. The majority of the samples were unlabeled, but data from six equines and six activities including eating, standing, trotting, galloping, walking-rider, and walking-natural were labeled extensively (87,621 2 s samples in total) and were used to classify equine activities in previous studies  [7, 34]. In this study, data from the triaxial accelerometer and gyroscope among the 87,621 samples were exploited separately, forming up to two tensors with a size of 1 × 3 × 200 for each sample.
e. , NB, DT, and SVM) and two deep learning methods used in equine activity recognition (i. e. , CNN and ConvNet7)  [14, 15] , based on the same public dataset. Specifically, the hand-crafted features used in machine learning were the same as those used by Kamminga et al. [7]. To further explore the performance of our CMIM, we ran the network without CMIM and with it inserted after the 1st, 2nd, and 3rd max-pooling layers to obtain four different Sensors 2021, 21, 5818 variants, i. e. , Variant0, Variant1, Variant2, and Variant3, respectively. The softmax CE loss was used as the loss function for all variants. All experiments were executed using the PyTorch framework on an NVIDIA Tesla V100 GPU. The developed source code will be available at https://github. com/Max-1234-hub/CMI-Net from 1 September 2021.
Results and Discussion Overall, experiments conducted on the public dataset demonstrated that our proposed CMI-Net outperformed the existing algorithms. Ablation studies were then carried out to verify the effectiveness of CMIM and that applying the CMIM in the upper layer of CMI-Net could obtain better performance. Different loss functions were adopted to validate that CB focal loss performed better than any class-level or sample-level reweighted loss used alone, and it effectively improved the overall precision, recall, and F1-score, although the overall accuracy decreased due to the imbalanced dataset used. Furthermore, recognition performance analysis was presented to help us probe the predicted performance on each activity using our CMI-Net with CB focal loss. The details are described as follows.
& denotes all networks presented in this table were trained using softmax CE loss; # denotes the network without a cross-modality interaction module (CMIM); * denotes the network where the CMIM was inserted after the 1st, 2nd, and 3rd max-pooling layers, respectively. The results above have proven that the inclusion of the CMIM in the network provided quantifiable improvements in identification performance. This was also reflected in the qualitative visualization of the embeddings and the corresponding clusters in Figure  3 , with the help of t-distributed stochastic neighbor-embedding (t-SNE), a technique for visualizing high-dimensional data by giving each data point a location in a two-or threedimensional map  [42]. Figure  3  shows the two-dimensional embedded features from the part test dataset after the fully connected layers of both CNN branches under the network without and with CMIM by using the t-SNE technique with an init of 'pca' and perplexity of 30.
This explained that the overall classification performance increased mainly due to the increase in walking-natural, as it focused more on difficult samples and samples of minority classes. However, the overall accuracy of CMI-Net with CB focal loss decreased by 2. 69% (Table  3 ), which was related to the different variations of recall values in different activities and the current imbalanced dataset. In particular, the overall accuracy could also be presented as the weighted average of the recall value for each activity according to the sampling frequency of each activity. As shown in Figure  5 , the recall increases were 35. 92% for walking-natural, 1. 17% for standing, and 0. 91% for galloping, and the recall decreases were 8. 41% for walking-rider, 4. 26% for eating, and 0. 36% for trotting when using CB focal loss.
It can be observed that all activities with increased recall belonged to the minority class, while the remaining activities with decreased recall belonged to the majority class, resulting in a decrease in overall accuracy. Thus, it is necessary to collect a more balanced dataset in the future. In addition, experiments under different loss functions were conducted to verify the effectiveness of the CB focal loss, as illustrated in Table  4. The contrasting losses mainly included CS_CE loss, CB_CE loss, focal loss, and ACS loss, as mentioned in the "Optimization" section. We found that CB focal loss combining CB loss and focal loss performed better than any of them used alone, which indicated that adding the CB term to the focal loss function improved the overall classification performance on the imbalanced dataset.
Sensors 2021,  21, 5818  In addition, experiments under different loss functions were conducted to verify the effectiveness of the CB focal loss, as illustrated in Table  4. The contrasting losses mainly included CS_CE loss, CB_CE loss, focal loss, and ACS loss, as mentioned in the "Optimization" section. We found that CB focal loss combining CB loss and focal loss performed better than any of them used alone, which indicated that adding the CB term to the focal loss function improved the overall classification performance on the imbalanced dataset. In addition, the precision, recall, and F1-score of CS_CE loss and CB focal loss increased by different degrees, while both accuracies decreased compared with softmax CE loss. Specifically, the accuracy was only 83. 79%, although the recall reached the highest value of 85.
11%. This was because the recall of walking-rider was only 72. 49%, although that of walking-natural was 69. 16% (Figure  6 ). This result further verified that decreased accuracy occurred when using balancing techniques on the imbalanced dataset. In addition, we found that the recall of majority classes decreased while that of minority classes increased when using CS_CE loss and CB focal loss (Figure  6 ). This result revealed that both losses effectively focused on the samples of minority classes during training, but it is inevitable that more samples in majority classes were misclassified as minority classes so that overall accuracy would decrease.
Classification Performance Analysis In Figure  7 , we show the precision and recall confusion matrix aggregating the classification results under 6-fold cross-validation when using CMI-Net with CB focal loss (γ = 0. 5). Both precision and recall values of all activities had more than 90% accuracy (i. e. , the precision and recall for eating were 92. 86% and 90. 89%, for galloping were 91. 41% and 92. 89%, for standing were 95. 18% and 95. 11%, for trotting were 97. 34% and 97. 46%, and for walking-rider were 93. 49% and 90. 01%, respectively), except for the walking-natural activity, which only obtained low precision and recall (Figure  7 ). This low classification precision and recall occurred for two main reasons. The first reason was class imbalance. Walking-natural as the minority class in the dataset only occupied 3.
Limitations and Future Works The first limitation of our proposed method is that our model was trained on a public dataset that contained only six labeled activities, i. e. , eating, standing, trotting, galloping, walking-rider, and walking-natural. Indeed, there are some other activities such as head shaking, scratch biting, rubbing, and rolling, all of which, although infrequent, are physiologically critical to equine health and welfare, and should have been labeled and included in the dataset. Due to the missing of these infrequent activities in the dataset, inevitably, as a typical open-set recognition problem  [44] , these unlabeled activities that occur in real behavior monitoring scenarios will be easily misclassified as the six defined activities, resulting in loss of some key information.
Thus, as a next step to further improve classification performance for equine activities, we will investigate some feasible techniques such as classification-reconstruction learning and weightless neural networks  [44] [45] [46]  to enable our activity classifiers to not only accurately classify the defined classes appearing in training but also effectively deal with unlabeled ones generated in practice. The second limitation is that the algorithms we developed and adopted in this study were based on supervised learning, which relied on a large number of annotated samples. Data annotation is a labor-intensive and time-consuming task, and well-annotated data is often limited as reflected by the fact that we can only find one public dataset for equine activities. With regard to the found dataset  [32] , in fact, there are still vast amounts of unlabeled samples that can be used to alleviate the overfitting problem and improve the generalization ability of models.
Dataset Description A publicly available dataset of dairy cows was used in this research  [16]. The dataset contains data collected using three microphones (Nady 151 VR, Nady Systems, Oakland, CA, USA) attached to the foreheads of three 4-to 6-year-old lactating Holstein cows weighing 608 ± 24. 9 kg; and ingesting sounds (i. e. , chews, bites, and chew-bites) from four types of microswards were continuously recorded for five days. The microswards consisted of sets of 4-L plastic pots with either alfalfa (Meicago sativa) or tall fescue (Lolium arundinaceum, Schreb. ) with two heights, tall (24. 5 ± 3. 8 cm) or short (11. 6 ± 1. 9 cm). An illustration of the two forages used in the dataset is provided in Figure  1. The acoustic signals were saved as WAV files and labeled for observed ingestion behavior by the technicians from the same research team with a labeling agreement of over 99%.
A total of 52 labeled WAV files totaling 54 min 24 s were in the dataset, and the files were processed and segmented based on ingestive behaviors, forage species, and forage height, resulting in 3038 segments with a duration of 1647. 37 s being used for model evaluation and experiments (Table  1 ). Sample illustrations of acoustic signals in frequency and time domains are presented in Figure  2  for the three ingestive behaviors. The three behaviors showed apparent differences in temporal and spatial signal patterns, and the acoustic signals of the behaviors were mostly in low frequencies (<1 kHz)  [14]. Additionally, based on the observation reports from the technicians, each ingestive behavior event commonly lasted for less than 1 s  [16]. The recorded acoustic signals are in 16 bit sample depth format taken at a 22.
Statstical Analysis for Evaluating Effects of Forage on Acoustic Features Statistical analyses were conducted to investigate the relationships between forage species and heights and acoustic features of the ingestive behaviors. The data used for statistical analysis were all in the time domain. The normalized amplitude and duration of the acoustic signals were extracted for each of the 3038 segments. The acoustic feature labels provided with the dataset were processed by the research team  [16]. A larger normalized amplitude indicates that louder sounds were produced around cow mouths; and duration is the length of a segment, with a longer duration indicating dairy cows spent more time ingesting forage  [16]. The effects of forage species, forage heights, and their interaction on the amplitude and duration of the segments corresponding to the three ingestive behaviors were analyzed with ANOVA using PROC MIXED in Statistical Analysis Software (version 9.
Overall Deep Learning Algorithm Workflow As shown in Figure  3 , the overall workflow consisted of four steps. The first step was to filter background noises (i. e. , beeping sounds) from the input acoustic data in order to reduce interference for classification. The second step was to remove the low power signals, which can be considered as uninformative data. This may help to constrain model attention to learn important features and improve inference efficiency and accuracy. The first two steps were to clean data based on physical characteristics in the dataset. However, unwanted signals and low-power signals still existed after the initial two steps of data cleaning. The third step was to convert cleaned data into Mel-frequency cepstral coefficient (MFCC) features which are used to highlight high-power data and transform the original acoustic spectrogram into the human perception level  [18].
The first three steps all involved filtering, but only the first step was named "filtering" to differentiate the data cleaning procedures. The final step was to classify the ingestive behaviors using the processed data and deep learning models. Details of the four steps are elaborated in Sections 2. 4-2. 6. The processing was conducted in a local machine with the processor of Intel(R) Core (TM) i9-10900KF CPU @ 3. 7GHz, installed memory (RAM) of 128 GB, graphics processing unit (GPU) of NVIDIA GeForce RTX 3080, and Python-based computing environments.