dataesr/scientific-paragraphs-categorization huggingface dataset:
Timestacks Images Preprocessing Images from the site of Zarautz were annotated by hand. This method of annotation was preferred over semi-automatic annotation because: (i) the position of the seawall varied between timestack images, making the semi-automatic method more laborious, and (ii) the presence of strong winds and gust negatively impacted the quality of certain images, making the semi automatic method less robust. A simple web application was developed to facilitate the annotation for the operator and is accessible in a public GitHub repository (link in Data Availability Statement section). After classification by hand, the result of the annotation was 19,596/2776/162 (Swash/Collision/Overwash). Like the images of Biarritz, images of Zarautz present class imbalance, and they were resized to fit to the input dimensions of the CNNs before the training.
Finally, the output layer estimates the probabilities of belonging to the specified categories for the input image with a "softmax" activation function. The CNNs are trained with backpropagation in the same manner as classical neural networks: the weights in the convolutional and fully connected layers are updated iteratively to minimize the errors between the prediction of the network and the ground truth. The ground truth dataset for such a network is made by annotating images. Details on the annotation of the timestacks can be found in the "Study Sites" section. Only the general ideas about CNN have been presented above; for a detailed description on CNN and their training, the reader is referred to the work of Bengio et al. [33]. There are many CNN architectures, each with different complexity and characteristics.
• A custom architecture inspired by the work of LeCun et al. [34]  adapted for bigger images. The architecture is presented in the appendix (Table  A1 ). • AlexNet  [35] , which won the ImageNet challenge in 2012. Its architecture contains more convolutional layers and dense layers (Table  A2 ). The number of filters is also larger than that of the custom architecture. • VGG16  [36] , which is a very deep CNN that uses 13 convolutional layers and three dense layers (Table  A3 ). • Inception v3, an improved version of the GoogleNet from Szegedy et al. [37]  which won the ILSVRC in 2014. It relies on inception modules, which perform convolutions with filters of multiple size and concatenate their results (Table  A4 ). In addition, the convolution operation with filters of large size inside an inception module are made by using 1 × n filters to reduce computational cost.
Data Processing The datasets of both sites were divided into training, validation and testing sets containing, respectively, 65%, 15%, and 20% of the data (common proportions in the literature). Stratified random sampling was used to ensure that each part contains the same class proportions. The training set is used to fit the CNN. The validation set is used to stop the training for the CNN (early stopping). At last, the test part is used to evaluate the performance of the neural network on unseen data (not used in the training step). During the training, each training image is seen multiple times by the CNN. This can be a problem as the network can learn exactly the characteristics of the training images and might not generalize to new data. To avoid this problem, called overfitting, data augmentation is employed during the training of the CNN.
This method consists in making small changes to images in the training set before feeding them to the CNN. By generating modified images, this method artificially increases the number of images in the minority classes and makes the models more robust to overfitting. The following changes were made: The datasets from both sites suffer from the class imbalance problem. Indeed, the distributions of storm impact regimes are highly imbalanced. For the Biarritz site, 96. 8% of the images display swash regimes, 2. 6% collision regimes, and 1% of overwash regimes. For Zarautz, 87% of the images are swash regimes, 12% collision regimes, and 1% overwash regimes. This class imbalance problem was expected as we are studying rare events. It has been proven that class imbalance can negatively affect the performances of machine learning models in classification tasks  [38].
Transfer Learning For complex and deep CNN, it is common to use transfer learning to speed up the learning process and to improve performances. Transfer learning methods consist in using knowledge gained on a specific task to solve a different task. There are different methods of transfer learning for CNN; for an exhaustive listing, readers are referred to the work of Pan and Yang  [28]. The method used in this article is "pre-training". It consists of using the weights of a CNN trained on a first task as initialization weights for a second CNN that will perform on a second task. The efficiency of pre-training was tested by using the pre-trained weights of VGG16 and Inceptionv3 on the ImageNet dataset, which is one of the largest labeled image dataset  [41]. Then, transfer learning was performed between sites to see if the knowledge gained on one site is beneficial for the learning on the second site.
Application to the Datasets The workflow for this study is presented in Figure  5. For each site, the four CNNs with different architectures were fitted without and with the two methods related to the class imbalance problem: oversampling and cost-sensitive learning (class weights). Transfer learning was used on the more complex architectures (VGG16, Inceptionv3) and only for the best performing method to cope with class imbalance. Data augmentation was used during the training of all the CNNs. The networks were trained on a laptop equipped with a GPU (Quadro RTX 4000) using Keras (tensorflow GPU 1. 12. 0/Keras 2. 3. 1/Python 3. 6. 1), an open-source python library designed for building and training neural networks. The scripts used in this article are available on a public GitHub repository (link in Data Availability Statement section).
Architectures Table  1  regroups the training time, number of epochs, and also the performance metrics (accuracy, recall, F 1 -score) for different CNN architectures, methods to cope with class imbalance, and with or without pre-training with ImageNet dataset for both sites. For both sites and for every methods used to cope with class imbalance (or not), CNNs with deeper and more complex architectures yielded better results (higher values of precision, recall, and F 1 -score). Indeed, these kind of architectures tend to learn more complex features that lead to better performance in harder tasks. The downside of these networks is the training time, which is significantly higher than simpler and shallower models. Without coping with class imbalance problem, CNNs tend to predict all the images as the majority class, resulting in poor classification results.
Pre-Training Finally, models using pre-trained weights (transfer learning) train faster (fewer epochs) and yield better classification results than models trained from scratch. Indeed, the F1-scores obtained with the pre-trained models are, respectively, 6 to 8% higher for the VGG16 and Inceptionv3 models. Even though the images from the ImageNet dataset have different characteristics than the timestack images that are being classified, the pre-trained weights might contain knowledge about general features that are helpful to better classify the timestacks.
The misclassification errors were corrected for the test and validation sets, and the best network was trained once again, resulting in slightly lower results but this time without human misclassification error (Table  3 ). The errors made on the Zarautz dataset were also analyzed (Table  A6 ). A large number of errors were made on images that were in between the categories of storm impact regimes: either the images displayed a swash regime, which was very close to the collision regime, or the images displayed a regime impact, with one small overtopping of the wall. Some misclassification errors were made. The rest of the errors may have come from lighting conditions (large horizontal band, lighter in the images). The misclassification errors were corrected for the test and validation, and the best network was trained once again, resulting in a slightly better result for this site (Table  3 ).
Transferability between Sites The interest in transfer learning for CNN training has been highlighted in Section 4. 1. 3. The models using pre-trained weights from ImageNet trained faster (fewer epochs) and yielded better classification results. In this section, we investigate if the knowledge acquired on one site (CNN weights) could be transferred to another site with pre-training. Pretraining is the most common way to transfer knowledge between tasks. It consists in using the weights of a neural network trained on one site as initialization weights for the training on the second site. The weights of the best CNN for each site are used for the pre-training of a CNN on the other site. The performances of these CNN are presented in Table  4. The weights of the best model on Zarautz data (VGG16 transfer) were used as initialization weights for the training on Biarritz data.
This resulted in classification results better than the learning from scratch with a higher precision and F 1 -score (Table  4 ). However, the values of precision, recall, and F 1 -score obtained with pre-training on Zarautz data remained slightly lower than the ones obtained with pre-training on ImageNet data. Pre-training method was also applied on Zarautz data, where the weights of the best model on the Biarritz site were used as initialization weights. The classification results were better than learning from scratch and learning with pre-trained weights from ImageNet with higher F 1 -score (Table  4 ).
Sensitivity Analysis A sensitivity analysis was performed on the dataset of Zarautz to highlight the effect of the size of the training images dataset on the classification accuracy. The dataset of Zarautz was divided into three smaller datasets. Each of these datasets was divided into the training/validation/test sets with the proportions described in Section 3. 2. 1. Finally, a CNN model was trained on each smaller datasets (VGG16 with transfer learning ImageNet and oversampling). The averaged F1-metric for these three models was 0. 805. This value is slightly lower than the one obtained with the full dataset, which is 0. 858 (Table  1 ). These results confirm what was already known in the literature: CNN performances tend to increase with the training set size  [42].
Discussion Even though we showed the strong potential of CNN to automatically generate storm impact regime database, the proposed methodology can be improved in several ways. More attention could be paid to the choice of CNN architectures and hyperparameters. Other CNN architectures need to be tested, especially recent architectures such as ResNet, MobileNet, or Xception. They could perform better than the architectures presented in this work. For instance, the ResNet architecture contains skip connections between layers, which allows the training of much deeper and performant networks. Hyperparameters are parameters whose values are specified by the user before the training process begins; they affect the structure of a CNN and how well it trains. They have a non negligible impact on the final results.
With more training images, the minority classes will contain more images, and this will lead to less classification errors for these classes. Moreover, if enough timestacks are collected, intermediate storm impact regime classes could be created. These classes could reduce the errors on the images displaying impact regimes not corresponding to the three regimes presented in this work. One very interesting feature of CNNs models is their transferability. We showed that using the knowledge acquired from another site can lead to improved classification results when using pre-training (especially for Zarautz site). The weights of the best CNNs for both sites are available in a GitHub repository (link in Data Availability Statement section) and could be used as initialization weights for a CNN applied to a new site.
Conclusions In this paper, we presented an innovative methodology based on convolutional neural networks and coastal imagery that could be used to collect storm impact data routinely. We described the methodologies associated with CNNs, including the annotation of the dataset, the training of the networks, or transfer learning. We also introduced the problem of class imbalance, which is due to the extreme nature of the storm impact regimes, and we proposed and compared different solutions such as oversampling or cost-sensitive learning. The proposed methodology was tested on two sites: Biarritz and Zarautz. We showed that convolutional neural networks are well adapted for the classification of timestacks into storm impact regimes. Overall, we found that more complex and deeper architectures yielded better results.
Best performances were achieved with the VGG16 architecture for both sites with F-scores of 0. 866 for the site of Biarritz and 0. 858 for the site Zarautz. For the class imbalance problem, the method of oversampling showed better classification accuracy than the cost-sensitive learning method, with F-scores on average 30% higher. Finally, we showed that the method can be easily applied to a new site with optimal efficiency using transfer learning. Indeed, training a CNN using pre-trained weights (ImageNet or weights of another site) resulted in better accuracy than training a CNN from scratch (F-scores on average 6 to 8% higher). With convolutional neural networks, we can take full advantage of the large number of data produced by video monitoring systems. We showed that they are able to transform images into usable qualitative data about storm impact.
Data Availability Statement: The images used to train the CNN are not publicly available due to the large size of the files. SIAME laboratory owns the Biarritz images data; they can be provided upon request. Images of Zarautz's site used in this work are from Azti and the Zarautz Town Council (shared). They can also be provided upon request. The python scripts and weights of the best CNN are available here: https://github. com/AurelienCallens/CNN_Timestacks, accessed on 1 May 2021. The web application used to label the images of Zarautz site (R programming language) is available here: https://github. com/AurelienCallens/Shiny_Classifier, accessed on 1 May 2021.
The author mentioned that, according to AHP tool, the comparison results presented that Onshape was most preferred with the final priority of 59%, and SolidWorks as a runner up with its priority of 41% approximately, which further confirms the choice of our criterion "Software familiarisation". Junk and Spannbauer (2018)  realized a technical comparison between Onshape and Fusion 360 based criteria related to daily use (e. g. userfriendliness and mobility), the evaluation of functionality (e. g. part design, virtual assembly and drawings) and functions that are rarely used or that run in the background (e. g. storage and getting started). Therefore, the design was taken into account as a comparison criterion, while considering the different functionalities to be acquired in TN20. They concluded that, Onshape offers the user a simple user surface, high mobility and flexibility in the extension of additional applications.
Compared to Fusion 360 which has a large range of functions in a full package that contains many useful features, however it can have minimal extension. Among numerous CAD software today, and numerous comparisons between them, it hard to provide a performed evaluation. That's why,  Galimova (2020)  presented a summary table of the advantages and disadvantages, on the basis of which the author developed a method facilitating the choice of the optimal approach to CAD testing. Also,  Wu et al. (2017)  presented a technological guide for commercial customers to select the most appropriate application on the cloud or to use it during the whole product process development. This guide is generated from the state-of-the art that they realized on digital design and manufacturing software and services currently available on the cloud.
For educational aim, onshape is experimented by students the University of Kaiserslautern, where they organized the development process by themselves using a full cloud CAD software. They deduced that Onshape is very powerful for collaborative work since it offers best data management functions like branching, versioning or merging of CAD models, hence they obtained complete results because all the data is already at the database  (Eigner and al. , 2017). Barrie (2016)  presented Fusion 360 and Onshape as the two current and well know cloud-based CAD packages distributed as Software as-a-Service (SaaS) in the market. The author, points to the potential of such Software in an academic setting processed to a specific evaluation between both systems, by looking at the platforms, without expressing preference.
Advantages and limitations of using cloud-based CAD software in education In the educational context, cloud-based CAD software system, which perform as a Software-as-a-Service (SaaS) have important benefits such as:  Models and visualizations are supported by browsers enabling display on any device (Barrie, 2016)  Wok file are replaced by compound documents, which are stored on the cloud as database entities  Buying a costly node-locked annual license is not required.
ICED21 Fusion 360 and Onshape are distributed as a Software-as-a-Service (SaaS) and as mentioned on section 2 they served for comparison to explore cloud-based CAD benefits. In fact, Fusion 360 is a cloud-based 3D CAD, CAM, and CAE platform for product development, according to  Barrie (2016)  can offers industrial and mechanical design applications in the form of surfacing and solid modelling, operate on multiple platforms such as PC's, Macs and mobile devices and keep up to date with model and project changes in the cloud Fusion 360. Wu et al. (2017) , mentioned that fusion 360 allows creating smooth and precise surfaces with T-Splines technology or with sketch curves, patches, and extrusions via a web browser. However, it does not facilitate global collaboration to unlike Onshape, which claims 'full-cloud' based CAD and collaboration.
Intuitive interface The user interface in Onshape has similarities with Creo. In fact, features in the interface part studio and the tree structure are the same as on Creo and the feature search function is highlighted. Also, the system version is accessible without prior research and upgrading a part and its history are easily understandable. For Fusion 360, the novice user find that the interface is simpler than a classic software (such as Catia V5). Indeed, he is not overwhelmed by a multitude of functions on the screen, only the most used are displayed directly on the screen (sketch, extrusion, etc. ). The surface ("Surface") and volume ("Solid") workshops make it easy to determine the type of design desired and thus avoid certain errors in the use of features. Also, it is noticed that there is the "Sheet metal" workshop but it has not been used here.
However, the use becomes a little more complicated because many functions are "hidden" in the menus "create", "modify" or "assemble". Which can confuse a new CAD user. The tree is less accessible than OnShape's, but better ordered: it contains a few peculiarities that may confuse the user at first. Indeed, the tree is represented by functions (sketches, bodies or components) and the features for adding or modifying material are located at the bottom of the screen, which requires a little research during the first use. In addition, the list of these features is not very large and is rather difficult to read. However, to go back and modify a revolution for example, you have to go back to the bottom of this screen and click on the feature. Upgrading parts is easy because all you have to do is register the part, just like with any conventional CAD software.
Accessibility In terms of accessibility, Onshape does not require installation on computer. It is 100% accessible on the web browser, and regardless the characteristics of the computer, the software will have the same fluidity on a computer with poorer characteristics than on a more powerful computer. However, Fusion 360 requires a light installation, but it is still CAD software designed on the principle of SAAS. Hence, a stable internet connection is required for Onshape, even though, while the light installation allows Fusion 360 to be used without an internet connection, although the user can, as well, use it online. This is convenient because even with an unstable connection, the software did not ask to refresh the page when a connection was lost. Also, from the accessibility point of view, Onshape is available as a mobile application, where 3D modelling can be realized in the same way as on a computer, but this poses ergonomic problems.
Software familiarization The software is taken up fairly quickly, especially in terms of the basic functions of extrusion, revolution, scanning, Boolean, etc. where the system is always the same. Indeed, features, Surfaces and parts can be manipulated at the same interface. Thus, some subtleties are to be grasped (assemblies are not based on plans but on points and connectors) because all functions are based on sketches. Also, Fusion 360 gets started quickly when creating parts for the first time. However, some subtleties like the tree structure and the presence of features at the bottom of the screen can be unsettling. In addition, the software uses concepts that are specific to it such as "bodies" and "components". Moreover, in Fusion 360, the user has the possibility to create assemblies of parts from parts created in the same "design" (workspace), the parts are all found in the same space and are not distributed in a directory.
From Fusion 360 side, the collaborative part is quite close to a PLM system. It integrates two concepts: hubs and projects. The projects are integrated into hubs. Each hub has a team in which users have different roles (team administrator, team member, project contributor). Each role incorporates certain rights. Likewise, each project includes three roles, which authorize some actions (administrator, editor, viewer). These roles are issued based on the user's role in the hub. In terms, of real-time collaboration, the software shows the other users working on the project or part. Unlike OnShape, it is not enough to create a sketch for the other user to see it, therefore the part has to be saved, so that the other logged in user can see what has been done.
Designing with Fusion 360 The design is done in a workshop called "design". It can contain several parts in the same space (see the notions of bodies and components). The part, automotive spring damper (figure1), is modelled based on the skeleton method. The Fusion360 software is mainly based on sketches such as with Onshape. Solid Fusion 360 works as the same way of Onshape (and regular CAD software). Thus, the sketches are the basis of features, and the features are the same as those on classic software. As on Onshape, the notions of bodies and components must be taken into consideration when using features. Although they can be annoying at first, these functions are interesting because they allow to create an assembly of parts themselves designed in the same workspace ("design"). The advantage is that all the parts are in same workspace and that the user does not have to navigate through the files of the Hub to insert created parts.