dataesr/scientific-paragraphs-categorization huggingface dataset:
Building the Dataset Maker Library Using the WORDNET Hierarchy To re-train a deep convolutional network (like VGG 16) for a specific task, one of the most important components is the dataset. We needed a tool that would allow us to generate datasets suitable for answering our question. Therefore, we created a library that, from a keyword, generates a dataset with image folders containing (target) or not containing (distractor) this keyword  [37]. For this, we will use the corresponding set of labels from the IMAGENET database  [24] , which is based on a large lexical database of the English language: WORDNET  [33]. The nouns, verbs, adjectives, and adverbs in this database are grouped into a graphical set of cognitive synonyms, synset, each of which expresses a different concept. These synsets are linked to each other using a few conceptual relations (see Figure  2 ).
For example, if we set the dataset maker with the keyword 'animal', we used the hyperonym link to determine that a German Shepherd is a type of dog and that a dog is a type of 'animal', thus defining a hyperonym path. In this example, the synset 'animal' from WORDNET is in the hyperonym path of the label 'German Shepherd' in IMAGENET. Based on this relationship, the dataset creator selected a specific subset of labels in the IMAGENET database to build our datasets. Once the list of labels corresponding to our task was selected, the dataset maker randomly selected from the URLs provided for the IMAGENET challenge  [24]  to download the images that make up the dataset. With this tool, we generated datasets according to two given tasks. In particular, we generated the dataset necessary to train the network to answer our question, "Is there an animal in this scene?", by selecting the 'animal' synset.
To answer the question, "Is there an artifact in this scene?", we followed the same protocol. As a control, we also created a 'random' dataset, which was generated by randomly selecting 500 labels from the IMAGENET database. The latter was generated to infer the role of the possible links between arbitrary labels by measuring the resulting efficiency of categorization by a deep convolutional network. In summary, we used Dataset Maker to generate three datasets: One based on the 'animal' synset, one based on the 'artifact' synset, and a 'random' one. Each newly generated dataset contains a 'test', 'validation', and 'train' set (with 1200, 800, and 2000 images, respectively). Each set contained a 'target' and a 'distractor' category (both with the same number of images). All networks were trained on the 'training' set and tested during training on the corresponding 'validation' set.
Transfer Learning We used the transfer learning method to re-train networks  [26]. This method takes the knowledge gained from one task and applies it to a different but related task. We used an existing network that had been pre-trained on a specific task: VGG 16  [25]. This architecture is loaded thanks to the PYTORCH library  [38]  and trained on the database used to solve the IMAGENET  [24]  task. We had previously found that this model provided the best trade-off between accuracy and complexity  [39]. It also achieves a good model of biological function as measured by the Brain score  [40]. Compared to other architectures such as ResNet, VGG 16 stands out as an ideal candidate. Two notable advantages of the transfer learning method are the robustness and the convergence speed for learning the network.
This results in lower total execution time and energy consumption. In particular, this method allowed us to save computational time during the learning process and thus experiment with several possible strategies (see Figure  3 ). We first validated this hypothesis by training a network with random weights: VGG SLS (Supervised Learning from Scratch) as a control. During transfer learning, we kept all layers of the VGG 16 network, since it is already capable of performing feature extraction on natural scenes, and re-trained only the last fully connected layer. In particular, we replaced this last layer trained on IMAGENET (i. e. , with a vector of dimension K = 1000 that captures the predicted probability of detection for each of the labels in the IMAGENET database) with a layer whose output dimension is simply K = 1 and which represents the predicted probability of detection of a new object of interest, i.
e. , a target rather than a distractor. We then re-trained this fully connected layer, while freezing the weights of the other layers, to match the pre-trained features (the output of the convolutional layers) with the synset corresponding to the new task implemented by the dataset maker. Following this process, we re-trained a network that we call VGG TLC (Transfer Learning on Classification layers). As a control, we also tested the effectiveness of freezing the remaining layers by completely re-training all layers of the pre-trained network, VGG TLA (Transfer Learning on All layers). Note that these two networks were trained without any form of data augmentation.
Convolution + Relu blocks Since the network is asked to make a binary decision during training ("Is this synset present in this scene?"), we implemented the loss using the binary cross-entropy loss with logits from the PYTORCH library. We used the stochastic gradient descent (SGD) optimizer from the PYTORCH library and validated parameters such as batch size, learning rate, and momentum by performing a sweep of these parameters for each network. During the sweep, we varied one of these parameters over a given range while leaving the others at their default values for 25 epochs. We chose the parameters' values that gave the best average accuracy on the validation set: batch size = 8, learning rate = 0. 00005, momentum = 0. 99. Then, to increase the generality of our results, we implemented various preprocessing steps on the inputs to introduce more variation into the training dataset: data augmentation.
From the VGG TLC protocol, we tested the effectiveness of this data augmentation using two strategies: first, by re-training a pre-trained network with a set of custom transformations from the PYTORCH library: random horizontal flipping (with p = 0. 5), random vertical flipping (with p = 0. 5), a random rotation (p = 1), and random grayscale (with p = 0. 5), such that we trained the VGG TLDA (Transfer Learning with Data Augmentation) model. Then, the input images were distorted using the auto-augment function from the PYTORCH library. This function implements a total of 16 randomly parameterized affine transformations on the inputs to perform data augmentation  [41] , thus defining the VGG TLAA (Transfer Learning with Auto Augment function) model. Finally, we studied a VGG RANDOM model trained on the 'random' dataset (that is, consisting of two categories defined by randomly chosen labels among the 1000 labels of IMAGENET).
Performances on Natural Scenes Containing Animals without Transfer Learning Obviously, testing the initial pre-trained net should be one of the first experiments before re-training the neural networks. If we were to test it on the dataset on which it was trained to categorize an animal, it would indeed perform very well, with a mean accuracy of 0. 99 for categorizing an animal (and 0. 98 for an artifact; see Figure  4 ). The goodness of these results is quite stunning compared to human behavioral results and highlights one difference between human and machine intelligence. [37]. For each dataset, the network is tested with original images (left) or after applying a random rotation (right). The dotted line represents the chance level for all graphs. However, as soon as we added a perturbation such as a random rotation (images similar to  [42] ) into the same dataset, the performance dropped to 0.
Performances on Natural Scenes Containing Animals with Transfer Learning We then tested different variations of transfer learning on the task, "Is there an animal in this visual scene?", and show the mean accuracies for different datasets, as summarized in Table  1. First, we have seen that the VGG LUT network seems to be robust, as validated on the dataset used by Serre et al. [19] , on which the network achieves a mean accuracy of 0. 95. Note that it achieves better performances compared to about 0. 84 obtained by the model designed in Serre et al. [19]  and about 0. 80 in psychophysics, and this is without any retraining process. Now, let us focus on the network after the transfer learning process, as the VGG TLC, VGG TLA, VGG TLDA, and VGG TLAA reached similar levels of performance on the test set (with 0.
97, 0. 96, 0. 97, and 0. 95, respectively) and also maintained robust categorization on the Serre et al. [19]  dataset (with 0. 94, 0. 92, 0. 91, and 0. 88, respectively). Compared to the VGG SLS, which could only reach 0. 64 on the same task, these results show that transfer learning allows us to obtain highly accurate networks for the categorization of a synset of interest. Note that this low performance is only due to the computational limits that we imposed in our study. We then focused on the robustness of the categorization of the different data augmentation strategies (VGG TLC, VGG TLA, VGG TLDA, and VGG TLAA) compared to the state of the art VGG LUT and the expected performance in neurobiological models.
Robustness of the Categorization with Different Geometric Transformations Since we were looking for the best robustness for this task, we tested VGG TLC, VGG TLA, VGG TLDA, VGG TLAA, and VGG LUT on the newly constructed dataset using our dataset maker library with the synset 'animal'. We applied either a grayscale filter or a vertical or a horizontal reflection to the input (see Table  2 ). We also tested the robustness to rotation by rotating the image around the center by an angle ranging from -180 • to +180 • (see Figure  5 ). All these networks maintained good average accuracy on the returned dataset and on the grayscale dataset (see Table  2 ). These results were consistent with psychophysical results showing that ultrafast categorization is robust to a grayscale transformation  [10].
In the following, we therefore focused on exploring the features that this model relies on to perform its categorization. Table  2. Mean accuracies for ultrafast image categorization of an animal in a scene using various geometric transformation on the input: vertical flip, horizontal flip, grayscale filter. These transformations were implemented using our dataset maker library with the synset 'animal' for four re-trained networks: VGG TLC, VGG TLA, VGG TLDA, and VGG TLAA. It was compared with the state-of-the-art network VGG LUT. All the transformations used here were performed using the PYTORCH library  [38].
What Features Are Necessary to Achieve the Task? We designed an experiment in which we gradually removed layers from a pre-trained network VGG 16 for 12 different "depth" factors. For each level, we tested the re-trained pruned networks to categorize an animal in a scene for our dataset IMAGENET. VGG LUT and VGG TLAA achieved the best accuracy for this task (see Figure  6 ). The accuracies of the networks remained similar to the performance found by Serre et al. [19]  with a slight drop between VGG-9 and VGG-12. This is not a surprise, as their model relied on low-level features  [34]. Note that the computational time required to perform the categorization decreased with the depth of the network (in seconds on a Quadro RTX 5000 GPU, we obtained VGG TLAA = 0. 005 ± 0. 0001 and VGG-8 = 0. 003 ± 0.
L U T T L A A v g g -1 v g g -2 v g g -3 v g g -4 v g g -5 v g g -6 v g g -7 v g g -8 v g g -9 v g g -1 0 v g g -1 1 v g g - We also tested all pruned networks on our IMAGENET dataset by rotating the image around the center from -180 • to +180 • ; however, the categorization may lose robustness with fewer layers (see Figure  7 ). Indeed, as the number of layers and the mean accuracy after rotation decreased, the standard deviation of the mean accuracy increased (VGG TLAA = 0. 91 ± 0. 02, VGG-8 = 0. 73 ± 0. 04). Although the networks seem to be able to categorize an animal with fewer layers, they seem to trade this advantage for a lower robustness to transformations such as rotations. To get a better idea of the size of the feature maps needed to categorize an animal in a scene, we tested the networks on a new "shuffled" dataset, where the image had been divided into square patches of different sizes and then blended to generate a new image  [43].
For example, VGG TLAA appears to rely on feature map sizes between 32 × 32 and 64 × 64 pixels, as its accuracy drops when we exceed this threshold (see Figure  8 ); however, further study is needed to quantify this feature map size. In a future application, we could extract feature maps from these low-level layers to better understand the features needed to perform this task. This would allow us to design a stimulus set for a psychological task such as in Thorpe et al. [3]. Such a test could be relevant to whether these features are sufficient to categorize an animal in a flashed scene. Average accuracy over the test dataset of the re-trained networks for rotations around the center from -180 • to +180 •. These networks were tested on a dataset based on IMAGENET images constructed using the 'animal' synset.
The index after "vgg-" indicates the number of convolutional layers pruned in the networks (with vgg-1 being the deepest and vgg-12 being the shallowest). test dataset, where we applied a shuffled transformation to the input image. We show the results as we decreased the size of the shuffled patches on the images. The networks were retrained to categorize animals and tested on datasets based on IMAGENET images created using the 'animal' synset. The index after "vgg-" indicates the number of convolutional layers pruned in the networks. The dotted line represents the chance level for all plots.
Dependence of Accuracy Scores between the Two Tasks We examined dependence of learning performance of VGG TLAA between two tasks by introducing a variation of the synset of interest in the construction of the dataset. We used our dataset maker tool with the keyword 'artifact', thus generating a new network trained to categorize the presence of the 'artifact' synset in a natural scene: VGG ARTI-FACT. We displayed the average accuracy of the networks trained to detect the 'animal' synset (here VGG ANIMAL stand for our VGG TLAA) on the dataset constructed with the 'animal' synset (respectively trained to detect the artifact synset tested on the dataset constructed with the artifact synset). Next, we tested the networks trained to detect the 'animal' synset on the dataset constructed with the 'artifact' synset and vice versa.
Here, by exposing the predictions for the 'animal' and 'artifact' synsets, we highlight a bias in the composition of the dataset. Although the outputs are independent, the 'animal' images confidently match the 'non-artifact' images (and vice versa), thus facilitating global detection (see Figure  9A, B ). (B) on the dataset built with the 'animal' synset where the distractor "is not an artifact" thus constitutes the "strictly animal" dataset; (C) on the dataset built with the 'artifact' synset; (D) on the dataset built with the 'artifact' synset where the distractor "is not an animal" thus constitutes the "strictly artifact" dataset; (E) on a dataset built with a random distribution of synset (see Figure  3 ). The dotted line represents the chance level for all graphs. To infer the influence of this bias on the performance of the network, we generated through the dataset maker a dataset based on the 'animal' synset where, in addition to not being animals, the distractors would also not be 'artifacts'.
This defines the 'strictly animal' set (respectively, one defines the 'strictly artifact' set based on the 'artifact' synset where, in addition to not being an artifact, the distractors would also not be an 'animal'). Once this distinction was made, although there is a loss in performance for both networks, they remained fit for their respective tasks by maintaining an accuracy above 0. 8. On the other hand, they did not seem to be able to predict the absence of their respective sentences once the ensemble was modified (see Figure  9B, D ). These results reinforce the argument that, despite task independence, the composition of the dataset can generate bias in network categorization. As a control, we tested the VGG RANDOM network on the corresponding dataset (see Section 2. 1 for details).
As it obtains an average accuracy close to the one obtained with the VGG SLS network, its poor performance can be explained by the fact that the pre-trained weights of the VGG 16 network do not match the new task. Incidentally, this bias is also present in the dataset used by Serre et al. [19]. However, when we compared the performance of the humans on this dataset with the performance achieved by the network on a frame-by-frame basis, we found a high correspondence (about 0. 84) in their correct predictions. Indeed, for some images, the networks failed at categorizing but the human succeeded, and vice versa. For some images, both the network and the human succeeded or failed in categorizing an animal, and there were cases where the network was wrong but the humans responded correctly on average (see Figure  10 ).
This kind of bias is also found in humans and seems to impact the categorization as well  [45]  and could be linked to top-down influences  [46]. The question of detecting an animal in an image is indeed tightly linked to that of detecting an artifact, allowing for the possibility of the less likely appearances of an animal object (like a teddy bear) or of a non-animal non-object (like a mountain). The study of this kind of bias could possibly allow for building ecologicallyrelevant datasets to maximize the learning process of the networks in order to discover more about the features needed for categorization  [47]. While the level of 80% correct categorization between humans and machines in this type of task is similar, both could be driven to make different "mistakes", and these particular examples could then be used as subjects for studies in the design of psychophysical tests.
Average accuracy obtained as a function of depth in pruning networks. The networks are re-trained to categorize animals and tested on test datasets based on IMAGENET images constructed using the 'animal' synset. The prefix "vgg-" indicates the number of convolutional layer blocks pruned from the original network (see Section 2. 3 for more details). The dotted line represents the chance level for all graphs. Table showing the average time in seconds required for networks to perform a prediction for a 256 × 256 resolution image with a Quadro RTX 5000 GPU.
Data Availability Statement: This work is made reproducible using the following tools. First, the code reproducing all figures is available at GitHub https://github. com/SpikeAI/2022-09_ UltraFastCat/blob/main/Readme. md  [55]  (accessed on 15 March 2023), and in particular the code at DataSetMaker https://github. com/SpikeAI/DataSetMaker [37]  (accessed on 15 March 2023) was used to retrieve images. The paper is available as an arXiv preprint https://arxiv. org/abs/2205. 03635 with links to previous versions and to the code (accessed on 15 March 2023). Also find the associated zotero group https://www. zotero. org/groups/4560566/ultrafastcat (accessed on 15 March 2023) used to regroup relevant literature on the subject.
, 2021; Turner et al. , 2021). Historically, the use of EO data for coastal applications has been limited by data access restrictions, computational requirements, and low spatial resolution of EO imagery. The opening of the Landsat archive in 2008  (Wulder et al. , 2016)  followed by the introduction of Sentinel-2A in 2015 and Sentinel-2B data in 2017, has allowed the coastal community access to these powerful datasets at no cost leading to the emergence of several data access platforms offering additional high-performance cloud computing capabilities such as Google Earth Engine (GEE)  (Gorelick et al. , 2017)  and Data Access Cube  (Dhu et al. , 2017). The development of innovative analysis tools such as CASSIE  (Almeida et al. , 2021) , SHOREX  (Palomar-Vázquez et al. , 2018) , and CoastSat  (Vos et al.
Understanding the influence of beach type on the accuracy of the derived shorelines will increase our confidence in applying EO data to describe shoreline evolution, extract future shoreline change projections, and determine links between large scale climate patterns and coastal response. In this paper, we address the influence of tidal elevation and waveinduced water-level fluctuations on the accuracy of satellite-derived waterlines (SDW), and corresponding satellite-derived shorelines (SDS), which result from the translation of the SDWs to a standard reference elevation. We use two sites representing end members of beach morphological type: Perranporth (macrotidal, dissipative, exposed, unidirectional wave incidence, cross-shore dominant transport) and Slapton Sands (macrotidal, reflective, semi-sheltered, bidirectional wave incidence, longshore dominant transport).
Satellite data and waterline detection algorithm We use CoastSat  (Vos et al. , 2019b) , a python-based satellite shoreline detection toolkit available on GitHub (https://github. com/kvos /CoastSat), to extract the shorelines from Landsat-5 (L5), Landsat-7 (L7), Lansat-8 (L8), and Sentinel-2 (S2) images at the two sites. The tool allows the user to extract the land/water interface, that represents the instantaneous waterline, with an accuracy of 10-15 m  (Vos et al. , 2019a)  and has been successfully used to extract the land/water interface in different coastal settings  (Cuttler et al. , 2020; Castelle et al. , 2021; Heimhuber et al. , 2021). A detailed description of the approach is provided in  (Vos et al. , 2019b)  and a simplified schematic representation of the methodology is provided in Fig.
Perranporth beach CPRG conducts monthly intertidal and supratidal beach surveys covering the southern part of Perranporth beach (Perran Sands). The surveys are conducted with an All-Terrain Vehicle (ATV) equipped with a real-time kinematic global satellite navigation system (RTK-GNSS) and data is collected along a predefined grid (alongshore spacing: 250 m; cross-shore spacing: 25 m). The data is quality checked and reprocessed to a 5 m interpolated grid. Profile data was extracted from the gridded data along shore-normal transects (Fig. 3d ) and combined with the quasi-biannual RTK-GNSS profile surveys conducted by the PCO on foot. A total of 155 CPRG and 31 PCO surveys between September 2006 and December 2019 were used (Fig. 4i ).
Slapton sands All data at Slapton Sands is collected using a pole-mounted RTK-GNSS. CPRG conducts monthly surveys, though not all profiles are included in every survey round. Once again, these data were combined with the quasi-biannual data collected by the PCO as part of the South West Coastal Monitoring Program. In all, data from 193 CPRG and PCO surveys conducted between November 2006 and December were used (Fig. 4j ).
Wave and water level data Tidal elevation prediction time series for the two sites were extracted using the UK Hydrographic Office (UKHO) TotalTide™ v. 5. 0. 4. 9. 4 (ATT, 2019) which provides harmonic tide predictions globally based on observed tidal records (Fig. 4e, f ). Wave data was provided by the Copernicus Marine Environment Monitoring Service North West Shelf Wave Reanalysis hindcast (CMEMS, 2021), henceforth referred to as CMEMS. The CMEMS data is generated with a WaveWatch III (v. 4. 18) 1. 5-km grid Atlantic Margin Model forced by the ECMWF-ERA5 wind model (Fig. 4a, b ). The lateral boundary conditions are supplied by the UK Met Office global wave model hindcast and the model bathymetry is derived from EMODnet bathymetry (http://www. emodnet-bathymetry. eu). Wave parameters are available at a temporal resolution of 3 h on a regular 1.