dataesr/scientific-paragraphs-categorization huggingface dataset:
We manually labeled a sample of 4157 tweets into three categories: misinformation, not-misinformation, and irrelevant tweets. The labeling was performed separately by two social scientists working on hazard and risk communication in the field of seismology. The tweets were selected randomly. Disagreements between the classifications were resolved by a third person. The Cohen's kappa coefficient of κ = 0. 995 for the classified dataset shows an almost perfect level of agreement  60. The misinformation category includes all tweets that claimed to be able to predict a future earthquake, according to the state-of-the-art summarized in the recently published Communication Guide 1. The not-misinformation category includes general earthquake notifications, tweets clarifying that earthquakes cannot be predicted, and tweets explaining what information is available and how certain services work (e.
g. , earthquake forecasting, EEW). All other tweets, such as tweets unrelated to earthquakes and discussions about secondary hazards triggered by earthquakes, were classified as irrelevant. To avoid bias in the model when fine-tuning, we applied a preprocessing methodology similar to Nguyen et al. 61  and removed tweets that were identical or very similar to each other from the manually labeled dataset. We limited the dataset to tweets that were 350 characters or shorter after preprocessing, thus removing a negligible 0. 5% of the manually labeled dataset and reducing the maximum length of tweets by about 50%. The final data comprises 81,862 tweets, of which 3584 were manually labeled. The labeled dataset has 698 misinformation tweets, 1328 not-misinformation tweets, and 1558 irrelevant tweets.
Second, after establishing the model as viable, we were able to fine-tune a model without using a test set, therefore maximizing the data available for training. First, we fine-tuned a RoBERTa-base model using the labeled dataset five times for 10 epochs, each with a dropout of 0. 2, weight decay of 0. 01, learning rate of 1e-5, and a batch size of 16, using stratified fivefold cross validation on the labeled tweets. The test set in each fold was split into 10% for the validation set and 10% for the test set. The models converged at epoch 5 with an average evaluation loss of 0. 446 ± 0. 042 and an average F1 score of 0. 846 ± 0. 018 on the validation set. The results for the test set at epoch 5 were evaluated using weighted F1, precision, and recall scores. The average F1, precision, and recall scores of the five models on the respective test sets were 0.
845 ± 0. 019, 0. 85 ± 0. 016, and 0. 85 ± 0. 016, respectively. These results show that the model is viable. Second, for the final classification model of the data, we fine-tuned a RoBERTa model once more using 90% of the labeled dataset for the training set and 10% for the validation set, without a test set. Considering the model had already been established as viable, this approach enabled the use of a larger dataset for training and fine-tuning. The model converged at epoch 4 with a loss of 0. 3574, an evaluation loss of 0. 4919, and an F1 of 0. 8494 on the validation set. The fine-tuned model was used for the classification of the complete dataset into one of three categories. The results of the classification of the full data set with the fine-tuned model are described in Table  2. Step 4: Analysis.
Acknowledgements The authors thank the handling editor and the two reviewers for their valuable comments, which improved the clarity of our article. We also thank the proofreader for the corrections and suggestions for improvement. This research has received funding from the  European Union 's  Horizon 2020 research and innovation program  under Grant agreement No. 101021746 , CORE (science and human factor for resilient society). Opinions expressed in this paper solely reflect the authors' view; the EU is not responsible for any use that may be made of information it contains.
, 2014; Di Filippo et al. , 2006; Juphard et al. , 2004; Zoccolotti et al. , 2005). Novel to our approach, we examine within-grade variability on a LD task. We obtained LD data from a large sample of students in 6 th grade (N= 1,500, mean age of 11y. o. ), which is the first year of middle school in France, a turning point given the increasing importance of silent reading skills and decreasing teacher-led explicit reading instruction. We subdivided our participants based on groups of reading ability from their national fluency scores, then investigated how these groups perform on a LD task. We provide a detailed account of group lexical and sublexical reading procedures through the prism of accuracy and RT. Given that students in this grade have had five years of reading exposure, we hypothesized that, ideally, all students should show no length effect for frequent words, and that length effects would persist for less frequent and rare words, predominantly for poor readers.
Participants In the context of the French national evaluations given to all students in 6th grade (the first year of middle school), we deployed our LD task in a panel of students. Test administration and data collection were performed by the ministerial service in charge of education statistics, the Direction of Evaluation, Prospective and Performance (DEPP, www. education. gouv. fr/direction-de-l-evaluation-de-la-prospective-et-de-laperformance-depp-12389). 3,614 students, selected by the DEPP as representative population of France, were included in the panel. Due to a software error (leading to missing data and negative RTs in some students) only 1,500 participants with a complete dataset were retained for our analysis (806 girls and 694 boys, mean age = 11. 0 y. o. ). We received from the DEPP anonymized data from a standardized fluency test and our LD task.
Word stimuli We first extracted all mono-lemmatic and mono-morphemic words from the Lexique 3. 83 database  (New et al. , 2004)  with a length of four to eight letters and a frequency higher than three per million. We manually excluded all potentially offending, inappropriate or foreign words, thus resulting in a stimulus set of 3,656 words. These items were then separated into four different frequency bands: very frequent, frequent, rare and very rare (see table  1  for details). 12 modules were designed using this database. For each module, we randomly selected 3 words from each frequency category and each length, resulting in a factorial design with length (5 levels, 4-8 letters) and frequency (4 levels) as factors, and a total of 60 words per module.
Acknowledgements We gratefully acknowledge the  Direction de l'Evaluation, de la prospective et de la performance (DEPP)  of the  French National Ministry for Education  for its assistance in the deployment of the tests and the data collection and  Johannes Ziegler ,  Mathias Sablé-Meyer  and  Aakash Agrawal  for their advice on data analysis. This work was funded by  INSERM ,  Collège de France ,  CEA ,  DEPP ,.
CNN Architecture Let us now describe the CNN that was tested. We recorded 500 speckle patterns for each particle shape (and thus a total of 3000 patterns) to create our database. We used 2400 patterns in grayscale for the training phase (actually, the central selections of 256 × 256 pixels in Cartesian coordinates were converted to polar coordinates, as previously explained), and each image was normalized between -1 and 1. The choice of the dimensions of the selected part (i. e. , 256 × 256) was a balance between computing time, memory capacity, and the number of speckle grains present in the selected part: -Concerning the computing time and memory capacity, the network was trained on a GPU with 8 GB of memory capacity. With this capacity, the GPU could store the model, the batch of images, and the gradient calculation for each epoch.
CNN Architecture Let us now describe the CNN that was tested. We recorded 500 speckle patterns for each particle shape (and thus a total of 3000 patterns) to create our database. We used 2400 patterns in grayscale for the training phase (actually, the central selections of 256 × 256 pixels in Cartesian coordinates were converted to polar coordinates, as previously explained), and each image was normalized between -1 and 1. The choice of the dimensions The role of the CNN is to recognize the characteristics of the patterns presented in polar coordinates for different shapes of rough particles and for different sizes of these particles.
CNN Architecture Let us now describe the CNN that was tested. We recorded 500 speckle patterns for each particle shape (and thus a total of 3000 patterns) to create our database. We used 2400 patterns in grayscale for the training phase (actually, the central selections of 256 × 256 pixels in Cartesian coordinates were converted to polar coordinates, as previously explained), and each image was normalized between -1 and 1. The choice of the dimensions In summary, the choice of the dimensions of the selected part was a compromise between the computation capabilities and the geometry of the setup. The selected part could of course be larger using more powerful computers. The model shape of the CNN was based on two successive custom "inception" modules  [30, 31]. Figure  7  shows the CNN architecture.
Each inception module was composed of three branches: top, middle, and bottom. The top branch was composed of three successive convolution blocks with filter sizes of (1,1), (1,3), and (3,1); the middle branch was composed of five successive convolution blocks with filter sizes of (1,1), (1,3), (3,1), (1,3), and (3,1); and the bottom branch was composed of a single convolution block with a (1,1) filter. In each convolution layer, we set the number of output filters to 24. The network was trained on a computer with eight Tesla M10 GPUs with Pytorch 1. 10. 2. It took approximately 45 min for 50 epochs to occur. The learning rate was set to 5 × 10 -4 with an L2 regularization of 10 -4 , and the batch size was set to 32 images. The loss function that was used was the sparse categorical cross entropy, and we used the accuracy metric to measure learning evolution during training.
In each convolution layer, we set the number of output filters to 24. The network was trained on a computer with eight Tesla M10 GPUs with Pytorch 1. 10. 2. It took approximately 45 min for 50 epochs to occur. The learning rate was set to 5. 10 -4 with an L2 regularization of 10 -4 , and the batch size was set to 32 images. The loss function that was used was the sparse categorical cross entropy, and we used the accuracy metric to measure learning evolution during training. We initially tested more simple architectures of CNNs using successive convolution, ReLU correction, and batch normalization, but the best results were obtained using inception modules, as described previously.
As would be observed in Cartesian coordinates, the size of the speck of light is larger, and the number of specks of light is reduced when the particle is smaller. Figure  1  shows the six families of shapes that were programmed. However, during training and validation, there were no identical particles in our database. First, the size of each particle was random. Figure  9  shows a typical size histogram of the programmed particles in the case of the stick-like particles. Similar histograms could be plotted for the other families of particle shapes. In addition, the widths of the branches of each particle were random (in the range [30 µm, 80 µm]). The choice of the micromirrors programmed in the "on-state" in the global shape of the particle was random. Finally, the orientations of these particles in the transverse (X,Y) plane were also random.
Obtained Results Figure  10  shows the model loss versus the number of epochs. From epoch 30, there was good convergence between the training data and validation data. For the last iteration, the validation loss was 0. 0018. Using only 50 epochs, the model allowed the loss of three decades. Moreover, the training and validation curves converged with the same speed until the last epoch: we observed no under-fitting or over-fitting. For this reason, the model can easily generalize the classification problem. Figure  11  shows the accuracy metrics of the model. As the model learned, the accuracy increased, as expected, until it reached 100% accuracy over the validation dataset. This is a very satisfying result for image classification. The addition the panel of non-centrosymmetric shapes was important because interferometric particle imaging can suffer from the twin image problem, as discussed using phase retrieval algorithms in a recent study  [24].
Obtained Results Figure  10  shows the model loss versus the number of epochs. From epoch 30, there was good convergence between the training data and validation data. For the last iteration, the validation loss was 0. 0018. Using only 50 epochs, the model allowed the loss of three decades. Moreover, the training and validation curves converged with the same speed until the last epoch: we observed no under-fitting or over-fitting. For this reason, the model can easily generalize the classification problem. Figure  11  shows the accuracy metrics of the model. As the model learned, the accuracy increased, as expected, until it reached 100% accuracy over the validation dataset. This is a very satisfying result for image classification. The test dataset was composed of 300 images. Figure  12  shows the confusion matrix of the whole test dataset.
The neural network was trained in conditions similar to the previous ones. The validation results show that stick-like particles were still recognized without error, regardless of the sizes and orientations of the sticks. However, the error could exceed 30% of the tested cases for more complex objects such as cross-like or dendrite-like particles. The conversion to polar coordinates avoided the non-invariant rotational problem of convolution in the neural network to make the classification of interferometric images easier. The test dataset was composed of 300 images. Figure  12  shows the confusion of the whole test dataset. The model classified these images with 100% accuracy. The test dataset was composed of 300 images. Figure  12  shows the confusion of the whole test dataset. The model classified these images with 100% accuracy.
With the CNN, a Y-shaped particle (see Figure  1f ) is never confused with a dendrite-like particle with six branches (see Figure  1c ). In the same way, an L-like particle (see Figure  1d ) is never confused with a cross-like particle (see Figure  1b ). The CNN can thus be a very powerful tool to complete analyses carried out using algorithms of reconstruction or to replace them. It could actually be very powerful to combine both procedures through the development of a network inspired by physics-informed neural networks in radiative transfer and fluid mechanics  [35, 36]. The results that we have presented in this study are experimental results that contain different sources of noise. The creation of our experimental database required a few hours. During this time, different sources of noise were present: dust deposits on the optics and slight variations in the laser intensity and the optical alignment.
In addition, we realized our database by randomly programming particles of different shapes and sizes with random numbers of micromirrors. For example, the smallest stick was composed of only 100 on-state micromirrors, while the biggest dendrite was composed of 17,200 on-state micromirrors. For all shapes of particles, the intensity of the interferometric images could thus vary very significantly. Nevertheless, particle classification from the interferometric images was possible with the CNN. In the future, a complete analysis of the influence of noise will require performing IPI in real conditions, i. e. , with real rough particles and illumination with a pulsed laser. In the present study, the database could be limited to 500 images per particle, which gave convincing results and a 100% recognition rate.
Audio-visual DKF Generative model We follow the DKF generative model  [11] , and propose to extend it to the audio-visual case. Given some clean AV training data u1:T = {st, vt} T t=1 , with vt being the visual feature vector at time frame t extracted using a video encoder, the generative model is defined as follows:    p θ (st|zt, vt) = Nc 0, diag(σ 2 θs (zt, vt)) , p θ (zt|zt-1, vt) = N µ θz (zt-1, vt), diag(σ 2 θz (zt-1, vt)) , where Nc(0, Σ) denotes a circularly symmetric complex Gaussian distribution, σ θs , µ θz , σ 2 θz are parametric non-linear functions realized by some DNN architectures, and θ = {θs, θz}. The approximate posterior takes the following form: q ψ (z1:T |u1:T ) = T t=1 q ψ (zt|rt) = T t=1 N µ ψ (rt), diag(σ 2 ψ (rt)) (1) where rt = {zt-1, ut:T } collects all the conditioning variables, and µ ψ , σ 2 ψ are DNN-parameterized non-linear functions, i.
e. , the encoder. As with the previous works, the encoder takes the modulus square of STFT data as input. Learning the set of parameters, i. e. , Φ = {θ, ψ}, amounts to optimizing the ELBO: L(Φ; u1:T ) = T t=1 E q ψ (z t |u 1:T ) {log p θ (st|zt, vt)} - T t=1 E q ψ (z t-1 |u 1:T ) {DKL(q ψ (zt|rt)∥p θ (zt|zt-1, vt))} , (2) where DKL(q∥p) denotes the Kullback-Leibler (KL) divergence between q and p. The two expectations can be computed recursively, as detailed in  [11]. A single-sample Monte-Carlo approximation of the two expectations is computed followed by the reparametrization trick  [10]  before optimizing the parameters. The proposed AV-DKF architecture is shown in Fig. 1.
Datasets. For training all the VAE variants, we used the TCD-TIMIT corpus  [22]. This dataset contains AV speech data from 56 English speakers (39 for training, 8 for validation, and 9 for testing) with an Irish accent, uttering 98 different sentences, each ∼ 5-second long, and sampled at 16 kHz. This amounts to ∼ 8 hours of data. There is a corresponding video file for each utterance that captures a frontal view of the speaker at a rate of 30 frames per second. For all the videos, the lip region of interest (ROI) are already extracted as 67 × 67 images (a sample is shown in Fig. 1 ). The STFT of the speech data is computed with a 1024 samples-long (64 ms) sine window, 75% overlap, without zero-padding, yielding STFT frames of length F = 513. The ROI images for each video were upsampled across time to make visual and audio frame rates equal.
ACKNOWLEDGMENTS We thank  Michiel Burgelman ,  Mark Dykman ,  Mazyar Mirrahimi , and  Alain Sarlette  for fruitful discussions. This work was supported by  NSERC , the  Canada First Research Excellence Fund  and the  U. S. Army Research Office  Grant No. W911NF-18-1-0411. This material is based upon work supported by the  U. S. Department of Energy, Office of Science, National Quantum Information Science Research Centers, Quantum Systems Accelerator.
Introduction While the proliferation of large datasets has been a driving force behind the progress and success of machine learning methods, many application domains remain constrained by relatively small low-dimensional datasets. For instance, collecting useful data can be computationally expensive in domains, like photonics design  [1] , that rely on simulationbased-optimizations interacting with physics equation solvers. Gene expression data is often obtained from experimental processes with very limited data due to costs and ethical considerations  [2]. Materials design problems can involve running computationally intensive models as well as expensive experiments, all resulting in small useful datasets  [3] [4] [5]. Yet, these datasets can often benefit from dimensionality reduction techniques to analyze the low dimensional space for future experiment design or to assist in some other downstream tasks.
Principal Component Analysis (PCA)  [6]  is a classical dimensionality reduction approach with provably optimal performance for linearly dependent data  [7]. It is often the first, if not the only, method used for datasets where the number of features is smaller than the number of data points (n < m). However, its efficiency clearly suffers when data exhibits curvature as PCA cannot distinguish between nonlinear structure and lack of structure in data. In many domains where dimensionality reduction is desired, the data is inherently nonlinear and therefore nonlinear methods are in principle expected to yield superior results. Autoencoders  [8]  are commonly used for nonlinear dimensionality reduction as they offer a convenient way of controlling the reduction by setting network's bottleneck size.
Another work has previously explored a different PCA-based initialization of autoencoders  [10]. However, the proposed initialization is a heuristic, and the main objective of that work is to speed up a slow training process on large datasets rather than improve performance on scarce data. Linear variational autoencoders to address data scarcity problems in collaborative filtering was previously proposed  [11]. However, this model cannot capture nonlinear relationship between parameters. Allerbo and Jörnsten  [12]  present techniques for inducing sparsity in dimensionality reduction using autoencoders. However, the sparsity regularization is meant to primarily address interpretability issues rather than improve projection error performance. Also, sparsity regularization introduces extra parameters that require tuning and is therefore less suitable to problems with small datasets.
To study the impact of the proposed approach we first analyze its performance on carefully designed synthetic data where we can control the degree of nonlinearity. We then follow up with a variety of datasets coming from the nanophotonic component design domain as well as gene expression data and a benchmark breast cancer dataset. Our results demonstrate that there exists a synergy between linear and nonlinear methods in that the proposed approach improves upon both baselines in most low data regimes while being comparable to the best performing method in all other situations.
(reproduced with permission [1]). A vertical grating coupler is a device that diffracts light injected in-plane vertically upwards, or couples light signal in the photonic chip to and from optical fibers or free space. It is a necessary device for connecting photonic chips to the surrounding environment. A schematic of the device that is considered here is shown in Figure  5  (Top). The complete dataset of optimized structures for this vertical grating coupler consists of 540 designs, also referred to as a set of good designs. These were obtained from computationally intensive simulation-based optimizations and selected from a set consisting of more than 30,000 candidate designs based on an optical performance criterion (a waveguide-fiber coupling efficiency of at least 74%). The designs are characterized by five segment values (L 1 to L 5 ) representing different widths of silicon blocks, which are the parameters to the design problem (see Figure  5  (Top)).
The experiments are further repeated 50 times creating different data splits to measure accurate statistics of the results. The error bars represent the uncertainty of an average estimate. Figure  6  (Left) shows the performance of different methods evaluated on the test set. We notice that our proposed Robust model consistently outperforms randomly-initialized autoencoders as well as PCA across all dataset sizes. As expected, the performance of randomly-initialized autoencoders slowly catches up and is expected to match the performance of Robust model when sufficient data is provided. Note that with only 20 samples the PCA-Robust model has lower error than the standard Autoencoder with 100 samples in Figure  6 , while PCA requires around double the amount of samples to catch up to the same error.