dataesr/scientific-paragraphs-categorization huggingface dataset:
In the current study we make a first step towards simulating the acquisition of language via grounding in perception in a more naturalistic scenario. Our main focus is on learning the meaning of linguistic expressions from spoken utterances grounded in video. We use the well-known children's cartoon Peppa Pig as a case study. Compared to commonly used video datasets, this dataset has a number of interesting characteristics. The visual modality is very schematic, the language is simple in terms of vocabulary size and syntactic complexity, and analysis of its linguistic features suggests its suitability for beginner learners of English  (Kokla, 2021; Scheffler et al. , 2021). Crucially, however, most of the speech in the videos consists of naturalistic dialogs between the characters in which they do not only discuss the here and now, but also often use displaced language.
1 Thus, the utterances are only loosely and noisily correlated to the scenes and actions depicted in the videos. This choice of data thus allows us to directly address the ecological limitations of the current approaches. In addition, the cartoon videos contain comments interjected by the narrator. We use these for evaluating the acquisition of meaning as they are more descriptive and less noisy and allow us to measure performance, while controlling for speaker characteristics. We implement a simple bi-modal architecture that learns spoken language embeddings from videos, and train it on the Peppa Pig dataset. Our contributions are the following: • We evaluate model performance in terms of video fragment retrieval and additionally design controlled evaluation protocols inspired by the intermodal preferential looking paradigm  (Hirsh-Pasek and Golinkoff, 1996).
Spoken Language Grounded in Images The availability of datasets of images associated with spoken captions such as Flickr Audio Captions  (Harwath and Glass, 2015) , Places  (Zhou et al. , 2014) , and Spoken COCO  (Hsu et al. , 2019)  led to a rapid development of neural models of grounded language learning; see  Chrupała (2022)  for a comprehensive overview. In contrast to earlier approaches, these models are trained end-to-end directly on large datasets. Following the architecture proposed in  Karpathy et al. (2014) , the visual and speech modality are usually encoded using separate pathways, and subsequently mapped into a joint representation space. Visual features are extracted from a pre-trained image classification model that processes the whole or a specific region of an image (however, see  Harwath et al.
[2018] , who train the model end-to-end on images and their spoken captions on the Places dataset). The audio encoder component in most models is either an adaptation of  Harwath et al. (2016) , which feeds a spectrogram of the speech signal to a convolutional architecture, or a hybrid architecture of convolutional followed by recurrent layers using Mel-Frequency Cepstral Coefficient (MFCC) features from the audio signal as input, as introduced by  Chrupała et al. (2017). The majority of models of speech grounded in images are optimized for and evaluated on image retrieval via spoken caption and vice versa. Additionally, a range of diagnostic analyses have been performed on the hidden representations of these models to study whether they encode the identity and boundaries of subword units such as phonemes and syllables  (Alishahi et al.
Spoken Language Grounded in Video There have also been recent attempts to learn spoken language grounded in video instead of static images. Boggust et al. (2019)  sample audio-visual fragments from cooking videos; their grounded model treats video frames as still images ignoring the temporal dimension. Rouditchenko et al. (2021)  integrate the temporal information when encoding videos from the Howto100m dataset  (Miech et al. , 2019) , and perform better than previous work in language and video clip retrieval. Models trained on instructional video datasets often do not generalize well to other domains. Monfort et al. (2021)  highlight this limitation and show that training on their larger and more diverse Spoken Moments in Time dataset leads to better generalization. But the point remains that these video datasets contain descriptive speech, thus ensuring that there is a strong correlation between the spoken language and their visual context, a characteristic that is not representative of the experience of learning language in real world.
Dataset We use the dataset provided by  Papasarantopoulos and Cohen (2021) , which consists of metadata for the set of 209 episodes (seasons 1-5) of the English-language version of Peppa Pig. 2 he annotations created by  Papasarantopoulos and Cohen (2021)   This means our data consists mostly of video clips that contain some speech. 4 Table  1  shows the sizes of the training and validation splits. The vocabulary size of transcriptions corresponding to the training data is 5,580.
3 second duration. This length is close to the mean duration of audio aligned to a single line of subtitles. The number of clips for each dataset split is shown in Table  1. Jitter In this approach the mean duration of the segments is the same (2. 3 seconds) but we randomly vary the length of the video, and, independently, of the corresponding audio around this average duration. This means that (i) the segments can be partially overlapping and (ii) the video and the audio it is paired with are of different length. Specifically, we sample the fragment duration d (in seconds) from the following distribution: d ∼ min(6, max(0. 05, N (2. 3, 0. 5))) (1) The video is subsampled to 10 frames per second, and to 180 × 100 resolution. 5 The audio is converted to mono by averaging the two channels and the raw waveform is used as input.
Video Encoder As a video encoder we use the 18-layer ResNet (2+1)D architecture  (Tran et al. , 2018) , pretrained on the action recognition dataset Kinetics-400  (Kay et al. , 2017). The pre-trained model is available via PyTorch. 7 This architecture implements 3D convolution by decomposing it into a 2D spatial convolution followed by 1D temporal convolution. The output of this module is aggregated using the attention mechanism with the same architecture as for the audio module, linearly projected to the same dimensionality as the audio (512) and L 2 normalized. For our experiments we also use a version of the video encoder without pre-training. STATIC Baseline As a baseline to investigate the contribution of temporal information to video modeling, we swap the video ResNet (2+1)D with the 2D ResNet pre-trained on ImageNet, which embeds each video frame separately.
Evaluation The most common approach to evaluation for visually grounded models trained on spoken image captions is caption-to-image retrieval (often combined with image-to-caption retrieval); this technique has been carried over from text-based image-caption modeling. With the standard spoken caption datasets this approach is unproblematic because the content of the captions is not correlated with extra-linguistic clues in the speech signal, such as speaker identity (since speakers are randomly assigned to captions) or non-speech environmental sounds. In such an artificial setting, a retrieval metric measures the ability of the model to match spoken utterances to images based on their semantic content. This is not the case for the Peppa Pig dataset: Here we can expect that when a video segment depicts a particular character (e.
Video Retrieval For the retrieval evaluation, as for training, we also use both the FIXED and the JITTER segmentation strategies; however, for most conditions, we only report retrieval for the FIXED evaluation data. We encode each audio clip in a candidate set sampled from the validation (or test) data using the speech encoder part of the model; similarly, we encode each video clip using the video encoder. We then measure cosine similarity between the encodings of the audio clip and all the video clips. If the video clip corresponding to the audio is among the n most similar video clips, we count that as a success. The proportion of successes across all audio clips gives us the retrieval metric known as recall@n. In Section 5 we report recall@N of the complete model on narration test data for values of N between 1 and 10; for the rest of the experiments in this paper we focus on n = 10.
We use the model's audio encoder to encode the anchor, and the video encoder to encode both video clips. We then check whether the anchor is more similar to the positive or to the negative clip in terms of cosine similarity (see Figure  1  for an example). More precisely, triplet accuracy is the mean over all triplets of the following quantity: signum(cosine(A, P ) -cosine(A, N )) + 1 2 (4) where A is the anchor, P is the positive and N is the negative video clip. For this metric, we expect random-guessing performance to be at 0. 5, and perfect performance to be at 1. 0, regardless of the specific set of test items. We also quantify uncertainty by resampling the triplets 500 times from the dataset, and display the score distribution.
9 Following Nikolaus and Fourtassi (2021), we pair every such triplet example with a corresponding counter-example to control the evaluation for linguistic biases in the dataset. Figure  2  shows an example of how two counterbalanced test trials are constructed from audio and video clips. Here, the anchor A example of the example triplet is the audio of Peppa loves jumping, the positive video P example is the corresponding video, and the negative video N example is the video corresponding to George loves jumping. In the counter-example triplet, the anchor A counterex is the audio of George loves jumping, and the positive and negative videos are flipped. We measure word accuracies by calculating the triplet accuracy for all triplets that contain a given word (e. g. , Peppa in the previous example) either as target or distractor.
Experimental Settings We implement the architecture in PyTorch  (Paszke et al. , 2019). We use the Adam optimizer (Kingma and Ba, 2015) with the scheduling described in  Devlin et al. (2019). We train every configuration on a single GPU and stop training after 48 hours, with batch-size 8 and accumulating gradients over 8 batches, in 16 bit precision mode. For each model configuration we save model weights after each epoch and report results for the checkpoint which gets the best triplet accuracy on the narration validation data. Our code is publicly available at https:// github. com/gchrupala/peppa, and can be consulted for further details of the experimental setup.
Conclusion We simulate grounded language learning in a naturalistic setting, where the connection between the linguistic and visual modalities is not always strong and is potentially confounded by correlations with non-semantic aspects of the speech signal. Our experimental results suggest that despite the challenges inherent to the naturalistic aspects of our training dataset, a simple bimodal architecture can capture aspects of visual meaning of individual words as well as full utterances, and generalize well to narrative utterances featuring a single unseen speaker and a descriptive rather than conversational style. Our analyses show that generalization is substantially boosted by fine-tuning audio representations pretrained on unlabeled single-modality speech data. Fine-tuning a pretrained video encoder also makes a contribution, but is less crucial to generalization from dialog to narration.
Limitations and Future Work To better understand what aspects of language are learning in our setting, we need to carry out in-depth analyses of learned representations on sub-word, lexical, and phrasal levels. It would also be worthwhile to figure out the details of how specifically temporal information in video contributes to acquiring linguistic knowledge. Some analyses in this direction are currently constrained by the size of the evaluation dataset, and more large-scale datasets are needed in the future. We model the acquisition of spoken language from language-internal correlations as well as from grounding in vision by fine-tuning an audio encoder pretrained on read speech. This approach is rather simplistic and does not match the real experience of language learners. It would be interesting to make the setting more realistic by using pretraining data which reflect a young learner's experience more closely, and to realistically interleave learning via self-supervision from speech and via grounding in vision.
feature written transcriptions aligned with the audio as well as segmentation into dialog and narration. 3 Dialogs are the parts spoken by the characters, while narrations are comments inserted by the narrator, which are more descriptive in nature. All the narration segments Duration in hours and number of clips (FIXED condition) for all dataset splits. are uttered by the same voice actor. We use the dialogs for training the model, and set aside the narrations for evaluation purposes only. A small portion of the dialog data is also used for validation. Specifically, out of the total 209 episodes, we use dialog from episodes 1-196 for training, and 197-209 for validation. We set aside narrations from episodes 1-104 for validation and 105-209 for testing. We disregard portions of the video that are annotated as neither dialog nor narration:.
Acknowledgments We would like to thank  Nikos Papasarantopoulos  and  Shay B. Cohen  for creating the Peppa Pig annotations and for sharing them with us. Part of the  NWO/E-Science Center  grant number  027. 018. G03 was used to purchase the video data on DVD. Thanks to  Bertrand Higy  for taking care of this purchase, as well as for sharing his ideas with us in the initial stages of this work. We also thank  Abdellah Fourtassi , and three anonymous reviewers for their useful feedback. This work was supported by grants  ANR-16-CONV-0002 AMX-19-IET-009  ( Archimedes Institute ), the  Excellence Initiative of Aix-Marseille University  (A*MIDEX).
Contrary to the domain-based and space-filling approaches where the training database is selected according to its point-to-point distance distribution, the adaptive selection is based on the surrogate model's information. The solutions to the CPU challenge of repetitive costly simulations reviewed in this paper aim at reducing the training and evaluation costs of metamodels. On the one hand, the MFMs combine variable simulation costs and, on the other hand, the ROMs allow to handle high-dimensional outputs while taking into account the vector representation of the simulation data (typically the discretized solutions of fluid state variables in CFD). Compared to scalar approximation models of Quantities of Interest (QoI), they are supposed to provide better insight into the physical model  [47].
S = T  (18)  where and T matrices are respectively composed of the left and right singular vectors, of size n × M and M × n, is a diagonal matrix containing the singular values of S and equivalent when squared to eigen values of the "big" (n × n) C = SS T or "small" M × M correlation matrix c = S T S. Note that the decomposition may be performed by different approaches, the most common being "economical" SVD. In the offline phase, this method is used to obtain the decomposition for the initial dataset, and once in the online phase, its incremental version  [155]  uses only the newly added points instead of the overall DoE at each iteration. Often the basis is truncated to lower the computational burden thus speed-up computation. The idea behind the truncation is to consider the most relevant modes only.
Greedy approaches The principle of a greedy algorithm is to choose the most promising elements (or to suppress the least promising ones) iteratively to obtain the best prediction. Once the models are built in the off-line phase, the most (or least) predictive features may be selected during the online phase based on error criteria. The ROM database model's accuracy can be adjusted in the offline phase, where a local ROM database is built by a greedy procedure  [162]. The greedy algorithm is used for the selection of snapshots for the approximation of the parameterized PDE solution  [147, 216]  and in a non-intrusive framework  [67]. Such approach was applied by  [10]  to time-dependent non linear prob-lems using RBF-based NIROM. This model adaptively generates snapshots based on a greedy approach to minimize the overall computational cost.
Adaptive Gappy POD  [32]  use an adaptive Gappy POD (GPOD) (see "Reduced-order modeling" section) for two-dimensional airfoil inverse design. This approach aims at finding the optimal sensor locations and adaptively improve the POD basis. At each iteration, the criterion used to achieve the selection is based on the Gappy POD maximum error. The complete model information is then generated by solving the system between known, and unknown data at this position, the database is enriched using the resulting snapshot, and the reduced basis is updated. These steps are then repeated, using the updated model to calculate the reduced states at each locations around the airfoil. Active-learning approaches The NIPOD was adapted to an active-learning  4  framework using the partitioning approach in the LDM method  [65].
The final interpolation of the field is obtained by projecting the MF field on a reduced MF basis MF = [ HF | LF ]. Unlike the MF NIPOD extension of  [139] , the MFNIPOD enriches the reduced space obtained from scarce HF information with LF related orthogonal modes, giving a hierarchical ROM. The method was compared to a model based on a single-fidelity RBF Neural Network (RBFNN) and co-Kriging on a constrained 2D analytical optimization and the 19-D optimization of an industrial compressor blade and its non-axisymmetric hub. The results show that MFNIPOD outperforms models based on co-Kriging and RBFNN in terms of costs and accuracy. Combining different data simulations from multiple fidelity snapshots presents difficulties because the two sets of fields lie most likely in different spaces.
The predicted classes include binary malignancy, cancer probability estimation, breast density, and image laterality. Since few samples have all classes available and confirmed, we propose to introduce the uncertainty related to the classes as a per-sample weight during training. Such weighting prevents updating the network's parameters when training on uncertain or missing labels. We evaluate our approach on the public INBreast and private datasets, showing statistically significant improvements compared to baseline and independent state-of-the-art approaches. Moreover, we use mammograms from Susan G. Komen Tissue Bank for fine-tuning, further demonstrating the ability to improve the performances in our multi-task learning setup from raw clinical data. We achieved the binary classification performance of AUC = 80.
Most recent CAD algorithms  (9, (16) (17) (18) , based on supervised Deep Learning (DL) approaches, rely on a set of samples for training. The model's success during test time will largely depend on this dataset, which should be representative of the variations, clinically relevant, and preferably include explicit ground truth annotations. As we discuss next, these requirements are challenging in the context of mammography imaging analysis. First, access to clinical data is restricted due to patient privacy concerns. Healthcare providers holding imaging databases are not allowed to share the data, either for free or for a fee, without patients' consent. That is, patients shall be informed and confirm that they agree to their data being used for research or software development purposes. Hence, a protocol compliant with regulations needs to be carefully designed before sharing images with third parties.
Second, clinical data from screening protocols are strongly imbalanced. With breast cancer prevalence in a range between 1 and 2% 1 , screening imaging databases are mainly composed of negative cases while the true positive (biopsy-proven) cases are generally under-represented. Third, ground truth for the images is not always easy to collect. Generally, each clinical case is composed of a set of images and a descriptive clinical report. Often, reports contain information about the images (number and type of acquisitions), physiological details (breast density), and the most important information about the diagnosis. Depending on radiologists' practice in each site, the level of details may significantly vary. Also, while the clinical report of a mammography usually contains an ACR classification indicating the probability of cancer, the actual confirmation of the malignancy may not be available; for instance, the biopsy reports can be stored in a different database (e.
g. , paper hard-copies), or be effectively missing. Besides, there is a certain number of false-negative mammography exams, estimated to 10 -15% in  (9, 19). Hence, despite the substantial volume of screening data, image databases relying on labels from clinical repots can be poorly and sometimes erroneously annotated, leading to significant label uncertainties. The issues above make the design and training of deep learning CAD solutions difficult. To better exploit raw clinical databases, in this work, we propose a Multi-Task Learning (MTL) approach allowing us to maximize the quantity of data used for training, without the need to carefully curate sample annotations. To this end, we propose to simultaneously train a neural network for multiple tasks relevant to mammography. Our main purpose is to mitigate the data (epistemic) uncertainty by encouraging the network to learn common features despite missing or noisy labels, thereby improving the performances of each particular task.
Related Work Deep learning applied to mammography imaging has been broadly studied in past years  (21) (22) (23). Recent works include deep neural networks for classification  (17, 18, 20, 24) , detection  (16, 25) , and segmentation  (26) (27) (28). Several successful methods for breast cancer classification from mammograms have been recently proposed. Shen et al. (17)  describe a binary-classification method trained on two datasets, CBIS-DDSM  (29)  and INBreast  (30)  reaching AUC = 95. 00 on selected validation samples from INBreast. Ribli et al. (16)  approach the classification task with a detection method. The detector is trained on two datasets: DDSM  (31)  and privately collected by the authors. Both datasets includes manually delineated Regions of Interest (ROI) around lesions. Using the entire INBreast dataset for validation (16) claim a similar AUC = 95.
00. Other works, such as  (18, 32, 33)  use substantially larger private datasets for training and testing, with AUC scores of 89. 50, 93. 00, and 94. 50, respectively. Both  (18)  and  (32)  rely only on image-wise labels (i. e. , without ROI). Wu et al. (18)  classify mammography cases (composed of four views) by combining features extracted from each view. Shen et al. (32)  perform image-wise classification and introduce an additional step that evaluates patches extracted from the image using saliency map to get the most relevant patches. Alternatively, Lotter et al. (33)  train a network using explicit pixel-wise ground truth for detection, before fine-tuning with image-wise labels on a larger dataset. There is a trend toward algorithms processing multiple mammography views of the same patient simultaneously  (34) (35) (36) (37) (38).
g. , last) feature level. In this way, we leave the network select the features relevant to a given task automatically during learning. Finally, we propose to introduce a measure of uncertainty in the training process. Contrary to  (54) , who quantify the uncertainty of output predictions, we propose to use the uncertainty estimates within the optimization function similar to  (55). However, we propose to rely on prior knowledge about the dataset to scale the uncertainty, instead of using the uncertainty coming from the network as in  (55). In a way, our approach relates to the work of  (56) , who use inter-rater agreement as an uncertainty indicator.