dataesr/scientific-paragraphs-categorization huggingface dataset:
We then automatically apply rotating, flat-disk 3D models to each cubelet using both FAT and 3D-Barolo in the same manner across all selected sources. The two algorithms differ significantly in initialisation and optimisation even when applying the same underlying model to a dataset  (Di Teodoro & Fraternali 2015; Kamphuis et al. 2015). We therefore use differences between the code outputs to judge model reliability and estimate uncertainties. We visually examine the output models to assess whether or not they are successful, and only sources for which both 3D-Barolo and FAT produce physically plausible models are further considered. For those sources, we average the best-fitting disk geometries of the successful model pairs to derive the optimised disk geometry. We then derive an interpolated average rotation curve from the individual model outputs, inclination-corrected to the optimised disk geometry, and adopt half the difference between the two models as the uncertainty.
Access via CADC CADC also offers Virtual Observatory services for querying catalogues and data products, but unlike CASDA does not require user j https://astroquery. readthedocs. io/en/latest/casda/casda. html. registration. CADC provides user guides for using TAP services k and ADQL queries. l The CADC Advanced Search tool m is a browser interface to all of the multi-wavelength, multi-facility data collections that it stores. The interface itself builds ADQL queries based on the inputs to the search engine that can be used elsewhere. CADC Advanced Search is also integrated with a powerful 'sky view' that shows the coverage of a particular area by WALLABY and other surveys in the archive. k https://www. cadc-ccda. hia-iha. nrc-cnrc. gc. ca/en/doc/tap/. l https://github. com/opencadc/notebook-tutorials.
Lastly, some modal-based methods have in effect been developed  [22] , but the corresponding tools are and will remain proprietary (see for example  [23] ) Therefore, we implemented an approach solving all the aforementioned shortcomings. It involves working within a general-purpose Finite Elements Model (in our case: ANSYS Mechanical APDL, see Fig. 1 ), and extending its capabilities using its recently introduced matrix manipulation language (APDL Math). Specifically, two additional capabilities have been implemented: (1) thermal modal analysis and (2) thermal harmonic analysis (see Code 1-4, Supplementary Materials). These additional tools allow for rigorously deriving the thermal modes and the Thermal Transfer Function (TTF) of the system. Knowing the statistical properties of the external temperature fluctuations by means of their Power Spectral Density (PSD), it is possible to quantitatively estimate the temperature response properties.
Introduction The objective of this work is to develop a sign spotting model that can identify and localise instances of signs within sequences of continuous sign language (see Fig. 1 ). Sign languages represent the natural means of communication for deaf communities (Sutton-Spence and Woll 1999) and sign spotting has a broad range of practical applications. Exam-ples include: indexing videos of signing content by keyword to enable content-based search; gathering diverse dictionaries of sign exemplars from unlabelled footage for linguistic study; automatic feedback for language students via an "autocorrect" tool (e. g. "did you mean this sign?"); making voice activated wake word devices available to deaf communities; and building sign language datasets by automatically labelling examples of signs.
Recently, deep neural networks, equipped with largescale, labelled datasets produced considerable progress in audio  (Coucke et al. 2019; Véniat et al. 2019 ) and visual  (Momeni et al. 2020; Stafylakis and Tzimiropoulos 2018)  keyword spotting in spoken languages. However, a direct replication of these keyword spotting successes in sign language requires a commensurate quantity of labelled data [(note that modern audiovisual spoken keyword spotting datasets contain millions of densely labelled examples  (Chung et al. 2017; Afouras et al. 2018) ], but such datasets are not available for sign language. It might be thought that a sign language dictionary would offer a relatively straightforward solution to the sign spotting task, particularly to the problem of covering only a limited vocabulary in existing large-scale corpora.
To address these challenges, we propose a unified framework in which sign spotting embeddings are learned from the dictionary (to provide broad coverage of the lexicon) in combination with two additional sources of supervision. In aggregate, these multiple types of supervision include: (1) watching sign language and learning from existing sparse annotations obtained from mouthing cues  (Albanie et al. 2020) ; (2) exploiting weak-supervision by reading the subtitles that accompany the footage and extracting candidates for signs that we expect to be present; (3) looking up words (for which we do not have labelled examples) in a sign language dictionary. The recent development of a large-scale, subtitled dataset of continuous signing providing sparse annotations  (Albanie et al. 2020)  allows us to study this problem setting directly.
We make the following contributions, originally introduced in  Momeni et al. (2020) : (1) We provide a machine readable British Sign Language (BSL) dictionary dataset of isolated signs, BslDict, to facilitate study of the sign spotting task; (2) We propose a unified Multiple Instance Learning framework for learning sign embeddings suitable for spotting from three supervisory sources; (3) We vali-date the effectiveness of our approach on a co-articulated sign spotting benchmark for which only a small number (low-shot) of isolated signs are provided as labelled training examples, and (4) achieve state-of-the-art performance on the BSL-1K sign spotting benchmark  (Albanie et al. 2020 ) (closed vocabulary). We show qualitatively that the learned embeddings can be used to (5) automatically mine new signing examples, and (6) discover "faux amis" (false friends) between sign languages.
Related Work Our work relates to several themes in the literature: sign language recognition (and more specifically sign spotting), sign language datasets, multiple instance learning and low-shot action localization. We discuss each of these themes next. Sign language recognition. The study of automatic sign recognition has a rich history in the computer vision community stretching back over 30 years, with early methods developing carefully engineered features to model trajectories and shape  (Kadir et al. 2004 ;  Tamura and Kawasaki 1988; Starner 1995; Fillbrandt et al. 2003). A series of techniques then emerged which made effective use of hand and body pose cues through robust keypoint estimation encodings  (Buehler et al. 2009; Cooper et al. 2011; Ong et al. 2012; Pfister et al. 2014). Sign language recognition also has been considered in the context of sequence prediction, with HMMs  (Agris et al.
The concurrent work of  Li et al. (2020)  also seeks to align representation of isolated and continuous signs. However, our work differs from theirs in several key aspects: (1) rather than assuming access to a large-scale labelled dataset of isolated signs, we consider the setting in which only a handful of dictionary examples may be used to represent a word; (2) we develop a generalised Multiple Instance Learning framework which allows the learning of representations from weakly-aligned subtitles whilst exploiting sparse labels from mouthings  (Albanie et al. 2020 ) and dictionaries (this integrates cues beyond the learning formulation in  Li et al. (2020) ); (3) we seek to label and improve performance on co-articulated signing (rather than improving recognition performance on isolated signing).
Also related to our work,  (Pfister et al. 2014 ) uses a "reservoir" of weakly labelled sign footage to improve the performance of a sign classifier learned from a small number of examples. Different to  Pfister et al. (2014) , we propose a multiple instance learning formulation that explicitly accounts for signing variations that are present in the dictionary (Fig. 2 ). Sign language datasets. A number of sign language datasets have been proposed for studying Finnish  (Viitaniemi et al. 2014 ), German (Koller et al. 2015; Agris et al. 2008) , American  (Athitsos et al. 2008; Joze and Koller 2019; Li et al. 2019; Wilbur and Kak 2006)  and Chinese  (Chai et al. 2014; Huang et al. 2018 ) sign recognition. For British Sign Language (BSL),  (Schembri et al. 2013)  gathered the BSL Corpus which represents continuous signing, labelled with fine-grained linguistic annotations.
More recently  (Albanie et al. 2020 ) collected BSL-1K, a large-scale dataset of BSL signs that were obtained using a mouthing-based keyword spotting model. Further details on this method are given in Sect. 3. 1. In this work, we contribute BslDict, a dictionary-style dataset that is complementary to the datasets of  Schembri et al. (2013) ,  Albanie et al. (2020) -it contains only a handful of instances of each sign, but achieves a comprehensive coverage of the BSL lexicon with a 9K English vocabulary (vs a 1K vocabulary in  Albanie et al. (2020) ). As we show in the sequel, this dataset enables a number of sign spotting applications. While BslDict does not represent a linguistic corpus, as the correspondences to English words and phrases are not carefully annotated with glosses 2  , it is significantly larger than its linguistic counterparts (e.
3 , Left, Stage 2). For building the BSL-1K dataset,  Albanie et al. (2020)  use a probability threshold of 0. 5 and runs the visual keyword spotter with a vocabulary of 1350 keywords across 1000 h of signing. A further filtering step is performed on the vocabulary to ensure that each word included in the dataset is represented with high confidence (at least one instance with confidence 0. 8) in the training partition, which produces a final dataset vocabulary of 1064 words. The resulting BSL-1K dataset has 273K mouthing annotations, some of which are illustrated in Fig. 3  (right). We employ these annotations directly to form the set M in this work.
Implementation Details In this section, we provide details for the learning framework covering the embedding architecture, sampling protocol and optimization procedure. Embedding architecture. The architecture comprises an I3D spatio-temporal trunk network (Carreira and Zisserman 2017) to which we attach an MLP consisting of three linear layers separated by leaky ReLU activations (with negative slope 0. 2) and a skip connection. The trunk network takes as input 16 frames from a 224 × 224 resolution video clip and produces 1024-dimensional embeddings which are then projected to 256-dimensional sign spotting embeddings by the MLP. More details about the embedding architecture can be found in. Joint pretraining. The I3D trunk parameters are initialised by pretraining for sign classification jointly over the sparse annotations M of a continuous signing dataset (BSL-1K  Albanie et al.
2020 ) and examples from a sign dic-tionary dataset (BslDict) which fall within their common vocabulary. Since we find that dictionary videos of isolated signs tend to be performed more slowly, we uniformly sample 16 frames from each dictionary video with a random shift and random frame rate n times, where n is proportional to the length of the video, and pass these clips through the I3D trunk then average the resulting vectors before they are processed by the MLP to produce the final dictionary embeddings. We find that this form of random sampling performs better than sampling 16 consecutive frames from the isolated signing videos (see for more details). During pretraining, minibatches of size 4 are used; and colour, scale and horizontal flip augmentations are applied to the input video, following the procedure described in  Albanie et al.
(2020). The trunk parameters are then frozen and the MLP outputs are used as embeddings. Both datasets are described in detail in Sect. 4. 1. Minibatch sampling. To train the MLP given the pretrained I3D features, we sample data by first iterating over the set of labelled segments comprising the sparse annotations, M, that accompany the dataset of continuous, subtitled sampling to form minibatches. For each continuous video, we sample 16 consecutive frames around the annotated timestamp (more precisely a random offset within 20 frames before, 5 frames after, following the timing study in  Albanie et al. (2020) ). We randomly sample 10 additional 16-frame clips from this video outside of the labelled window, i. e. , continuous background segments. For each subtitled sequence, we sample the dictionary entries for all subtitle words that appear in V L (see Fig.
Experiments In this section, we first present the datasets used in this work (including the contributed BslDict dataset) in Sect. 4. 1, followed by the evaluation protocol in Sect. 4. 2. We then illustrate the benefits of the Watch, Read and Lookup learning framework for sign spotting against several baselines (Sect. 4. 3) with a comprehensive ablation study that validates our design choices (Sect. 4. 4). Next, we investigate three applications of our method in Sect. 4. 5, showing that it can be used to (i) not only spot signs, but also identify the specific sign variant that was used, (ii) label sign instances in continuous signing footage given the associated subtitles, and (iii) discover "faux amis" between different sign languages. We then provide experiments on sign language recognition, significantly improving the state of the art by applying our labelling technique to obtain more training examples automatically (Sects.
Datasets Although our method is conceptually applicable to a number of sign languages, in this work we focus primarily on BSL, the sign language of British deaf communities. We use BSL-1K  (Albanie et al. 2020 ), a large-scale, subtitled and sparsely annotated dataset of more than 1000 h of continuous signing which offers an ideal setting in which to evaluate the effectiveness of the Watch, Read and Lookup sign spotting framework. To provide dictionary data for the lookup component of our approach, we also contribute BslDict, a diverse visual dictionary of signs. These two datasets are summarised in Table  1  and described in more detail below. We further include experiments on a new dataset, BOBSL  (Albanie et al. 2021 ), which we describe in Sect. 4. 7 together with results. The BOBSL dataset has similar properties to BSL-1K.
BSL-1K comprises over 1000 hours of video of continuous sign-language-interpreted television broadcasts, with accompanying subtitles of the audio content  (Albanie et al. 2020). In  Albanie et al. (2020) , this data is processed for the task of individual sign recognition: a visual keyword spotter is applied to signer mouthings giving a total of 273K sparsely localised sign annotations from a vocabulary of 1064 signs (169K in the training partition as shown in Table  1 ). Please refer to Sect. 3. 1 and  (Albanie et al. 2020 ) for more details on the automatic annotation pipeline. We refer to Sect. 4. 6 for a description of the BSL-1K sign recognition benchmark (Test Rec 2K and Test Rec 37K in Table  1 ). In this work, we process this data for the task of retrieval, extracting long videos with associated subtitles.
With this process, the average number of frames per  video drops from 78 to 56 (still significantly larger than coarticulated signs). To the best of our knowledge, BslDict is the first curated, BSL sign dictionary dataset for computer vision research. A collection of metadata associated for the BslDict dataset is made publicly available, as well as our pre-computed video embeddings from this work. For the experiments in which BslDict is filtered to the 1064 vocabulary of BSL-1K, we have 3K videos as shown in Table  1. Within this subset, each sign has between 1 and 10 examples (average of 3).
Comparison to Baselines In this section, we evaluate different components of our approach. We first compare our contrastive learning approach with classification baselines. Then, we investigate the effect of our multiple-instance loss formulation. Finally, we report performance on a sign spotting benchmark. I3D baselines. We start by evaluating baseline I3D models trained with classification on the task of spotting, using the embeddings before the classification layer. We have three variants in Table 2: (i) I3D BSL-1K provided by  Albanie et al. (2020)  which is trained only on the BSL-1K dataset, and we also train (ii) I3D BslDict and (iii) I3D BSL-1K,BslDict. Training only on BslDict (I3D BslDict ) performs significantly worse due to the few examples available per class and the domain gap that must be bridged to spot co-articulated signs, suggesting that dictionary samples alone do not suffice to solve the task.
We observe improvements with fine-tuning I3D BSL-1K jointly on the two datasets (I3D BSL-1K,BslDict ), which becomes our base feature extractor for the remaining experiments to train a shallow MLP. Loss formulation. We first train the MLP parameters on top of the frozen I3D trunk with classification to establish a baseline in a comparable setup. Note that, this shallow architecture can be trained with larger batches than I3D. Next, we investigate variants of our loss to learn a joint sign embedding between BSL-1K and BslDict video domains: (i) standard single-instance InfoNCE  (Oord et al. 2018; Wu et al. 2018 ) loss which pairs each BSL-1K video clip with one positive BslDict clip of the same sign, (ii) Watch-Lookup which considers multiple positive dictionary candidates, but does not consider subtitles (therefore limited to the annotated video clips).
We conclude from Fig. 5b  that setting τ to values between [0. 04-0. 10] does not impact the performance significantly; therefore, we keep τ = 0. 07 following the previous work  (Wu et al. 2018; He et al. 2020 ) for all other experiments. However, values outside this range negatively impact the performance, especially for high values, i. e. , {0. 50, 1. 00}; we observe a major decrease in performance when τ approaches 1. Mouthing confidence threshold at training. As explained in Sect. 3. 1, the sparse annotations from the BSL-1K dataset are obtained automatically by running a visual keyword spotting method based on mouthing cues. The dataset provides a confidence value associated with each label ranging between 0. 5 and 1. 0. Similar to  Albanie et al. (2020) , we experiment with different thresholds to determine the training set.
Applications In this section, we investigate three applications of our sign spotting method. Sign variant identification. We show the ability of our model to spot specifically which variant of the sign was used. In Fig. 6 , we observe high similarity scores when the variant of the sign matches in both BSL-1K and BslDict videos. Identifying such sign variations allows a better understanding of regional differences and can potentially help standardisation efforts of BSL. Dense annotations. We demonstrate the potential of our model to obtain dense annotations on continuous sign language video data. Sign spotting through the use of sign dictionaries is not limited to mouthings as in  Albanie et al. (2020)  and therefore is of great importance to scale up datasets for learning more robust sign language models.
We retrieve nearest neighbors according to visual embedding similarities between BslDict which has a 9K vocabulary and WLASL  (Li et al. 2019 ), an ASL isolated sign language dataset with a 2K vocabulary. We provide some examples in Fig. 8. We automatically identify several signs with similar manual features some of which correspond to different meanings in English (left), as well as same meanings, such as "ball", "stand", "umbrella" (right).
Results on the BOBSL Dataset BOBSL is a dataset similar to BSL-1K; however, unlike BSL-1K, BOBSL is publicly available  (Albanie et al. 2021). The dataset consists of 1400 h of BSL-interpreted BBC broadcast footage accompanied by written English subtitles. We repeated our sign spotting techniques on this data using mouthing and dictionary cues in combination with subtitles. Keyword spotting with mouthing follows our previous work  (Albanie et al. 2020 ) and obtains 502K sign localisations over 0. 5 confidence (M 0. 5 ). Sign spotting with dictionaries is similar to the procedure described in Sect. 4. 6, resulting in 727K sign localisations over 0. 75 similarity (D 0. 75 ). In Table  8 , we present sign recognition results using these automatic annotations for classification training over a vocabulary of 2281 categories.
Limitations In this section, we investigate failure modes of our sign spotting mechanism, in particular by using the data obtained through manual verifications. More specifically, we make statistics from 10K annotations on the BOBSL test set that were obtained via dictionary spotting through subtitles. From these, 76% were marked as correct. In Fig. 10 , we present a breakdown for per-word accuracy to check whether certain signs fail more than others. We note two main failure modes: (i) fingerspelled words (e. g. , 'dvd', 'dna') are difficult for the model, perhaps due to sparse frame sampling from long dictionary videos, (ii) common words such as 'even' and 'able' may have context-dependent meanings; querying these words due to occurrence in subtitles lead to false positives. In Fig. 11 , we further visualise samples from this manually verified set of spottings.
Conclusions We have presented an approach to spot signs in continuous sign language videos using visual sign dictionary videos, and have shown the benefits of leveraging multiple supervisory signals available in a realistic setting: (i) sparse annotations in continuous signing (in our case, from mouthings), (ii) accompanied with subtitles, and (iii) a few dictionary samples per word from a large vocabulary. We employ multiple-instance contrastive learning to incorporate these signals into a unified framework. We finally propose several potential applications of sign spotting and demonstrate its ability to scale up sign language datasets for training strong sign language recognition models. Here, we investigate removing such class-balancing constraint. In that case, we make sure we do not mark samples with the same labels as negatives, instead we discard them.
B. 3 Language-Aware Negative Sampling Working with a large vocabulary of words brings the additional challenge of handling synonyms. We consider two types of similarities. First, two different categories in the BslDict sign dictionary may belong to the same sign category if the corresponding English words are synonyms. Second, the meta-data we have collected with the BslDict dataset provides similarity labels between sign categories, which may be used to group certain signs. In this work, we have largely ignored this issue by associating each sign to a single word. This results in constructing negative pairs for We use the last 20 frames before the mouthing annotations with confidence above 0. 5. We do not obtain gains with the S3D architecture; therefore, we use I3D in all the experiments to compute video features two identical signs such as 'happy' and 'content'.