dataesr/scientific-paragraphs-categorization huggingface dataset:
Leafy automata are a model of computation over an infinite alphabet. This area has been explored extensively, partly motivated by applications to database theory, notably XML  [43]. In this context, nested data first appeared in  [8] , where the authors considered shuffle expressions as the defining formalism. Later on, data automata  [10]  and class memory automata  [9]  have been adapted to nested data in  [16, 14]. They are similar to leafy automata in that the automaton is allowed to access states related to previous uses of data values at various depths. What distinguishes leafy automata is that the lifetime of a data value is precisely defined and follows a question and answer discipline in correspondence with game semantics. Leafy automata also feature run-time "zero-tests", activated when reading answers.
Leafy automata We would like to be able to represent the game semantics of FICA using automata. To that end, we introduce leafy automata (LA). They are a variant of automata over nested data, i. e. a type of automata that read finite sequences of letters of the form (t, d 0 d 1 • • • d j ) (j ∈ N) , where t is a tag from a finite set Σ and each d i (0 ≤ i ≤ j) is a data value from an infinite set D. In our case, D will have the structure of a countably infinite forest and the sequences d 0 • • • d j will correspond to branches of a tree. Thus, instead of -Infinite branching: pred -1 ({d ⊥ }) is infinite for any d ⊥ ∈ D ∪ {⊥}. d 0 • • • d j , -Well-foundedness: for any d ∈ D, there exists i ∈ N, called the level of d, such that pred i+1 (d) = ⊥. Level-0 data values will be called roots. In order to define configurations of leafy automata, we will rely on finite subtrees of D, whose nodes will be labelled with states.
In Section 2, we introduce the concept of IDE as Code and the specifications required for its implementation. Section 3 puts the emphasis on the need for a higher level specification of language protocols. A possible implementation through microservices orchestration is presented and discussed in Section 4. Some related works are referenced in Section 5. And finally, we propose a research agenda for the questions raised by this paper in Section 6. 5 cf. https://www. eclipse. org/glsp/ 6 cf. https://obeonetwork. github. io/GraphicalServerProtocol/.
VISION Our vision aims at providing more customizable and adaptable IDEs for the end users through language services independent of any client. It is driven both by the need to support domain-specific language services in multiple environments and to contribute in the trend of turning IDEs into web platforms. When we consider programming languages in the large, including both general-purpose and domain-specific languages, we need to account for an ever-growing number of language services. In contrast, at a given time, an IDE can only manage a subset of these. Since protocols such as LSP and DAP were designed around the features available in Visual Studio Code, it made perfect sense for Microsoft to also fix the set of the capabilities they offer to ensure their full support in their environment.
SPECIFYING LANGUAGE PROTOCOLS The first step before attaining an IDE as Code is to unify the specifications of language protocols. Current language protocols are basically provided as bi-directional APIs. While the data-flow between servers and clients is precisely specified, the control-flow is not explicit and can only be deduced from processing the documentation available in natural language. As a consequence, even though maintainers for servers and clients should be free of any specific implementation, they end up relying on Visual Studio Code as a reference. 7 Here, we propose that the control-flow between language services should be formalized and become a part of the protocol specification. The implementation of language service packages needs to separate this concern from the actual implementation.
In order to support resource scaling for web IDEs that need to serve multiple clients at the same time, the concept of language service packages is implemented as microservices. An event broker lets both language microservices and UI microservices interact as part of a choreography. In the figure, UI microservices are all included in the same platform, but technically they could also be deployed and integrated into others: the service displaying the execution trace could very well be embedded inside a completely independent web page, or as part of a Jupyter Notebook. Other implementation approaches such as OSGi plugins or Visual Studio Code extensions could also be considered, but comparing them is out of the scope of this paper. Here, we also made the decision to use JSON serialization and websockets for the transport in order to stay close to LSP implementations, but we do not argue that these are the most efficient nor the best choices.
Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analysing this knowledge in LMs specifically trained for different languages (English, French, Spanish and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words' polysemy level and their partitionability into senses. Polysemyrelated information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels.
Whether these models encode knowledge about lexical polysemy and sense distinc-tions is, however, still an open question. Our work aims to fill this gap. We propose methodology for exploring the knowledge about word senses in contextualised representations. Our approach follows a rigorous experimental protocol proper to lexical semantic analysis, which involves the use of datasets carefully designed to reflect different sense distributions. This allows us to investigate the knowledge models acquire during training, and the influence of context variation on token representations. We account for the strong correlation between word frequency and number of senses  (Zipf, 1945) , and for the relationship between grammatical category and polysemy, by balancing the frequency and part of speech (PoS) distributions in our datasets and applying a frequency-based model to polysemy prediction.
Importantly, our investigation encompasses monolingual models in different languages (English, French, Spanish and Greek) and multilingual BERT (mBERT). We demonstrate that BERT contextualised representations encode an impressive amount of knowledge about polysemy, and are able to distinguish monosemous (mono) from polysemous (poly) words in a variety of settings and configurations (cf. Figure  1 ). Importantly, we demonstrate that representations derived from contextual LMs encode knowledge about words' polysemy acquired through pretraining which is combined with information from new contexts of use (Sections 3-6). Additionally, we show that BERT representations can serve to determine how easy it is to partition a word's semantic space into senses (Section 7). Our methodology can serve for the analysis of words and datasets from different topics, domains and languages.
Related Work The knowledge pre-trained contextual LMs encode about lexical semantics has only recently started being explored. Works by  Reif et al. (2019)  and  Wiedemann et al. (2019)  propose experiments using representations built from Wikipedia and the SemCor corpus  (Miller et al. , 1993) , and show that BERT can organise word usages in the semantic space in a way that reflects the meaning distinctions present in the data. It is also shown that BERT can perform well in the word sense disambiguation (WSD) task by leveraging the senserelated information available in these resources. These works address the disambiguation capabilities of the model but do not show what BERT actually knows about words' polysemy, which is the main axis of our work. In our experiments, sense annotations are not used to guide the models into establishing sense distinctions, but rather for creating controlled conditions that allow us to analyse BERT's inherent knowledge of lexical polysemy.
Dataset Creation We build our English dataset using SemCor 3. 0  (Miller et al. , 1993) , a corpus manually annotated with WordNet senses  (Fellbaum, 1998). It is important to note that we do not use the annotations for training or evaluating any of the models. These only serve to control the composition of the sentence pools that are used for generating contextualised representations, and to analyse the results. We form sentence pools for monosemous (mono) and polysemous (poly) words that occur at least ten times in SemCor. 3 For each mono word, we randomly sample ten of its instances in the corpus. For each poly word, we form three sentence pools of size ten reflecting different sense distributions: • Balanced (poly-bal). We sample a sentence for each sense of the word in SemCor until a pool of ten sentences is formed.
The controlled composition of the poly sentence pools allows us to investigate the behaviour of the models when they are exposed to instances of polysemous words describing the same or different senses. There are 1,765 poly words in Sem-Cor with at least 10 sentences available. 5 We randomly subsample 418 from these in order to balance the mono and poly classes. Our English dataset is composed of 836 mono and poly words, and their instances in 8,195 unique sentences. Table  1  shows a sample of the sentences in different pools. For French, Spanish and Greek, we retrieve sentences from the Eurosense corpus  (Delli Bovi et al. , 2017)  which contains texts from Europarl automatically annotated with BabelNet word senses  (Navigli and Ponzetto, 2012). 6 We extract sentences from the high precision version 7  of Eurosense, and create sentence pools in the same way as in English, balancing the number of monosemous and polysemous words (418).
Contextualised Word Representations We experiment with representations generated by three English models: BERT  (Devlin et al. , 2019)   9  , ELMo  (Peters et al. , 2018)  and con-text2vec  (Melamud et al. , 2016). BERT is a Transformer architecture  (Vaswani et al. , 2017)  that is jointly trained for a masked LM and a next sentence prediction task. Masked LM inolves a Cloze-style task, where the model needs to guess randomly masked words by jointly conditioning on their left and right context. We use the bert-base-uncased and bert-base-cased models, pre-trained on the BooksCorpus  (Zhu et al. , 2015)  and English Wikipedia. ELMo is a bi-LSTM LM trained on Wikipedia and news crawl data from WMT 2008-2012. We use 1024-d representations from the 5. 5B model. 10 Context2vec is a neural model based on word2vec's CBOW architecture  (Mikolov et al.
, 2013)  which learns embeddings of wide sentential contexts using a bi-LSTM. The model produces representations for words and their context. We use the context representations from a 600-d context2vec model pre-trained on the ukWaC corpus (Baroni et al. , 2009). 11  For French, Spanish and Greek, we use BERT models specifically trained for each language: Flaubert (flaubert_base_uncased)  (Le et al. , 2020) , BETO (bert-base-spanishwwm-uncased)  (Cañete et al. , 2020) , and Greek BERT (bert-base-greek-uncased-v1)  (Koutsikakis et al. , 2020). We also use the bert-base-multilingual-cased model for each of the four languages. mBERT was trained on Wikipedia data of 104 languages. 12 All BERT models generate 768-d representations.
The Self-Similarity Metric All models produce representations that describe word meaning in specific contexts of use. For each instance i of a target word w in a sentence, we extract its representation from: (i) each of the 12 layers of a BERT model; 13 (ii) each of the three context2vec 12 The mBERT model developers recommend using the cased version of the model rather than the uncased one, especially for languages with non-Latin alphabets, because it fixes normalisation issues. More details about this model can be found here: https://github. com/google-research/bert/ blob/master/multilingual. md. 13 We also tried different combinations of the last four layers, but this did not improve the results. When a word is split ELMo layers; (iii) context2vec. We calculate selfsimilarity (Self Sim)  (Ethayarajh, 2019)  for w in a sentence pool p and a layer l, by taking the average of the pairwise cosine similarities of the representations of its instances in l: Self Sim l (w) = 1 |I| 2 -|I| i∈I j∈I j =i cos(x wli , x wlj ) (1) In formula 1, |I| is the number of instances for w (ten in our experiments); x wli and x wlj are the representations for instances i and j of w in layer l.
Mono-Poly in Other Languages The top row of Figure  2  shows the average Self Sim obtained for French, Spanish and Greek words using monolingual models. Flaubert, BETO and Greek BERT representations clearly distinguish mono and poly words, but average Self Sim values for different poly pools are much closer than in English. BETO seems to capture these fine-grained distinctions slightly better than the French and Greek models. The second row of the Figure shows results obtained with mBERT representations. We observe the highly similar average Self Sim values assigned to different poly pools, which show that distinction is harder than in monolingual models. Statistical tests show that the difference between Self Sim values in mono and poly-rand is significant in all layers of BETO, Flaubert, Greek BERT, and mBERT for Spanish and French.
Overall, our results highlight that BERT encodes higher quality knowledge about polysemy. We test the significance of the inter-band differences detected in poly-rand using the same approach as in Section 3. 4. 1. These are significant in all but a few 20 layers of the models. The bands are also correctly ranked in the other three languages but with smaller inter-band differences than in English, especially in Greek where clear distinctions are only made in a few middle layers. This variation across languages can 19 Average Self Sim values for context2vec in the poly-rand setting: low: 0. 37, mid: 0. 36, high: 0. 36. 20 low→mid in ELMo's third layer, and mid→high in context2vec and in BERT's first layer. be explained to some extent by the quality of the automatic EuroSense annotations, which has a direct impact on the quality of the sentence pools.
Anisotropy Analysis In order to better understand the reasons behind the smaller inter-band differences observed with mBERT, we conduct an additional analysis of the models' anisotropy. We create 2,183 random word pairs from the English mono, low, mid and high bands, and 1,318 pairs in each of the other languages. 22 We calculate the cosine similarity between two random instances of the words in each pair and take the average over all pairs (RandSim). The plots in the left column of Figure  6  show the results. We observe a clear difference in the scores obtained by monolingual models (solid lines) and mBERT (dashed lines). Clearly, mBERT assigns higher similarities to random words, an indication that its semantic space is more anisotropic than the one built by monolingual models. High anisotropy means that representations occupy a narrow cone in the vector space, which results in lower quality similarity estimates and in the model's limited potential to establish clear semantic distinctions.
Dataset Composition We perform this analysis in English using frequency information from Google Ngrams  (Brants and Franz, 2006). For French, Spanish and Greek, we use frequency counts gathered from the OS-CAR corpus  (Suárez et al. , 2019). We split the words into four ranges (F ) corresponding to the quartiles of frequencies in each dataset. Each range f in F contains the same number of words. We provide detailed information about the composition of the English dataset in Figure  7. 23.
Controlling for Frequency and PoS We conduct an additional experiment where we control for the composition of the poly bands Figure  9 : Average Self Sim inside the poly bands balanced for frequency (FREQ-bal) and part of speech (POSbal). Self Sim is calculated using representations generated by monolingual BERT models from sentences in each language-specific pool. We do not balance the Greek dataset for PoS because it only contains nouns. in terms of grammatical category and word frequency. We call these two settings POS-bal and FREQ-bal. We define n pos , the smallest number of words of a specific PoS that can be found in a band. We form the POS-bal bands by subsampling from each band the same number of words (n pos ) of that PoS. For example, all POS-bal bands have n n nouns and n v verbs.
We follow a similar procedure to balance the bands by frequency in the FREQ-bal setting. In this case, n f is the minimum number of words of a specific frequency f that can be found in a band. We form the FREQ-bal dataset by subsampling from each band the same number of words (n f ) of a given range f in F. Table  2  shows the distribution of words per PoS and frequency range in the POS-bal and FREQbal settings for each language. The table reads as follows: the English POS-bal bands contain 198 nouns, 45 verbs, 64 adjectives and 7 adverbs; similarly for the other two languages. Greek is not included in this POS-based analysis because all sense-annotated Greek words in EuroSense are nouns. In FREQ-bal, each English band contains 40 words that occur less than 7. 1M times in Google Ngrams, and so on and so forth.
We train a binary logistic regression classifier for each type of representation and feature. As explained in Section 4, the three poly bands (low, mid and high) and mono contain a different number of words. For classification into polysemy bands, we balance each class by randomly subsampling words from each band. In total, we use 1,168 words for training, 252 for development and 252 for testing (70/15/15%) in English. In the other languages, we use a split of 840/180/180 words. We train multi-class logistic regression classifiers with the two types of features, Self Sim and pairCos. We compare the results of the classifiers to a baseline that predicts always the same class, and to a frequencybased classifier which only uses the words' log frequency in Google Ngrams, or in the OSCAR corpus, as a feature.
Instances of different senses, such as: Put granola bars in a bowl vs. That's not a very high bar, present no overlap in their in-context substitutes: {snack, biscuit, block, slab} vs. {pole, marker, hurdle, barrier, level, ob-struction}. Semantically related instances, on the contrary, share a different number of substitutes depending on their proximity. The need for manual annotations, however, constrains the method's applicability to specific datasets. We propose to extend and scale up the  Mc-Carthy et al. (2016)  clusterability approach using contextualised representations, in order to make it applicable to a larger vocabulary. These experiments are carried out in English due to the lack of evaluation data in other languages.
Data We run our experiments on the usage similarity (Usim) dataset  (Erk et al. , 2013)  for comparison with previous work. Usim contains ten instances for 56 target words of different PoS from the Se-mEval Lexical Substitution dataset  (McCarthy and Navigli, 2007). Word instances are manually annotated with pairwise similarity scores on a scale from 1 (completely different) to 5 (same meaning). We represent target word instances in Usim in two ways: using contextualised represen-tations generated by BERT, context2vec and ELMo (BERT-REP, c2v-REP, ELMo-REP); 27  using substitute-based representations with automatically generated substitutes. The substitutebased approach allows for a direct comparison with the method of  McCarthy et al. (2016). They represent each instance i of a word w in Usim as a vector i, where each substitute s assigned to w over all its instances (i ∈ I) becomes a dimension (d s ).
For a given i, the value for each d s is the number of annotators who proposed substitute s. d s contains a zero entry if s was not proposed for i. We refer to this type of representation as Gold-SUB. We generate our substitutebased representations with BERT using the simple "word similarity" approach in  Zhou et al. (2019). For an instance i of word w in context C, we rank a set of candidate substitutes S = {s 1 , s 2 ,. , s n } based on the cosine similarity of the BERT representations for i and for each substitute s j ∈ S in the same context C. We use representations from the last layer of the model. As candidate substitutes, we use the unigram paraphrases of w in the Paraphrase Database (PPDB) XXL package  (Ganitkevitch et al. , 2013; Pavlick et al. , 2015). 28 or each instance i of w, we obtain a ranking R of all substitutes in S.
Sense Clustering The clusterability metrics that we use are metrics initially proposed for estimating the quality of the optimal clustering that can be obtained from a dataset; the better the quality of this clustering, the higher the clusterability of the dataset it is derived from  (Ackerman and Ben-David, 2009). In order to estimate the clusterability of a word w, we thus need to first cluster its instances in the data. We use the k-means algorithm which requires the number of senses for a lemma. This is, of course, different for every lemma in our dataset. We define the optimal number of clusters k for a lemma in a data-driven manner using the Silhouette coefficient (SIL)  (Rousseeuw, 1987) , without recourse to external resources. 29 For a data point i, SIL compares the intra-cluster distance (i.
Acknowledgements This work has been supported by the  French National Research Agency  under project  ANR-16-CE33-0013. The work is also part of the  FoTran  project, funded by the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme  (grant agreement № 771113). We thank the anonymous reviewers and the TACL Action Editor for their careful reading of our paper, their thorough reviews and their helpful suggestions.
For the information content of microscopy images to be appropriately interpreted, reproduced, and meet FAIR (Findable Accessible Interoperable and Reusable) principles, they should be accompanied by detailed descriptions of microscope hardware, image acquisition settings, image pixel and dimensional structure, and instrument performance. Nonetheless, the thorough documentation of imaging experiments is significantly impaired by the lack of community-sanctioned easy-to-use software tools to facilitate the extraction and collection of relevant microscopy metadata. Here we present Micro-Meta App, an intuitive open-source software designed to tackle these issues that was developed in the context of nascent global bioimaging community organizations, including BioImaging North America (BINA) and QUAlity Assessment and REProducibility in Light Microscopy (QUAREP-LiMi), whose goal is to improve reproducibility, data quality and sharing value for imaging experiments.
The App provides a user-friendly interface for building comprehensive descriptions of the conditions utilized to produce individual microscopy datasets as specified by the recently proposed 4DN-BINA-OME tiered-system of Microscopy Metadata model. To achieve this goal the App provides a visual guide for a microscope-user to: 1) interactively build diagrammatic representations of hardware configurations of given microscopes that can be easily reused and shared with colleagues needing to document similar instruments. 2) Automatically extracts relevant metadata from image files and facilitates the collection of missing image acquisition settings and calibration metrics associated with a given experiment. 3) Output all collected Microscopy Metadata to interoperable files that can be used for documenting imaging experiments and shared with the community.