dataesr/scientific-paragraphs-categorization huggingface dataset:
From a technical point of view, we used the IRaMuTeQ software  (Ratinaud & Déjean, 2009)  to perform dictionary-based processing, without disambiguation, also called endogenous lemmatisation. After lemmatisation, we filtered out only the verbal, nominal and adjectival forms. The CA visualizations were produced using the R FactoMineR package  (Lê et al. , 2008). As our dataset contains 8460 preprint citation contexts, to compare them with nonpreprint citation contexts we need to consider collections of similar sizes. From the entire set of more than 16M citations to peer-reviewed publications in PLOS, we constructed 3 different collections (c1, c2 and c3) of 8460 citations to peer-reviewed publications chosen randomly. The linguistic difference of citing sentences between preprint and peer-reviewed papers may be biased by journals.
Progression of citations over time We have plotted the number of preprint citations received by the different preprint databases by year. Figure  1  shows that the overall number of preprint citations grows rapidly since 2011 and this growth is consistent with the creation and development of new preprint repositories. For example, the bioRxiv database was created in 2013 and its presence in the citations is visible from 2014 and has been growing ever since. In order to compare the progression of preprint citation over time with that of the rest of citations, we have considered three different random collections of citation contexts of peer-reviewed publications (c1, c2 and c3). "Analysis of citation contexts" section provides details on the construction of collections c1, c2 and c3. Figure  2  shows the progression of citations over time in these three collections.
Contribution of the different preprint databases The disciplinary nature of the different preprint databases is one of the factors that result in the different number of preprint citations among the journals. Figure  7  presents the relative parts of the contribution of the different preprint databases to the total preprint citations in the PLOS journals. We can observe that BioRxiv is present in all PLOS journals, and is most frequently cited in PLOS Biology, PLOS Genetics, PLOS Neglected Tropical Diseases and PLOS Pathogens. Thus, the disciplinary speciality in the biomedical field leads to the wide use of this preprint database by researchers. These data corroborate the results obtained by  Li et al. (2015)  who identified the point at which repositories are cited in their own domain.
Conclusion and perspectives In this paper we study preprint citations in several different aspects: the progression of preprint citations over time, their relative frequencies in relation to the IMRaD structure of articles, their distributions over time, per preprint database and per PLOS journal. We have also compared the lexical content of preprint citation contexts with that of citation contexts of peer-reviewed publications. For our experiment, we have processed the PLOS corpus that covers 7 journals and a total of about 240,000 articles up to January 2021. From this corpus we have extracted 8460 preprint citations that were analysed in this paper. We have shown that preprint citations have grown rapidly during the last years and that this growth is consistent with the creation and use of preprint repositories.
As a joint effort from various communities involved in the Worldwide LHC Computing Grid, the Operational Intelligence project aims at increasing the level of automation in computing operations and reducing human interventions. The distributed computing systems currently deployed by the LHC experiments have proven to be mature and capable of meeting the experimental goals, by allowing timely delivery of scientific results. However, a substantial number of interventions from software developers, shifters, and operational teams is needed to efficiently manage such heterogenous infrastructures. Under the scope of the Operational Intelligence project, experts from several areas have gathered to propose and work on "smart" solutions. Machine learning, data mining, log analysis, and anomaly detection are only some of the tools we have evaluated for our use cases.
OPINT AREAS OF INTEREST One of the most important aspects of the OpInt project is enabling the collaboration between the different experiments and teams that take part in it. High-energy physics computing is a complex and diverse world, but many components used in software development can be shared across projects. As we are a diverse community, some of the activities aim toward modest evolution and harmonization, rather than revolution, with a highly visible impact on users. Nevertheless, one of the challenges we are facing as we approach the HL-LHC era is the nurturing of cross-team, cross-experiment, and cross-domain collaborations, to share ideas and tools, working toward a common goal, together, making our community stronger and sustainable on the long term. In the following paragraphs, we summarize the areas of the challenges that we are addressing.
Monitoring the Computing Infrastructures-Tools and Their Unification The OpInt activity heavily relies on the monitoring infrastructure that consists of experiment-agnostic and specific components described later in this section. One of the challenges we address is finding the correct balance of generalization versus customization in the infrastructure configuration, in the tools, the data formats, interfaces, access protocols, and visualization. We believe it is important to offer a shared development platform for all the OpInt-related projects in order to foster code re-usability and enable a quick prototyping of new ideas by using pre-created components. Further description of the monitoring infrastructure is given in Section 2. 1.
CMS Intelligent Alert System The intelligent alert system for the CMS experiment, further described in Section 2. 5, aggregates alerts from different experiment-specific monitoring systems and augments them with additional information from the multi-experiment monitoring tool Site Status Board (SSB)  (Andreeva et al. , 2012a)  and Global Grid User Support (GGUS)  (Antoni et al. , 2008). The GGUS is a ticketing system used in the HEP-distributed computing domain (and others too) that facilitates a standardized issuereporting and communication interface among the experiment teams (e. g. , operators, experts, developers, and users) and the sites. In the CMS intelligent alert system, focusing mainly on network and database interventions and outages, annotations are created to assist the experiment's operators.
FTS Logs, Errors, and Failures Analysis The distribution of the large volumes of data collected by the LHC is mainly managed by the file transfer service (FTS)  (Karavakis et al. , 2020). Each data transfer operation among grid sites is tracked by FTS, and the related log files contain large volumes of information related to performance and errors. Analysis of the FTS logs represents a typical OpInt workflow, processing logs of the FTS service with a ML model and visualizing the clusters and their updates in a monitoring dashboard. It is written in an experiment-agnostic manner (developed in the CMS experiment), and one of the future challenges will be to integrate it with the infrastructure of another experiment (notably the ATLAS experiment). Analysis of FTS errors and failures represents a challenge to distributed data management (DDM) infrastructure operations.
CERN MONIT Infrastructure The CERN IT department offers a stack of monitoring services, referred to as MONIT  (Aimar et al. , 2017) , based on different open-source technologies, supporting a variety of data sources. Clients can either actively push data to MONIT or pull it. All the collected data are processed by Apache Kafka at the MONIT core, with Apache Spark used to enrich and aggregate the data (Apache Software Foundation). Data are written into three different data stores: InfluxDB 1 for time-series data, ElasticSearch 2 for more detailed data with a limited retention, and Apache Hadoop (Apache Software Foundation) for long-term archival. Data visualization is offered via Grafana 3 dashboards.
ATLAS Monitoring Infrastructure The ATLAS monitoring infrastructure  (Beermann et al. , 2020)  is based on MONIT. The main components are the DDM transfer monitoring and accounting, the jobs monitoring and accounting, and the SSB. The DDM dashboard is based on the Rucio events and traces, which are collected from Apache ActiveMQ (Apache Software Foundation). They are enriched with topology information from the Computing Resource Information Catalogue (CRIC)  (Anisenkov et al. , 2020) , aggregated and written both to InfluxDB and ElasticSearch. The DDMaccounting dashboards are based on preprocessed accounting reports from Rucio, enriched with topology information and written to ElasticSearch. The jobs monitoring and accounting dashboards data are collected directly from the PanDA  (Maeno et al.
OpInt Framework In order to share expertise and re-use code and approaches across the board, we started developing the OpInt framework 4  and have already created the following components: • Abstract data fetchers to enable getting data from various sources, and a demo implementation of a fetcher that gets data from an Apache Hadoop endpoint. • A module to schedule and execute data-fetching tasks either periodically or on demand. • A module to construct REST APIs to share processed data with other services. • database integration to store fetched and aggregated data. • An abstract template for NLP-processing pipelines. Two of the pipelines that are used in our projects have already been adapted to follow the abstract implementation. • A sample application that can be used to kick-start new projects.
Predictive Site Maintenance Log data are often unstructured, in which the format and semantics may vary significantly from system to system, making an approach toward a general-purpose log-based anomaly detection system extremely challenging. Supervised approaches provide more accurate results at the cost of requiring a large dataset of labeled entries, which is a hard restrictive requirement in the majority of real-world scenarios. Generally, labeled anomalous logs are not available and because of this supervised approaches are seldom used without expensive data pretreatment. In addition, to generate ad hoc solutions for each existing system is counterproductive since a single data center can incorporate hundreds of computing sub-systems. For this reason, general-purpose solutions are preferable even if, initially, they are expected to perform worse than dedicated solutions.
Automated log parsing offers promising results through standard off-the-shelf NLP algorithms  (Bertero et al. , 2017). Further improvement is possible  (Wang et al. , 2020 ) by applying word2vec  (Mikolov et al. , 2013)  algorithms to log events. Whilst log parsing is often used to improve the effectiveness of anomaly detection tools, NLP is a very expensive technique since it was originally envisioned to approach a much more complex class of problems, such as handling natural language. The OpInt effort is focused on NLP solutions.
Jobs Buster Jobs Buster is an application built within the OpInt framework to detect large groups of job failures and identify the most probable cause. We used a CatBoost library  (Prokhorenkova et al. , 2019)  and the CatBoostClassifier to train a gradient-boosted tree, which were able to predict the outcome status of a job by its features. The CatBoost library provides embedded support of categorical variables and efficient learning. Using a model previously trained on a particular data slice, Jobs Buster extracts the list of features with importance above a threshold of 0. 5 and performs a ranking of its values. Groups of feature-value pair sets for failing jobs define clusters that might be identified with a specific source of failure (a particular task fails on a selected grid site due to misconfiguration or some rack in a data center is out of order).
This is to transform the textual information into a convenient numeric representation and serves as preprocessing for the next steps. A clustering algorithm is the applied to get groups of related errors: we pre-train a word2vec model on a big dataset and run a k-means++ algorithm  (Arthur and Vassilvitskii, 2007)  online during the monitoring shifts. In order to demonstrate the approach, we report an analysis of FTS data from one full day of operation. Figure  2  shows an example of a summary table for the biggest cluster found by the model. The first three columns provide numeric summaries: 1) the cluster size, 2) the number of unique strings within the cluster, and 3) the number of unique patterns: unique strings after the removal of parametric parts such as paths, IP addresses, and URLs.
NLP Applications in Rucio Data in Rucio is organized using data identifiers (DIDs), which have three levels of granularity: files, datasets, and containers. Datasets and containers are used to organize sets of files in various potentially overlapping groups and to facilitate bulk operations such as transfers or deletions. Users are permitted to perform certain actions on the DIDs: downloads, uploads, or transfers. Due to the large sizes of the collaborations, the experts support a noteworthy amount of requests daily. To reduce the required efforts, we are looking into methods to assist the support team in answering these requests. Ideally, the support would be provided  by an intelligent bot able to process and understand the user's requests and, if possible, trigger appropriate automated action.
The NLP assists in processing questions from users, where the objective is to provide satisfactory answers. A prototype bot capable of handling user's requests up to a certain level of complexity has been developed, acting with emails as inputs. When a new email is received, it goes through the QuestionDetector, and a question object is created. Originally, the support bot was meant to answer incoming support emails. After an initial data analysis, we expanded that idea and introduced new data sources and the option to query the bot directly. A question from a user is treated as a query by the search engine in the next step. Once a question has been detected from a new email or given by a user query, the search engine looks into the support bot data storage and retrieves the most similar documents.
Documents are archived questions from emails or GitHub issues and Rucio documentation pages from GitHub as well as FAQ question-answer pairs created by Rucio experts. The search engine subclasses exist for each data source. Once the most relevant documents have been retrieved, we use them as context for the new question in a SQuAD-like  (Rajpurkar, 2021)  manner to generate answers with a given confidence. The current implementation caches and uses DistilBERT  (Sanh et al. , 2020)  and BERT large-cased/uncased models, fine-tuned on the SQuAD dataset. The NLP applications described in this section are in different maturity states, from prototypes to production deployment. They aim to reduce human person-power cost in DDM operations, targeting partly DDM operation experts, partly suggestion, and chat systems for the user community.
FUTURE DEVELOPMENTS We foresee activities developing in a variety of areas: • Predictive Site Maintenance: We foresee tuning of anomaly detection approaches applied to data centers' metrics as well as further exploration and exploitation of granular computing classifiers in predictive site maintenance. We intend to introduce specialized debugging tests to probe further the root cause of the failure with HammerCloud dynamic job shaping, and provide suggestions to the operators, or even assist in the automation of the issue reporting and mitigation. • Jobs Buster: We plan to add long-term models into Jobs Buster, enriching jobs features with information extracted from failures that occurred in the past. • Error Messages Clustering: Further framework developments involve code parallelization of the timeexpensive stages.
• FTS Logs, Errors and Failures Analysis: We wish to build a reference dataset to store labels for error categories, root causes, priority, and solving actions, assisting in performance optimization. Additionally, further NLP tools such as Question Answering or Named Entity Recognition (NER) look promising to support our target of understanding the root causes and suggesting solving actions for the issues. • NLP Applications in Rucio: Creating a user interface will allow us to deploy the support bot on a test server to a wider set of users, expanding the reach from the beta-testers and from the developers and experts community to the user communities. Their questions and the answers given can then be supervised and examined to create a dataset for the fine-tuning of the models. Additionally, we wish to create a NER tagger to detect Rucio-specific language entities, exploring potential to boost the tool performance significantly, and provide a way to create dynamic answers.
CONCLUSION The operational intelligence initiative brings together experts from various WLCG communities and is aiming to reduce the person-power cost of operating distributed computing systems. We presented an overview of activities in varying stages of completeness in the areas of computing centers operation and the workflow and data management, which represent only three of the areas, where innovative approaches can bring substantial improvement while we benefit from the state-of-the-art technologies. The projects described in Section 2 and Section 3 are summarized in Table  1. In the collaborative spirit, in compliance with a variety of computer security and data privacy guidelines and policies applicable to our environment, we have been sharing the code developed in the scope of the various operational intelligence initiative projects in a GitHub repository 5.
I study its impact on the risk premium in particular. My article therefore contributes to the integration of deeper psychological traits (not just psychological biases) into existing asset pricing theories in this burgeoning literature at the intersection of psychology and behavioral finance  (Akerlof & Shiller, 2009; Kahneman, 2003). I use data from 1980Q1 to 2021Q4 including the COVID period with low interest rates, increased wealth, and consumption volatility. Both the CC model and the sentiment model remain robust and are able to model both the first and second moments of the risk premium, risk-free rate, and excess returns with a comparable 12% absolute error. 2 The results show that the sentiment model subsumes a lower risk aversion coefficient (1 to 1. 5) than the CC model. This is consistent with research showing that sentiment states affect risk aversion  (Aren & Koten, 2019; Conte et al.
ACKNOWLEDGMENTS I thank  Laurent Calvet  and  Abraham Loiui  for their guidance,  Harrison Hong  for the independent examiner remarks, and seminar participants in the EDHEC Phd seminar, in particular,  Ichiro Tange. I also thank the anonymous referee for helpful comments. All errors are solely mine. Open access publishing facilitated by  James Cook University , as part of the  Wiley -James Cook University agreement via the Council of Australian University Librarians.
, 2006; Srinivas et al. , 2010. ). This context has fostered the development of a fairly large number of open-source packages to facilitate the use of GPs. Some of the popular choices are the Python modules scikit-learn  (Pedregosa et al. , 2011) , GPy (Sheffield machine learning group, 2020), GPflow  (Matthews et al. , 2017) , GPyTorch  (Gardner et al. , 2018) , OpenTURNS  (Baudin et al. , 2017) ; the R package DiceKriging  (Roustant et al. , 2012) ; and the Matlab/GNU Octave toolboxes GPML  (Rasmussen and Nickisch, 2010) , STK  (Bect et al. , 2021)  and GPstuff  (Vanhatalo et al. , 2012). In practice, all implementations require the user to specify the mean and covariance functions of a Gaussian process prior under a parameterized form. Out of the various methods available to estimate the model parameters, we can safely say that the most popular approach is the maximum likelihood estimation (MLE) method.
However, a simple numerical experiment consisting in interpolating a function (see Table  1 ), as Table  1 : Inconsistencies in the results across different Python packages. The results were obtained by fitting a GP model, with constant mean and a Matérn kernel (ν = 5/2), to the Branin function, using the default settings for each package. We used 50 training points and 500 test points sampled from a uniform distribution on  [-5, 10 ] × [0, 15]. The table reports the estimated values for the variance and length scale parameters of the kernel, the empirical root mean squared prediction error (ERMSPE) and the minimized negative log likelihood (NLL). The last row shows the improvement using the recommendations in this study. is usually done in Bayesian optimization, shows that different MLE implementations from different Python packages produce very dispersed numerical results when the default settings of each implementation are used.
These significant differences were also noticed by  Erickson et al. (2018)  but the causes and possible mitigation were not investigated. Note that each package uses its own default algorithm for the optimization of the likelihood: GPyTorch uses ADAM  (Kingma and Ba, 2015) , OpenTURNS uses a truncated Newton method  (Nash, 1984)  and the others generally use L-BFGS-B  (Byrd et al. , 1995). It turns out that none of the default results in Table  1  are really satisfactory compared to the result obtained using the recommendations in this study 3. Focusing on the case of GP interpolation (with Bayesian optimization as the main motivation), the first contribution of this article is to understand the origin of the inconsistencies across available implementations. The second contribution is to investigate simple but effective strategies for improving these implementations, using the wellestablished GPy package as a case study.
Stopping condition Most GP implementations rely on well-tested gradient-based optimization algorithms. For instance, a popular choice in Python implementations is to use the limited-memory BFGS algorithm with box constraints (L-BFGS-B; see  Byrd et al. , 1995)  of the SciPy ecosystem. (Other popular optimization algorithms include the ordinary BFGS, truncated Newton constrained, SQP, etc. ; see, e. g. ,  Nocedal and Wright (2006). ) The L-BFGS-B algorithm, which belongs to the class of quasi-Newton algorithms, uses limitedmemory Hessian approximations and shows good performance on non-smooth functions  (Curtis and Que, 2015). Regardless of which optimization algorithm is chosen, the user usually has the possibility to tune the behavior of the optimizer, and in particular to set the stopping condition.
Conclusions and recommendations Our numerical study has shown that the parameterization of the covariance function has the most significant impact on the accuracy of MLE in GPy. Using restart / multi-start strategies is also very beneficial to mitigate the effect of the numerical noise on the likelihood. The two other levers have second-order but nonetheless measurable influence. These observations make it possible to devise a recommended combination of improvement levers-for GPy at least, but hopefully transferable to other software packages as well. When computation time matters, an improved optimization procedure for MLE consists in choosing the combination of a log reparameterization, with a gridsearch initialization, the soft (GPy's default) stopping condition, and a small number, say N opt = 5, of restarts.