dataesr/scientific-paragraphs-categorization huggingface dataset:
A dialog management system (e. g. a decision tree) then exploits the intents and named entities extracted from user requests to define the behaviour of the chatbot. Thus, at every step during design, a dataset labelled with intents and named entities is required to generate and maintain the classification model forming the core of the chatbot. Updating and annotating a dataset is usually considered an iterative process involving steps described by the acronym MATTER (Model, Annotate, Train, Test, Evaluate, and Revise) (see  Stubbs (2013) ). Following this approach, phases of dataset design and annotation are most often the result of manual work, and this methodology has several limitations: • Prior to starting annotation, a model grouping data in relevant intents has to be defined: when performed manually, this task relies exclusively on the knowledge of the types of requests detained by one or several business experts; • Once the modelling of intents is achieved, the task of labelling data requires thorough understanding of this model to avoid errors: when the knowledge of the intents model is not well integrated by one of the annotators, or uniformly shared between them, there is a risk of introducing intra-individual or inter-individual inconsistencies in the dataset; • During initialization or maintenance of the model, the scope of the chatbot may change: in this case, any change to the intents model implies an additional cost of relabelling the dataset accordingly.
As an example,  (Lampert et al. , 2019)  propose a collaborative clustering approach integrating user constraints to improve performance of a topographical clustering task from satellite images. However, such constrained annotation methods are harder to apply to natural language: data cannot be observed all at once by the human when defining constraints, as can be done with an image. In  Schild et al. (2021a Schild et al. ( , 2021b)) , the authors proposed an active learning method adapted from images constrained clustering to assist questions annotation task. This approach, coined Interactive Clustering, is composed of several iterations involving data sampling heuristics to guide expert annotations and a constrained clustering algorithm to partition data with expert corrections. The study showed that adequate selection of implementation parameters could lead to a relevant iterative definition of intents on a small dataset.
While the article defines a methodology that could be suited to define intents set iteratively, this conclusion has to be confirmed on a larger dataset containing more 3 data and classes. The usability of this method in a realistic setting was also not explored in this study, for instance in terms of time consumption or classification performance of the resulting intents model. In this article, the authors propose an extension of the previous study to investigate the suitability of Interactive Clustering for the iterative design of conversational agents. Stemming from practical organization constraints encountered when developing chatbots in a business context (see  Finlayson & Erjavec (2016) ), the methodology is applied on a realistic French dataset representative of requests in the banking domain (including ground truth labels proposed by domain experts).
This step involves business experts who can validate the relevance of information considered for the chatbot. Step 2: Collecting a dataset of questions related to this area. Questions can come from different sources: user surveys, experts' insights, database extractions, or web scrapping. Data collection should include a filtering step to exclude out of scope data (e. g. questions not related to the specified area). Step 3: Modelling the set of intents from the dataset. Intents can be defined as the action or information expected by the user when formulating a request. Business experts define the set of intents according to the data collected and their domain knowledge. A chatbot dedicated to requests related to mortgage loans may for example include intents like "loan subscription", "contract negotiation", or "customer state consultation".
Modelling intents can be complex, because of the elasticity of business knowledge: in most cases, experts will become aware of certain requests and incorporate them in the model only when presented with actual questions. The types and number of requests that need to be integrated therefore varies greatly as experts explore the dataset and interact with other experts. From the authors' experience, depending on the level of detail desired during modelling, experts can define up to 100 intents from a 10. 000 questions dataset.
4 Step 4: Labelling the dataset. Labelling the dataset is a necessary step towards training the intents model. During annotation, business experts must maintain in memory a representation of the entire set of intents. The complexity of the model, in addition to the task being time consuming and repetitive, makes annotation an errorprone task, that can be subjective and introduce inconsistencies. In practice, the annotation step questions the modelling of intents, and experts may decide to revise the set and the definition of intents (thus rolling back to step 3). Step 5: Training and implementing the chatbot. Once labelling is complete, a supervised classification model is trained and the dialog behaviour is defined from the intents (e. g. using a dialog decision tree). At this stage, training the model often yields poor performance due to the complexity of the intents model or inconsistencies in annotation.
Constraints Sampling: A heuristic suggests pairs of questions that the expert has to discriminate. Heuristics The process is initialized with an unconstrained clustering to provide a first (and likely little relevant) data partitioning. During each iteration of Interactive Clustering, the computer selects a pair of questions using a predetermined heuristic, the expert discriminates them by annotating binary constraints, and the clustering uses the constraints to perform a corrected data partitioning. Thus, the relevance of clustering results should increase over iterations. Using this methodology, the annotator does not need to manage a dataset and requires little knowledge in data science: they simply have to express their knowledge of the relevant business domain. Furthermore, a preliminary definition of possible intents is no longer necessary to start annotating: the intent structure will be determined over iterations through clustering.
7 Step 2: Collect a dataset of questions related to this area. See above, no changes. Step 3: Perform iterative and interactive annotation with Interactive Clustering. The annotator runs several iterations of constraints sampling (1), constraints annotation (2) and constrained clustering (3). Over iterations, the business relevance of the clustering should improve by incorporating more constraints from the expert. This task does not use any predefined intent modelling and uses binary constraints based on response similarity. Step 4: Perform statistical and semantic validation. This step focuses on validating the relevance of results obtained after a few iterations of Interactive Clustering. The simplest check is a statistical validation of the data partitioning a k-fold cross validation checks the statistical consistency.
Poor performance can be a sign that too many or inconsistent constraints have been placed on the data. Then, another check is necessary to confirm the clusters business relevance: This analysis cannot be automated because it requires clusters inspection (lexical fields' analysis, relevant patterns detection, semantic consistency analysis, etc. ). If the analysis is not satisfying, there are probably not enough annotated constraints for the Interactive Clustering to converge. Step 5: Train and implement the chatbot with the labelled dataset. After annotating with Interactive Clustering, clusters are named according to the discovered intents, and a supervised classification model is trained with these intents. Using this model, the dialog behaviour of the chatbot can be defined. This task is made easier since this method tends to group together questions that yield similar responses.
Methods To test the article hypotheses, the authors propose an experiment of chatbot dataset annotation. This experiment consists in performing Interactive Clustering iterations in order to annotate an unlabelled dataset, starting from no known constraints and ending when all the possible constraints between questions are defined. The human annotator is simulated by the algorithm, and annotations are made by comparing with ground truth labels: two questions are annotated with a "MUST-LINK" if they come from the same intent, and with a "CANNOT-LINK" constraint otherwise. With this automatic annotation, no conflict can occur. In this article, the influence of the following parameters is studied: i. Four Levels of Data Preprocessing: (a) no preprocessing; (b) simple preprocessing (lowercasing, accent deletion, punctuation deletion, whitespace deletion); (c) lemmatized preprocessing (simple preprocessing and token lemmatization); and (d) filtered preprocessing (simple preprocessing and restriction to first-order token in the syntactic dependency tree).
• Implementation: use of spacy lemmatizer and dependency parser  (Honnibal & Montani, 2017). ii. Two Levels of Data Vectorization: (a) tfidf (vectors based on terms frequency) and (b) fr-corenews-md model (pre-trained spacy language model). • NB: A model of pre-trained vectors on a banking corpus can be used, but no French model is available for this experiment. Such a model could be considered in a later study. • Implementation: use of Scikit-Learn TFIDF vectoriser  (Pedregosa et al. , 2011)  and spaCy French language model  (Honnibal & Montani, 2017). • NB: For this study, it is assumed the number of clusters is known and set the ground truth intents number. Altogether, there were 192 possible combinations of parameters. Each configuration was repeated 5 times. The relevance of data segmentation is measured using homogeneity, completeness, and v-measure, computed on the clustering results of each iteration.
Measures are obtained through comparison with a ground truth, corresponding to annotations by business experts prior to the experiment (with no computer guidance). In this study, the following performance thresholds are considered: • Complete Annotation: The number of iterations required to annotate all constraints. In this case, the annotator defines the link between every pair of questions in the dataset. The clustering therefore becomes a deterministic graph traversal problem. • Sufficient Annotation: The number of iterations required to reach a v-measure of 100%, corresponding to complete agreement with the ground truth. • Partial Annotation: The number of iterations required to reach a v-measure of 80%. To analyse the convergence speed and the effect size of the implementation parameters on the number of annotations required, the authors perform repeated measures ANOVA in R (R Core Team, 2017).
Dataset Description The ground truth used for this experiment is available in  Schild (2021c). It relies on a French dataset of 500 questions dealing with typical bankcard management requests. Prior to the experiment, the dataset was analysed and annotated manually by business experts. It was divided into 10 intents of 50 questions each. Sample questions from each intent are shown in Table  1. To comply with the previously defined instructions on constraints annotation, intents were created by grouping questions requiring similar responses. Thus, a pair of questions coming from the same intent can be annotated by a "MUST-LINK" constraint (similar responses according to an expert), or a "CANNOT-LINK" constraint otherwise. ). Posthoc analysis of these effects shows that the best average setting is made of simple preprocessing, TFIDF vectorization, sampling of closest pairs from different clusters, and average-link or single-link hierarchical constrained clustering.
Discussion The aim of this article was to propose and experiment on a realistic dataset and methodology for the development of conversational agents in a business context. The method, adapted from  Lampert et al. (2019)  and coined Interactive Clustering, cycles through iterations composed of a constrained clustering phase (proposing a segmentation of unannotated requests samples), and a binary constraint annotation phase during which a business expert discriminates pairs of requests based on the expected response. Constraints take the form of "MUST-LINK" labels when the expected response is the same, and "CANNOT-LINK" otherwise. Based on a realistic dataset of credit card-related questions, this article proposed to validate the capacity of the method to converge to a business-relevant segmentation of requests, and to test the influence of implementation parameters (preprocessing, vectorization, sampling heuristic, and clustering algorithm) on the convergence.
The results of the experiment confirm that regardless of the choice of implementation parameters, cycling through iterations of Interactive Clustering causes the data segmentation to converge to the human-made business-relevant segmentation used as a Ground Truth (see Figure  2 ). However, the shape and speed of convergence seems to depend on implementation parameters. A full dataset annotation is not possible because it requires too many constraints (4431 on average, 1630 at the optimum, see Table  4  for optimal settings). The authors therefore seek to perform a sufficient annotation (full subjective clustering correction) or a partial annotation (agreement between human annotation and clustering exploration). As shown by the analysis of parameters proposed in Table  2  and Table 3 , faster convergence can be obtained by using a simple (only character normalization) or lemmatized preprocessing, TFIDF vectors, and by annotating in priority pairs of data points selected at the border between clusters.
These results are in agreement with optimal implementation parameters obtained on a smaller dataset in  Schild (2021a Schild ( , 2021b)). COP K-Means and hierarchical clustering (single or average links) are the most adapted algorithms for faster convergence. However, even if COP K-Means faces some unstable iterations due to cluster assignation conflicts (cf. computation time variations), it is faster at first iterations, and is therefore more suited for quick integration of new constraints. On the other hand, hierarchical clustering algorithms are slower at first iterations, but have a decreasing computation time over iterations (see Figure  2 ). As the methodology is based on mixed initiative from the human (to impose constraints) and the computer (to propose data segmentation), Interactive Clustering proposes an optimal compromise between business relevance of the segmentation and performance obtained during intent classification.
On this experiment on a realistic dataset of 500 questions, optimal segmentation with 80% of v-measure was reached after about 650 binary annotations, which is greater than the size of the dataset. However, the user experience associated with these annotations differs from direct annotation: while direct annotations require to place an abstract label chosen among a large set, Interactive Clustering constraints require simple discrimination between two sentences. By replacing complex categorization between multiple categories with discrimination between two cases, this approach therefore reduces the burden placed on the annotator's working memory. Consequently, the task requires less mental resources  (Norman, 2013)  and would facilitate the annotation process, in particular when the dataset and intent set are complex.
Perspectives While the Interactive Clustering method seems successful at providing a fluid user experience during initial definition of a chatbot on this use case, the authors have left for future investigation the study of the influence of potential annotation conflicts on convergence (for instance if conflicting constraints are defined). The provision of a cluster analysis tool could also be considered to help the task of labelling clusters. To go further, although the authors recommend the use of optimal parameters described in Table  2  and Table 3 , the exploration of other implementation parameters could potentially improve the convergence speed and shape. Their influence could also differ when applied to languages other than French. Finally, several hypotheses concerning the annotator experience while performing interactive clustering still need to be verified in order to confirm the benefits of the proposed method.
CoNCLUSIoN This study was conducted in a context of growing automation in the treatment of customer requests, and the increasing efforts to develop conversational agents in a business operations context. It proposes a novel methodology of Interactive Clustering to support the efficient development of such conversational agents: • It exploits computer suggestions from a constrained clustering algorithm to make simpler user experience during annotation by focusing on placing constraints the dataset rather than modelling the intents set upfront; • It avoids error-prone situations occurring when the intents model is complex, by providing an evolving representation of intents: at each step of the process, a segmentation of the dataset is automatically kept up to date according to the constraints imposed by the expert, without requiring re-annotating the dataset; • The experiment described in this article shows that the methodology can be tuned by choosing the optimal algorithms for text preprocessing, vectorization, clustering, and selection of constraints.
The polarization calibration was performed by observing of J1058+0133. The raw data were reduced at the East Asian ALMA Regional Center (EA-ARC) using CASA (version 5. 6. 1)  52. We further performed interactive CLEAN deconvolution imaging with self-calibration. The Stokes I, Q, and U maps were CLEANed with an appropriate number of CLEAN iterations after the final round of selfcalibration. The off-source rms levels in I, Q, and U are consistent with the expectations for thermal noise alone. The quantities that can be derived from the polarization maps are the polarized intensity ( Q 2 + U 2 ), polarization degree (100 Q 2 + U 2 /I %, PD), and polarization position angle (1/2 arctan(U/Q), PA). By applying the polarization calibration to the phase calibrator J1036-3744 and creating Stokes maps for 6, 9, and 18 epochs during the 3-hr observing period, we confirm that the stability of linear polarization degree is <0.
For GRB 190114C, Jordana-Mitjans et al. (2020)  23  reported the detection of polarization induced by the dust in the host galaxy. The X-ray data obtained by the Swfit/XRT were collected from the UK Swift Science Data Centre 58, 59. We rebinned the spectra so that each spectral bin contains more than five counts. Using the software XSPEC 12, we perform spectral fitting with a single power law modified with intrinsic and Galactic absorptions, the latter of which are fixed at values calculated from Willingale et al. (2013)  60. TBabs and zTBabs that incorporate three absorption elements (i. e. , gas, molecules, and grains)  61  were used to describe the spectral absorptions. The derived best-fitting values are summarized in Extended Data Table  2. Using Schady et al. (2010)  62  , the measured N H are converted to A V.
The number of hidden layers and neurons that make up each layer of the neural network has an influence on the performance of system predictions. There are different methods for finding the optimal architecture like grid search or based on empirical equations. All these techniques can be very time-consuming. In this paper, we will present the two packages that we have developed, the genetic algorithm (GA) and the particle swarm optimization (PSO) to optimize the parameters of the neural network for the prediction of the pathogenicity of the BRCA1 and BRCA2 genes. We will compare the results obtained by the two algorithms. We used datasets collected from our NGS analysis of BRCA1 and BRCA2 genes to train deep learning models. This represents a data collection of 11,875 BRCA1 and BRCA2 variants (BRCA1 benign 2,632, BRCA1 pathogenic 2,660, BRCA2 benign 3,446, BRCA2 pathogenic 3,137).
From them, the network can learn automatically by adapting and correcting itself to fit observed patterns in the data. The ability to automatically construct data representations is a key advantage of deep neural nets over conventional machine learning  [1]. There are many open-source deep learning projects, such as the H2O platform, Theano, Torch, Tensorflow, Caffe, and Keras. For the needs of the project, we used Keras Tensorflow version 2. 2 with the R language version 3. 6. The network architecture is made by the choice of how many hidden layers L = {L 1 , L 2 ,. , L n } the network will have and how many hidden units N = {N 1 , N 2 ,. , N n } each hidden layer will have  [2]. Parameters L and N are very important and have a major influence on the performance of deep machine learning  [3].
Tuning these parameters can be done by hand with the tune grid search method or by choosing an empirical architecture like, for example, minimization of quadratic search  [4]. Grid search can save time in setting L and N. With this method, the structure could be tuned until a suitable number of nodes and/or layers is found to reduce or remove overfitting for the problem. Alternately, the model could be overfitting and pruned by removing nodes until it achieves suitable performance on a validation dataset  [5]. However, the grid search method is still time-consuming as the number of combinations is exponential. Moreover, if the list of the tuning parameters is chosen badly or poorly, the network may not be efficient in prediction or learn slowly and perhaps not at all  [3]. This paper proposes two parameters selection methods for deep learning models: genetic algorithm (GA) and particle swarm optimization (PSO) to tune the architecture of deep neural networks.
We will tune, the number of hidden layers, the number of hidden neurons, dropouts, and activation functions that compose each hidden layer. GA and PSO were written in R language and re-adapted to work with Keras Tensorflow API. To perform our tuning test, we worked on our data collection of BRCA1 / BRCA2 genes (BRCA1 benign 2,632, BRCA1 pathogenic 2,660, BRCA2 benign 3446, BRCA2 pathogenic 3137), which represents from January 2018 until April 2021 a total of 11,875 genes of BRCA1 and BRCA2. These genes are involved in many forms of cancer, mainly breast cancer, where their names come from (BReast CAncer) but also breast cancer ovaries or prostate. However, cancers caused by a BRCA mutation have the advantage of responding well to treatment-inhibiting PARP, which makes it possible to prevent heavier treatments like chemotherapy.
Data selection BRCA1/BRCA2 dataset contains clinical information of 5,292 mutation variants of BRCA1 and 6,538 mutation variants of BRCA2. This represents a collection of 11,875 variants composed of Single Nucleoid Variant (SNV), MNV, and INDEL (Figure  2 ). All variants were labeled by the biologist as benign or pathogenic in the function of the pathology of the patient.
We divided the main dataset of 11,875 variant calls into two subsets. The deep neural network was trained on 70% of the dataset and 30% remaining for the validation (unseen data). We used the "validation split" function from the Keras library to divide the dataset. This function was designed to ensure that the data are separated in such a way that it always trains on the same portion of the data for each epoch. BRCA1/BRCA2 mutations from NGS data. We used the categorical cross-entropy loss function as it fits for multi-class classification tasks. These are tasks where the model must decide which one belongs to the possible categories. It quantifies the difference between two probability distributions (Figure  5 ).