dataesr/scientific-paragraphs-categorization huggingface dataset:
76±0. 23 mag, consistent with the result using the LDT reference. We extracted sources using Source Extractor and determined a photometric zero point using 23 PS1 stars in the field. Using the ZOGY  142  algorithm-based python pipeline, we performed image differencing on Gemini data. The pipeline makes use of Source Extractor, PSFEx 143 , SWarp, SCAMP, and PyZOGY 144 to perform image subtraction. We took a 1300 pix × 1300 pix sized cutout for both the science and reference images, centered at the ZTF20abzwysqy position in the image in order to achieve good subtraction quality, as the images were affected by background variation. Sources were extracted using Source Extractor from both science and reference images with a 5-σ detection threshold. The resulting catalogues were fed to SCAMP, which calculates and corrects for the astrometric errors in both science and reference catalogues with the help of a Gaia data release 2 catalogue  145  for the same field.
5 +4. 8 -2. 3 × 10 -2. The posterior probability distributions are shown in Extended Data Figure  5  and the parameter estimates are listed in Extended Data Table  7. Bayesian model selection. In addition to an afterglow-only model, we consider an afterglow plus a KN and an afterglow plus a SN. Kilonova. We use SEDs simulated by the multi-dimensional Monte Carlo radiative transfer code POSSIS  185. The simulations are performed over a grid of KN parameters: dynamical ejecta M dyn ej , disk wind ejecta M wind ej , opening angle Φ, and the observation angle Θ obs (see Ref. 20  for details). We use Gaussian process regression  186, 187  to interpolate the model, enabling rapid parameter inference. Supernova. The SN model starts with a bicubic spline in time and frequency, L SN ν (t, ν), that interpolates a K-corrected SN1998bw template  22.
5 , strongly favoring the presence of a SN. To better understand the source of discriminating power between models, we carry out a posterior predictive check. Here, we sample the posterior while excluding the GMOS i-band data point and then predict its value using the rest of the data. The posterior predictive distribution of the AB magnitude for the GMOS i-band detection both with and without the inclusion of a SN contribution is shown in Extended Data Figure  6. The fit with the SN is consistent with the observation, while the fit without the SN is inconsistent at the ∼ 5σ level. Therefore, inclusion of the GMOS i-band data point requires a SN component, confirming a collapsar origin. We compared our GMOS-N detection against extinction-corrected i-band fluxes of three GRB-KN candidates found in the literature  191  ; using photometry from GRB 130603  192  and GRB 160821  193, 194  , and the compiled light curve of AT2017gfo  5, 196  , we correct to the redshift of GRB 200826A.
Acknowledgements This work was supported by the  GROWTH (Global Relay of Observatories Watching Transients Happen)  project funded by the  National Science Foundation  under  PIRE  Grant No  1545949. GROWTH is a collaborative project among  California Institute of Technology (USA) ,  University of Maryland College Park (USA) ,  University of Wisconsin Milwaukee (USA) ,  Texas Tech University (USA) ,  San Diego State University (USA) ,  University of Washington (USA) ,  Los Alamos National Laboratory (USA) ,  Tokyo Institute of Technology (Japan) ,  National Central University (Taiwan) ,  Indian Institute of Astrophysics (India) ,  Indian Institute of Technology Bombay (India) ,  Weizmann Institute of Science (Israel) ,  The Oskar Klein Centre at Stockholm University (Sweden), Humboldt University (Germany) ,  Liverpool John Moores University (UK)  and  University of Sydney (Australia).
Based on observations obtained with the  Samuel Oschin Telescope 48-inch  and the 60-inch Telescope at the  Palomar Observatory  as part of the  Zwicky Transient Facility  project. ZTF is supported by the  National Science Foundation  under Grant No. AST-1440341  and a collaboration including  Caltech , IPAC, the  Weizmann Institute for Science , the  Oskar Klein Center at Stockholm University , the  University of Maryland , the  University of Washington (UW) ,  Deutsches Elektronen-Synchrotron and Humboldt University , We thank the staff of the GMRT that made these observations possible. GMRT is run by the  National Centre for Radio Astrophysics of the Tata Institute of Fundamental Research. We thank  D. Bhattacharya ,  A. Vibhute , and  V. Shenoy  for help with CZTI analysis.
Los Alamos National Laboratories, the TANGO Consortium of Taiwan, the University of Wisconsin at Milwaukee, and Lawrence Berkeley National Laboratories. Operations are conducted by Caltech Optical Observatories, IPAC, and UW. The work is partly based on the observations made with the Gran Telescopio Canarias (GTC), installed in the Spanish Observatorio del Roque de los Muchachos of the Instituto de Astrofisica de Canarias, in the island of La Palma. The material is based upon work supported by NASA under award number 80GSFC17M0002. AJCT acknowledges all co-Is of the GTC proposal and the financial support from the State Agency for Research of the Spanish MCIU through the "Center of Excellence Severo Ochoa" award to the Instituto de Astrofísica de Andalucía (SEV-2017-0709). The ZTF forced-photometry service was funded under the Heising-Simons Foundation grant #12540303 (PI: Graham).
S. McB. and J. M. acknowledge support from Science Foundation Ireland under grant number 17/CDA/4723. R. D. acknowledges support from the Irish Research Council (IRC) under grant GOIPG/2019/2033 Analysis was performed on the YORP cluster administered by the Center for Theory and Computation, part of the Department of Astronomy at the University of Maryland. Resources supporting this work were provided by the NASA High-End Computing (HEC) Program through the NASA Advanced Supercomputing (NAS) Division at Ames Research Center. These results also made use of Lowell Observatory's Lowell Discovery Telescope (LDT), formerly the Discovery Channel Telescope. Lowell operates the LDT in partnership with Boston University, Northern Arizona University, the University of Maryland, and the University of Toledo.
Partial support of the LDT was provided by Discovery Communications. LMI was built by Lowell Observatory using funds from the National Science Foundation (AST-1005313). M. W. Coughlin acknowledges support from the National Science Foundation with grant number PHY-2010970. S. Anand gratefully acknowledges support from the GROWTH PIRE grant (1545949). Part of this research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. E. C. Kool acknowledges support from the G. R. E. A. T research environment and the Wenner-Gren Foundations. P. T. H. P. is supported by the research program of the Netherlands Organization for Scientific Research (NWO). H. Kumar thanks the LSSTC Data Science Fellowship Program, which is funded by LSSTC, NSF Cybertraining Grant #1829740, the Brinson Foundation, and the Moore Foundation; his participation in the program has benefited this work.
This paper presents WebT-IDC, an end-to-end Web Tool for Intelligent Dataset Creation, able to construct noiseless corpora of feedback on different topics in any language, from web forums and blogs. The method is based on a unique extraction model of pagination element pattern, independent of DOM structure. WebT-IDC is a holistic tool covering all stages, from user's query, crawling and scraping web pages and extracting relevant data, to building text corpora, free from boilerplate and repetitive data, useful in machine learning tasks. The output is a partially labeled dataset, accurate to reflecting a realistic vision of the market, tested in a Transformer based model in order to show the relevance for immediate use in ML task, offering high accuracy in polarity, language and product recognition.
Introduction The World Wide Web is considered as a massive depository of all kind of constantly increasing data (text, image, video, sound), widely used as an inexhaustible source for corpora creation by mining it  [2] ,  [11]. Before highlight the ever-increasing need for creating and using all kinds of corpora and datasets, it is necessary to provide precision about the use of 'Corpus' and 'Dataset' terms. While both are similar considering their aspect of representative samples of language production, often both providing information of annotations, we may slightly.
Introduction The World Wide Web is considered as a massive depository of all kind of constantly increasing data (text, image, video, sound), widely used as an inexhaustible source for corpora creation by mining it  [2] ,  [11]. Before highlight the ever-increasing need for creating and using all kinds of corpora and datasets, it is necessary to provide precision about the use of 'Corpus' and 'Dataset' terms. While both are similar considering their aspect of representative samples of language production, often both providing information of annotations, we may slightly differentiate 'corpus' as a collection of gathered text and 'dataset' as a structured collection of textual data with attributes and values. Concerning linguistic corpus, researchers spent time to analyze and evaluate "the web as a corpus"  [12] ,  [16] , but this notion lately has changed, including flexibility and customization by crawling more controlled parts of the web  [6].
There were controversial statements about the theoretical approach of "what a corpus is or could be" besides a big collection of texts in a machine-readable format when questions entered the debate about representativeness, size, sampling, balance, design and purpose. The applied requirements to call a text data collection a corpus are the size and the representativeness. We do not consider the balance because of multilingualism. Some languages are less present than others on the web. [12]  argued about the practical aspect with the question "Is corpus x good for task y?" and encouraged the exploitation of the web potential. For the WebT-IDC, the term 'dataset' includes the further use for specific research in machine learning tasks, as 'training' and 'test' datasets. As for the increasing need of corpora and datasets let us remind that they are the main material of many AI projects that rely on data, using Machine Learning (ML) techniques in all aspects of teaching the machines the ability to perform complex natural language operations.
Since datasets are at the core of these projects, we constantly need to create large and high-quality text collections composed of quick, flexible specialized and customized multilingual data from the web  [9]. Even a search engine was created (https://datasetsearch. research. google. com/) to help unravel all existing datasets in the web, without specifying whether the use of the dataset is free or not. 1. However, not many languages have corpora made from web resources such as forums and blogs, and even less have corpora made only by reviews and opinions, written in any language, about a specific product or service available anywhere in the world. The potentiality of further treatment for such kind of context accurate enough to reflect realistic vision of the market (about products and services) is limitless.
Despite the existence of some datasets, we still need to create large text datasets for training. Problems related to topic data scarcity could prevent some AI projects from being accomplished. Furthermore, gathering data is not enough; filtering boilerplate and repetitive data often requires a large-scale cleaning, an essential step of managing and analyzing data, as well as classifying and labelling them which is time-consuming. That is why datasets are still manually created and it is a challenge to generate and maintain them. Since WebT-IDC creates automatically noiseless corpus and datasets, it is essential to mention the difficulties of such an operation. Creating noiseless web corpora implies specific tools for multilingual resources  [5]. This task is always relevant as long as multiple steps are involved for preprocessing textual data  [12].
Among them, cleaning is crucial for all layers of linguistic and machine learning processing and remains an essential task in data preprocessing by filling in missing values, smoothing noisy data, identifying or removing outliers, resolving inconsistencies  [20]. We present WebT-IDC, an intelligent web tool able to crawl web pages and extract relevant information about users reviews and feedbacks in order to build a noiseless thematic dataset. The method used is based on a unique language independent extraction model for forums DOM structures containing pagination architecture. The performance of WebT-IDC computation time is due to multithreading parallel processing and direct access to the relevant page's regions. This paper is organized as follows: Section 2 gives an overview of existing scraping frameworks and tools.
Web Content Mining related works Over the last years, the world wide web has become a huge source of available data, and many research efforts have been made to take advantage of this wealth in several domains. We distinguish three subcategories for web mining: web content mining  [8] , web structure mining and web usage mining  [23]. Also known as Web Log Mining, Web Usage Mining aims to detect behavioural patterns of web users focusing on web server log files. Web Structure Mining analyses the web network structure by studying hyperlinks and navigational relationships between web pages. 1 https://www. blog. google/products/search/discovering-millions-datasets-web/ Finally, Web Content Mining (WCM) focuses on the web page's content and helps extract needful data such as text, image or audio files in a ready-to-use format based on users queries.
Web corpora creation belongs to the Web Content Mining (WCM) category. It is commonly based on web scraping technique, which provides specifically coded programs to extract data relevant to users requests and transform them into structured or semi-structured formats  [14]. Among the state-of-art frameworks used for web scraping, we cite Apache Nutch, Bootcat, JusText and Scrapy. We briefly present them below. Apache Nutch  [10] , is an open-source framework written in JAVA. It has four main components: a web page crawler, a web content fetcher, a web page indexer used to avoid duplicate URL among users requests and a data storing processor. These components implement a multithreading paradigm. Bootcat  [1]  is a Perl open-source toolkit helping to build thematic corpora based on the search engine's results and an input list of seed terms describing what user needs.
This setting step is not trivial, making it time-consuming since the user should respect a set of predefined rules. Justext  [19]  is a python library that works on both text and HTML contents. It is designed to remove boilerplate data by defining a list of stop words and a list of most probable boilerplate tags using a clustering algorithm. Scrapy  [18] : is an open-source Python framework for large scale web scraping. It provides a set of libraries to both crawl and extract data from a web page. This framework is actively updated and helps to extract data from websites. It can also process and store them in the preferred structure and format. Nevertheless, these frameworks and toolkits do not ensure noiseless extraction nor a completely automatic extraction process. Moreover, based on web scraping techniques and available libraries, multiple web content mining tools have been created to ease the data gathering process, according to users needs and queries.
Among the commonly used scraping tools, we cite : Screen-scraper is a web content mining tool that interfaces with SQL database server. It offers a graphical interface to ask the user to provide the URL to mine, target data elements and define scraping logic implemented either in Java,. net or PHP. The processing is then running concurrently on multiple websites. [13]  Web Info Extractor is a tool for extracting both structured and unstructured data from web pages data into a local file or a database. It keeps traces of visited web pages adding new content once the page is updated. It deals with all languages and supports multi-task and recursive processing. Mozenda  [17]  is another web data extraction tool where users can set up agents to process URLs. It provides a web console to view and organize results then export and publish them.
The main drawback is that Mozenda's agent component can run only on the Windows platform. Web Content Extractor is another data extraction tool for Web scraping. It allows users to mine various websites such as online stores, online auctions, shopping sites, real estate sites, financial sites and business directories. It stores the extracted data in many formats, including Microsoft Excel (CSV), Access, TXT, HTML, XML, SQL script, MySQL script, and any ODBC data source. Eliminating noise from constructed corpora is considered one of the essential steps when implementing a WCM tool. Authors in  [15]  proposed a data extraction tool that detects target data section using different string distance measurements such as Levenstein and edit distance. [21]  employed a sequence-labelling approach based on Conditional Random Fields (CRF) to classify each page section into either "main content" or "noisy segment" classes.
[7]  has used the DOM tree tags and attributes to feed data input to a backpropagation artificial neural network (ANN) to cluster web page part into noisy and relevant sections. These tools, unlike WebT-IDC, do not ensure the entire process chain. Besides, they are based on a semi-automatic extraction process requiring human intervention to identify relevant regions through a sophisticated user interface. Thus, they become time-consuming when applied to large dataset creation. In the next section, we describe the two main components implemented by WebT-IDC : the web crawling and filter component and the web scraping and corpus creation component.
System description To meet the urging need of large multilingual corpora ready-to-use in machine learning tasks, we developed a web tool able to construct a noiseless textual corpus composed of reviews and opinions about products and services posted in forums and blogs websites. The design of WebT-IDC is user query driven, without any knowledge about the web page architecture or the language used to express the reviews. It is composed by two modules ensuring, respectively, web crawling with filtering and web scraping with corpus creation component. The former searches for web pages over the world wide web that matches best the user query via search engines and filters boilerplate and repetitive data. The latter extracts users' reviews text and complementary information such as date of the post, location, ranking rate and saves them into semi-structured output files ready to feed machine learning tasks.
The crawling and filter component As shown in figure  1 , the crawler and filter component takes as inputs the user query q and a topic t. It starts by browsing over the net to collect seed web pages relevant to the user's need via the method crawl for query() that uses GoogleScraper  [22]  tool, an open-source tool extracting all found links. Then, it validates the collected list of URLs using a filter component removing duplicate HTTP links (validate remove duplicates()), and consequently avoiding extracting the same data multiple times within the final output dataset. At the end of this step, we obtain a clean list of seed URLs ready to be processed by the scraping and corpus creation component described in the next section. Algorithm 1 WebT-IDC : web crawling and filter component Require: user query q, topic t Ensure: A list of matching URLs candidate urls = crawl for query( q, t) list urls = validate remove duplicates(candidate urls) for all url in list urls do scrap url(url , t) end for.
The relevant data depth is most likely to match either the same depth level as the pagination element node, or the n previous depth where 0 < n < 2. Direct access to the relevant target section considerably reduces the scraping execution time since it avoids browsing all the elements of the DOM tree. The scraping process extracts from the relevant page section the review text and the information related to the reviewer's ID, evaluation score, publishing date, and location when they are provided. These additional features are useful for training dataset annotation in machine learning tasks, such as predicting temporal and geographic trends, positive and negative scores. In our algorithm, the presence of theses features is detected by using structural pattern matching elements, detailed in section 3.
Defining the scraping logic The scraping processing is mainly based on both the DOM tree elements (tags) and style sheet classes. It is written in Python programming language using partly the previously introduced Scrapy framework  [18]  and XPath querying language  [3]  to selecting nodes in semi-structured files. XPath simplifies the direct access to the target nodes by introducing three concepts: the navigation axis, the node's test and the predicates. It is worth noting that Scrapy represents only a part of the developing framework while the logic and the extraction rules are implemented by our algorithm.
Thematic multilingual corpus/dataset creation To make the collected reviews valuable information for further machine learning (or other NLP) tasks, we store them into separate semi-structured CSV files grouped by theme and language. Therefore, each file represents a topic about either a product or a service expressed in a different language. Each line presents the user's feedback described by the comment's text, the user name, the publication date and location when provided, and some extra annotation such as the corresponding theme and the language in which the comment is expressed. In order to keep our scraper efficient in term of execution time, we applied the multi-threading paradigm on two levels: • Seed URLs mining: The scraping component takes as input the flat file, containing a list of relevant URLs after the user's query, which is the output from the previous crawling and filtering step.
Experiments and Results Evaluation We evaluated the performance of WebT-IDC using the precision and recall metrics, as well as the execution time. Table  1  represents the comparison between WebT-IDC and three scraping tools that do not provide the crawling feature. Whereas table 2 compares the presented tool to existing tools build upon frameworks that allow both crawling and scraping. All the tested tools are implemented with a semi-automated extraction approach that requires human expertise to identify the relevant data nodes. This selection process must be adjusted each time following the inherent architecture of the web page changes. As these tools do not provide the crawling mechanism, we used our model to fix the input seed URLs to scrap. The results show that WebT-IDC can achieve 1OO% precision with a fully automatic extraction process.
Execution time performance is also competitive. In table  2 , we evaluated the quality of the scraped data comparing to the experiment presented in  [15]  since the tool is not available for new tests. So, we repeated the same experiment by testing twenty web pages from the domain "ubuntu-fr. org". WebT-IDC outperforms all the considered tools. It gives a noiseless dataset with 1OO% precision and recall and obtains the best execution time. For this test the crawling mechanism was disabled for WebT-IDC since the input seed URLs were fixed. In another example of WebT-IDC performance, with a topic such as "airlines services", we obtained a global number of 44348 reviews dispatched as follows: 17083 comments are expressed in French, 16668 in English, 2240 in Spanish, 3300 in Italian and 5057 in German.