dataesr/scientific-paragraphs-categorization huggingface dataset:
The acquisition and reconstruction units are similar (Fig. 2c-d ). Each of them implements a file watcher thread that monitors a folder selected by the user and then adds the files to a queue. Each file is processed sequentially, and a signal is emitted when each execution is finalized. The acquisition unit deals with python scripts that execute the experiments (runScript) and emits the signal sigScriptExecutionFinished. The reconstruction handles raw images that are reconstructed, followed by emitting the signal sigExecutionFinished. The user can select between saving the raw images in Zarr or HDF5 files in the acquisition unit. The metadata is also included as an attribute (ImSwitchData) and contains all the experimental details needed for reproducibility. The reconstruction unit saves the data in both TIFF and Zarr files.
The file watcher functionality is implemented in the model layer of ImSwitch, and the orchestrator as an independent plugin distributed in the napari hub. The widgets can be easily applied to other python software packages and monitor any file extension added to the initialization function, providing generalization to multiple applications. A logger file is written into the folder once the experiment is finalized. It contains information about the computer name and starting date, all the files processed, and the time between arrival and end of execution of each item. Therefore, the performance of each computer can be easily evaluated for different reconstruction algorithms and experimental applications using resource monitor software, such as the Resource Manager in Windows.
ImSwitch installation and configuration We have released a new version of ImSwitch (v2. 0. 0) that supports the features presented in this article, which is included in the Python Package Index (PyPI). Therefore, the only necessary steps in order to install ImSwitch are to install conda and run the following commands: conda create -n imswitch-env conda activate imswitch-env pip install imswitch imswitch After installing ImSwitch, a configuration folder is generated in ''Documents" > ''ImSwitchConfig". A standard procedure is to add the setup file containing the hardware device list and software widgets to display in ''imcontrol_setups" folder (see more in the documentation). Upon installation, there is already a list of example files, and upon initialization, the user can choose one of the available options.
After ImSwitch is initialized, the setup file to load can be changed in ''Tools" > ''Pick hardware setup. ". The widget with the functionality for the file watcher needs to be included in the ''availableWidgets" by the end of the setup file as ''Watcher". Fig. 3  shows the ImSwitch GUI in the acquisition unit for the microscope that we employed in the experimental part of this work (see Validation and Characterization). The file ''config"> ''modules" contains a list of the GUI modules to be displayed. We recommend only having ''imcontrol" and ''imscripting" in the acquisition unit and ''imreconstruct" in the reconstruction unit. Fig. 4  shows the reconstruction unit ImSwitch GUI. The reconstruction module is implemented for parallelized RESOLFT strategies. However, different modules can be implemented and added to ImSwitch.
3. In the acquisition unit, in the recording settings widget (bottom left of the GUI), the user can select the folder where the experimental data will be saved, which will be a folder in the shared disk. Also, select Zarr as the file extension. 4. In the reconstruction unit, in the bottom-left corner and the widget ''File Watcher," the data folder should be selected, and then check the ''Watch and run" box. 5. The reconstructed data will be saved in a folder named ''rec" inside the data folder. Select that folder in the file watcher widget of the ''napari-file-watcher" plugin, and then press ''Watch and run. " 6. Finally, select the scripting folder (same as in 1. ) from the scripting widget in the ''napari-file-watcher" plugin, write scripts or open existing ones, and then start the experiment by pressing ''Add".
The script can access any API exported function, and new functions can be easily added using the @APIExport decorator in any controller. A list of the accessible functions is shown on the documentation page. Fig. 6  shows an example script for performing timelapse imaging of N = 10 lapses in the (x,y,z) coordinates. Each of the lapses is a scan-based experiment that saves the raw data on disk. The scripting engine is in charge of executing the files in the acquisition unit (imscripting). It also has a GUI, which we display in Fig. 7. It provides basic script editing functionality and the possibility to run and stop experiments. The GUI can be helpful first to test, debug and implement the experimental scripts locally before using the full framework. After the scripts are implemented, the orchestrator unit will be used to adapt the scripts further and send them to the acquisition unit.
Validation and characterization In order to validate and characterize the proposed framework, we have used the MoNaLISA  [31]  microscope. MoNaLISA is a parallelized RESOLFT method that offers a large FOV (up to 50x50lm 2 ) with an increased speed (0. 3-1. 3 Hz). We chose two use cases related to timelapse and tiling images of extended regions for different biological structures: mitochondria and actin in U2OS and neuronal cells. The main focus is to automate the acquisition using scripts and increase the microscope throughput by distributing the acquisition and reconstruction workload using different devices.
The focus lock keeps the focus automatically over time by using the cover glass reflection and successfully adjusts to the 2x2 tiling movement. The experimental design and results are shown in Fig. 8. A script contains the experimental instructions to be sent to the microscope. In this case, we perform a MoNaLISA scan in every tile and save the files on the hard drive. In between scans, the motorized stage moves to the following area. We use the digital outputs of a data analog card (NIDAQ PCI 6371) to synchronize hardware devices (e. g. , lasers and stage axis) with the camera (externally triggered) using a pulse scheme optimized for RESOLFT imaging. The script can successfully wait for the end of the scan by using exported signals in the imscripting module of ImSwitch. By combining MoNaLISA with the presented multiunit framework, higher throughput can be achieved by distributing the reconstruction workload (limited mainly by the RAM usage) and improving the computation time while keeping the lateral and temporal resolution introduced by the technique.
Selective tiling in neurons and U2OS cells with a user-driven approach As a second application, we extended the FOV of the MoNaLISA microscope by applying the proposed framework in tiling experiments to achieve higher throughput. We imaged extended sample regions of neurons and U2OS cells. Neurons are highly polarized cells that communicate with each other via specialized sites called synapses, which occur along neuritic processes such as axons and dendrites. These processes grow from the cell body and expand over large areas with high morphological and functional complexity relevant to synaptic organization and transmission  [51]. Therefore, tiling in superresolution microscopy is a method that can highly benefit neuronal imaging by massively expanding the FOV  [25]. We developed a tiling script that interacts with the user in order to perform selective tiling.
The script consists of two steps. Firstly, user-driven registration of focus and tiles is performed, with the possibility of annotating the tiles that can be skipped because they do not contain biological information. Secondly, the microscope performs automatic tiling of the selected tiles, performing a MoNaLISA scan in each position (Fig. 9a ). The stage moves automatically between the tiles, and the script interacts with the user through a dedicated widget (TilingWidget). The MoNaLISA optical setup used has a field of view of 38 Â 38 lm 2 and collects the raw data by scanning the sample with piezoelectric actuators (3-axis NanoMax stage from Thorlabs). To move in between fields of view, we employed the stepper motors of the stage controlled through USB and integrated into ImSwitch using the BSC203Manager (which calls the python library thorlabs-apt-devices), and a 14 % overlap between tiles.
The raw data consist of a stack of 324 frames and the acquisition time is 8 ms per frame (2. 6 s per image), and we added an extra time of 1 s after each stage movement to prevent from drift. Because of the tilt between the sample and the stage, the z position was slightly different for each tile, which is why we added a widefield-based registration. This process could be further improved by characterizing the tilt or automatically finding the focus in each tile. We used the Fiji plugin MosaicJ for stitching the images  [56]. We performed 5x5 tiles (Fig. 9b ) to extend the FOV from 38 Â 38 lm 2 (Fig. 9c ) to 160 Â 160 lm 2 (Fig. 9d-e ) while keeping the super-resolution features introduced by the microscope (Fig. 9f-h ). Furthermore, we extracted the reconstruction time of the remote unit from the logger files.
Evaluation of acquisition and reconstruction latency and resources The system can be employed either by using multiple computers or running ImSwitch instances within the same machine. In order to evaluate whether multiple computers are necessary, the workload of each process needs to be assessed for each application. In Table  1. we characterize the occupied resources of the acquisition and reconstruction units in terms of RAM, CPU usage, and computation time for the proposed use cases. We have done so by using the Resource Monitor in Windows for the RAM and CPU usage and the datetime python library directly in ImSwitch to precisely compute the timings. Results show that CPU usage is always low and close to idle (control) since our experiments use digital triggering from an external data acquisition card.
Understanding the veracity of statements is important when consuming information on the Web. Whereas fact-checking sites have provided a large corpus of already verified claims, matching a given utterance to already fact-checked claims remains a challenging task. Verified claim retrieval has been approached through a variety of different methods, among them approaches relying on supervised neural models. Whereas such models tend to perform strongly, they require significant training effort and their robustness towards unseen data distributions may vary heavily. Also, prior works demonstrate the capability of unsupervised models to provide state-of-the-art performance. In this paper, we assess established claim retrieval benchmark datasets and experimentally evaluate and compare different state-of-the-art supervised and unsupervised methods with regard to performance, but also computational effort Robust and Efficient Claim Retrieval for Online Fact-Checking Applications and run time.
This problem is known as verified claim retrieval and has been recognised by the Check That! initiative  [5] [6] [7] , that has advanced research in this field by producing benchmarks and baselines and organizing shared tasks. A variety of approaches has since been proposed. However, those were evaluated on different individual datasets, making it hard to compare their performance, their strengths, and weaknesses, and assess their robustness. Generally, while supervised approaches usually dominate regarding task performance, they tend to overfit the data distributions prevalent in the training corpus at hand, making them less robust on new data and evolved data distributions. Also, they are prone to picking up spurious correlations in the data, causing them to perform well on a given dataset rather than performing well on the task  [8].
It is thus crucial to understand the characteristics of benchmark datasets, to compare the performance of methods across different datasets, and to understand how well the data reflects the task in the real-world. Another important aspect that has so far achieved little attention is the question whether proposed approaches are actually useful in practice  [9] , in this case, to link unverified statements to a database of fact-checked claims, such as ClaimsKG  [10, 11] , as proposed by  [12]  with their ClaimLinker system. This means that beyond the quality of predictions, method performance with respect to run time and computational costs needs to be understood. This involves both efficiency during inference as well as offline computation costs during training, especially since dynamically emerging and evolving vocabulary has proven a significant challenge, requiring frequent updates of models to ensure stable performance over time  [13].
This does not only pose challenges for practical reasons, but also leads to high energy consumption, exacerbating environmental issues  [14] , especially when unnecessarily large language models are applied  [15]. With this work, we investigate the state-of-the-art in the retrieval of previously fact-checked claims, assessing used benchmark datasets and the performance of claim retrieval models. In particular, we aim at understanding whether the performance gains on particular datasets justify the required effort and computational and environmental costs for training and fine-tuning large supervised models. We also compare the efficiency of approaches and their scalability. Given the growing amount of previously fact-checked claims and the importance of fast response times in real-world fact-checking scenarios, fast run time at inference is a crucial criterion.
To investigate these questions, we (a) assess established benchmark datasets for the retrieval of previously fact-checked claims to aid the interpretation of evaluation results, (b) compare 22 (15 supervised and 7 unsupervised) state-of-the-art methods with respect to performance, and (c) replicate and newly generate additional results on more datasets for 8 of these systems (4 supervised and 4 unsupervised) to compare them with respect to overall performance, robustness, scalability and run time. We find that fine-tuning and supervision can indeed increase performance at the cost of risking overfitting. For instance, the overall best approach, RIET Lab, is supervised and relies on large language models. It scores between 0. 008 and 0. 039 points higher in MAP@5 on all datasets but one than the overall second best approach, unsupervised SimBa 2023 which uses considerably smaller pre-trained models.
However, SimBa 2023 scores 0. 108 points higher on the remaining dataset. The choice of pre-trained models and embeddings turns out to be more crucial than fine-tuning and the high computational costs for supervision may not be worth the performance gain for real applications, given that the performance difference between the best supervised approach and the best unsupervised approach averages to -0,07 points in MAP@5 over all datasets while the unsupervised approach achieves lower retrieval times on a laptop than the supervised approach on a high-performance GPU server, which usually is not available for online fact-checking applications. The main contributions of our work are: • an assessment of established benchmark datasets for the retrieval of previously fact-checked claims • a comparative evaluation of different supervised and unsupervised approaches on established datasets for the claim retrieval task and extensive experimental results w.
r. t. robustness of the approaches comparing the performances when training and testing on claims of different discourse types • the assessment of the applicability of different approaches for realworld claim retrieval applications based on performance, efficiency and computational expenses • modifications of existing approaches to allow cross-evaluation and application on different datasets. All modifications and all scripts to replicate this study are released to the community. The paper is structured as follows. We start with the problem definition and introduction of our research questions (Section 2), before giving an overview of related work and the state of the art (Section 3). Section 4 details the compared claim retrieval approaches and our experimental setup. Our data analysis follows in Section 5 before we present and discuss the results of our comparative evaluation in Section 6.
Problem Definition and Research Questions From an application perspective, the task of retrieving previously fact-checked claims represents the initial step of fact-checking, e. g. in journalistic or news contexts, aiming at identifying if a given statement has been already factchecked by trusted sources and finding fact-checked statements that are relevant to determining the veracity of the input claim. More formally, given an input query q (e. g. the text of a tweet, or a sentence in a political debate) and a dataset D of previously fact-checked verified claims (with each verified claim denoted as c), the claim retrieval task aims at providing a ranked list of top-n claims C from D. Note here that a retrieved claim c consists of not only the text of the utterance but also additional metadata such as the title of the fact-checking article.
For example, consider the sentence from a political debate as input query: "You know, interest on debt will soon exceed security spending. " Table  1  displays the top-3 list of previously fact-checked claims provided by the ClaimLinker  [12]  system. In this context, the size of the dataset D (number of previously fact-checked claims) is important because it can affect both the retrieval effectiveness and the retrieval efficiency. In particular, a larger dataset might mean more semantically similar claims (which can make retrieval more difficult and affect Table  1 : Top-3 verified claims retrieved by the ClaimLinker system for the input query "You know, interest on debt will soon exceed security spending. " precision) and higher response times, especially for supervised methods that often require pair-wise comparisons.
Operating on a small dataset might lead to more unmatchable input claims, reducing a system's recall. In this paper, we investigate the following research questions: 1. RQ1: What are the characteristics of the task of retrieving previously fact-checked claims and existing benchmark datasets? 2. RQ2: Which supervised and unsupervised state-of-the-art approaches perform best on these established datasets and why? 3. RQ3: How robust is their performance depending on the training data? 4. RQ4: How efficient are the different approaches and how well do they scale depending on the number of verified claims in the dataset? 5. RQ5: Which approach is the best choice for (online) claim retrieval applications? 3 Related Work In the following, we provide an overview of existing claim retrieval tasks, datasets, approaches and studies giving insights regarding dataset characteristics that are relevant to this work.
Claim retrieval tasks and datasets A number of shared tasks and datasets have been proposed that evolve around fact-checking and the linking of input claims to information that can aid with fact-checking. FEVER  [16] , for Fact Extraction and VERification, is a shared task focusing on checking factual claims. It consists of the retrieval of evidence from Wikipedia and using it to verify the claims. The FEVER dataset  [17]  consists of 185,445 claims generated by modifying Wikipedia excerpts that are subsequently verified independently from the original part of the text from which they are derived. The contained claims and their corresponding evidence (sentences that support or refute them) are categorized into three classes: Supported, Refuted, or NotEnoughInfo. Also dealing with the problem of claim verification and the categories Supports, Refutes, NoInfo/NotEnoughInfo, the dataset SciFact  [18]  and its recent extension SciFact Open  [19]  focus on claims and evidence from the scientific domain.
While these are related tasks, the linked evidence and articles cannot be used to directly infer veracity labels for a claim as is expected for verified claim retrieval. Rather, the contained evidence has to be reviewed and assessed by experts. The MultiFC  [23]  and the ClaimsKG  [10, 11]  datasets both provide structured data of and about claims coming from reputed fact-checking portals, where claims are stored together with rich metadata (such as authors, sources, claim reviews and other contextual information, including veracity labels). While both datasets retrieve and store claims from fact-checking portals, they are complementary in a couple of aspects. MultiFC focuses on evidence-based fact-checking in terms of downstream tasks, where via the Google Search API the 10 most highly ranked search results per claim are retrieved and stored.
ClaimsKG, on the other hand, provides a rich data model (an RDFS ontology) to represent check-worthy (or fact-checked) claims and related metadata, which is an important effort towards standardization and enables federated access to distributed data, where a specific search engine is provided 3  in addition to a public Sparql endpoint  [24]. MultiFC contains data in English, ClaimsKG is multilingual, harvesting data from fact-checking portals in English, French, Russian, Spanish, Italian, and Hindi. These datasets can be used to provide a pool of verified claims with additional metadata for fact-checking applications and to extract links to claims that are mentioned in fact-checking articles. One initiative that directly focuses on the linking of unverified claims to previously fact-checked claims is the CheckThat! Lab  [6, 7, 25].
Both titles refer to the same ranking task. The task was offered for multiple languages. The CheckThat! Lab features real claims uttered by humans: Task 2A treats check-worthy tweets as input claims, Task 2B contains claims made in political debates or speeches. In both cases, real fact-checks from fact-checking portals are used to construct the set of verified claims to match against, partly drawn from ClaimsKG. Since this task definition matches our problem definition, we choose the CheckThat! Lab datasets and the participating systems for our study.
Claim retrieval approaches Early work has discussed the task of retrieving fact-checked claims as part of an end-to-end fact-checking pipeline, without providing detail or an evaluation  [27]. Hence, the two ground-laying works in that field are  [26]  and  [3]. In  [26] , the authors provide two datasets: one using Politifact articles and one using tweets mentioned in Snopes articles. The latter and the annotation protocol later became the basis for the CLEF CheckThat Lab! claim retrieval datasets. They evaluate a standard information retrieval method (BM25) and provide a definition of the task. In  [3] , the authors extend this method and incorporate global and local contextual information contained within the source texts, i. e. the political debate containing the input claims, and the target texts, i.
e. the fact-checking articles containing the verified claims. They find that contextual information, especially on the source-side, helps the claim retrieval performance. A multi-modal approach was proposed by  [28] , whose neural ranking model uses both text and images for the retrieval and recommendation of claims about images, based on mining tweets containing images and links to fact-checking articles. On the same dataset and on data from the Chinese platform Weibo,  [29]  developed and evaluated their memory-enhanced transformer-based approach to select key sentences from fact-checking articles to estimate the relevance of a verified claim for a given input claim. Systems submitted to the CheckThat! Lab shared claim retrieval tasks range from relying on pre-trained or fine-tuned language models and using supervised methods to efficient index-based systems relying on traditional information retrieval techniques such as BM25 in supervised or unsupervised setups.