dataesr/scientific-paragraphs-categorization huggingface dataset:
1. Project name: A string encoding the LHC beam type with the year of data taking, 2. RunID: The run number, 3. Stream name: Events passing specific triggers are written to one or more data streams, 4. Data format: The stage of processing at which the data are indexed (usually AOD), 5. AMI Tag: a string encoding the processing steps these events have undergone, i. e. effectively the processing version for the events in this run and stream. 6. Production Step: A short string to distinguish between an intermediate or final processing stage. Since the Datasets table is the parent table for all other tables of the schema, this table has an integer primary key associated with each indexed dataset name as is common in relational database design. For each dataset, all events are stored in the Events table.
We store up to 3 GUIDs per event, which we found to be sufficient. References are numbered starting at 0 (the GUID of the indexed file), with subsequent references 1 and/or 2 used for its upstream file formats. Since available GUID types (RAW, AOD, and DAOD) are common to all events in the dataset, GUID types are stored only once in the Datasets table (another advantage of relational design). Fig. 6  Relational tables of the EventIndex Oracle (EIO) schema. The "Datasets" table contains one row for each imported dataset. The unique events for each dataset are stored in the "Events" tables and its duplicated events (if any) are stored in the "Event Duplicates" table. Aggregated information about each dataset is stored in "LBN Counts" (a count of events per Luminosity Block) and "Dataset Overlaps" (the number of common events between datasets of the same run number) Also, at the event level, we store the event's luminosity block number (LBN) and LHC bunch crossing identifier (BCID), which are useful for use cases described in "Web Interface".
The Event Duplicates table keeps the list of all duplicated events found. Duplication can (and has been shown to) occur at any stage in processing. In the initial 2015 loading of the data, hundreds of datasets were found with duplicate events; using data stored in this table combined with event counts at each stage of processing (from the AMI database), we could identify the stages at which duplication occurred. Subsequent refinements in upstream systems have considerably reduced the occurrence of duplicated events in current data. The Dataset Overlaps table stores the number of events in common between different datasets of the same run. This data serve a secondary use case of providing these overlaps to experts in DAOD production for refinement of the ATLAS Derivation Framework  [47]. The LBN Counts table stores the event counts and the number of associated unique GUIDs by LBN.
Data in this table have multiple secondary uses including many forms of integrity checks, determining the probable LBN for an event, investigating missing events and files, and understanding the splitting of luminosity blocks at file boundaries. Both the 'Overlaps' and the 'LBN Counts' table content could be computed dynamically using the data in the two primary tables, but we chose to materialize this data in these tables since the computation can take more than a few seconds, the data volume is minimal, and some of the aggregated data are used in multiple services. When a new dataset appears in Hadoop storage, it is considered for import into Oracle if the run exists in the COMA system (which contains only runs of potential physics interest), and if the Stream meets similar selection criteria (e.
g. there are no known use cases for indexing datasets in calibration streams). If the dataset passes these requirements, the "Import process" stage imports the dataset and its events into "Staging tables" which are similar to the final tables, but without indexes or constraints. The "Oracle scheduler Jobs" stage runs verification checks such as checking that the events are consistent with belonging in the same dataset and flagging if any duplicate events are found. If the data pass verification checks, the data are moved to the "Destination Tables", writing any duplicate events to a separate table, while keeping one copy for the Events table. Subsequently, supplemental information is added to the Datasets table including data from other repositories (COMA and AMI) as well as aggregated information from Events table loading.
This includes dataset status flags, various relevant dates, event counts both within the system as well as related counts in ATLAS file systems (upstream dataset files), counts of unique GUIDs associated with the dataset, and counts of total and unique duplicated events. In addition, the datasets with Run, Stream, and Data Format in common are ranked by dataset creation date, which is useful to help users find the latest processing of a set of events. These columns are used to enhance various services, and/or are included in user interfaces and reports. A number of additional database optimization techniques deployed in this system, which further minimize storage volume (beyond relational normalization mentioned previously), transaction volume and database load, and optimize query performance for use cases, strongly deserve mention: -The Events table is "list" partitioned by DATASET _ID.
The main advantage is that sets of events can be deleted by simply dropping the associated partition. This operation is needed more often than we initially expected because datasets are sometimes re-indexed because of constituent file loss on the grid (which invalidates the associated GUIDs). -For the Events table we use Oracle's "basic" compression for table data and key compression on its primary key index. Moreover for data loading we use Oracle's direct data load interface. In combination, storage utilization is reduced by a factor of about 3. 5 which has the added advantage of reducing similarly the I/O footprint for writing data, undo and redo to the storage subsystem. -Up to three GUID reference columns per event in the source data are 36-character strings (for example "21EC2020-3AEA-4069-A2DD-08002B30309D").
The savings of 4 TB of disk space, in itself, is not the foremost point but has a knock-on effect which is particularly beneficial for query performance: it enables the caching of a larger fraction of the database rows into the database data cache (buffer pool) which yields real performance gains in query response (around 10 ms for simple queries). In summary, using a relational model and a number of carefully chosen techniques available in Oracle RDBMS results in an impressive minimization of resources while exceeding performance goals.
Web Interface The central part of the user interface is the EIO (Event Index in Oracle) Browser shown in Fig. 7 , which allows users to easily find indexed datasets and their properties. The browser offers dynamic filtering of datasets by any of the dataset name fields and/or other dataset characteristics. With each iteration of selection criteria, the system shows the number of remaining datasets meeting the criteria and displays the remaining criteria. Once the user has selected their dataset(s) of interest, they can choose from the following services: -Event Lookup serves the primary use of returning GUIDs for user specified events (RunID/EventID pairs). In the absence of a user specified dataset version, GUIDs from the highest EIO-derived ranked dataset are returned. The report provides additional details about the events found as well as information to help determine why events were not found.
-The Dataset Report includes a table displaying details about each selected dataset: the collected and derived information in the Datasets table. EIO event counts are compared to counts of the corresponding upstream dataset files which helps to understand event losses/filtering at each stage of processing. Links are provided to related AMI dataset and COMA run reports and to other EIO services described herein. -The Dataset Overlaps Report shows the count and percentage of events in common between selected datasets of any run that is useful for the resource optimization of the offline production of DAOD  [47]. Results are displayed in a colour-enhanced 2-D matrix (as in Fig. 8 ) showing datasets which overlap by more than a 70% threshold. This threshold and a choice of two overlap computation algorithms are configurable in the interface.
-The Duplicate Event Report displays all copies of events with any duplicated event identifiers in a dataset. It shows clearly the LBN(s) where duplication occurred and the associated GUIDs, from which, combined with event counts at each stage in processing, we can unambiguously determine the processing stage in which the duplication occurred. -A Missing Event Report can be generated when a dataset has fewer unique events than expected. The cause may be intentional filtering or an unintentional error resulting in in-file event loss or entire files of events being lost; reports show event counts (and computed losses) at each processing stage. If those events have been completely processed and indexed in another version of processing, the report shows lost event ranges and associated LBN(s).
-The Count by BCID Report displays event counts in each LHC bunch crossing (BCID). During collision operations, one clearly observes the correlation in the peaks of recorded events with BCID with the LHC fill configuration of the run. Fig. 7  The EIO dataset browser entry page. Users can set search filters by clicking on predefined options or typing in the text boxes on the left, and then select the kind of report by choosing from the menu on the right. The reports are described in "Web Interface" Fig. 8  The EIO dataset overlaps report, as an example of the functionalities provided by the EIO dataset browser. This report shows the count and percentages of events in common between selected datasets, as described in "Web interface" -The Count by LB and GUID Reports both display event and GUID counts along with EventID ranges per LBN which have been aggregated in the LBN Counts table.
System Monitoring The successful operation of the EventIndex system depends on a number of different components. Each component has different sets of parameters and states and requires a dedicated approach for monitoring. A first version of the EventIndex monitoring tools  [22]  based on Kibana  [48]  was developed in late 2014, but it suffered from performance issues, so a new version based on InfluxDB [49] as data store and Grafana  [50]  for the display was developed  [51]. The monitoring infrastructure consists of two parts, producer and viewer. The producer part is responsible for collecting data and transferring them to the database; it includes the scheduler, a number of Python scripts and the database. The scheduler uses a cron utility to run the Python scripts at fixed times. The Python scripts collect data from CERN and Grid sites and insert them into the database.
Several different modules monitor different system sub-components: Data Production, Consumer Processes, Hadoop Imports, Hadoop Cluster, Trigger Database, Web Interface, Server Status, Event Picking Tests and Data Volumes. Each of these modules requires a different approach for data collection and processing, thus every module has its own Python script and scheduler to run it. The viewer part is responsible for the graphical presentation of data. Figure  9  shows the functional schema of the EventIndex monitoring system and the data flow. Grafana supports different back-end databases; it was decided to use InfluxDB as a front-end database because support for InfluxDB+Grafana is provided by the CERN-IT Monitoring group. Although the group policy does not allow writing data directly to InfluxDB, an HTTP endpoint to the middleware that moves data to the database is provided for records in JSON format.
This JSON format has a common part that is the same for all databases supported by the CERN-IT Monitoring group, and custom parts that are different for each database and carry the specific information for each service to be monitored. The visualization component has a status dashboard for all modules, dashboards for the most important parameters of each module and links to the module details pages. The current status for each module is calculated using its own algorithm based on the module critical parameters. The status can have one of following values: -"available" (green)-the module works correctly -"degraded" (yellow)-the module has some non-critical problem -"unavailable" (red)-the module has a critical problem -"N/A" (white)-monitoring data are not available for this module. The details page of each component usually contains additional dashboards.
Operations and Performance Before the start of LHC Run 2 operations, we indexed all LHC Run 1 datasets in AOD format on the Tier-0 cluster. In this way we collected information on the RAW dataset provenance and on the trigger parameters of each Run 1 event. After that, real time operations started. The indexing jobs are distributed on all sites available to ATLAS, selecting primarily those that are closest to the input dataset location from the network point of view, as shown in Fig. 10. The indexing jobs are fast, as the producer transformation only reads the header of each event and takes between 10 and 50 ms/ event, depending on whether the trigger record is needed or not; each job indexes several files and runs for 30 to 60 min. The total CPU consumption of EventIndex Producer jobs is well below 10 -4 of the total CPU power used by the ATLAS experiment world-wide.
The percentage of jobs failing has been around 7% , with the main causes of errors being problems related to the input files and the sites operation, such as corrupted files, sites with storage or disk configuration issues, stage-in problems, etc. The indexing jobs are the first jobs run on just-produced datasets, so they are useful to detect at an early stage any problem with data corruption or unavailability. After contacting site administrators and the ATLAS data management operations team, the problems are usually solved promptly, so that simply re-running the problematic jobs is sufficient to achieve consistency. The number of stored event records increased approximately linearly as shown in Fig. 11. Some datasets, especially those with type DAOD, do not have an infinite lifetime but are periodically replaced by newer versions generated with better calibrations or improved algorithmic code; after some time the old versions are deleted.
The down steps in Fig. 11  correspond to periodic clean-up operations that remove the information regarding obsoleted datasets. The current amount of disk space used by the Event Index data in the Hadoop cluster is shown in Fig. 12. Most of the disk space is used by real data and MC AODs, which contain the trigger records. The event generator datasets (EVNT) and derived analysis formats (DAOD) contain many more event records, but without trigger information, as it is either not yet available (in case of the generator-level EVNT datasets) or retrievable from the corresponding AOD datasets (for the derived formats DAOD). The Hadoop system runs a variety of tasks, importing and cataloguing data, running consistency checks, establishing links between related datasets, and last but not least responding to user queries.
The EventIndex data stored in Oracle increase in size in parallel to the growth of the main Hadoop store, as shown in Fig. 16. Due to the relational database nature of Oracle, a large amount of indexes is stored together with the actual payload data, so that a similar amount of disk space is used by actual payload data and the indexes. Most of the event picking requests are for single events in RAW format; other requests are placed from time to time from physics analysis groups who need to extract their complete highly selected data sample for further processing and/or more detailed analyses; so far the EventIndex system could cope very well with all requests. Figure  17  shows the statistics of the event picking jobs run between January 2019 and June 2021. During this period 9. 5% of the jobs were automatically "closed" and rescheduled to another site while waiting for the input file (or files) to be staged from tape, and 6.
System Evolution The described storage implementation reflects the state of the art for BigData storage tools in 2012-2013 when the project started, but several different options appeared since, even within the Hadoop ecosystem. With the   15  Response times of the EventIndex Hadoop server to queries using Map/Reduce jobs to retrieve information on all events from datasets containing 50k, 100k, 1M and 10M events as a function of time, recorded between February and June 2021. The discontinuity corresponds to a day of Hadoop cluster maintenance, during which no statistics were collected increase of data-taking and simulation production rates foreseen for LHC Run 3 (2022-2025) and even more for LHC Run 4 (High-Luminosity LHC, from 2028 onwards), a re-design of the core systems is needed. In order to be safe, a new system should be able to absorb a factor 10 more event rate than the current one, i.
Additional tools like Apache Phoenix [36] can provide SQL access to HBase tables, if the tables are designed appropriately upfront, which can be done in our case. HBase works best for random access, which is perfect for the event picking use case where we want low latency access to a particular event to get its location information. Use cases where we need information retrieval (trigger info, provenance) for particular events are served by fast HBase gets, with good performance. In addition, analytic use cases where we need to access a range of event information for one or several datasets (derivation or trigger overlaps calculation), can be solved with scans on these data. They can be optimized with a careful table and key design in order to maintain related data close within the storage, reducing access time.
EventIndex data rarely need schema changes, so we can benefit from Phoenix access, designing the required schema and tables accordingly. The table schemas and their relations  [54]  closely resemble those implemented for the Oracle version of the data store ("Data Storage in Oracle"). While updating the core storage system, other components can be revised and if necessary updated or replaced: -The Producer implementation is currently done in python with a single thread. It will be upgraded to work with the latest data analysis software and external libraries like stomp. py [55] , boto  [56]  and Protocol Buffers  [30]. -The Data Collection system will use modern data processing technologies like Spark  [57]. It will also allow to simplify all procedures, reducing data duplication and using common job management tools over the stored data.
-The Supervisor will be expanded to cover the entire workflow, from the selection of datasets to be indexed to the storage of data in HBase. -The detection of duplicated events and the calculation of statistics for each dataset will be done "on the fly" during the import process. -A new implementation of the Trigger Counter will make direct use of the Hbase/Phoenix infrastructure, which provides fields and families to store the six trigger masks of the event. -A graph database layer working on top of any SQL database has been implemented to deliver a graphical and highly interactive view of the EventIndex data stored in the Phoenix SQL database. Thanks to its SQL genericity, this layer can work with all ATLAS data stored in SQL databases, thus providing a global navigable overview of all ATLAS data.
Acknowledgements This work was done as part of the distributed computing and databases applications research and development programme of the ATLAS Collaboration, and we thank the collaboration for its support and cooperation. Part of this work was funded through  PRIN Project  " STOA-LHC 20108T4XTM ", CUP:  I11J12000080001 , of the  Italian Ministry of Education, University and Research (MIUR). In Spain , MICINN partially supported this work under grants  FPA2016-75141-C2-1-R  and  PID2019-104301RB-C21 , which include FEDER funds from the  European Union.
Conclusions The ATLAS EventIndex was designed to hold the catalogue of all ATLAS events in advance of LHC Run 2 in 2012-2013, and all system components were developed and deployed in their first implementation as described in this paper by the start of Run 2 in 2015. As any software project, it went through several stages of development and optimization through the years. Thanks to the partitioned project architecture, each new component version could be tested in parallel with the production version and phased in when its performance was considered stable, and better than the previous version. The EventIndex operation and performance during and after the LHC Run 2 period has been satisfactory. The significant increases in the data rates expected in LHC Run 3 and the subsequent HL-LHC runs required a transition to a new technology for the main EventIndex data store.
Introduction Social media platforms like Twitter have become integral to modern communication, facilitating the exchange of information and ideas on a global scale. Within these platforms, the central gateway for users navigating the vast amount of user-generated content is the Newsfeed, which is thoughtfully curated to maximize user engagement. These systems, which filter content from one's social environment, function as attention-allocators  1  , wielding substantial influence over the content encountered by users and thereby present a potential systemic risks to society. A notable study by Huszár et al 2 demonstrated how Twitter's recommender system amplified the reach of political tweets depending on their ideological alignment. To demonstrate this, they leveraged proprietary data on Twitter users and conducted a years-long experiment, which included a controlled group of nearly two million users not exposed to Twitter's recommender system.
Furthermore, we present how political-group-wise, our set of participant does not significantly differ from a random sampling of Twitter French users, in terms of their friends political leaning distribution. Taking the participants having been active on the desktop version of Twitter between March 3, 2023, and April 6, 2023 as our only objects of study, the analysis has been performed on N = 463 participants. On average, our participants followed 682 [22,2712] accounts (5-95 percentiles). In conjunction with the crowd-sourced data collection, we leveraged the Twitter API to fetch additional information, in particular the number of tweets published by participants's friend within the considered timeframe. Balancing the server-side data collection burden while encompassing a large portion of the participants' friends, we restricted our data collection to the 42k accounts followed by at least two participants, within the pool of 182k unique accounts followed by at least one participants.
Toxicity and Sentiment Analysis of tweets We investigated the algorithmic amplification of tweets either sentimentally valenced or deemed "toxic", such as tweets featuring insults, threats, and obscenities. The sentiment analysis was conducted using the XLM-T model, a multilingual language model trained on tweets. For the identification of toxic tweets, we leveraged Detoxify  13  , an open-source natural language processing model trained on Google's Jigsaw toxic comments database. Particular emphasis was placed on the mitigation of unintentional biases in the training process, including racial biases, as discussed by Davidson et al  14. The validity of employing this two models for this study was established by manually annotating randomly sampled tweets from the collected pool.