OpenRaiser/Intern-Atlas huggingface dataset:
Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation
Venue: arXiv cs.CL 2024
Year: 2024

With the recent advances of large language models (LLMs), it is no longer infeasible to build an automated debate system that helps people to synthesise persuasive arguments. Previous work attempted this task by integrating multiple components. In our work, we introduce an argument mining dataset that captures the end-to-end process of preparing an argumentative essay for a debate, which covers the tasks of claim and evidence identification (Task 1 ED), evidence convincingness ranking (Task 2 ECR), argumentative essay summarisation and human preference ranking (Task 3 ASR) and metric learning for automated evaluation of resulting essays, based on human feedback along argument quality dimensions (Task 4 SQE). Our dataset contains 14k examples of claims that are fully annotated with the various properties supporting the aforementioned tasks. We evaluate multiple generative baselines for each of these tasks, including representative LLMs. We find, that while they show promising results on individual tasks in our benchmark, their end-to-end performance on all four tasks in succession deteriorates significantly, both in automated measures as well as in human-centred evaluation. This challenge presented by our proposed dataset motivates future research on end-to-end argument mining and summarisation. The repository of this project is available at https://github.com/HaoBytes/ArgSum-Datatset
Ethical considerations of use of hold-out sets in clinical prediction model management
Venue: arXiv cs.LG 2024
Year: 2024

Clinical prediction models are statistical or machine learning models used to quantify the risk of a certain health outcome using patient data. These can then inform potential interventions on patients, causing an effect called performative prediction: predictions inform interventions which influence the outcome they were trying to predict, leading to a potential underestimation of risk in some patients if a model is updated on this data. One suggested resolution to this is the use of hold-out sets, in which a set of patients do not receive model derived risk scores, such that a model can be safely retrained. We present an overview of clinical and research ethics regarding potential implementation of hold-out sets for clinical prediction models in health settings. We focus on the ethical principles of beneficence, non-maleficence, autonomy and justice. We also discuss informed consent, clinical equipoise, and truth-telling. We present illustrative cases of potential hold-out set implementations and discuss statistical issues arising from different hold-out set sampling methods. We also discuss differences between hold-out sets and randomised control trials, in terms of ethics and statistical issues. Finally, we give practical recommendations for researchers interested in the use hold-out sets for clinical prediction models.
Brief Announcement: Distributed Unconstrained Local Search for Multilevel Graph Partitioning
Venue: arXiv cs.DC 2024
Year: 2024

Partitioning a graph into blocks of roughly equal weight while cutting only few edges is a fundamental problem in computer science with numerous practical applications. While shared-memory parallel partitioners have recently matured to achieve the same quality as widely used sequential partitioners, there is still a pronounced quality gap between distributed partitioners and their sequential counterparts. In this work, we shrink this gap considerably by describing the engineering of an unconstrained local search algorithm suitable for distributed partitioners. We integrate the proposed algorithm in a distributed multilevel partitioner. Our extensive experiments show that the resulting algorithm scales to thousands of PEs while computing cuts that are, on average, only 3.5% larger than those of a state-of-the-art high-quality shared-memory partitioner. Compared to previous distributed partitioners, we obtain on average 6.8% smaller cuts than the best-performing competitor while being more than 9 times faster.
Fuzzing Frameworks for Server-side Web Applications: A Survey
Venue: arXiv cs.SE 2024
Year: 2024

There are around 5.3 billion Internet users, amounting to 65.7% of the global population, and web technology is the backbone of the services delivered via the Internet. To ensure web applications are free from security-related bugs, web developers test the server-side web applications before deploying them to production. The tests are commonly conducted through the interfaces (i.e., Web API) that the applications expose since they are the entry points to the application. Fuzzing is one of the most promising automated software testing techniques suitable for this task; however, the research on (server-side) web application fuzzing has been rather limited compared to binary fuzzing which is researched extensively. This study reviews the state-of-the-art fuzzing frameworks for testing web applications through web API, identifies open challenges, and gives potential future research. We collect papers from seven online repositories of peer-reviewed articles over the last ten years. Compared to other similar studies, our review focuses more deeply on revealing prior work strategies in generating valid HTTP requests, utilising feedback from the Web Under Tests (WUTs), and expanding input spaces. The findings of this survey indicate that several crucial challenges need to be solved, such as the ineffectiveness of web instrumentation and the complexity of handling microservice applications. Furthermore, some potential research directions are also provided, such as fuzzing for web client programming. Ultimately, this paper aims to give a good starting point for developing a better web fuzzing framework.
Linking Named Entities in Diderot's \textit{Encyclopédie} to Wikidata
Venue: arXiv cs.CL 2024
Year: 2024

Diderot's \textit{Encyclopédie} is a reference work from XVIIIth century in Europe that aimed at collecting the knowledge of its era. \textit{Wikipedia} has the same ambition with a much greater scope. However, the lack of digital connection between the two encyclopedias may hinder their comparison and the study of how knowledge has evolved. A key element of \textit{Wikipedia} is Wikidata that backs the articles with a graph of structured data. In this paper, we describe the annotation of more than 10,300 of the \textit{Encyclopédie} entries with Wikidata identifiers enabling us to connect these entries to the graph. We considered geographic and human entities. The \textit{Encyclopédie} does not contain biographic entries as they mostly appear as subentries of locations. We extracted all the geographic entries and we completely annotated all the entries containing a description of human entities. This represents more than 2,600 links referring to locations or human entities. In addition, we annotated more than 9,500 entries having a geographic content only. We describe the annotation process as well as application examples. This resource is available at https://github.com/pnugues/encyclopedie_1751
Object Manipulation in Marine Environments using Reinforcement Learning
Venue: arXiv cs.RO 2024
Year: 2024

Performing intervention tasks in the maritime domain is crucial for safety and operational efficiency. The unpredictable and dynamic marine environment makes the intervention tasks such as object manipulation extremely challenging. This study proposes a robust solution for object manipulation from a dock in the presence of disturbances caused by sea waves. To tackle this challenging problem, we apply a deep reinforcement learning (DRL) based algorithm called Soft. Actor-Critic (SAC). SAC employs an actor-critic framework; the actors learn a policy that minimizes an objective function while the critic evaluates the learned policy and provides feedback to guide the actor-learning process. We trained the agent using the PyBullet dynamic simulator and tested it in a realistic simulation environment called MBZIRC maritime simulator. This simulator allows the simulation of different wave conditions according to the World Meteorological Organization (WMO) sea state code. Simulation results demonstrate a high success rate in retrieving the objects from the dock. The trained agent achieved an 80 percent success rate when applied in the simulation environment in the presence of waves characterized by sea state 2, according to the WMO sea state code
Soft GPGPU versus IP cores: Quantifying and Reducing the Performance Gap
Venue: arXiv cs.AR 2024
Year: 2024

eGPU, a recently-reported soft GPGPU for FPGAs, has demonstrated very high clock frequencies (more than 750 MHz) and small footprint. This means that for the first time, commercial soft processors may be competitive for the kind of heavy numerical computations common in FPGA-based digital signal processing. In this paper we take a deep dive into the performance of the eGPU family on FFT computation, in order to quantify the performance gap between state-of-the-art soft processors and commercial IP cores specialized for this task. In the process, we propose two novel architectural features for the eGPU that improve the efficiency of the design by 50\% when executing the FFTs. The end-result is that our modified GPGPU takes only 3 times the performance-area product of a specialized IP core, yet as a programmable processor is able to execute arbitrary software-defined algorithms. Further comparison to Nvidia A100 GPGPUs demonstrates the superior efficiency of eGPU on FFTs of the size studied (256 to 4096-point).
Error-preserving Automatic Speech Recognition of Young English Learners' Language
Venue: arXiv cs.CL 2024
Year: 2024

One of the central skills that language learners need to practice is speaking the language. Currently, students in school do not get enough speaking opportunities and lack conversational practice. Recent advances in speech technology and natural language processing allow for the creation of novel tools to practice their speaking skills. In this work, we tackle the first component of such a pipeline, namely, the automated speech recognition module (ASR), which faces a number of challenges: first, state-of-the-art ASR models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners' speech. Second, most ASR systems contain a powerful language model, which smooths out errors made by the speakers. To give corrective feedback, which is a crucial part of language learning, the ASR systems in our setting need to preserve the errors made by the language learners. In this work, we build an ASR system that satisfies these requirements: it works on spontaneous speech by young language learners and preserves their errors. For this, we collected a corpus containing around 85 hours of English audio spoken by learners in Switzerland from grades 4 to 6 on different language learning tasks, which we used to train an ASR model. Our experiments show that our model benefits from direct fine-tuning on children's voices and has a much higher error preservation rate than other models.
Temperature Illusions in Mixed Reality using Color and Dynamic Graphics
Venue: arXiv cs.HC 2024
Year: 2024

Sensory illusions - where a sensory stimulus causes people to perceive effects that are altered by a different sensory stimulus - have the potential to enrich mixed-reality based interactions. The well-known colour-temperature illusion is a sensory illusion that causes people to, somewhat counterintuitively, perceive blue objects to feel warmer and red objects to feel colder. There is currently little information about whether this illusion can be recreated in mixed reality (MR). Additionally, it is unknown whether dynamic graphical effects made possible by mixed-reality systems could create a similar or potentially stronger effect to the color-temperature illusion. The results of our study (n=30) support that the color-temperature illusion can be recreated in MR and that dynamic graphics can create a new temperature-sensory illusion. Our dynamic-graphics-temperature illusion creates a stronger effect than the color-temperature illusion and has more intuitive relationship between the stimulus and the effect: cold graphical effects (a virtual ice ball) are perceived as colder and hot graphical effects (a virtual fire ball) as hotter. Our results demonstrate that mixed reality has the potential to create novel and stronger temperature-based illusions and encourage further investigation into graphical effects to shape user perception.
Variational Pseudo Marginal Methods for Jet Reconstruction in Particle Physics
Venue: arXiv cs.LG 2024
Year: 2024

Reconstructing jets, which provide vital insights into the properties and histories of subatomic particles produced in high-energy collisions, is a main problem in data analyses in collider physics. This intricate task deals with estimating the latent structure of a jet (binary tree) and involves parameters such as particle energy, momentum, and types. While Bayesian methods offer a natural approach for handling uncertainty and leveraging prior knowledge, they face significant challenges due to the super-exponential growth of potential jet topologies as the number of observed particles increases. To address this, we introduce a Combinatorial Sequential Monte Carlo approach for inferring jet latent structures. As a second contribution, we leverage the resulting estimator to develop a variational inference algorithm for parameter learning. Building on this, we introduce a variational family using a pseudo-marginal framework for a fully Bayesian treatment of all variables, unifying the generative model with the inference process. We illustrate our method's effectiveness through experiments using data generated with a collider physics generative model, highlighting superior speed and accuracy across a range of tasks.
Near-field Beam training for Extremely Large-scale MIMO Based on Deep Learning
Venue: arXiv cs.LG 2024
Year: 2024

Extremely Large-scale Array (ELAA) is considered a frontier technology for future communication systems, pivotal in improving wireless systems' rate and spectral efficiency. As ELAA employs a multitude of antennas operating at higher frequencies, users are typically situated in the near-field region where the spherical wavefront propagates. The near-field beam training in ELAA requires both angle and distance information, which inevitably leads to a significant increase in the beam training overhead. To address this problem, we propose a near-field beam training method based on deep learning. We use a convolutional neural network (CNN) to efficiently learn channel characteristics from historical data by strategically selecting padding and kernel sizes. The negative value of the user average achievable rate is utilized as the loss function to optimize the beamformer. This method maximizes multi-user networks' achievable rate without predefined beam codebooks. Upon deployment, the model requires solely the pre-estimated channel state information (CSI) to derive the optimal beamforming vector. The simulation results demonstrate that the proposed scheme achieves a more stable beamforming gain and significantly improves performance compared to the traditional beam training method. Furthermore, owing to the inherent traits of deep learning methodologies, this approach substantially diminishes the near-field beam training overhead.
VWise: A novel benchmark for evaluating scene classification for vehicular applications
Venue: arXiv cs.CV 2024
Year: 2024

Current datasets for vehicular applications are mostly collected in North America or Europe. Models trained or evaluated on these datasets might suffer from geographical bias when deployed in other regions. Specifically, for scene classification, a highway in a Latin American country differs drastically from an Autobahn, for example, both in design and maintenance levels. We propose VWise, a novel benchmark for road-type classification and scene classification tasks, in addition to tasks focused on external contexts related to vehicular applications in LatAm. We collected over 520 video clips covering diverse urban and rural environments across Latin American countries, annotated with six classes of road types. We also evaluated several state-of-the-art classification models in baseline experiments, obtaining over 84% accuracy. With this dataset, we aim to enhance research on vehicular tasks in Latin America.
Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation
Venue: arXiv cs.CV 2024
Year: 2024

Semantic segmentation is an essential component of medical image analysis research, with recent deep learning algorithms offering out-of-the-box applicability across diverse datasets. Despite these advancements, segmentation failures remain a significant concern for real-world clinical applications, necessitating reliable detection mechanisms. This paper introduces a comprehensive benchmarking framework aimed at evaluating failure detection methodologies within medical image segmentation. Through our analysis, we identify the strengths and limitations of current failure detection metrics, advocating for the risk-coverage analysis as a holistic evaluation approach. Utilizing a collective dataset comprising five public 3D medical image collections, we assess the efficacy of various failure detection strategies under realistic test-time distribution shifts. Our findings highlight the importance of pixel confidence aggregation and we observe superior performance of the pairwise Dice score (Roy et al., 2019) between ensemble predictions, positioning it as a simple and robust baseline for failure detection in medical image segmentation. To promote ongoing research, we make the benchmarking framework available to the community.
Analyzing and Estimating Support for U.S. Presidential Candidates in Twitter Polls
Venue: arXiv cs.SI 2024
Year: 2024

Polls posted on social media have emerged in recent years as an important tool for estimating public opinion, e.g., to gauge public support for business decisions and political candidates in national elections. Here, we examine nearly two thousand Twitter polls gauging support for U.S. presidential candidates during the 2016 and 2020 election campaigns. First, we describe the rapidly emerging prevalence of social polls. Second, we characterize social polls in terms of their heterogeneity and response options. Third, leveraging machine learning models for user attribute inference, we describe the demographics, political leanings, and other characteristics of the users who author and interact with social polls. Finally, we study the relationship between social poll results, their attributes, and the characteristics of users interacting with them. Our findings reveal that Twitter polls are biased in various ways, starting from the position of the presidential candidates among the poll options to biases in demographic attributes and poll results. The 2016 and 2020 polls were predominantly crafted by older males and manifested a pronounced bias favoring candidate Donald Trump, in contrast to traditional surveys, which favored Democratic candidates. We further identify and explore the potential reasons for such biases in social polling and discuss their potential repercussions. Finally, we show that biases in social media polls can be corrected via regression and poststratification. The errors of the resulting election estimates can be as low as 1%-2%, suggesting that social media polls can become a promising source of information about public opinion.
CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planning
Venue: arXiv cs.AI 2024
Year: 2024

Large Language Models (LLMs) possess extensive foundational knowledge and moderate reasoning abilities, making them suitable for general task planning in open-world scenarios. However, it is challenging to ground a LLM-generated plan to be executable for the specified robot with certain restrictions. This paper introduces CLMASP, an approach that couples LLMs with Answer Set Programming (ASP) to overcome the limitations, where ASP is a non-monotonic logic programming formalism renowned for its capacity to represent and reason about a robot's action knowledge. CLMASP initiates with a LLM generating a basic skeleton plan, which is subsequently tailored to the specific scenario using a vector database. This plan is then refined by an ASP program with a robot's action knowledge, which integrates implementation details into the skeleton, grounding the LLM's abstract outputs in practical robot contexts. Our experiments conducted on the VirtualHome platform demonstrate CLMASP's efficacy. Compared to the baseline executable rate of under 2% with LLM approaches, CLMASP significantly improves this to over 90%.
LncRNA-disease association prediction method based on heterogeneous information completion and convolutional neural network
Venue: arXiv cs.LG 2024
Year: 2024

The emerging research shows that lncRNA has crucial research value in a series of complex human diseases. Therefore, the accurate identification of lncRNA-disease associations (LDAs) is very important for the warning and treatment of diseases. However, most of the existing methods have limitations in identifying nonlinear LDAs, and it remains a huge challenge to predict new LDAs. In this paper, a deep learning model based on a heterogeneous network and convolutional neural network (CNN) is proposed for lncRNA-disease association prediction, named HCNNLDA. The heterogeneous network containing the lncRNA, disease, and miRNA nodes, is constructed firstly. The embedding matrix of a lncRNA-disease node pair is constructed according to various biological premises about lncRNAs, diseases, and miRNAs. Then, the low-dimensional feature representation is fully learned by the convolutional neural network. In the end, the XGBoot classifier model is trained to predict the potential LDAs. HCNNLDA obtains a high AUC value of 0.9752 and AUPR of 0.9740 under the 5-fold cross-validation. The experimental results show that the proposed model has better performance than that of several latest prediction models. Meanwhile, the effectiveness of HCNNLDA in identifying novel LDAs is further demonstrated by case studies of three diseases. To sum up, HCNNLDA is a feasible calculation model to predict LDAs.
HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits
Venue: arXiv cs.LG 2024
Year: 2024

Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development. Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating new evaluation data is tedious and may result in temporally inconsistent results. We introduce HelloFresh, based on continuous streams of real-world data generated by intrinsically motivated human labelers. It covers recent events from X (formerly Twitter) community notes and edits of Wikipedia pages, mitigating the risk of test data contamination and benchmark overfitting. Any X user can propose an X note to add additional context to a misleading post (formerly tweet); if the community classifies it as helpful, it is shown with the post. Similarly, Wikipedia relies on community-based consensus, allowing users to edit articles or revert edits made by other users. Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web. We backtest state-of-the-art LLMs supplemented with simple web search access and find that HelloFresh yields a temporally consistent ranking. To enable continuous evaluation on HelloFresh, we host a public leaderboard and periodically updated evaluation data at https://tinyurl.com/hello-fresh-LLM.